Обсуждение: Database migration across different operating systems
We are in an unusual circumstance, where we need to move all of our Postgres 10 databases from their homes on servers runningFreeBSD 11 with ZFS to servers in another data center running Red Hat Linux 7.6 (also with ZFS), with minimal downtime. I understand that the recommended, safest, approach would be to dump/restore. We intend to do that wherever possible. However, we have some databases that require high availability/minimal downtime that are also very large. (50 TB, for example) Because we use ZFS on all of our current FreeBSD database hosts, one idea that we are exploring is to use filesystem replicationto move the data ahead of time, and keep it in sync. Then we could start the databases on the Linux hosts at anytime and it would be up to date. With that option, there’s almost no downtime. We’ve tested this procedure on a small database, and it starts, it runs, returns queries. There seems to be no issues. Is this a reasonable approach? Are there any known concerns/things to investigate before choosing this course? If it helps: The cpu architecture is the same on both hosts, and we’ll be running Postgres with the same version and same build options. Many thanks, Natalie
On Tue, Jan 8, 2019, 2:28 AM Natalie Wenz <nataliewenz@gmail.com wrote:
We are in an unusual circumstance, where we need to move all of our Postgres 10 databases from their homes on servers running FreeBSD 11 with ZFS to servers in another data center running Red Hat Linux 7.6 (also with ZFS), with minimal downtime.
I understand that the recommended, safest, approach would be to dump/restore. We intend to do that wherever possible.
However, we have some databases that require high availability/minimal downtime that are also very large. (50 TB, for example)
Because we use ZFS on all of our current FreeBSD database hosts, one idea that we are exploring is to use filesystem replication to move the data ahead of time, and keep it in sync. Then we could start the databases on the Linux hosts at any time and it would be up to date. With that option, there’s almost no downtime.
We’ve tested this procedure on a small database, and it starts, it runs, returns queries. There seems to be no issues.
Is this a reasonable approach? Are there any known concerns/things to investigate before choosing this course?
If it helps:
The cpu architecture is the same on both hosts, and we’ll be running Postgres with the same version and same build options.
Many thanks,
Natalie
Hey ! Le lun. 7 janv., vers 22:51, Natalie Wenz exprimait : > > We are in an unusual circumstance, where we need to move all of our > Postgres 10 databases from their homes on servers running FreeBSD 11 > with ZFS to servers in another data center running Red Hat Linux 7.6 > (also with ZFS), with minimal downtime. > > I understand that the recommended, safest, approach would be to > dump/restore. We intend to do that wherever possible. It's not anymore since PostgreSQL have logical replication. You can initialize you FreeBSD cluster with a pg_basebackup and you follow updates with logical streaming. > However, we have some databases that require high > availability/minimal downtime that are also very large. (50 TB, for > example) With logical replication, is it's permanently up to date in miliseconds, you can switch when you want. > If it helps: The cpu architecture is the same on both hosts, and > we’ll be running Postgres with the same version and same build > options. Your OS or cpu architecture doesn't matter, really :-) Regards, -- Stéphane KANSCHINE - https://www.hexack.fr./ - https://www.nuajik.io./ @ stephane@hexack.fr +33 6 64 31 72 52
Hello, On 8/1/19 6:51 π.μ., Natalie Wenz wrote: > We are in an unusual circumstance, where we need to move all of our Postgres 10 databases from their homes on servers runningFreeBSD 11 with ZFS to servers in another data center running Red Hat Linux 7.6 (also with ZFS), with minimal downtime. any technical reason behind the switch to linux? > > I understand that the recommended, safest, approach would be to dump/restore. We intend to do that wherever possible. > > However, we have some databases that require high availability/minimal downtime that are also very large. (50 TB, for example) > > Because we use ZFS on all of our current FreeBSD database hosts, one idea that we are exploring is to use filesystem replicationto move the data ahead of time, and keep it in sync. Then we could start the databases on the Linux hosts at anytime and it would be up to date. With that option, there’s almost no downtime. Not at any time, you should : - pause (all) the application(s)/service(s) accessing the DB - stop the filesystem replication - start the new cluster - point the apps/services to the new server - resume the apps/services So, you should not start the new cluster with the filesystem replication still active. There are many other solutions as well, you should give logical replication a try. But you have to test first on a similarsized db with similar sized tables, ideally with a copy of the production db. Some tuning might be needed. > > We’ve tested this procedure on a small database, and it starts, it runs, returns queries. There seems to be no issues. > > Is this a reasonable approach? Are there any known concerns/things to investigate before choosing this course? > > If it helps: > The cpu architecture is the same on both hosts, and we’ll be running Postgres with the same version and same build options. > > Many thanks, > Natalie -- Achilleas Mantzios IT DEV Lead IT DEPT Dynacom Tankers Mgmt
>>>>> "Natalie" == Natalie Wenz <nataliewenz@gmail.com> writes: Natalie> We are in an unusual circumstance, where we need to move all Natalie> of our Postgres 10 databases from their homes on servers Natalie> running FreeBSD 11 with ZFS to servers in another data center Natalie> running Red Hat Linux 7.6 (also with ZFS), with minimal Natalie> downtime. [...] Natalie> We’ve tested this procedure on a small database, and it Natalie> starts, it runs, returns queries. There seems to be no issues. Natalie> Is this a reasonable approach? Are there any known Natalie> concerns/things to investigate before choosing this course? Locales will bite you unless your databases all have lc_collate=C or you're using ICU for all collations); you will need, at minimum, to reindex every index containing a collatable column with an OS-provided collation (i.e. not ICU and not C / POSIX). -- Andrew (irc:RhodiumToad)
On 8/1/19 11:01 π.μ., Stéphane KANSCHINE wrote: > Hey ! > > Le lun. 7 janv., vers 22:51, Natalie Wenz exprimait : >> We are in an unusual circumstance, where we need to move all of our >> Postgres 10 databases from their homes on servers running FreeBSD 11 >> with ZFS to servers in another data center running Red Hat Linux 7.6 >> (also with ZFS), with minimal downtime. >> >> I understand that the recommended, safest, approach would be to >> dump/restore. We intend to do that wherever possible. > It's not anymore since PostgreSQL have logical replication. You can > initialize you FreeBSD cluster with a pg_basebackup and you follow > updates with logical streaming. Hmm, maybe you mean pg_dump --schema-only? I cannot see how taking physical copy with pg_basebackup will help with logicalreplication. Logical replication has its own initial snapshot phase. Doing a pg_basebackup would work if followed by truncating all tables, but that's duplicate work and equivalent to pg_dumpall --schema-only| psql -f - > >> However, we have some databases that require high >> availability/minimal downtime that are also very large. (50 TB, for >> example) > With logical replication, is it's permanently up to date in > miliseconds, you can switch when you want. > >> If it helps: The cpu architecture is the same on both hosts, and >> we’ll be running Postgres with the same version and same build >> options. > Your OS or cpu architecture doesn't matter, really :-) > > Regards, -- Achilleas Mantzios IT DEV Lead IT DEPT Dynacom Tankers Mgmt
I thought about logical replication, but it seems that the initial sync is just a plain copy for each table, which will getus into xid trouble on the publisher side while large tables copy. Unless someone can say conclusively that the copy doesNOT hold an xid open that would prevent autovacuums from completing on the publisher. Then of course there’s the excitement of having to freeze everything on the subscriber side, which can take weeks. > On Jan 8, 2019, at 3:01 AM, Stéphane KANSCHINE <stephane@hexack.fr> wrote: > > > Hey ! > > Le lun. 7 janv., vers 22:51, Natalie Wenz exprimait : >> >> We are in an unusual circumstance, where we need to move all of our >> Postgres 10 databases from their homes on servers running FreeBSD 11 >> with ZFS to servers in another data center running Red Hat Linux 7.6 >> (also with ZFS), with minimal downtime. >> >> I understand that the recommended, safest, approach would be to >> dump/restore. We intend to do that wherever possible. > > It's not anymore since PostgreSQL have logical replication. You can > initialize you FreeBSD cluster with a pg_basebackup and you follow > updates with logical streaming. > >> However, we have some databases that require high >> availability/minimal downtime that are also very large. (50 TB, for >> example) > > With logical replication, is it's permanently up to date in > miliseconds, you can switch when you want. > >> If it helps: The cpu architecture is the same on both hosts, and >> we’ll be running Postgres with the same version and same build >> options. > > Your OS or cpu architecture doesn't matter, really :-) > > Regards, > -- > Stéphane KANSCHINE - https://www.hexack.fr./ - https://www.nuajik.io./ > @ stephane@hexack.fr > +33 6 64 31 72 52
Le mar. 8 janv., vers 11:26, Achilleas Mantzios exprimait : > On 8/1/19 11:01 π.μ., Stéphane KANSCHINE wrote: > > > > Le lun. 7 janv., vers 22:51, Natalie Wenz exprimait : > > > > > > I understand that the recommended, safest, approach would be to > > > dump/restore. We intend to do that wherever possible. > > > > It's not anymore since PostgreSQL have logical replication. You can > > initialize you FreeBSD cluster with a pg_basebackup and you follow > > updates with logical streaming. > > Hmm, maybe you mean pg_dump --schema-only? I cannot see how taking physical > copy with pg_basebackup will help with logical replication. You're right, it's useless and counterproductive. I'm still on holydays. But still, take a look at logical replication, it's way more efficient than pg_dump. -- Stéphane KANSCHINE - https://www.hexack.fr./ - https://www.nuajik.io./ @ stephane@hexack.fr +33 6 64 31 72 52
Thanks, Andrew. All of our databases do have lc_collate=C. To check my understanding, if that were not the case, we’d need to reindex, butsince it is all lc_collate=C, we shouldn’t run into that particular problem. Did I get that right? On Jan 8, 2019, at 3:06 AM, Andrew Gierth <andrew@tao11.riddles.org.uk> wrote: >>>>>> "Natalie" == Natalie Wenz <nataliewenz@gmail.com> writes: > > Natalie> We are in an unusual circumstance, where we need to move all > Natalie> of our Postgres 10 databases from their homes on servers > Natalie> running FreeBSD 11 with ZFS to servers in another data center > Natalie> running Red Hat Linux 7.6 (also with ZFS), with minimal > Natalie> downtime. > > [...] > > Natalie> We’ve tested this procedure on a small database, and it > Natalie> starts, it runs, returns queries. There seems to be no issues. > > Natalie> Is this a reasonable approach? Are there any known > Natalie> concerns/things to investigate before choosing this course? > > Locales will bite you unless your databases all have lc_collate=C or > you're using ICU for all collations); you will need, at minimum, to > reindex every index containing a collatable column with an OS-provided > collation (i.e. not ICU and not C / POSIX). > > -- > Andrew (irc:RhodiumToad)
>>>>> "Natalie" == Natalie Wenz <nataliewenz@gmail.com> writes: Natalie> Thanks, Andrew. Natalie> All of our databases do have lc_collate=C. To check my Natalie> understanding, if that were not the case, we’d need to Natalie> reindex, but since it is all lc_collate=C, we shouldn’t run Natalie> into that particular problem. Did I get that right? To be completely precise, it's not enough to have lc_collate=C, you also need to not have indexed columns with individually specified non-default collations. i.e. what matters is not so much the database setting, but the actual collation being used for indexed columns (because it's the collation that determines the order in which values appear in the index). -- Andrew (irc:RhodiumToad)