Обсуждение: User/Roles, Owner, and privileges migration strategy
Hi Team,
Can you please let us know what is the best strategy or different approaches to migrate users/roles, privileges and owner to different PaaS PostgreSQL instance like AWS RDS or GCP cloud SQL from on-premises? There are different use cases and how this can be handled like “Alter Owner” of an object runs first before grant all privileges in pg_dump and it fails in the PaaS services.
Thanks & Regards,
Aditya
I do not know what is your source database.
In postgres, roles are instance basis unlike database basis in other database flavours.
Good luck,
Sarwar
From: ADITYA DUVURI <adilove1987@gmail.com>
Sent: Friday, October 20, 2023 6:20 AM
To: pgsql-admin@postgresql.org <pgsql-admin@postgresql.org>
Subject: User/Roles, Owner, and privileges migration strategy
Sent: Friday, October 20, 2023 6:20 AM
To: pgsql-admin@postgresql.org <pgsql-admin@postgresql.org>
Subject: User/Roles, Owner, and privileges migration strategy
Hi Team,
Can you please let us know what is the best strategy or different approaches to migrate users/roles, privileges and owner to different PaaS PostgreSQL instance like AWS RDS or GCP cloud SQL from on-premises? There are different use cases and how this can be handled like “Alter Owner” of an object runs first before grant all privileges in pg_dump and it fails in the PaaS services.
Thanks & Regards,
Aditya
Hi Team,
Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.
In this example at the database level the owner for some objects are -
When I perform pg_dump with owners and ACL. the statements formed are -
- Create Table2
- Alter Owner Table2 to user2
- Grant All privileges on Table2 to user2
The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.
One of the solutions is to have multiple pg_dump statements like -
- pg_dump - schema only with no owner no ACL
- pg_dump -schema only with ACL | sed "Grant All"
- pg_dump -schame only with owner | sed "Alter owner"
The above solution might have a performance impact since the source might contain millions of objects and performing pg_dump is an expensive operation multiple times. Is there any other way to perform different ownership of objects and grant migration from source to target PostgreSQL instances for that specific database and in an automated way?
Source version of PostgreSQL instance can be - 10,11,12
Target version of PostgreSQL instance can be - 14, 15
Thanks & Regards,
Aditya D
On Fri, 20 Oct 2023 at 20:39, David G. Johnston <david.g.johnston@gmail.com> wrote:
On Friday, October 20, 2023, ADITYA DUVURI <adilove1987@gmail.com> wrote:Hi Team,Can you please let us know what is the best strategy or different approaches to migrate users/roles, privileges and owner to different PaaS PostgreSQL instance like AWS RDS or GCP cloud SQL from on-premises?Execute the migration scripts you have sitting in version control against the new server then restore the dumped data.David J.
Вложения
Am 22.10.23 um 04:51 schrieb ADITYA DUVURI:
Hi Team,Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.In this example at the database level the owner for some objects are -When I perform pg_dump with owners and ACL. the statements formed are -
- Create Table2
- Alter Owner Table2 to user2
- Grant All privileges on Table2 to user2
The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.One of the solutions is to have multiple pg_dump statements like -
- pg_dump - schema only with no owner no ACL
- pg_dump -schema only with ACL | sed "Grant All"
- pg_dump -schame only with owner | sed "Alter owner"
The above solution might have a performance impact since the source might contain millions of objects and performing pg_dump is an expensive operation multiple times. Is there any other way to perform different ownership of objects and grant migration from source to target PostgreSQL instances for that specific database and in an automated way?Source version of PostgreSQL instance can be - 10,11,12Target version of PostgreSQL instance can be - 14, 15Thanks & Regards,Aditya DOn Fri, 20 Oct 2023 at 20:39, David G. Johnston <david.g.johnston@gmail.com> wrote:
On Friday, October 20, 2023, ADITYA DUVURI <adilove1987@gmail.com> wrote:Hi Team,Can you please let us know what is the best strategy or different approaches to migrate users/roles, privileges and owner to different PaaS PostgreSQL instance like AWS RDS or GCP cloud SQL from on-premises?Execute the migration scripts you have sitting in version control against the new server then restore the dumped data.David J.
Ordinary Dump files are made to be restored by a superuser role, so that GRANT and ALTER statements can be executed in any order.
Dump files created with the option --no-owner can be played back by any user capable of creating objects, but the created objects will be owned by the creating user.
-- Holger Jakobs, Bergisch Gladbach, Tel. +49-178-9759012
Вложения
On 10/21/23 21:51, ADITYA DUVURI wrote:
Did you first run "pg_dumpall --globals-only" against the on-prem server? (Maybe you did, and I missed it.)
That has all of the CREATE and ALTER ROLE statements. You'll have to edit it first, though, to remove references to role "postgres", and other statements that are outdated, or irrelevant to AWS RDS.
Hi Team,Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.In this example at the database level the owner for some objects are -When I perform pg_dump with owners and ACL. the statements formed are -
- Create Table2
- Alter Owner Table2 to user2
- Grant All privileges on Table2 to user2
The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.
Did you first run "pg_dumpall --globals-only" against the on-prem server? (Maybe you did, and I missed it.)
That has all of the CREATE and ALTER ROLE statements. You'll have to edit it first, though, to remove references to role "postgres", and other statements that are outdated, or irrelevant to AWS RDS.
--
Born in Arizona, moved to Babylonia.
Born in Arizona, moved to Babylonia.
Вложения
Thanks a lot Holger and Ron for the reply and valuable inputs. AWS RDS or any other PaaS PostgreSQL instances does not support Superuser and to avoid multiple pg_dump statements from on-premises to PaaS, is there any recommended way?
Query the catalog tables and form the alter owner statements which will run post pg_dump - - no owner or any other suggested method?
Regards,
Aditya D
On Mon, 23 Oct 2023 at 01:14, Ron <ronljohnsonjr@gmail.com> wrote:
On 10/21/23 21:51, ADITYA DUVURI wrote:Hi Team,Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.In this example at the database level the owner for some objects are -When I perform pg_dump with owners and ACL. the statements formed are -
- Create Table2
- Alter Owner Table2 to user2
- Grant All privileges on Table2 to user2
The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.
Did you first run "pg_dumpall --globals-only" against the on-prem server? (Maybe you did, and I missed it.)
That has all of the CREATE and ALTER ROLE statements. You'll have to edit it first, though, to remove references to role "postgres", and other statements that are outdated, or irrelevant to AWS RDS.--
Born in Arizona, moved to Babylonia.
Вложения
I regularly logged into an AWS RDS Postgresql instance as user "postgres", where I created and altered roles.
On 10/24/23 06:18, Aditya D wrote:
Thanks a lot Holger and Ron for the reply and valuable inputs. AWS RDS or any other PaaS PostgreSQL instances does not support Superuser and to avoid multiple pg_dump statements from on-premises to PaaS, is there any recommended way?Query the catalog tables and form the alter owner statements which will run post pg_dump - - no owner or any other suggested method?Regards,Aditya DOn Mon, 23 Oct 2023 at 01:14, Ron <ronljohnsonjr@gmail.com> wrote:On 10/21/23 21:51, ADITYA DUVURI wrote:Hi Team,Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.In this example at the database level the owner for some objects are -When I perform pg_dump with owners and ACL. the statements formed are -
- Create Table2
- Alter Owner Table2 to user2
- Grant All privileges on Table2 to user2
The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.
Did you first run "pg_dumpall --globals-only" against the on-prem server? (Maybe you did, and I missed it.)
That has all of the CREATE and ALTER ROLE statements. You'll have to edit it first, though, to remove references to role "postgres", and other statements that are outdated, or irrelevant to AWS RDS.--
Born in Arizona, moved to Babylonia.
--
Born in Arizona, moved to Babylonia.
Born in Arizona, moved to Babylonia.
Вложения
You should be able to restore a pg_dump as long as the user role you are leveraging has the correct permissions. The user role created with the RDS instance (whatever you name it) is as close as you can get to superuser (member of rds_superuser role). That said, you can login as that user (or another user with the same permissions) and create / alter roles. Also, check if rds.restrict_password_commands is true; if so you may also need to be a member of rds_password role.
In my opinion, having migrations for the creation of Roles (without credentials [those do not belong in VCS]), DCL, and DDL is also helpful as this allows you to quickly review code used to deploy and arrive at the current state. The code for these roles and migrations is also a convenient place to comment on any differences between on-prem and cloud configuration; things never match one to one (rds group roles, authentication methods, and parameter groups vs configuration files are often different). Further this supports local development / testing with Docker or Kubernetes with a bit more work (paying dividends if you are supporting software teams or need to quickly test).
On Tue, Oct 24, 2023 at 8:47 AM Ron <ronljohnsonjr@gmail.com> wrote:
I regularly logged into an AWS RDS Postgresql instance as user "postgres", where I created and altered roles.On 10/24/23 06:18, Aditya D wrote:Thanks a lot Holger and Ron for the reply and valuable inputs. AWS RDS or any other PaaS PostgreSQL instances does not support Superuser and to avoid multiple pg_dump statements from on-premises to PaaS, is there any recommended way?Query the catalog tables and form the alter owner statements which will run post pg_dump - - no owner or any other suggested method?Regards,Aditya DOn Mon, 23 Oct 2023 at 01:14, Ron <ronljohnsonjr@gmail.com> wrote:On 10/21/23 21:51, ADITYA DUVURI wrote:Hi Team,Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.In this example at the database level the owner for some objects are -When I perform pg_dump with owners and ACL. the statements formed are -
- Create Table2
- Alter Owner Table2 to user2
- Grant All privileges on Table2 to user2
The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.
Did you first run "pg_dumpall --globals-only" against the on-prem server? (Maybe you did, and I missed it.)
That has all of the CREATE and ALTER ROLE statements. You'll have to edit it first, though, to remove references to role "postgres", and other statements that are outdated, or irrelevant to AWS RDS.--
Born in Arizona, moved to Babylonia.--
Born in Arizona, moved to Babylonia.
Best regards,
Dan Smith
Dan Smith
Вложения
Thanks All.
For this scenario where ownership of the Object in PostgreSQL is getting changed and while doing pg_restore with the admin user (non-superuser) since PaaS PostgreSQL services does not provide superuser access in the target, getting the following error " pg_restore: error: could not execute query:ERROR: permission denied for schema event_tracking. Command was: ALTER TABLE event_tracking.notifications_responses OWNER TO user3"
In the pg_dump file the Grants to the event_tracking.notifications_responses are coming after the Alter owner command.
Can you please help me here?
On Tue, Oct 24, 2023 at 7:08 PM Dan Smith <j.daniel.smith1@gmail.com> wrote:
You should be able to restore a pg_dump as long as the user role you are leveraging has the correct permissions. The user role created with the RDS instance (whatever you name it) is as close as you can get to superuser (member of rds_superuser role). That said, you can login as that user (or another user with the same permissions) and create / alter roles. Also, check if rds.restrict_password_commands is true; if so you may also need to be a member of rds_password role.In my opinion, having migrations for the creation of Roles (without credentials [those do not belong in VCS]), DCL, and DDL is also helpful as this allows you to quickly review code used to deploy and arrive at the current state. The code for these roles and migrations is also a convenient place to comment on any differences between on-prem and cloud configuration; things never match one to one (rds group roles, authentication methods, and parameter groups vs configuration files are often different). Further this supports local development / testing with Docker or Kubernetes with a bit more work (paying dividends if you are supporting software teams or need to quickly test).On Tue, Oct 24, 2023 at 8:47 AM Ron <ronljohnsonjr@gmail.com> wrote:
I regularly logged into an AWS RDS Postgresql instance as user "postgres", where I created and altered roles.On 10/24/23 06:18, Aditya D wrote:Thanks a lot Holger and Ron for the reply and valuable inputs. AWS RDS or any other PaaS PostgreSQL instances does not support Superuser and to avoid multiple pg_dump statements from on-premises to PaaS, is there any recommended way?Query the catalog tables and form the alter owner statements which will run post pg_dump - - no owner or any other suggested method?Regards,Aditya DOn Mon, 23 Oct 2023 at 01:14, Ron <ronljohnsonjr@gmail.com> wrote:On 10/21/23 21:51, ADITYA DUVURI wrote:Hi Team,Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.In this example at the database level the owner for some objects are -When I perform pg_dump with owners and ACL. the statements formed are -
- Create Table2
- Alter Owner Table2 to user2
- Grant All privileges on Table2 to user2
The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.
Did you first run "pg_dumpall --globals-only" against the on-prem server? (Maybe you did, and I missed it.)
That has all of the CREATE and ALTER ROLE statements. You'll have to edit it first, though, to remove references to role "postgres", and other statements that are outdated, or irrelevant to AWS RDS.--
Born in Arizona, moved to Babylonia.--
Born in Arizona, moved to Babylonia.--Best regards,
Dan Smith
Вложения
On 2023-Oct-24, Aditya D wrote: > Thanks a lot Holger and Ron for the reply and valuable inputs. AWS RDS or > any other PaaS PostgreSQL instances does not support Superuser and to avoid > multiple pg_dump statements from on-premises to PaaS, is there any > recommended way? > > Query the catalog tables and form the alter owner statements which will run > post pg_dump - - no owner or any other suggested method? I would do this using the following steps: 1. take a "pg_dumpall -g". 2. take a pg_dump (in -Fc or -Fd mode) of each database to migrate. Do not use "--no-owner" or anything that breaks or incompletely dumps the ownership or ACLs or objects. 3. write a script to filter each of the dumps in step 2, to obtain the list of roles needed in GRANT commands or as object owners. 4. Filter the dump produced by step 1 down to the creation of just the roles obtained by step 3. 5. Restore each of the dumps from step 2. If step 5 finishes cleanly, then you're done. If it throws errors about ownership or GRANTs, then your filtering scripts in steps 3 or 4 have some mistake. Go back and fix the script, clean up from steps 4 and 5 and restart from 3. Lather, rinse, repeat. Note that you only need to dump each database exactly twice (first to produce the dumps you'll test with, second to do the actual migration once your filtering script for steps 3 and 4 have been perfected.) If you have objects owned by "postgres" or some other superuser, I'd change them ahead of time to something else. -- Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/ "In fact, the basic problem with Perl 5's subroutines is that they're not crufty enough, so the cruft leaks out into user-defined code instead, by the Conservation of Cruft Principle." (Larry Wall, Apocalypse 6)
Alvaro Herrera <alvherre@alvh.no-ip.org> writes: > On 2023-Oct-24, Aditya D wrote: >> Thanks a lot Holger and Ron for the reply and valuable inputs. AWS RDS or >> any other PaaS PostgreSQL instances does not support Superuser and to avoid >> multiple pg_dump statements from on-premises to PaaS, is there any >> recommended way? > I would do this using the following steps: > ... I kind of suspect that that's going to be a dead end. If you can't run dump and restore as superuser, it's going to be very hard to deal with a multi-user database, unless you use --no-owner which of course doesn't restore the multiple object ownerships. You might have to resort to separately dumping the objects belonging to each user, and then running each restore as that user. regards, tom lane
On 2023-Oct-25, Tom Lane wrote: > I kind of suspect that that's going to be a dead end. If you can't > run dump and restore as superuser, it's going to be very hard to deal > with a multi-user database, unless you use --no-owner which of course > doesn't restore the multiple object ownerships. > > You might have to resort to separately dumping the objects belonging > to each user, and then running each restore as that user. In Postgres you're probably right, but I understood that this is RDS, which offers a superuser-of-sorts. It allows some alien stuff to be done. Though ... re-reading the question, I see now he's saying it may be RDS or maybe *other* DBaaS providers, so maybe it works and maybe it doesn't, depending on the specifics. -- Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
On 10/25/23 14:32, Alvaro Herrera wrote:
You can definitely run CREATE ROLE, ALTER ROLE and GRANT statements in AWS RDS Postgresql. You're just not going to get password hashes.On 2023-Oct-25, Tom Lane wrote:I kind of suspect that that's going to be a dead end. If you can't run dump and restore as superuser, it's going to be very hard to deal with a multi-user database, unless you use --no-owner which of course doesn't restore the multiple object ownerships. You might have to resort to separately dumping the objects belonging to each user, and then running each restore as that user.In Postgres you're probably right, but I understood that this is RDS, which offers a superuser-of-sorts. It allows some alien stuff to be done. Though ... re-reading the question, I see now he's saying it may be RDS or maybe *other* DBaaS providers, so maybe it works and maybe it doesn't, depending on the specifics.
--
Born in Arizona, moved to Babylonia.
Born in Arizona, moved to Babylonia.