Обсуждение: User/Roles, Owner, and privileges migration strategy

Hi Team,

Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.

In this example at the database level the owner for some objects are - 

When I perform pg_dump with owners and ACL. the statements formed are -

• Create Table2
• Alter Owner Table2 to user2
• Grant All privileges on Table2 to user2

The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.

One of the solutions is to have multiple pg_dump statements like -

• pg_dump - schema only with no owner no ACL
• pg_dump  -schema only with ACL | sed "Grant All"
• pg_dump -schame only with owner | sed "Alter owner"

The above solution might have a performance impact since the source might contain millions of objects and performing pg_dump is an expensive operation multiple times. Is there any other way to perform different ownership of objects and grant migration from source to target PostgreSQL instances for that specific database and in an automated way?

Source version of PostgreSQL instance can be - 10,11,12

Target version of PostgreSQL instance can be - 14, 15

Thanks & Regards,

Aditya D

On Fri, 20 Oct 2023 at 20:39, David G. Johnston <david.g.johnston@gmail.com> wrote:

On Friday, October 20, 2023, ADITYA DUVURI <adilove1987@gmail.com> wrote:

Hi Team,

Can you please let us know what is the best strategy or different approaches to migrate users/roles, privileges and owner to different PaaS PostgreSQL instance like AWS RDS or GCP cloud SQL from on-premises?

Execute the migration scripts you have sitting in version control against the new server then restore the dumped data.

David J.

Ordinary Dump files are made to be restored by a superuser role, so that GRANT and ALTER statements can be executed in any order.

Dump files created with the option --no-owner can be played back by any user capable of creating objects, but the created objects will be owned by the creating user.

-- 
Holger Jakobs, Bergisch Gladbach, Tel. +49-178-9759012

Hi Team,

Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.

In this example at the database level the owner for some objects are - 

When I perform pg_dump with owners and ACL. the statements formed are -

• Create Table2
• Alter Owner Table2 to user2
• Grant All privileges on Table2 to user2

The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.

Thanks a lot Holger and Ron for the reply and valuable inputs. AWS RDS or any other PaaS PostgreSQL instances does not support Superuser and to avoid multiple pg_dump statements from on-premises to PaaS, is there any recommended way?

Query the catalog tables and form the alter owner statements which will run post pg_dump - - no owner or any other suggested method?

Regards,

Aditya D

On Mon, 23 Oct 2023 at 01:14, Ron <ronljohnsonjr@gmail.com> wrote:

On 10/21/23 21:51, ADITYA DUVURI wrote:

Hi Team,

Let me take a user scenario here, I have an On-premises PostgreSQL instance with 50 databases. Out of which I need to migrate only 15 databases. There are around 10 different users and roles and millions of different database objects.

In this example at the database level the owner for some objects are - 

When I perform pg_dump with owners and ACL. the statements formed are -

• Create Table2
• Alter Owner Table2 to user2
• Grant All privileges on Table2 to user2

The issue while restoring the above dump file created in any on-premises/AWS/GCP PostgreSQL instance fails since the grant statements have to be executed first before the Alter owner statement.

Did you first run "pg_dumpall --globals-only" against the on-prem server?  (Maybe you did, and I missed it.)

That has all of the CREATE and ALTER ROLE statements.  You'll have to edit it first, though, to remove references to role "postgres", and other statements that are outdated, or irrelevant to AWS RDS.

--
Born in Arizona, moved to Babylonia.

On 2023-Oct-24, Aditya D wrote:

> Thanks a lot Holger and Ron for the reply and valuable inputs. AWS RDS or
> any other PaaS PostgreSQL instances does not support Superuser and to avoid
> multiple pg_dump statements from on-premises to PaaS, is there any
> recommended way?
> 
> Query the catalog tables and form the alter owner statements which will run
> post pg_dump - - no owner or any other suggested method?

I would do this using the following steps:

1. take a "pg_dumpall -g".

2. take a pg_dump (in -Fc or -Fd mode) of each database to migrate.  Do
not use "--no-owner" or anything that breaks or incompletely dumps the
ownership or ACLs or objects.

3. write a script to filter each of the dumps in step 2, to obtain the
list of roles needed in GRANT commands or as object owners.

4. Filter the dump produced by step 1 down to the creation of just the
roles obtained by step 3.

5. Restore each of the dumps from step 2.


If step 5 finishes cleanly, then you're done.  If it throws errors about
ownership or GRANTs, then your filtering scripts in steps 3 or 4 have
some mistake.  Go back and fix the script, clean up from steps 4 and 5
and restart from 3.  Lather, rinse, repeat.

Note that you only need to dump each database exactly twice (first to
produce the dumps you'll test with, second to do the actual migration
once your filtering script for steps 3 and 4 have been perfected.)

If you have objects owned by "postgres" or some other superuser, I'd
change them ahead of time to something else.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"In fact, the basic problem with Perl 5's subroutines is that they're not
crufty enough, so the cruft leaks out into user-defined code instead, by
the Conservation of Cruft Principle."  (Larry Wall, Apocalypse 6)

Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> On 2023-Oct-24, Aditya D wrote:
>> Thanks a lot Holger and Ron for the reply and valuable inputs. AWS RDS or
>> any other PaaS PostgreSQL instances does not support Superuser and to avoid
>> multiple pg_dump statements from on-premises to PaaS, is there any
>> recommended way?

> I would do this using the following steps:
> ...

I kind of suspect that that's going to be a dead end.  If you can't
run dump and restore as superuser, it's going to be very hard to deal
with a multi-user database, unless you use --no-owner which of course
doesn't restore the multiple object ownerships.

You might have to resort to separately dumping the objects belonging
to each user, and then running each restore as that user.

            regards, tom lane

On 2023-Oct-25, Tom Lane wrote:

> I kind of suspect that that's going to be a dead end.  If you can't
> run dump and restore as superuser, it's going to be very hard to deal
> with a multi-user database, unless you use --no-owner which of course
> doesn't restore the multiple object ownerships.
> 
> You might have to resort to separately dumping the objects belonging
> to each user, and then running each restore as that user.

In Postgres you're probably right, but I understood that this is RDS,
which offers a superuser-of-sorts.  It allows some alien stuff to be
done.  Though ... re-reading the question, I see now he's saying it may
be RDS or maybe *other* DBaaS providers, so maybe it works and maybe it
doesn't, depending on the specifics.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/

On 2023-Oct-25, Tom Lane wrote:

I kind of suspect that that's going to be a dead end.  If you can't
run dump and restore as superuser, it's going to be very hard to deal
with a multi-user database, unless you use --no-owner which of course
doesn't restore the multiple object ownerships.

You might have to resort to separately dumping the objects belonging
to each user, and then running each restore as that user.

In Postgres you're probably right, but I understood that this is RDS,
which offers a superuser-of-sorts.  It allows some alien stuff to be
done.  Though ... re-reading the question, I see now he's saying it may
be RDS or maybe *other* DBaaS providers, so maybe it works and maybe it
doesn't, depending on the specifics.