Обсуждение: Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)

On Mon, 2025-07-14 at 11:19 +0530, Amol Inamdar wrote:
> I'm currently running PostgreSQL version 16.6 inside a Docker container
> (base image: UBI 9), using Docker Compose. The PostgreSQL data directory
> is mounted from an NFS volume hosted on a z/OS NFS server.
>
> The environment has a few constraints:
>
> - It’s a highly secure and access-controlled setup.
> - Due to platform restrictions on z/OS, the mounted NFS directory cannot
>   be owned by the PostgreSQL user (e.g., `postgres`) inside the container.
> - As a result, PostgreSQL fails to start because of the directory
>   ownership validation check.

It is not a good idea to have a mount point be the data directory.
The proper solution is to create the data directory inside the
mount point.  That way, the permissions of the data directory don't
have to be the same as the permissions of the mount point.

Yours,
Laurenz Albe

On Mon, 2025-07-14 at 17:59 +0530, Amol Inamdar wrote:
> If I am not mistaken, below is my understanding of your suggestion. 
>
> Suppose that My mount point on the NFS server is say /nfs-mount/postgres/ 
> and you are suggesting to have a data directory as say /nfs-mount/postgres/db or something like that ? 
> and assign this value to the PGDATA ? 
>
> If that is the case, then when and who should be creating the directory DB ? 
>
> Please correct me if I am wrong about the understanding.

You understood me perfectly well.

The data directory can either be created by "initdb", in which case
the mount point must allow the PostgreSQL user to create a directory.
You could set the group of the mount point to the group of the
PostgreSQL user and use permissions 1770, which should be perfectly safe.

Alternatively, the root user could create the data directory with the
correct ownership and permissions prior to running "initdb".

Yours,
Laurenz Albe

Laurenz Albe <laurenz.albe@cybertec.at> writes:
> It is not a good idea to have a mount point be the data directory.

^^^ This. ^^^

That is primarily for safety reasons: if for some reason the
filesystem gets dismounted, or hasn't come on-line yet during
a reboot, you do not want Postgres to be able to write on the
underlying mount-point directory.  There is a sobering tale
in this old thread:

https://www.postgresql.org/message-id/flat/41BFAB7C.5040108%40joeconway.com

Now it didn't help any that they were using a start script that
would automatically run initdb if it didn't see a data directory
where expected.  But even without that, you are in for a world of
hurt if the mount drops while the server is running and the server
has any ability to write on the underlying storage; it will think
whatever it was able to write is safely down on disk.  To prevent
that, the server must not have write permissions on the mount
point, which dictates making a separate data directory (with
different ownership/permissions) just below the mount.

Do not bypass that ownership/permissions check.  It is there
for very good reasons.

            regards, tom lane

On Mon, 2025-07-14 at 18:32 +0530, Amol Inamdar wrote:
> > The data directory can either be created by "initdb", in which case
> > the mount point must allow the PostgreSQL user to create a directory.
> > You could set the group of the mount point to the group of the
> > PostgreSQL user and use permissions 1770, which should be perfectly safe.
>
> This exactly is the problem we are facing, to give you a summary, 
> our NFS server is enabled with AT-TLS authentication
> and we are accessing the server via a proxy server (Haproxy). 
> This acts as our NFS client and it is configured with the 
> required client certificates.
>
> The outcome of above configuration is that any directory created 
> in the NFS mount is always owned by the user in the certificates 
> and if that user isn't present in the proxy container it is marked 
> as nobody:nogroup, we tried various things like
> created the user similar to postgres user so that the users ids match but 
> always ended up giving error  “data directory “/var/lib” has wrong ownership 
>
> Hence, we thought of skipping this check (Directory owner and postgres user validation) and 
> wanted to understand the implication of the same.

No; don't.

Simply mount the directory once, create a subdirectory with the
appropriate ownership and permissions, and there you go.
Problem solved.

Yours,
Laurenz Albe

On 2025-07-14 10:07:20 -0400, Tom Lane wrote:
> Laurenz Albe <laurenz.albe@cybertec.at> writes:
> > It is not a good idea to have a mount point be the data directory.
>
> ^^^ This. ^^^
>
> That is primarily for safety reasons: if for some reason the
> filesystem gets dismounted, or hasn't come on-line yet during
> a reboot, you do not want Postgres to be able to write on the
> underlying mount-point directory.  There is a sobering tale
> in this old thread:
>
> https://www.postgresql.org/message-id/flat/41BFAB7C.5040108%40joeconway.com
>
> Now it didn't help any that they were using a start script that
> would automatically run initdb if it didn't see a data directory
> where expected.  But even without that, you are in for a world of
> hurt if the mount drops while the server is running and the server
> has any ability to write on the underlying storage; it will think
> whatever it was able to write is safely down on disk.  To prevent
> that, the server must not have write permissions on the mount
> point, which dictates making a separate data directory (with
> different ownership/permissions) just below the mount.

Be careful: There are two different directorys involved in a mount
point. The one in the parent filesystem and the one in the mounted file
system.

E.g.:

# mkdir /mnt/demo
# stat /mnt/demo
  File: /mnt/demo
  Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: 254,0    Inode: 317         Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

So here we have inode 317 on device 254,0 and unsurprisingly it belongs
to root and is only writable by root.

Now let's mount a filesystem:

# mount /dev/vgroot/demo /mnt/demo
# stat /mnt/demo
  File: /mnt/demo
  Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: 254,13    Inode: 2           Links: 3
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

This is now inode 2 on device 254,13 (i.e. the root directory on
/dev/vgroot/demo). Since I just created that it also belongs to root.
Let's change that:

# chown postgres: /mnt/demo
# stat /mnt/demo
  File: /mnt/demo
  Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: 254,13    Inode: 2           Links: 3
Access: (0755/drwxr-xr-x)  Uid: (  108/postgres)   Gid: (  114/postgres)

Ok, now it belongs to postgres and postgres could create files,
subdirectories, etc.

But when I unmount it:

# umount /mnt/demo
# stat /mnt/demo
  File: /mnt/demo
  Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: 254,0    Inode: 317         Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

Lo and behold: The original Inode is back and unchanged.

The user postgres couldn't write to that directory if it tried to.

That said, there are of course some remaining failure modes:

1) An admin might notice the wrong permissions and "fix" them without
   investigating the cause
2) Some automated procedure (running as root) might "fix" the
   permissions at startup (like the initdb in the example) or during an
   upgrade.
3) use your imagination ;-).

        hjp

--
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

"Peter J. Holzer" <hjp-pgsql@hjp.at> writes:
> On 2025-07-14 10:07:20 -0400, Tom Lane wrote:
>> That is primarily for safety reasons: if for some reason the
>> filesystem gets dismounted, or hasn't come on-line yet during
>> a reboot, you do not want Postgres to be able to write on the
>> underlying mount-point directory.

> Be careful: There are two different directorys involved in a mount
> point. The one in the parent filesystem and the one in the mounted file
> system.

True, and the safety requirement really is only that the parent
filesystem's mount-point directory not be writable by us.
But normal practice is that both directories are root-owned,
or at least owned by highly privileged users.

(I have a vague idea that there are system-level security hazards,
not specific to Postgres, if mount-point directories are publicly
writable.  Don't feel like researching that though.)

            regards, tom lane

Benjamin Wang <benjamin.ahrtr@gmail.com> writes:
> I am not sure whether PostgreSQL depends on system call `fsyncdata` to
>  sync data to disk. If yes, then I don't think it's safe to use NFS.

Well, that's a whole other discussion.  The point about mount
directories applies to any sort of dismountable storage.

(But I too would not use Postgres-over-NFS for any critical data.
Too many moving parts.  It's tough enough to ensure crash safety
with local storage.)

            regards, tom lane

On Mon, 2025-07-14 at 14:30 -0400, Tom Lane wrote:
> (I have a vague idea that there are system-level security hazards,
> not specific to Postgres, if mount-point directories are publicly
> writable.  Don't feel like researching that though.)

Well, if you are using an ext? file system, there is a lost+found
directory where fsck places links to orphaned inodes.
If the PostgreSQL user owns the mount point and wants to use
"initdb" to create a data directory in it, the program will fail
and complain that the directory is not empty.  The danger is great
that the user removes the lost+found directory to solve the problem.

True, one could re-create it with "mklost+found", but if a DBA
is uneducated enough to remove the directory in the first place,
the risk is high that he wouldn't think of creating it again,
which is a problem if the file system ever becomes corrupted.

All this doesn't apply to NFS, but it is yet another reason
(apart from the safety of a subdirectory that doesn't exist
on the file system underlying the mount point) why we should
continue to recommend that the data directory be not a mount point.

Yours,
Laurenz Albe

On Wed, 2025-07-16 at 18:54 +0530, Amol Inamdar wrote:
> I would like to rephrase the question a little bit, below is how our setup going to be 
>    1. NFS mount point is for /nfs-mount/postgres (and permissions locked down so
>       that Postgres cannot create directories in here)
>    2. Postgres data directory is /nfs-mount/postgres/db
>    3. With secured NFS + AT-TLS setup Postgres will be able to write to data directory
>       but not parent dir, however the file ownership information Postgres sees from the
>       stat() call will not match the Postgres user in the container (even though the
>       AT-TLS strict access control will ensure only the Posgres user can read/write to
>       this directory)
> Considering the above scenario/setup, what is the danger of removing the ownership check
> in miscinit.c checkDataDir() function ? 

The danger is that somebody else than the PostgreSQL user has permissions on
the data directory.  You will argue that that somebody is root, and root has
these permissions anyway.

But there is another reason why PostgreSQL insists that the PostgreSQL user
owns the data directory: at startup, the postmaster checks if the data
directory belongs to the current user and fails if not.  This is a protection
against starting the postmaster with the wrong user.

There are certainly ways to do it differently, but I'd argue that they would
be more complicated, and the current simple solution is robust.

If you pre-create the data directory with the appropriate permissions,
what keeps you from giving ownership to the correct user too?

Yours,
Laurenz Albe

Amol Inamdar <amol.aai@gmail.com> writes:
> @Laurenz Albe <laurenz.albe@cybertec.at>
>> If you pre-create the data directory with the appropriate permissions,
>> what keeps you from giving ownership to the correct user too?

> Our NFS server is not a regular linux based server,
> it's on zOS (Mainframes) with AT-TLS security enabled,
> hence it doesn't allow changing of ownership.

Not only is that not a fit storage substrate for Postgres,
it's pretty hard to imagine that it's a fit substrate for
anything.  "Every file on this filesystem must belong to the
same owner" is a concept that should have gone out with
floppy disks.

You need some extremely fundamental re-examination of your
design decisions.  At the moment I am content to say that
Postgres does not support this storage mechanism and we
do not intend to do so in the future.

            regards, tom lane