BUG #17760: SCRAM authentication fails with "modern" (rsassaPss signature) server certificate

The following bug has been logged on the website:

Bug reference:      17760
Logged by:          Gunnar "Nick" Bluth
Email address:      gunnar.bluth@pro-open.de
PostgreSQL version: 13.8
Operating system:   Ubuntu 20.04
Description:

My client recently started rolling out new server certificates, which, when
dumped with "openssl x509 [...]", show slightly different information
regarding signature, hash etc.

Old:
Signature Algorithm: sha256WithRSAEncryption
New:
Signature Algorithm: rsassaPss
         Hash Algorithm: sha512
         Mask Algorithm: mgf1 with sha512
          Salt Length: 0x40
         Trailer Field: 0xBC (default)

When trying to authenticate on a server using such a certificate using
scram-sha-256, we receive an error message: "could not find digest for NID
UNDEF"

I tried to boil this down a bit. The error comes from
src/interfaces/libpq/fe-secure-openssl.c, pgtls_get_peer_certificate_hash(),
line 440+.
The comment there says "If something else is used, the same hash as the
signature algorithm is used."
And obviously, "EVP_get_digestbynid(NID_rsassaPss)" doesn't return a
result.

Now, openssl's "crypto/objects/obj_xref.txt" contains this:
# OID cross reference table.
# Links signatures OIDs to their corresponding public key algorithms
# and digests.
<snip>
sha256WithRSAEncryption sha256  rsaEncryption
<snip>
# For PSS the digest algorithm can vary and depends on the included
# AlgorithmIdentifier. The digest "undef" indicates the public key
# method should handle this explicitly.
rsassaPss               undef   rsassaPss


That explains the "UNDEF" in the error message (i.e., I *think* that's where
it's coming from).

I have to say that I'm not all too deep into crypto stuff. I couldn't even
tell how to create one such certificate, let alone what that PSS stuff is
all about ;-/

Maybe this is even fixed with recent OpenSSL versions (client has 1.1.1f,
Ubuntu 20.04)? Though that line was introduced in 2010...

I do think however that this is an oversight on our side and has to be
addressed. If not in code, the docs should point out that certain server
certificate types (PSS) may not work with SCRAM auth (or libpq needs to be
compiled against a minimum version of OpenSSL, if that's the root cause).

PS: the "obj_xref.txt" also lists "undef" for ED25519 and ED448

PPS: excerpt from "openssl s_client  -starttls postgres -showcerts -connect
[...]":
Requested Signature Algorithms:

ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Shared Requested Signature Algorithms:

ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, P-256, 256 bits