Re: Channel binding for post-quantum cryptography

On Mon, Oct 20, 2025 at 09:12:52AM +0200, Filip Janus wrote:
> The problem is caused by a difference between the currently used algorithms
> and post-quantum ones. For example, commonly used algorithms like RSA have
> a defined digest algorithm, but ML-DSA does not.
>
> PostgreSQL's channel binding implementation expects all signature
> algorithms to have a traditional digest mapping, but post-quantum
> algorithms such as ML-DSA use their hash function internally as part of the
> signature process.

Noted.

> As a result, the connection fails with the following error:
>
> could not find digest for NID UNDEF
>
> The issue can be worked around by disabling channel binding.
>
> Although the RFC is not entirely clear on how to handle this situation, in
> my patch I propose using SHA-256 as the default digest in such cases.

Based on the RFC at [1], we have indeed:
if the certificate's signatureAlgorithm uses a single hash function
and that hash function neither MD5 nor SHA-1, then use the hash
function associated with the certificate's signatureAlgorithm;

So it would be essential to reuse the hash function used by this
algorithm.  Except that you are saying that we have no way to retrieve
that, even if it's a different routine than EVP_get_digestbynid()?
Enforcing blindly SHA-256 does not seem to be the right move because
it could enforce an incorrect behavior, even if it would be more
user-friendly in some cases like this one.

So, X509_get_signature_info() is able to return an algo NID that we
can then use to decide which algo type we should take.  I can see
three NIDs associated to ML-DSA: NID_ML_DSA_44, NID_ML_DSA_65 and
NID_ML_DSA_87.  Could this list grow?

Based on ml-dsa.md, I am wondering if we could do something based on
ML_DSA_PARAM.  I am not sure, but OpenSSL, while being a spaghetti
code base, usually has some internal way to extract some of its
contents.  Saying that, they tend to hide more internals behind
pointers, 3.0 has added some of that stuff.

A good first step would be to design a reproducible test case to
investigate the issue.  Could it be possible to craft a test case that
could then be added into the tree?  We have automated tests in
src/test/ssl/.  See for example the level of craft done for a similar
past issue, as of commit 9244c11afe23.  That would be the minimum
required for a potential fix.

Note that the use of X509_get_signature_info() is conditional in our
code, so your patch would likely fail to compile depending on the
version of OpenSSL linked to.  The minimum version of OpenSSL
supported by PG on HEAD is 1.1.1.  On the oldest stable branches (v13
or v14), this requirement is..  Cough..  1.0.1.

[1]: https://datatracker.ietf.org/doc/html/rfc5929#section-4.1
--
Michael