Re: SSL tests fail on OpenSSL v3.2.0

Nazir,

Thanks for opening a thread. Was just about to start one, here what we
came up with so far.

Homebrew users discovered a regression[0] when using Postgres compiled
and linked against OpenSSL version 3.2.

$ psql "postgresql://$DB?sslmode=require"
psql: error: connection to server at "redacted" (redacted), port 5432 failed: ERROR:  Parameter 'user' is missing in
startuppacket. 
double free or corruption (out)
Aborted (core dumped)

Analyzing the backtrace, OpenSSL was overwriting heap-allocated data in
our PGconn struct because it thought BIO::ptr was a struct bss_sock_st
*. OpenSSL then called a memset() on a member of that struct, and we
zeroed out data in our PGconn struct.

BIO_get_data(3) says the following:

> These functions are mainly useful when implementing a custom BIO.
>
> The BIO_set_data() function associates the custom data pointed to by ptr
> with the BIO a. This data can subsequently be retrieved via a call to
> BIO_get_data(). This can be used by custom BIOs for storing
> implementation specific information.

If you take a look at my_BIO_s_socket(), we create a partially custom
BIO, but for the most part are defaulting to the methods defined by
BIO_s_socket(). We need to set application-specific data and not BIO
private data, so that the BIO implementation we rely on, can properly
assert that its private data is what it expects.

The ssl test suite continues to pass with this patch. This patch should
be backported to every supported Postgres version most likely.

[0]: https://github.com/Homebrew/homebrew-core/issues/155651

--
Tristan Partin
Neon (https://neon.tech)