Обсуждение: [HACKERS] PostgreSQL not setting OpenSSL session id context?
Dear hackers, a long-standing issue reported by users of the Npgsql .NET driver for PostgreSQL may have its roots on the PostgreSQL side. I'm far from being an SSL/OpenSSL expert so please be patient if the terms/analysis are incorrect.
When trying to connect with Npgsql to PostgreSQL with client authentication (PG has ssl_ca_file set), the first connection works just fine. The second connection, however, fails and the PostgreSQL logs contain the message session id context uninitialized". This occurs when using .NET's default SSL implementation, SslStream, which supports session resumption - the session connection's ClientHello message contains a session ticket from the first session, triggering the issue.
From some research, it seems that for session resumption/reuse to work, the SSL/TLS server must call SSL_CTX_set_session_id_context/and SSL_set_session_id_context with some arbitrary binary data, to distinguish between contexts/applications. A grep in the PostgreSQL source for "set_session_id_context" doesn't yield anything.
Can someone with more knowledge confirm whether an issue exists on the PostgreSQL side? If so, it seems completely trivial to fix this.
Thanks,
Shay
Shay Rojansky <roji@roji.org> writes:
> When trying to connect with Npgsql to PostgreSQL with client authentication
> (PG has ssl_ca_file set), the first connection works just fine. The second
> connection, however, fails and the PostgreSQL logs contain the message
> session id context uninitialized". This occurs when using .NET's default
> SSL implementation, SslStream, which supports session resumption - the
> session connection's ClientHello message contains a session ticket from the
> first session, triggering the issue.
AFAIK Postgres doesn't support session resumption. If I am correctly
understanding what that is supposed to provide, it would require saving
all of a backend's internal state on the off chance that somebody would
request resuming the session later. I do not think we are going there.
The idea makes sense for servers with relatively lightweight per-session
state, but that ain't us.
I think what you need to do is tell SslStream not to expect that PG
servers will do session resumption. (I'm a bit astonished that that
would be its default assumption in the first place, but whatever.)
regards, tom lane
I wrote:
> I think what you need to do is tell SslStream not to expect that PG
> servers will do session resumption. (I'm a bit astonished that that
> would be its default assumption in the first place, but whatever.)
Actually, after a bit of further googling, it seems that the brain
damage here may be on the server side. It seems that OpenSSL will
send a session ticket if requested, even though the surrounding
application has given it no means to identify the session (!?).
Apparently we need to pass SSL_OP_NO_TICKET to SSL_CTX_set_options
to prevent that from happening.
regards, tom lane
Hi Tom.
Again, I know little about this, but from what I understand PostgreSQL wouldn't actually need to do/implement anything here - the session ticket might be used only to abbreviate the SSL handshake (this would explain why it's on by default without any application support). In other words, simply setting the session context id may make the problem go away and at the same time unlock the abbreviated SSL handshake optimization. I could be wrong about this though.
Whether the above is correct or not, SSL resumption - which removes a network roundtrip from the connection process - may be a worthy optimization even for long-lived connections such as PostgreSQL, although obviously much less valuable than, say, short-lived HTTP connections.
But regardless, it seems that as you say: if you *don't* want to support resumption, you're required to explicitly disable it with SSL_OP_NO_TICKET.
Just to give some context, Npgsql has its own, internal TLS implementation which does not implement session tickets at the client side - this is the workaround currently used. However, it would be much better if the standard .NET SSL implementation could be used instead (i.e. I'm hoping a backport would be possible here).
On Sun, Jul 30, 2017 at 10:59 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I wrote:
> I think what you need to do is tell SslStream not to expect that PG
> servers will do session resumption. (I'm a bit astonished that that
> would be its default assumption in the first place, but whatever.)
Actually, after a bit of further googling, it seems that the brain
damage here may be on the server side. It seems that OpenSSL will
send a session ticket if requested, even though the surrounding
application has given it no means to identify the session (!?).
Apparently we need to pass SSL_OP_NO_TICKET to SSL_CTX_set_options
to prevent that from happening.
regards, tom lane
Just to continue the above, I can confirm that adding a simple call to SSL_CTX_set_session_id_
On Mon, Jul 31, 2017 at 1:15 AM, Shay Rojansky <roji@roji.org> wrote:
Hi Tom.Again, I know little about this, but from what I understand PostgreSQL wouldn't actually need to do/implement anything here - the session ticket might be used only to abbreviate the SSL handshake (this would explain why it's on by default without any application support). In other words, simply setting the session context id may make the problem go away and at the same time unlock the abbreviated SSL handshake optimization. I could be wrong about this though.Whether the above is correct or not, SSL resumption - which removes a network roundtrip from the connection process - may be a worthy optimization even for long-lived connections such as PostgreSQL, although obviously much less valuable than, say, short-lived HTTP connections.But regardless, it seems that as you say: if you *don't* want to support resumption, you're required to explicitly disable it with SSL_OP_NO_TICKET.Just to give some context, Npgsql has its own, internal TLS implementation which does not implement session tickets at the client side - this is the workaround currently used. However, it would be much better if the standard .NET SSL implementation could be used instead (i.e. I'm hoping a backport would be possible here).On Sun, Jul 30, 2017 at 10:59 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:I wrote:
> I think what you need to do is tell SslStream not to expect that PG
> servers will do session resumption. (I'm a bit astonished that that
> would be its default assumption in the first place, but whatever.)
Actually, after a bit of further googling, it seems that the brain
damage here may be on the server side. It seems that OpenSSL will
send a session ticket if requested, even though the surrounding
application has given it no means to identify the session (!?).
Apparently we need to pass SSL_OP_NO_TICKET to SSL_CTX_set_options
to prevent that from happening.
regards, tom lane
Вложения
On 07/31/2017 02:24 AM, Shay Rojansky wrote: > Just to continue the above, I can confirm that adding a simple call > to SSL_CTX_set_session_id_context() to be_tls_init() with some arbitrary > const value fixes the error for me. Attached is a patch (ideally a test > should be done for this, but that's beyond what I can invest at the moment, > let me know if it's absolutely necessary). I agree with Tom that we don't really want abbreviated SSL handshakes, or other similar optimizations, to take place. PostgreSQL connections are quite long-lived, so we have little to gain. But it makes the attack surface larger. There have been vulnerabilities related to SSL renegotiation, resumption, abbreviated handshakes, and all that. I think we should actually call SSL_CTX_set_session_cache_mode(ctx, SSL_SESS_CACHE_OFF), to disable session caching altogether. I'm not sure if we still need to call SSL_CTX_set_session_cache_mode() if we do that. I know next-to-nothing about .Net; is there some easy way to download a .Net client application and test this? - Heikki
Heikki Linnakangas <hlinnaka@iki.fi> writes:
> I agree with Tom that we don't really want abbreviated SSL handshakes,
> or other similar optimizations, to take place. PostgreSQL connections
> are quite long-lived, so we have little to gain. But it makes the attack
> surface larger. There have been vulnerabilities related to SSL
> renegotiation, resumption, abbreviated handshakes, and all that.
> I think we should actually call SSL_CTX_set_session_cache_mode(ctx,
> SSL_SESS_CACHE_OFF), to disable session caching altogether. I'm not sure
> if we still need to call SSL_CTX_set_session_cache_mode() if we do that.
AIUI (and I just learned about this stuff yesterday, so I might be wrong)
session caching and session tickets are two independent mechanisms for
SSL session reuse.
I have no objection to explicitly disabling session caching, but I think
it won't have any real effect, because no backend process could ever have
any entries in its session cache anyway. Maybe it'd result in a more
apropos error message, don't know.
But we need to disable session tickets separately from that. What's
happening right now in Shay's case, I believe, is that the client is
asking for a session ticket and getting one. The ticket contains enough
data to re-establish the same SSL context with a successor backend;
but it does not contain any data that would allow restoration of
relevant backend state. We could imagine "resuming" the session with
virgin backend state, but I think that violates the spirit if not the
letter of RFC 5077. In any case, implementing it with those semantics
would tie our hands if anyone ever wanted to provide something closer
to true session restoration.
regards, tom lane
Hi Tom and Heikki.
As Tom says, session caching and session tickets seem to be two separate things. However, I think you may be reading more into the session ticket feature than there is - AFAICT there is no expectation or mechanism for restoring *application* state of any kind - the mechanism is only supposed to abbreviate the SSL handshake itself, i.e. save a roundtrip in the full handshake process for agreeing on cypher etc. Here's an article I found useful about this: https://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077 (in addition to the RFC itself, of course).
Once again, I manged to make the error go away simply by setting the session id context, which seems to be a mandatory server-side step for properly support session tickets. So to summarize: at the moment PostgreSQL indeed provides a session ticket in the first connection, which is cached on the client side. On the second connection attempt, the (opaque) session ticket is included in the first SSL packet sent to the server (ClientHello), but the lack of a session id context causes OpenSSL to error. In effect, this seems to be a trivial server-side "misconfiguration".
I do understand the reluctance to deal with any SSL "optimizations", having experienced some of the headaches created by renegotiations. However, session tickets do seem like a simple and well-defined optimization that takes effect at connection only. Also, there is no risk of breaking any *current* clients, since at the moment session tickets simply aren't supported (because of the lack of session id context). So this seems to me like a rather low-risk thing to enable. On the other hand, I also understand that saving a connection-time handshake roundtrip is somewhat less relevant to PostgreSQL.
I'm a little busy at the moment but if you'd like I can whip up a trivial client implementation in .NET that demonstrates the issue.
On Tue, Aug 1, 2017 at 12:26 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Heikki Linnakangas <hlinnaka@iki.fi> writes:
> I agree with Tom that we don't really want abbreviated SSL handshakes,
> or other similar optimizations, to take place. PostgreSQL connections
> are quite long-lived, so we have little to gain. But it makes the attack
> surface larger. There have been vulnerabilities related to SSL
> renegotiation, resumption, abbreviated handshakes, and all that.
> I think we should actually call SSL_CTX_set_session_cache_mode(ctx,
> SSL_SESS_CACHE_OFF), to disable session caching altogether. I'm not sure
> if we still need to call SSL_CTX_set_session_cache_mode() if we do that.
AIUI (and I just learned about this stuff yesterday, so I might be wrong)
session caching and session tickets are two independent mechanisms for
SSL session reuse.
I have no objection to explicitly disabling session caching, but I think
it won't have any real effect, because no backend process could ever have
any entries in its session cache anyway. Maybe it'd result in a more
apropos error message, don't know.
But we need to disable session tickets separately from that. What's
happening right now in Shay's case, I believe, is that the client is
asking for a session ticket and getting one. The ticket contains enough
data to re-establish the same SSL context with a successor backend;
but it does not contain any data that would allow restoration of
relevant backend state. We could imagine "resuming" the session with
virgin backend state, but I think that violates the spirit if not the
letter of RFC 5077. In any case, implementing it with those semantics
would tie our hands if anyone ever wanted to provide something closer
to true session restoration.
regards, tom lane
Shay Rojansky <roji@roji.org> writes:
> Once again, I manged to make the error go away simply by setting the
> session id context, which seems to be a mandatory server-side step for
> properly support session tickets.
The fact that you made the error go away doesn't make this a good
solution. In particular, using a simple constant session ID is completely
insecure according to the TLS spec. RFC 5246, F.1.4, doesn't even care
for the idea of ever writing session IDs to stable storage; although
Apache seems to be content with a session ID that is unique per-server
(it looks like they just use a hash of the server's host name).
More generally, PG as currently configured can't do anything with a
session cache since each new backend would start with an empty cache.
So the question here is whether it's safe or worthwhile to allow use
of session tickets. I agree with Heikki's opinion that it's unlikely
to provide any meaningful performance gain for database sessions that
are of reasonable length. I'm also pretty concerned about the possibility
for security problems, eg a client being able to force a server into some
low-security SSL mode. Both RFC 5077 and the Apache people say that if
you use session tickets you'd better rotate the keys for them regularly,
eg in Apache's changelog we find
Session ticket creation uses a random key created during web
server startup and recreated during restarts. No other key
recreation mechanism is available currently. Therefore using session
tickets without restarting the web server with an appropriate frequency
(e.g. daily) compromises perfect forward secrecy. [Rainer Jung]
Since we have no mechanism for that, I think that we need to err on
the side of security.
Accordingly, what I think we should do is something more like the
attached. Could you see whether it fixes your problem?
regards, tom lane
diff --git a/src/backend/libpq/be-secure-openssl.c b/src/backend/libpq/be-secure-openssl.c
index dc307c1..fc6d0f7 100644
--- a/src/backend/libpq/be-secure-openssl.c
+++ b/src/backend/libpq/be-secure-openssl.c
@@ -290,6 +290,14 @@ be_tls_init(bool isServerStart)
SSL_OP_SINGLE_DH_USE |
SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3);
+ /* disallow SSL session tickets */
+#ifdef SSL_OP_NO_TICKET
+ SSL_CTX_set_options(context, SSL_OP_NO_TICKET);
+#endif
+
+ /* disallow SSL session caching, too */
+ SSL_CTX_set_session_cache_mode(context, SSL_SESS_CACHE_OFF);
+
/* set up ephemeral DH and ECDH keys */
if (!initialize_dh(context, isServerStart))
goto error;
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Shay Rojansky <roji@roji.org> writes:
> Once again, I manged to make the error go away simply by setting the
> session id context, which seems to be a mandatory server-side step for
> properly support session tickets.
The fact that you made the error go away doesn't make this a good
solution. In particular, using a simple constant session ID is completely
insecure according to the TLS spec. RFC 5246, F.1.4, doesn't even care
for the idea of ever writing session IDs to stable storage; although
Apache seems to be content with a session ID that is unique per-server
(it looks like they just use a hash of the server's host name).
I think there may be a confusion here - I'm not doing anything with session IDs, merely setting the session ID *context*. This seems to be an OpenSSL-specific feature (nothing to do with TLS) that simply allows distinguishing between different "applications" (or contexts) running in the same process. See https://wiki.openssl.org/index.php/Manual:SSL_CTX_set_session_id_context(3) for the docs.
This feature does not involve writing anything (and definitely not session IDs) to stable storage. The idea is to provide the client with an opaque "session ticket", which is passed by the client back to the server on subsequent connections, and which allows the skipping of a roundtrip in the SSL handshake.
More generally, PG as currently configured can't do anything with a
session cache since each new backend would start with an empty cache.
Again, there's no backend cache - RFC5077 is about having all state at the client side.
So the question here is whether it's safe or worthwhile to allow use
of session tickets. I agree with Heikki's opinion that it's unlikely
to provide any meaningful performance gain for database sessions that
are of reasonable length. I'm also pretty concerned about the possibility
for security problems, eg a client being able to force a server into some
low-security SSL mode. Both RFC 5077 and the Apache people say that if
you use session tickets you'd better rotate the keys for them regularly,
eg in Apache's changelog we find
Session ticket creation uses a random key created during web
server startup and recreated during restarts. No other key
recreation mechanism is available currently. Therefore using session
tickets without restarting the web server with an appropriate frequency
(e.g. daily) compromises perfect forward secrecy. [Rainer Jung]
Since we have no mechanism for that, I think that we need to err on
the side of security.
I may definitely be wrong about this, but I'm under the impression that management of the session ticket (as of the entire resumption mechanism) is OpenSSL's responsibility and does not require anything from PostgreSQL itself. However, if you're suspicious of OpenSSL itself that's another story (and I'd definitely understand).
Accordingly, what I think we should do is something more like the
attached. Could you see whether it fixes your problem?
I will be able to test this later tonight and confirm. I'm not sure why the SSL_OP_NO_TICKET is in an #ifdef, I would simply do it in all cases. I've seen people reporting that this issue is solved via setting SSL_OP_NO_TICKET (e.g. https://forums.aws.amazon.com/message.jspa?messageID=505895) so I'm not sure what SSL_CTX_set_session_cache_mode is supposed to add, but if the idea is to defensively disable other forms of caching than it makes sense.
Just to be clear, I don't necessarily have a problem with disabling RFC5077 session resumption as the benefits in the PostgreSQL scenario aren't big (although I don't think a handshake roundtrip is completely negligible either). I just think it's advisable we understand exactly what it is we're disabling - there seems to be a confusion between session IDs, session ID contexts, server/client-side state etc.
One more note: https://github.com/netty/netty/pull/5321/files is an equivalent PR setting the session ID context to a constant value in netty (which is also a server using OpenSSL). This is in line with the documentation on SSL_CTX_set_session_id_context (https://wiki.openssl.org/index.php/Manual:SSL_CTX_set_session_id_context(3)):
> Sessions are generated within a certain context. When exporting/importing sessions with i2d_SSL_SESSION/d2i_SSL_SESSION it would be possible, to re-import a session generated from another context (e.g. another application), which might lead to malfunctions. Therefore each application must set its own session id context sid_ctx which is used to distinguish the contexts and is stored in exported sessions. The sid_ctx can be any kind of binary data with a given length, it is therefore possible to use e.g. the name of the application and/or the hostname and/or service name ...
I tested the patch.
Doing SSL_CTX_set_session_cache_mode(context, SSL_SESS_CACHE_OFF) doesn't have any effect whatsoever - I still have the same issue (session id context uninitialized). I suspect session caching is an entirely different feature from session tickets/RFC5077 (although it might still be a good idea to disable).
Doing SSL_CTX_set_options(context, SSL_OP_NO_TICKET) indeed resolves the issue, as expected. As I wrote above, I'd remove the #ifdef and execute it always.
I'm still not convinced of the risk/problem of simply setting the session id context as I explained above (rather than disabling the optimization), but of course either solution resolves my problem.
On 2017-08-04 07:22:42 +0300, Shay Rojansky wrote: > I'm still not convinced of the risk/problem of simply setting the session > id context as I explained above (rather than disabling the optimization), > but of course either solution resolves my problem. How would that do anything? Each backend has it's own local memory. I.e. any cache state that openssl would maintain wouldn't be useful. If you want to take advantage of features around this you really need to cache tickets in shared memory... Greetings, Andres Freund
Shay Rojansky <roji@roji.org> writes:
> I tested the patch.
Thanks!
> Doing SSL_CTX_set_session_cache_mode(context, SSL_SESS_CACHE_OFF) doesn't
> have any effect whatsoever - I still have the same issue (session id
> context uninitialized). I suspect session caching is an entirely different
> feature from session tickets/RFC5077 (although it might still be a good
> idea to disable).
Right, we expected that that would have no visible effect, because there
is no way to cache sessions in Postgres anyway. The main point, if I
understand Heikki's concern correctly, is that this might save some
amount of low-level overhead from clients trying to cache connections.
> Doing SSL_CTX_set_options(context, SSL_OP_NO_TICKET) indeed resolves the
> issue, as expected.
Excellent. I'll push this patch tomorrow sometime (too late/tired
right now).
> As I wrote above, I'd remove the #ifdef and execute it always.
The reason I put the #ifdef in is that according to my research the
SSL_OP_NO_TICKET symbol was introduced in openssl 0.9.8f, while we
claim to support back to 0.9.8. I'd be the first to say that you're
nuts if you're running openssl versions that old; but this patch is not
something to move the compatibility goalposts for when it only takes
an #ifdef to avoid breaking older versions.
(I need to check how far back SSL_SESS_CACHE_OFF goes ... we might
need an #ifdef for that too.)
regards, tom lane
On 2017-08-04 07:22:42 +0300, Shay Rojansky wrote:
> I'm still not convinced of the risk/problem of simply setting the session
> id context as I explained above (rather than disabling the optimization),
> but of course either solution resolves my problem.
How would that do anything? Each backend has it's own local
memory. I.e. any cache state that openssl would maintain wouldn't be
useful. If you want to take advantage of features around this you really
need to cache tickets in shared memory...
Guys, there's no data being cached at the backend - RFC5077 is about packaging information into a client-side opaque session ticket that allows skipping a roundtrip on the next connection. As I said, simply setting the session id context (*not* the session id or anything else) makes this feature work, even though a completely new backend process is launched.
> Doing SSL_CTX_set_session_cache_mode(context, SSL_SESS_CACHE_OFF) doesn't
> have any effect whatsoever - I still have the same issue (session id
> context uninitialized). I suspect session caching is an entirely different
> feature from session tickets/RFC5077 (although it might still be a good
> idea to disable).
Right, we expected that that would have no visible effect, because there
is no way to cache sessions in Postgres anyway. The main point, if I
understand Heikki's concern correctly, is that this might save some
amount of low-level overhead from clients trying to cache connections.
OK, sounds right (i.e. this is a defensive measure that isn't directly connected to my problem but makes sense).
> Doing SSL_CTX_set_options(context, SSL_OP_NO_TICKET) indeed resolves the
> issue, as expected.
Excellent. I'll push this patch tomorrow sometime (too late/tired
right now).
Great. Do you think it's possible to backport to the other maintained branches as well, seeing as how this is quite trivial and low-impact?
> As I wrote above, I'd remove the #ifdef and execute it always.
The reason I put the #ifdef in is that according to my research the
SSL_OP_NO_TICKET symbol was introduced in openssl 0.9.8f, while we
claim to support back to 0.9.8. I'd be the first to say that you're
nuts if you're running openssl versions that old; but this patch is not
something to move the compatibility goalposts for when it only takes
an #ifdef to avoid breaking older versions.
(I need to check how far back SSL_SESS_CACHE_OFF goes ... we might
need an #ifdef for that too.)
Ah OK, thanks for the explanation - makes perfect sense.
Shay Rojansky <roji@roji.org> writes:
> Great. Do you think it's possible to backport to the other maintained
> branches as well, seeing as how this is quite trivial and low-impact?
Already done, will be in next week's minor releases. (You timed this
bug report well.)
regards, tom lane
Awesome, thanks!!
On Fri, Aug 4, 2017 at 11:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Shay Rojansky <roji@roji.org> writes:
> Great. Do you think it's possible to backport to the other maintained
> branches as well, seeing as how this is quite trivial and low-impact?
Already done, will be in next week's minor releases. (You timed this
bug report well.)
regards, tom lane
On 08/04/2017 08:48 PM, Shay Rojansky wrote: > On 2017-08-04 07:22:42 +0300, Shay Rojansky wrote: > > I'm still not convinced of the risk/problem of simply setting the session > > id context as I explained above (rather than disabling the optimization), > > but of course either solution resolves my problem. > > How would that do anything? Each backend has it's own local > memory. I.e. any cache state that openssl would maintain wouldn't be > useful. If you want to take advantage of features around this you really > need to cache tickets in shared memory... > > Guys, there's no data being cached at the backend - RFC5077 is about > packaging information into a client-side opaque session ticket that > allows skipping a roundtrip on the next connection. As I said, simply > setting the session id context (*not* the session id or anything else) > makes this feature work, even though a completely new backend process is > launched. Yes, session tickets are encrypted data which is stored by the client. But if we are going to support them I think we should do it properly with new GUCs for the key file and disabling the feature. Using a key file is less necessary for PostgreSQL than for a web server since it is less common to do round robin load balancing between different PostgreSQL instances. Andreas