Re: Reopen logfile on SIGHUP

On 11.04.2018 00:00, Tom Lane wrote:
> So we need a mechanism that's narrowly targeted
> to reopening the logfile, without SIGHUP'ing the entire database.

We can send SIGUSR1 to the syslogger process. To make its pid easier to 
find out, it can be published in "$PGDATA/logging_collector.pid", as 
suggested by Grigory. The attached patch does this. It also adds a brief 
description of how to use this with logrotate.

> Point 2: Depending on how you've got the log filenames configured,
> setting rotation_requested may result in a change in log filename

If logrotate only needs the file to be reopened, syslogger's rotation 
does just than when using a static log file name. I imagine logrotate 
can be configured to do something useful with changing file names, too. 
It is a matter of keeping the configuration of syslogger and logrotate 
consistent.

> BTW, another thing that needs to be considered is the interaction with
> rotation_disabled.  Right now we automatically drop that on SIGHUP, but
> I'm unclear on whether it should be different for logrotate requests.

The SIGUSR1 path is supposed to be used by automated tools. In a sense, 
it is an automatic rotation, the difference being that it originates 
from an external tool and not from syslogger itself. So, it sounds 
plausible that the rotation request shouldn't touch the 
rotation_disabled flag, and should be disabled by it, just like the 
automatic rotation.

Still, this leads us to a scenario where we can lose logs:
1. postgres is configured to use a static file name. logrotate is 
configured to move the file, send SIGUSR1 to postgres syslogger, gzip 
the file and delete it.
2. logrotate starts the rotation. It moves the file and signals postgres 
to reopen it.
3. postgres fails to reopen the file because there are too many files 
open (ENFILE/EMFILE), which is a normal occurrence on heavily loaded 
systems. Or it doesn't open the new file because the rotation_disable 
flag is set. It continues logging to the old file.
4. logrotate has no way to detect this failure, so it gzips the file and 
unlinks it.
5. postgres continues writing to the now unlinked file, and we lose an 
arbitrary amount of logs until the next successful rotation.

With dynamic file names, logrotate can be told to skip open files, so 
that it doesn't touch our log file if we haven't switched to the new 
one. With a static file name, the log file is always open, so this 
method doesn't work. I'm not sure how to make this work reliably.

-- 
Alexander Kuzmenkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company