Re: buildfarm logging versus embedded nulls

Tom Lane wrote:
> I was looking at this recent nonrepeatable buildfarm failure:
> http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=polecat&dt=2010-03-11%2021:45:10
> which has several instances of the known "pgstat wait timeout" problem
> during the parallel regression tests.
>
> I was disappointed to see that the postmaster log part of the report
> is truncated near the start of the run, so there's no way to see if
> anything interesting got logged near the point of the failure.
>
> When I run "make check" on my own OS X machine, I notice that the
> postmaster.log file usually has some runs of a few dozen null bytes in
> it.  I suspect this is an artifact of Apple's stdio buffering
> implementation when several backends are writing to the same log file.
> I suppose that what happened in the above case is that some nulls got
> embedded in postmaster.log, and then the file got truncated at the first
> null, perhaps during the upload to the buildfarm server, or maybe it's
> intact on the server but the web page is failing to display anything
> past that point.
>
> There's probably not much we can do about Apple's stdio (and I would bet
> that they inherited this behavior from the BSDen anyway).  What we
> *could* do is
>
> (1) encourage buildfarm owners to run the tests with logging_collector
> turned on, and/or
>
> (2) fix the buildfarm reporting mechanisms to not be fazed by nulls in
> the log files.
>
> I have no clear idea how hard either of these things is.
>
>             
>   

Well, the good news is that we actually have the data on the server, in 
a temp file that will be cleaned up, but hasn't been yet. I have placed 
a copy at <http://buildfarm.postgresql.org/polecat-check.log.gz>. And 
thus we know that the client does exactly what you want, and the problem 
is entirely on the server. That's more good news.

Now, the log_text field in our build_status_log table is text, so it's 
on insertion to the database that it gets truncated. I'm thinking that I 
should just escape nulls with a regex ( 's/\x00/\\0/g' or similar) 
before inserting the data.

Anyone got any better ideas?

(BTW, your idea of using logging_collector won't work without 
significant changes in the buildfarm client. Nice idea, though.)

cheers

andrew