Обсуждение: cfbot is failing all tests on FreeBSD/Meson builds
It looks like every recent cfbot run has failed in the
FreeBSD-13-Meson build, even if it worked in other ones.
The symptoms are failures in the TAP tests that try to
use interactive_psql:
Can't call method "slave" on an undefined value at /usr/local/lib/perl5/site_perl/IPC/Run.pm line 2889.
I suspect that we are looking at some bug in IPC::Run that exists in
the version that that FreeBSD release has (but not, seemingly,
elsewhere in the community), and that was mostly harmless until
c53859295 made all Perl warnings fatal. Not sure what we want
to do about this.
regards, tom lane
On Sat, Jan 13, 2024 at 9:32 AM Tom Lane <tgl@sss.pgh.pa.us> wrote: > It looks like every recent cfbot run has failed in the > FreeBSD-13-Meson build, even if it worked in other ones. > The symptoms are failures in the TAP tests that try to > use interactive_psql: > > Can't call method "slave" on an undefined value at /usr/local/lib/perl5/site_perl/IPC/Run.pm line 2889. > > I suspect that we are looking at some bug in IPC::Run that exists in > the version that that FreeBSD release has (but not, seemingly, > elsewhere in the community), and that was mostly harmless until > c53859295 made all Perl warnings fatal. Not sure what we want > to do about this. Right, I see this locally on my FreeBSD box. Reverting the fatal warnings thing doesn't help, it's more broken than that. I tried to understand https://github.com/cpan-authors/IPC-Run/blob/master/lib/IPC/Run.pm https://github.com/cpan-authors/IO-Tty/blob/master/Pty.pm but I am not good at perl. I think the error means that in ## Close all those temporary filehandles that the kids needed. for my $pty ( values %{ $self->{PTYS} } ) { close $pty->slave; } the variable $pty holds undef, so then we have to find out where undef was put into $self->{PTYS}. There is a place that does: ## Just flag the pyt's existence for now. It'll be ## converted to a real IO::Pty by _open_pipes. $self->{PTYS}->{$pty_id} = undef; We can see that this started a couple of days ago: https://github.com/postgres/postgres/commits/master/ But at older commits I see the problem locally too, so, yeah, it must be coming from where else. Looking at the relevant packages p5-IPC-Run and p5-IO-Tty I see that they recently moved to 20231003.0 and 1.18, respectively. Downgrading p5-IPC-Run to p5-IPC-Run-20220807.0.pkg was not enough. Downgrading p5-IO-Tty to p5-IO-Tty-1.17.pkg allowed psql/010_tab_completion to pass with either version of p5-IPC-Run. The answer may lie in the commits between those versions here: https://github.com/cpan-authors/IO-Tty/commits/master/ I see that Debian Bookwork is still using 1.17, which fits with your guess that FreeBSD is just breaking first because it is using a newer version.
Thomas Munro <thomas.munro@gmail.com> writes: > Looking at the relevant packages > p5-IPC-Run and p5-IO-Tty I see that they recently moved to 20231003.0 > and 1.18, respectively. Downgrading p5-IPC-Run to > p5-IPC-Run-20220807.0.pkg was not enough. Downgrading p5-IO-Tty to > p5-IO-Tty-1.17.pkg allowed psql/010_tab_completion to pass with either > version of p5-IPC-Run. > The answer may lie in the commits between those versions here: > https://github.com/cpan-authors/IO-Tty/commits/master/ > I see that Debian Bookwork is still using 1.17, which fits with your > guess that FreeBSD is just breaking first because it is using a newer > version. Yeah, I have nothing newer than 1.17 here either. Time for a bug report to IO::Tty's authors, I guess. regards, tom lane
On Sat, Jan 13, 2024 at 1:51 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > Time for a bug report to IO::Tty's authors, I guess. Ahh, there is one: https://github.com/cpan-authors/IO-Tty/issues/38 In the meantime, will look into whether I can pin that package to 1.17 somewhere in the pipeline, hopefully later today...
Thomas Munro <thomas.munro@gmail.com> writes: > On Sat, Jan 13, 2024 at 1:51 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Time for a bug report to IO::Tty's authors, I guess. > Ahh, there is one: https://github.com/cpan-authors/IO-Tty/issues/38 Just for the archives' sake: I hit this today on a fresh install of FreeBSD 14.0, which has pulled in p5-IO-Tty-1.18. Annoying... regards, tom lane
On Tue, Jan 30, 2024 at 5:06 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > Thomas Munro <thomas.munro@gmail.com> writes: > > On Sat, Jan 13, 2024 at 1:51 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > >> Time for a bug report to IO::Tty's authors, I guess. > > > Ahh, there is one: https://github.com/cpan-authors/IO-Tty/issues/38 > > Just for the archives' sake: I hit this today on a fresh install > of FreeBSD 14.0, which has pulled in p5-IO-Tty-1.18. Annoying... FWIW here's what I did to downgrade: # remove the problematic version (also removes p5-IPC-Run) pkg remove -y p5-IO-Tty # fetch the known good 1.17 package and install it curl -O "https://pkg.freebsd.org/freebsd:14:x86:64/release_0/All/p5-IO-Tty-1.17.pkg" pkg install -y p5-IO-Tty-1.17.pkg # put back p5-IPC-Run pkg install -y p5-IPC-Run # temporarily prevent future "pkg upgrade" from upgrading p5-IO-Tty pkg lock -y p5-IO-Tty
Thomas Munro <thomas.munro@gmail.com> writes: > On Tue, Jan 30, 2024 at 5:06 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Thomas Munro <thomas.munro@gmail.com> writes: >>> Ahh, there is one: https://github.com/cpan-authors/IO-Tty/issues/38 >> Just for the archives' sake: I hit this today on a fresh install >> of FreeBSD 14.0, which has pulled in p5-IO-Tty-1.18. Annoying... > FWIW here's what I did to downgrade: Thanks for the recipe --- this worked for me, although I noticed it insisted on installing perl5.34-5.34.3_3 alongside 5.36. Doesn't seem to be a problem though --- the main perl installation is still 5.36. FWIW, I spent some time yesterday staring at IPC/Run.pm and came to the (unsurprising) conclusion that there's little we can do to work around the bug. None of the moving parts are exposed to callers. Also, I'm not entirely convinced that the above-cited issue report is complaining about the same thing that's biting us. The reported error messages are completely different. regards, tom lane
On Thu, Feb 8, 2024 at 3:53 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > Thomas Munro <thomas.munro@gmail.com> writes: > > On Tue, Jan 30, 2024 at 5:06 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > >> Thomas Munro <thomas.munro@gmail.com> writes: > >>> Ahh, there is one: https://github.com/cpan-authors/IO-Tty/issues/38 > > >> Just for the archives' sake: I hit this today on a fresh install > >> of FreeBSD 14.0, which has pulled in p5-IO-Tty-1.18. Annoying... > > > FWIW here's what I did to downgrade: > > Thanks for the recipe --- this worked for me, although I noticed > it insisted on installing perl5.34-5.34.3_3 alongside 5.36. > Doesn't seem to be a problem though --- the main perl installation > is still 5.36. Looks like CI is broken in this way again, as of ~13 hours ago. Looking into that... > Also, I'm not entirely convinced that the above-cited issue report is > complaining about the same thing that's biting us. The reported error > messages are completely different. You could be right about that. It seems there was a clash between an upstream commit and a patch in FBSD's port tree, which has just been removed: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276535 So perhaps it's time for me to undo what I did before... looking now.
On Fri, Apr 19, 2024 at 10:36 AM Thomas Munro <thomas.munro@gmail.com> wrote: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276535 > > So perhaps it's time for me to undo what I did before... looking now. It turned out that I still needed the previous work-around, but I was too clever for my own boots last time. For the record, here is the new change to the image building script: https://github.com/anarazel/pg-vm-images/commit/faff91cd40d6af0cbc658f5c11da47e2aa88d332 I should have listened to Bilal's prediction[1] about this the first time. But this time, I know that the real fix is coming in the next package very soon, per bugzilla link above. One thing I noticed is that 010_tab_completion is actually being skipped, with that fix. It used to run, I'm sure. Not sure why, but I'll look more seriously when the 1.21 or whatever shows up. At least we should soon have green CI again in the meantime. [1] https://github.com/anarazel/pg-vm-images/pull/92