Hi,
On 2022-03-25 13:52:11 -0400, Robert Haas wrote:
> On Fri, Mar 25, 2022 at 12:36 PM Andres Freund <andres@anarazel.de> wrote:
> > Create a CF entry for it, or enable CI on a github repo?
>
> I created a CF entry for it. Then I had to try to Google around to
> find the URL from the cfbot, because it's not even linked from
> commitfest.postgresql.org for some reason. #blamemagnus
Yea, we really need to improve on that. I think Thomas has some hope of
improving things after the release...
> I don't think that the Windows CI is running the TAP tests for
> contrib. At least, I can't find any indication of it in the output. So
> it doesn't really help to assess how portable this test is, unless I'm
> missing something.
Yea. It's really unfortunate how vcregress.pl makes it hard to run all
tests. And we're kind of stuck finding a way forward. It's easy enough to work
around for individual tests by just adding them to the test file (like Thomas
did nearby), but clearly that doesn't scale. Andrew wasn't happy with
additional vcregress commands. The fact that vcregress doesn't run tests in
parallel makes things take forever. And so it goes on.
> I looked through the Linux output. It looks to me like that does run
> the TAP tests for contrib. Unfortunately, the output is not in order
> and is also not labelled, so it's hard to tell what output goes with
> what contrib module. I named my test 001_basic.pl, but there are 12 of
> those already. I see that 13 copies of 001_basic.pl seem to have
> passed CI on Linux, so I guess the test ran and passed there. It seems
> like it would be an awfully good idea to mention the subdirectory name
> before each dump of output.
Yea, the current output is *awful*.
FWIW, the way it's hard to run tests the same way across platforms, the crappy
output etc was one of the motivations behind the meson effort. If you just
compare the output from both *nix and windows runs today with the meson
output, it's imo night and day:
https://cirrus-ci.com/task/5869668815601664?logs=check_world#L67
That's a recent run where I'd not properly mirrored 7c51b7f7cc0, leading to a
failure on windows. Though it'd be more intersting to see a run with a
failure.
If one wants one can also see the test output of individual tests (it's always
logged to a file). But I typically find that not useful for a 'general' test
run, too much output. In that case there's a nice list of failed tests at the
end:
Summary of Failures:
144/219 postgresql:tap+vacuumlo / vacuumlo/t/001_basic.pl ERROR 0.48s (exit
status255 or signal 127 SIGinvalid)
Ok: 218
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Greetings,
Andres Freund