Обсуждение: List TAP test files in makefiles
meson.build files have to list TAP test files explicitly. Makefiles
have been relying on a t/*.pl wildcard. As a consequence, it is
traditional now that many developers who are primarily using make forget
to add any new TAP test files to the respective meson.build file.
The obvious solution is to also require the makefiles to list TAP files
explicitly, which is what I'm proposing here.
To list the test files, I'm re-using the make variable TAP_TESTS that is
already in use by extensions. Extensions up to now would just write
something like
TAP_TESTS = 1
Starting with this patch, they would have to write
TAP_TESTS = t/001_foo.pl t/002_bar.pl
This makes it somewhat backward compatible. (For example, with the
latter formulation, PG<=18 would still read it as enabling TAP tests,
but it would ignore the actual file list.)
(I have considered other variants: We already have the make variable
PROVE_TESTS that you can use to override the list of tests from the make
command line. But if you use that one to list the tests in an extension
makefile, then that extension would have to use both PROVE_TESTS and
TAP_TESTS (to enable TAP tests overall), which seems pretty confusing.
I think the solution I ended up with is compact and intuitive.)
In the attached patch, I have arranged it so that the interesting
changes are in the first three files, the rest is just mechanical.
Вложения
I agree that this difference between meson and autotools builds is kind of problematic. > On 23 Aug 2025, at 11:09, Peter Eisentraut <peter@eisentraut.org> wrote: > > The obvious solution is to also require the makefiles to list TAP files explicitly, which is what I'm proposing here. What is the downside of the approach where meson uses t/*.pl wildcard? Best regards, Andrey Borodin.
Hi, On 2025-08-23 11:57:37 +0500, Andrey Borodin wrote: > I agree that this difference between meson and autotools builds is kind of problematic. > > > On 23 Aug 2025, at 11:09, Peter Eisentraut <peter@eisentraut.org> wrote: > > > > The obvious solution is to also require the makefiles to list TAP files explicitly, which is what I'm proposing here. > > What is the downside of the approach where meson uses t/*.pl wildcard? In meson you can't do wildcards at "configure" time, since wildcards do not allow reliable detection of when re-configure is needed. You could do it by running prove to run the tap tests, like make does, but that would considerably slow down the tests, as prove has either no parallelism or independent parallelism from the make's (or ninjas). Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> On 2025-08-23 11:57:37 +0500, Andrey Borodin wrote:
>> What is the downside of the approach where meson uses t/*.pl wildcard?
> In meson you can't do wildcards at "configure" time, since wildcards do not
> allow reliable detection of when re-configure is needed.
But ... what "re-configuration" is needed when adding a new test file?
This proposed fix adds more manual maintenance effort, and I really
doubt that it will do much to solve the ostensible problem that
people forget to update the meson.build files. We should be striving
to make the meson system as easy to use as autoconf/make, not trying
to attain parity by making the latter experience worse.
regards, tom lane
Hi, On 2025-08-23 10:17:59 -0400, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: > > On 2025-08-23 11:57:37 +0500, Andrey Borodin wrote: > >> What is the downside of the approach where meson uses t/*.pl wildcard? > > > In meson you can't do wildcards at "configure" time, since wildcards do not > > allow reliable detection of when re-configure is needed. > > But ... what "re-configuration" is needed when adding a new test file? In meson, not autoconf/make. The test runner doesn't parse the meson.build files, that's done once when running configure. After that there's a table of tests that need to be run that gets used when you type ninja/meson test. > We should be striving to make the meson system as easy to use as > autoconf/make, not trying to attain parity by making the latter experience > worse. FWIW, I find the autoconf/make test run experience completely unusable. It literally is made me embark on getting away from it. I don't understand how people stand it. Greetings, Andres Freund
On 2025-08-23 Sa 10:38 AM, Andres Freund wrote: > Hi, > > On 2025-08-23 10:17:59 -0400, Tom Lane wrote: > >> We should be striving to make the meson system as easy to use as >> autoconf/make, not trying to attain parity by making the latter experience >> worse. > FWIW, I find the autoconf/make test run experience completely unusable. It > literally is made me embark on getting away from it. I don't understand how > people stand it. > Honestly, they both have annoyances and points where it's easy to get into the weeds. There are definitely cases where meson has left me scratching my head. But I think this is really analogous to adding a new object file. That needs to be added to the Makefile as well as the meson.build file, and I don't see why tests shouldn't work the same. That might be slightly annoying, but it would be way down on my list. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Andres Freund <andres@anarazel.de> writes: > FWIW, I find the autoconf/make test run experience completely unusable. It > literally is made me embark on getting away from it. I don't understand how > people stand it. Interesting perspective, because from where I sit the testing aspect is the weakest part of our meson setup: it seems to be very much all-or-nothing, whereas the makefiles allow you to run any particular test suite you want. Maybe this is just a lack of documentation? There is still not a word in https://www.postgresql.org/docs/devel/regress-run.html about how to accomplish any of those effects under meson. I find the lack of any equivalent to installcheck to be particularly distressing, since it means you can't repeat a test without paying for "make temp-install" every time. Admittedly that doesn't work for a test that needs particular non-default installation settings, but that's a minority of our tests. regards, tom lane
Hi, On 2025-08-23 12:11:51 -0400, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: > > FWIW, I find the autoconf/make test run experience completely unusable. It > > literally is made me embark on getting away from it. I don't understand how > > people stand it. > > Interesting perspective, because from where I sit the testing aspect > is the weakest part of our meson setup: it seems to be very much > all-or-nothing, whereas the makefiles allow you to run any particular > test suite you want. Meson definitely allows to run just some of the test. You can list all tests with meson test --list And you can run testsuites (like all the recovery tests) with meson test --suite recovery or individual tests with meson test test_misc/003_check_guc > Maybe this is just a lack of documentation? Yea. Partially because that's generic information about meson (i.e. not postgres specific stuff) and partially because there were some doc patches that we ended up not finding agreement upon approach-wise. > I find the lack of any equivalent to installcheck to be > particularly distressing, since it means you can't repeat > a test without paying for "make temp-install" every time. If you just want to skip the install (reusing the prior install), you can do that with meson test --no-suite setup That's basically the equivalent of NO_TEMP_INSTALL (which isn't documented either :( ) > Admittedly that doesn't work for a test that needs particular > non-default installation settings, but that's a minority > of our tests. You can run all tests against a running postgres by doing meson test --setup running Unless somebody else volunteers (please!), I guess I should write up a patch... Greetings, Andres Freund
On 2025-Aug-25, Andres Freund wrote: > Unless somebody else volunteers (please!), I guess I should write up a > patch... Maybe it doesn't have to be a patch -- we have some info on command lines to use for testing at https://wiki.postgresql.org/wiki/Meson#Test_related_commands which can perhaps be updated and filled in with additional content. There may be other outdated things in that page, which perhaps should be updated. For instance, I would rip out the entire "Why and What" section, which feels no longer relevant. I'm a bit short of time at present, but in a few weeks I can give this a look. I'm happy if somebody else does it though. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/ Si no sabes adonde vas, es muy probable que acabes en otra parte.
On 23.08.25 08:09, Peter Eisentraut wrote: > meson.build files have to list TAP test files explicitly. Makefiles > have been relying on a t/*.pl wildcard. As a consequence, it is > traditional now that many developers who are primarily using make forget > to add any new TAP test files to the respective meson.build file. > > The obvious solution is to also require the makefiles to list TAP files > explicitly, which is what I'm proposing here. > > To list the test files, I'm re-using the make variable TAP_TESTS that is > already in use by extensions. Extensions up to now would just write > something like > > TAP_TESTS = 1 > > Starting with this patch, they would have to write > > TAP_TESTS = t/001_foo.pl t/002_bar.pl > > This makes it somewhat backward compatible. (For example, with the > latter formulation, PG<=18 would still read it as enabling TAP tests, > but it would ignore the actual file list.) > > (I have considered other variants: We already have the make variable > PROVE_TESTS that you can use to override the list of tests from the make > command line. But if you use that one to list the tests in an extension > makefile, then that extension would have to use both PROVE_TESTS and > TAP_TESTS (to enable TAP tests overall), which seems pretty confusing. I > think the solution I ended up with is compact and intuitive.) > > In the attached patch, I have arranged it so that the interesting > changes are in the first three files, the rest is just mechanical. In the meantime, here is a new version to adapt to the addition of TAP tests for libpq-oauth.