Обсуждение: test runner (was Re: SQL-standard function body)
Hi, This started out as a reply to https://postgr.es/m/20210408170802.GA9392%40alvherre.pgsql but it's independent enough to just start a new thread... On 2021-04-08 13:08:02 -0400, Alvaro Herrera wrote: > Yes, coverage.pg.org runs "make check-world". > > Maybe it would make sense to change that script, so that it runs the > buildfarm's run_build.pl script instead of "make check-world". That > would make coverage.pg.org report what the buildfarm actually tests ... > it would have made this problem a bit more obvious. We desperately need to unify the different test run environments we have. I did spent some time trying to do that, and ended up with it being hard to do in a good way in the make / msvc environment. Not sure that I took the right path, but I end up doing experimental port of the buildsystem meson - which has a builtin test runner (working on all platforms...). andres@awork3:/tmp$ ccache --clear andres@awork3:/tmp$ ~/src/meson/meson.py setup ~/src/postgresql /tmp/pg-meson --prefix=/tmp/pg-meson-install The Meson build system Version: 0.57.999 Source dir: /home/andres/src/postgresql Build dir: /tmp/pg-meson Build type: native build Project name: postgresql Project version: 14devel ... Header <unistd.h> has symbol "fdatasync" : YES Header <fcntl.h> has symbol "F_FULLSYNC" : NO Checking for alignment of "short" : 2 Checking for alignment of "int" : 4 ... Configuring pg_config_ext.h using configuration Configuring pg_config.h using configuration Configuring pg_config_paths.h using configuration Program sed found: YES (/usr/bin/sed) Build targets in project: 116 Found ninja-1.10.1 at /usr/bin/ninja ... andres@awork3:/tmp/pg-meson$ time ninja [10/1235] Generating snowball_create with a custom command Generating tsearch script............................... [41/1235] Generating generated_catalog_headers with a custom command [1235/1235] Linking target contrib/test_decoding/test_decoding.so real 0m10.752s user 3m47.020s sys 0m50.281s ... andres@awork3:/tmp/pg-meson$ time ninja [1/1] Generating test clean with a custom command real 0m0.085s user 0m0.068s sys 0m0.016s ... andres@awork3:/tmp/pg-meson$ time ~/src/meson/meson.py install --quiet ninja: Entering directory `.' real 0m0.541s user 0m0.412s sys 0m0.130s ... andres@awork3:/tmp/pg-meson$ ninja test [1/2] Running all tests. 1/74 postgresql:setup / temp_install OK 0.52s 2/74 postgresql:setup / cleanup_old OK 0.01s 3/74 postgresql:tap+pg_archivecleanup / pg_archivecleanup/t/010_pg_archivecleanup.pl OK 0.29s 42 subtests passed 4/74 postgresql:tap+pg_checksums / pg_checksums/t/001_basic.pl OK 0.27s 8 subtests passed 5/74 postgresql:tap+pg_config / pg_config/t/001_pg_config.pl OK 0.26s 20 subtests passed ... 68/74 postgresql:tap+pg_dump / pg_dump/t/002_pg_dump.pl OK 28.26s 6408 subtestspassed ... 74/74 postgresql:isolation / pg_isolation_regress OK 114.91s Ok: 74 Expected Fail: 0 Fail: 0 Unexpected Pass: 0 Skipped: 0 Timeout: 0 Full log written to /tmp/pg-meson/meson-logs/testlog.txt And in cases of failures it'll show the failure when it happens (including the command to rerun just that test, without the harness in between), and then a summary at the end: 61/74 postgresql:tap+pg_verifybackup / pg_verifybackup/t/003_corruption.pl OK 10.65s 44 subtests passed 49/74 postgresql:tap+recovery / recovery/t/019_replslot_limit.pl ERROR 7.53s exit status 1 >>> MALLOC_PERTURB_=16 PATH=/tmp/pg-meson/tmp_install///usr/local/bin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin PG_REGRESS=/tmp/pg-meson/src/test/regress/pg_regressREGRESS_SHLIB=/tmp/pg-meson/src/test/regress/regress.so LD_LIBRARY_PATH=/tmp/pg-meson/tmp_install///usr/local/lib/x86_64-linux-gnu /home/andres/src/postgresql/src/tools/testwrap/tmp/pg-meson recovery t/019_replslot_limit.pl perl -I /home/andres/src/postgresql/src/test/perl-I /home/andres/src/postgresql/src/test/recovery /home/andres/src/postgresql/src/test/recovery/t/019_replslot_limit.pl ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― stderr: # Failed test 'check that required WAL segments are still available' # at /home/andres/src/postgresql/src/test/recovery/t/019_replslot_limit.pl line 168. # Looks like you failed 1 test of 14. (test program exited with status code 1) ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― 62/74 postgresql:tap+pg_basebackup / pg_basebackup/t/010_dump_connstr.pl OK 11.59s 14 subtests passed ... 74/74 postgresql:isolation / pg_isolation_regress OK 112.26s Summary of Failures: 49/74 postgresql:tap+recovery / recovery/t/019_replslot_limit.pl ERROR 7.53s exitstatus 1 Ok: 73 Expected Fail: 0 Fail: 1 Unexpected Pass: 0 Skipped: 0 Timeout: 0 Full log written to /tmp/pg-meson/meson-logs/testlog.txt FAILED: meson-test /usr/bin/python3 -u /home/andres/src/meson/meson.py test --no-rebuild --print-errorlogs ninja: build stopped: subcommand failed. It's hard to convey just how much nicer it is to see a progress report during the test, see the failing tests at the end, without needing to wade through reams of log output. The output of the individual tests is in testlog.txt referenced above. One can get a list of tests and then also just run subsets of them: andres@awork3:/tmp/pg-meson$ ~/src/meson/meson.py test --list postgresql:setup / temp_install postgresql:setup / cleanup_old postgresql:isolation / pg_isolation_regress postgresql:regress / pg_regress postgresql:tap+initdb / initdb/t/001_initdb.pl Can run "suites" of tests: ~/src/meson/meson.py test --suite setup --suite recovery Can run individual tests: ~/src/meson/meson.py test recovery/t/008_fsm_truncation.pl Obviously all very far from being ready, but this seemed like a good enough excuse to mention it ;) Greetings, Andres Freund
On 2021-04-08 10:50:39 -0700, Andres Freund wrote: > It's hard to convey just how much nicer it is to see a progress report > during the test, see the failing tests at the end, without needing to > wade through reams of log output. The output of the individual tests is > in testlog.txt referenced above. https://anarazel.de/public/t/pg-meson-test-screencap-2021-04-08_10.58.26.mkv Greetings, Andres Freund
On Thu, Apr 08, 2021 at 10:50:39AM -0700, Andres Freund wrote: > Obviously all very far from being ready, but this seemed like a good > enough excuse to mention it ;) This is nice. Are there any parallelism capabilities? -- Michael
Вложения
Hi, On 2021-04-09 08:39:46 +0900, Michael Paquier wrote: > On Thu, Apr 08, 2021 at 10:50:39AM -0700, Andres Freund wrote: > > Obviously all very far from being ready, but this seemed like a good > > enough excuse to mention it ;) > > This is nice. Are there any parallelism capabilities? Yes. It defaults to number-of-cores processes, but obviously can also be specified explicitly. One very nice part about it is that it'd work largely the same on windows (which has practically unusable testing right now). It probably doesn't yet, because I just tried to get it build and run tests at all, but it shouldn't be a lot of additional work. Greetings, Andres Freund
> This is nice. Are there any parallelism capabilities?
Yes. It defaults to number-of-cores processes, but obviously can also be
specified explicitly. One very nice part about it is that it'd work
largely the same on windows (which has practically unusable testing
right now). It probably doesn't yet, because I just tried to get it
build and run tests at all, but it shouldn't be a lot of additional
work.
The pidgin developers speak very highly of meson, for the same reasons already mentioned in this thread.