tests and meson - test names and file locations

Hi,

One of the things motivating me to work on the meson conversion is the ability
to run tests in an easily understandable way. Meson has a testrunner that
both allows to run all tests at once, and run subsets of tests.


= Test and testsuite naming =

Each test has a unique name, and 0-n labels (called 'suites'). One can run
tests by their name, or select tests by suites.

The way I've organized it so far is that tests are named like this:

main/pg_regress
main/isolation
recovery/t/018_wal_optimize.pl
pg_prewarm/t/001_basic.pl
test_shm_mq/pg_regress
psql/t/001_basic.pl

we could also name tests by their full path, but that makes the display very
unwieldy.


At the moment there's three suites differentiating by the type of test:
'pg_regress', 'isolation' and 'tap'. There's also a separate "axis" of suites,
describing what's being tested, e.g. 'main', 'test_decoding', 'recovery' etc.

That currently works out to each test having two suites, although I've
wondered about adding a 'contrib' suite as well.


Perhaps the pg_regress suite should just be named 'regress'? And perhaps the
t/ in the tap tests is superfluous - the display is already fairly wide?



= Log and Data locations =

To make things like the selection of log files for a specific test easier,
I've so far set it up so that test data and logs are stored in a separate
directory from the sources.

testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...>

The runner now creates a test.start at the start of a test and either
test.success or test.failure at the end. That should make it pretty easy for
e.g. the buildfarm and CI to make the logs for a failed test easily
accessible. I've spent far too much time going through the ~hundred logs in
src/test/recovery/ that the buildfarm displays as one thing.


I really like having all the test data separately from the sources, but I get
that that's not what we've done so far. It's doable to just mirror the current
choice, but I don't think we should. But I won't push too hard against keeping
things the same.


I do wonder if we should put test data and log files in a separate directory
tree, but that'd be a bit more work probably.


Any comments on the above?


= Example outputs =

Here's an example output that you mostly should be able to make sense of now:

$ m test --print-errorlogs
ninja: Entering directory `/tmp/meson'
ninja: no work to do.
  1/242 postgresql:setup / tmp_install                                                              OK
0.33s
  2/242 postgresql:tap+pg_upgrade / pg_upgrade/t/001_basic.pl                                       OK
0.35s  8 subtests passed
 
  3/242 postgresql:tap+recovery / recovery/t/011_crash_recovery.pl                                  OK
2.67s  3 subtests passed
 
  4/242 postgresql:tap+recovery / recovery/t/013_crash_restart.pl                                   OK
2.91s  18 subtests passed
 
  5/242 postgresql:tap+recovery / recovery/t/014_unlogged_reinit.pl                                 OK
3.01s  23 subtests passed
 
  6/242 postgresql:tap+recovery / recovery/t/022_crash_temp_files.pl                                OK
3.16s  11 subtests passed
 
  7/242 postgresql:tap+recovery / recovery/t/016_min_consistency.pl                                 OK
3.43s  1 subtests passed
 
  8/242 postgresql:tap+recovery / recovery/t/021_row_visibility.pl                                  OK
3.46s  10 subtests passed
 
  9/242 postgresql:isolation+tcn / tcn/isolation                                                    OK
3.42s
 10/242 postgresql:tap+recovery / recovery/t/023_pitr_prepared_xact.pl                              OK
3.63s  1 subtests passed
 
...
241/242 postgresql:isolation+main / main/isolation                                                  OK
46.69s
242/242 postgresql:tap+pg_upgrade / pg_upgrade/t/002_pg_upgrade.pl                                  OK
57.00s  13 subtests passed
 

Ok:                 242
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /tmp/meson/meson-logs/testlog.txt


The 'postgresql' is because meson supports subprojects (both to provide
dependencies if needed, and "real" subprojects), and their tests can be run at
once.


If a test fails it'll show the error output at the time of test:

39/242 postgresql:pg_regress+cube / cube/pg_regress                                                FAIL
3.74s  exit status 1
 
>>> REGRESS_SHLIB=/tmp/meson/src/test/regress/regress.so MALLOC_PERTURB_=44 PG_TEST_EXTRA='kerberos ldap ssl'
PG_REGRESS=/tmp/meson/src/test/regress/pg_regress
PATH=/tmp/meson/tmp_install/tmp/meson-install/bin:/tmp/meson/contrib/cube:/home/andres/bin/perl5/bin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/games
/usr/bin/python3/home/andres/src/postgresql/src/tools/testwrap --srcdir /home/andres/src/postgresql/contrib/cube
--basedir/tmp/meson --builddir /tmp/meson/contrib/cube --testgroup cube --testname pg_regress
/tmp/meson/src/test/regress/pg_regress--temp-instance /tmp/meson/testrun/cube/pg_regress/tmp_check --inputdir
/home/andres/src/postgresql/contrib/cube--expecteddir /home/andres/src/postgresql/contrib/cube --outputdir
/tmp/meson/testrun/cube/pg_regress--bindir '' --dlpath /tmp/meson/contrib/cube --max-concurrent-tests=20 --port=40012
cubecube_sci
 
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
# executing test in /tmp/meson/testrun/cube/pg_regress group cube test pg_regress, builddir /tmp/meson/contrib/cube
============== creating temporary instance            ==============
============== initializing database system           ==============
============== starting postmaster                    ==============
running on port 40012 with PID 354981
============== creating database "regression"         ==============
CREATE DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
============== running regression test queries        ==============
test cube                         ... FAILED      418 ms
test cube_sci                     ... ok           16 ms
============== shutting down postmaster               ==============

======================
 1 of 2 tests failed.
======================

The differences that caused some tests to fail can be viewed in the
file "/tmp/meson/testrun/cube/pg_regress/regression.diffs".  A copy of the test summary that you see
above is saved in the file "/tmp/meson/testrun/cube/pg_regress/regression.out".

# test failed

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

 40/242 postgresql:pg_regress+citext / citext/pg_regress                                            OK
3.77s
...

A list of failed tests is listed at the end of the test:


Summary of Failures:

 39/242 postgresql:pg_regress+cube / cube/pg_regress                                       FAIL              3.74s
exitstatus 1
 

Ok:                 241
Expected Fail:      0
Fail:               1
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /tmp/meson/meson-logs/testlog.txt


Greetings,

Andres Freund