Обсуждение: buildfarm + meson
Here's a progress report on adapting the buildfarm client to meson
There is a development branch where I'm working on the changes. They can be seen here:
<https://github.com/PGBuildFarm/client-code/compare/main...dev/meson>
On my Linux box (Fedora 37, where crake runs) I can get a complete run. There is work to do to make sure we pick up the right log files, and maybe adjust a module or two. I have adopted a design where instead of trying to know a lot about the testing regime the client needs to know a lot less. Instead, it gets meson to tell it the set of tests. I will probably work on enabling some sort of filter, but I think this makes things more future-proof. I have stuck with the design of making testing fairly fine-grained, so each suite runs separately.
On a Windows instance, fairly similar to what's running drongo, I can get a successful build with meson+VS2019, but I'm getting an error in the regression tests, which don't like setting lc_time to 'de_DE'. Not sure what's going on there.
meson apparently wants touch and cp installed, although I can't see why at first glance. For Windows I just copied them into the path from an msys2 installation.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On Wed, Feb 22, 2023 at 06:23:44PM -0500, Andrew Dunstan wrote: > On my Linux box (Fedora 37, where crake runs) I can get a complete run. > There is work to do to make sure we pick up the right log files, and maybe > adjust a module or two. I have adopted a design where instead of trying to > know a lot about the testing regime the client needs to know a lot less. > Instead, it gets meson to tell it the set of tests. I will probably work on > enabling some sort of filter, but I think this makes things more > future-proof. I have stuck with the design of making testing fairly > fine-grained, so each suite runs separately. Nice! > On a Windows instance, fairly similar to what's running drongo, I can get a > successful build with meson+VS2019, but I'm getting an error in the > regression tests, which don't like setting lc_time to 'de_DE'. Not sure > what's going on there. What's the regression issue? Some text-field ordering that ought to be enforced with a C collation? -- Michael
Вложения
Hi, On 2023-02-22 18:23:44 -0500, Andrew Dunstan wrote: > Here's a progress report on adapting the buildfarm client to meson > > There is a development branch where I'm working on the changes. They can be > seen here: > > <https://github.com/PGBuildFarm/client-code/compare/main...dev/meson> > > On my Linux box (Fedora 37, where crake runs) I can get a complete run. Nice! > There is work to do to make sure we pick up the right log files, and maybe > adjust a module or two. I have adopted a design where instead of trying to > know a lot about the testing regime the client needs to know a lot less. > Instead, it gets meson to tell it the set of tests. I will probably work on > enabling some sort of filter, but I think this makes things more > future-proof. I have stuck with the design of making testing fairly > fine-grained, so each suite runs separately. I don't understand why you'd want to run each suite separately. Serially executing the test takes way longer than doing so in parallel. Why would we want to enforce that? Particularly because with meson the tests log files and the failed tests can directly be correlated? And it should be easy to figure out which log files need to be kept, you can just skip the directories in testrun/ that contain test.success. > On a Windows instance, fairly similar to what's running drongo, I can get a > successful build with meson+VS2019, but I'm getting an error in the > regression tests, which don't like setting lc_time to 'de_DE'. Not sure > what's going on there. Huh, that's odd. > meson apparently wants touch and cp installed, although I can't see why at > first glance. For Windows I just copied them into the path from an msys2 > installation. Those should probably be fixed. Greetings, Andres Freund
On 2023-02-22 We 19:23, Michael Paquier wrote:
On a Windows instance, fairly similar to what's running drongo, I can get a successful build with meson+VS2019, but I'm getting an error in the regression tests, which don't like setting lc_time to 'de_DE'. Not sure what's going on there.What's the regression issue? Some text-field ordering that ought to be enforced with a C collation?
Here's the diff
diff -w -U3 C:/prog/bf/buildroot/HEAD/pgsql/src/test/regress/expected/collate.windows.win1252.out C:/prog/bf/buildroot/HEAD/pgsql.build/testrun/regress/regress/results/collate.windows.win1252.out --- C:/prog/bf/buildroot/HEAD/pgsql/src/test/regress/expected/collate.windows.win1252.out 2023-02-22 16:32:03.762370300 +0000 +++ C:/prog/bf/buildroot/HEAD/pgsql.build/testrun/regress/regress/results/collate.windows.win1252.out 2023-02-22 22:54:59.281395200 +0000 @@ -363,16 +363,17 @@ -- to_char SET lc_time TO 'de_DE'; +ERROR: invalid value for parameter "lc_time": "de_DE" SELECT to_char(date '2010-03-01', 'DD TMMON YYYY'); to_char ------------- - 01 MRZ 2010 + 01 MAR 2010 (1 row) SELECT to_char(date '2010-03-01', 'DD TMMON YYYY' COLLATE "de_DE"); to_char ------------- - 01 MRZ 2010 + 01 MAR 2010 (1 row) -- to_date
cheers andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2023-02-22 We 20:20, Andres Freund wrote:
There is work to do to make sure we pick up the right log files, and maybe adjust a module or two. I have adopted a design where instead of trying to know a lot about the testing regime the client needs to know a lot less. Instead, it gets meson to tell it the set of tests. I will probably work on enabling some sort of filter, but I think this makes things more future-proof. I have stuck with the design of making testing fairly fine-grained, so each suite runs separately.I don't understand why you'd want to run each suite separately. Serially executing the test takes way longer than doing so in parallel. Why would we want to enforce that? Particularly because with meson the tests log files and the failed tests can directly be correlated? And it should be easy to figure out which log files need to be kept, you can just skip the directories in testrun/ that contain test.success.
We can revisit that later. For now I'm more concerned with getting a working setup. The requirements of the buildfarm are a bit different from those of a developer, though. Running things in parallel can make things faster, but that can also increase the compute load. Also, running things serially makes it easier to report a failure stage that pinpoints the test that encountered an issue. But like I say we can come back to this.
On a Windows instance, fairly similar to what's running drongo, I can get a successful build with meson+VS2019, but I'm getting an error in the regression tests, which don't like setting lc_time to 'de_DE'. Not sure what's going on there.Huh, that's odd.
See my reply to Michael for details
meson apparently wants touch and cp installed, although I can't see why at first glance. For Windows I just copied them into the path from an msys2 installation.Those should probably be fixed.
Yeah. For touch I think we can probably just get rid of this line in the root meson.build:
touch = find_program('touch', native: true)
For cp there doesn't seem to be a formal requirement, but there is a recipe in src/common/unicode/meson.build that uses it, maybe that's what caused the failure. On Windows/msvc we could just use copy instead, I think.
I haven't experimented with any of this.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2023-02-23 06:27:23 -0500, Andrew Dunstan wrote:
>
> On 2023-02-22 We 20:20, Andres Freund wrote:
> >
> > > There is work to do to make sure we pick up the right log files, and maybe
> > > adjust a module or two. I have adopted a design where instead of trying to
> > > know a lot about the testing regime the client needs to know a lot less.
> > > Instead, it gets meson to tell it the set of tests. I will probably work on
> > > enabling some sort of filter, but I think this makes things more
> > > future-proof. I have stuck with the design of making testing fairly
> > > fine-grained, so each suite runs separately.
> > I don't understand why you'd want to run each suite separately. Serially
> > executing the test takes way longer than doing so in parallel. Why would we
> > want to enforce that?
> >
> > Particularly because with meson the tests log files and the failed tests can
> > directly be correlated? And it should be easy to figure out which log files
> > need to be kept, you can just skip the directories in testrun/ that contain
> > test.success.
> >
>
> We can revisit that later. For now I'm more concerned with getting a working
> setup.
My fear is that this ends up being entrenched in the design and hard to change
later.
> The requirements of the buildfarm are a bit different from those of a
> developer, though. Running things in parallel can make things faster, but
> that can also increase the compute load.
Sure, I'm not advocating to using a [high] concurrency by default.
> Also, running things serially makes it easier to report a failure stage that
> pinpoints the test that encountered an issue.
You're relying on running tests in a specific order. Instead you can also just
run tests in parallel and check test status in order and report the first
failed test in that order.
> But like I say we can come
> back to this.
>
> > > On a Windows instance, fairly similar to what's running drongo, I can get a
> > > successful build with meson+VS2019, but I'm getting an error in the
> > > regression tests, which don't like setting lc_time to 'de_DE'. Not sure
> > > what's going on there.
> > Huh, that's odd.
>
>
> See my reply to Michael for details
I suspect the issue might be related to this:
+ local %ENV = (PATH => $ENV{PATH}, PGUSER => $ENV{PGUSER});
+ @makeout=run_log("meson test --logbase checklog --print-errorlogs --no-rebuild -C $pgsql --suite setup
--suiteregress");
Greetings,
Andres Freund
On 2023-02-23 Th 10:58, Andres Freund wrote:
On a Windows instance, fairly similar to what's running drongo, I can get a successful build with meson+VS2019, but I'm getting an error in the regression tests, which don't like setting lc_time to 'de_DE'. Not sure what's going on there.Huh, that's odd.See my reply to Michael for detailsI suspect the issue might be related to this: + local %ENV = (PATH => $ENV{PATH}, PGUSER => $ENV{PGUSER}); + @makeout=run_log("meson test --logbase checklog --print-errorlogs --no-rebuild -C $pgsql --suite setup --suite regress");
I commented out the 'local %ENV' line and still got the error. I also got the same error running by hand.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2023-02-23 Th 16:12, Andrew Dunstan wrote:
On 2023-02-23 Th 10:58, Andres Freund wrote:On a Windows instance, fairly similar to what's running drongo, I can get a successful build with meson+VS2019, but I'm getting an error in the regression tests, which don't like setting lc_time to 'de_DE'. Not sure what's going on there.Huh, that's odd.See my reply to Michael for detailsI suspect the issue might be related to this: + local %ENV = (PATH => $ENV{PATH}, PGUSER => $ENV{PGUSER}); + @makeout=run_log("meson test --logbase checklog --print-errorlogs --no-rebuild -C $pgsql --suite setup --suite regress");
I commented out the 'local %ENV' line and still got the error. I also got the same error running by hand.
On drongo, this test isn't failing, and I think the reason is that it runs "make NO_LOCALE=1 check" so it never gets a database with win1252 encoding.
I'm going to try adding a win1252 test to drongo's locales.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On Fri, Feb 24, 2023 at 2:22 PM Andrew Dunstan <andrew@dunslane.net> wrote:
On drongo, this test isn't failing, and I think the reason is that it runs "make NO_LOCALE=1 check" so it never gets a database with win1252 encoding.
I'm going to try adding a win1252 test to drongo's locales.
What seems to be failing is the setlocale() for 'de_DE'. I haven't been able to reproduce it locally, but I've seen something similar reported for python [1].
As a workaround, can you please test "SET lc_time TO 'de-DE';"?
Regards,
Juan José Santamaría Flecha
On 2023-02-23 Th 10:58, Andres Freund wrote:
On 2023-02-23 06:27:23 -0500, Andrew Dunstan wrote:On 2023-02-22 We 20:20, Andres Freund wrote:There is work to do to make sure we pick up the right log files, and maybe adjust a module or two. I have adopted a design where instead of trying to know a lot about the testing regime the client needs to know a lot less. Instead, it gets meson to tell it the set of tests. I will probably work on enabling some sort of filter, but I think this makes things more future-proof. I have stuck with the design of making testing fairly fine-grained, so each suite runs separately.I don't understand why you'd want to run each suite separately. Serially executing the test takes way longer than doing so in parallel. Why would we want to enforce that? Particularly because with meson the tests log files and the failed tests can directly be correlated? And it should be easy to figure out which log files need to be kept, you can just skip the directories in testrun/ that contain test.success.We can revisit that later. For now I'm more concerned with getting a working setup.My fear is that this ends up being entrenched in the design and hard to change later.The requirements of the buildfarm are a bit different from those of a developer, though. Running things in parallel can make things faster, but that can also increase the compute load.Sure, I'm not advocating to using a [high] concurrency by default.
Perhaps the latest version will be more to your taste. This is now working on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. I do get a whole lot of annoying messages like this:
Unknown TAP version. The first line MUST be `TAP version <int>`. Assuming version 12.
Anyway, I think this is ready for any brave soul who wants to take it for a test run, not on a reporting animal just yet, though. To activate it you need the config to have 'using_meson => 1' and a meson_opts section - see the sample file. You can get the dev/meson version at <https://github.com/PGBuildFarm/client-code/archive/refs/heads/dev/meson.zip>
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi, On 2023-03-01 16:21:32 -0500, Andrew Dunstan wrote: > Perhaps the latest version will be more to your taste. I'll check it out. > This is now working > on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. > I do get a whole lot of annoying messages like this: > > Unknown TAP version. The first line MUST be `TAP version <int>`. Assuming > version 12. The newest minor version has fixed that, it was a misunderstanding about / imprecision in the tap 14 specification. Greetings, Andres Freund
On 2023-03-01 We 16:32, Andres Freund wrote:
This is now working on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. I do get a whole lot of annoying messages like this: Unknown TAP version. The first line MUST be `TAP version <int>`. Assuming version 12.The newest minor version has fixed that, it was a misunderstanding about / imprecision in the tap 14 specification.
Unfortunately, meson v 1.0.1 appears to be broken on Windows, I had to downgrade back to 1.0.0.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi On 2023-03-02 17:00:47 -0500, Andrew Dunstan wrote: > > On 2023-03-01 We 16:32, Andres Freund wrote: > > > This is now working > > > on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. > > > I do get a whole lot of annoying messages like this: > > > > > > Unknown TAP version. The first line MUST be `TAP version <int>`. Assuming > > > version 12. > > The newest minor version has fixed that, it was a misunderstanding about / > > imprecision in the tap 14 specification. > > > > Unfortunately, meson v 1.0.1 appears to be broken on Windows, I had to > downgrade back to 1.0.0. Is it possible that you're using a PG checkout from a few days ago? A hack I used was invalidated by 1.0.1, but I fixed that already. CI is running with 1.0.1: https://cirrus-ci.com/task/5806561726038016?logs=configure#L8 Greetings, Andres Freund
On 2023-03-02 Th 17:06, Andres Freund wrote:
Hi On 2023-03-02 17:00:47 -0500, Andrew Dunstan wrote:On 2023-03-01 We 16:32, Andres Freund wrote:This is now working on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. I do get a whole lot of annoying messages like this: Unknown TAP version. The first line MUST be `TAP version <int>`. Assuming version 12.The newest minor version has fixed that, it was a misunderstanding about / imprecision in the tap 14 specification.Unfortunately, meson v 1.0.1 appears to be broken on Windows, I had to downgrade back to 1.0.0.Is it possible that you're using a PG checkout from a few days ago? A hack I used was invalidated by 1.0.1, but I fixed that already. CI is running with 1.0.1: https://cirrus-ci.com/task/5806561726038016?logs=configure#L8
No, running against PG master tip. I'll get some details - it's not too hard to switch back and forth.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi, On 2023-03-01 13:32:58 -0800, Andres Freund wrote: > On 2023-03-01 16:21:32 -0500, Andrew Dunstan wrote: > > Perhaps the latest version will be more to your taste. > > I'll check it out. A simple conversion from an existing config failed with: Can't use an undefined value as an ARRAY reference at /home/bf/src/pgbuildfarm-client-meson/PGBuild/Modules/TestICU.pm line37. I disabled TestICU and was able to progress past that. ... piculet-meson:HEAD [19:12:48] setting up db cluster (C)... piculet-meson:HEAD [19:12:48] starting db (C)... piculet-meson:HEAD [19:12:48] running installcheck (C)... piculet-meson:HEAD [19:12:57] restarting db (C)... piculet-meson:HEAD [19:12:59] running meson misc installchecks (C) ... Branch: HEAD Stage delay_executionInstallCheck-C failed with status 1 The failures are like this: +ERROR: extension "dummy_index_am" is not available +DETAIL: Could not open extension control file "/home/bf/bf-build/piculet-meson/HEAD/inst/share/postgresql/extension/dummy_index_am.control":No such file or directory. +HINT: The extension must first be installed on the system where PostgreSQL is running. I assume this is in an interaction with b6a0d469cae. I think we need a install-test-modules or such that installs into the normal directory. Greetings, Andres Freund
On 2023-03-07 Tu 14:37, Andres Freund wrote:
Hi, On 2023-03-01 13:32:58 -0800, Andres Freund wrote:On 2023-03-01 16:21:32 -0500, Andrew Dunstan wrote:Perhaps the latest version will be more to your taste.I'll check it out.A simple conversion from an existing config failed with: Can't use an undefined value as an ARRAY reference at /home/bf/src/pgbuildfarm-client-meson/PGBuild/Modules/TestICU.pm line 37. I disabled TestICU and was able to progress past that.
Pushed a fix for that.
... piculet-meson:HEAD [19:12:48] setting up db cluster (C)... piculet-meson:HEAD [19:12:48] starting db (C)... piculet-meson:HEAD [19:12:48] running installcheck (C)... piculet-meson:HEAD [19:12:57] restarting db (C)... piculet-meson:HEAD [19:12:59] running meson misc installchecks (C) ... Branch: HEAD Stage delay_executionInstallCheck-C failed with status 1 The failures are like this: +ERROR: extension "dummy_index_am" is not available +DETAIL: Could not open extension control file "/home/bf/bf-build/piculet-meson/HEAD/inst/share/postgresql/extension/dummy_index_am.control": No such file or directory. +HINT: The extension must first be installed on the system where PostgreSQL is running. I assume this is in an interaction with b6a0d469cae. I think we need a install-test-modules or such that installs into the normal directory.
Exactly.
cheers
andrew
--
Andrew Dunstan EDB: https://www.enterprisedb.com
Hi, On 2023-03-07 15:47:54 -0500, Andrew Dunstan wrote: > On 2023-03-07 Tu 14:37, Andres Freund wrote: > > The failures are like this: > > > > +ERROR: extension "dummy_index_am" is not available > > +DETAIL: Could not open extension control file "/home/bf/bf-build/piculet-meson/HEAD/inst/share/postgresql/extension/dummy_index_am.control":No such file or directory. > > +HINT: The extension must first be installed on the system where PostgreSQL is running. > > > > I assume this is in an interaction with b6a0d469cae. > > > > > > I think we need a install-test-modules or such that installs into the normal > > directory. > > > > Exactly. Here's a prototype for that. It adds an install-test-files target, Because we want to install into a normal directory, I removed the necessary munging of the target paths from meson.build and moved it into install-test-files. I also added DESTDIR support, so that installing can redirect the directory if desired. That's used for the tmp_install/ installation now. I didn't like the number of arguments necessary for install_test_files, so I changed it to use --install target list of files which makes it easier to use for further directories, if/when we need them. Greetings, Andres Freund
Вложения
Hi,
On 2023-02-23 06:27:23 -0500, Andrew Dunstan wrote:
> Yeah. For touch I think we can probably just get rid of this line in the
> root meson.build:
>
> touch = find_program('touch', native: true)
Yep.
> For cp there doesn't seem to be a formal requirement, but there is a recipe
> in src/common/unicode/meson.build that uses it, maybe that's what caused the
> failure. On Windows/msvc we could just use copy instead, I think.
I don't know about using copy, it's very easy to get into trouble due to
interpreting forward slashes as options etc. I propose that for now we just
don't support update-unicode if cp isn't available - just as already not
available when wget isn't available.
Planning to apply something like the attached soon, unless somebody opposes
that plan.
Other unix tools we have a hard requirement on right now:
- sed - would be pretty easy to replace with something else
- tar, gzip - just for tests
I'm not sure it's worth working on not requiring those.
There's also flex, bison, perl, but those will stay a hard requirement for a
while longer... :)
Greetings,
Andres Freund
Вложения
On 2023-03-07 18:26:21 -0800, Andres Freund wrote:
> On 2023-02-23 06:27:23 -0500, Andrew Dunstan wrote:
> > Yeah. For touch I think we can probably just get rid of this line in the
> > root meson.build:
> >
> > touch = find_program('touch', native: true)
>
> Yep.
>
> > For cp there doesn't seem to be a formal requirement, but there is a recipe
> > in src/common/unicode/meson.build that uses it, maybe that's what caused the
> > failure. On Windows/msvc we could just use copy instead, I think.
>
> I don't know about using copy, it's very easy to get into trouble due to
> interpreting forward slashes as options etc. I propose that for now we just
> don't support update-unicode if cp isn't available - just as already not
> available when wget isn't available.
>
> Planning to apply something like the attached soon, unless somebody opposes
> that plan.
Done.
On 2023-03-07 Tu 20:29, Andres Freund wrote:
Hi, On 2023-03-07 15:47:54 -0500, Andrew Dunstan wrote:On 2023-03-07 Tu 14:37, Andres Freund wrote:The failures are like this: +ERROR: extension "dummy_index_am" is not available +DETAIL: Could not open extension control file "/home/bf/bf-build/piculet-meson/HEAD/inst/share/postgresql/extension/dummy_index_am.control": No such file or directory. +HINT: The extension must first be installed on the system where PostgreSQL is running. I assume this is in an interaction with b6a0d469cae. I think we need a install-test-modules or such that installs into the normal directory.Exactly.Here's a prototype for that. It adds an install-test-files target, Because we want to install into a normal directory, I removed the necessary munging of the target paths from meson.build and moved it into install-test-files. I also added DESTDIR support, so that installing can redirect the directory if desired. That's used for the tmp_install/ installation now. I didn't like the number of arguments necessary for install_test_files, so I changed it to use --install target list of files which makes it easier to use for further directories, if/when we need them.
So if I understand this right, the way to use this would be something like:
local $ENV{DESTDIR} = $installdir;
run_log("meson compile -C $pgsql install-test-files");
Is that right? I did that but it didn't work :-(
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi,
On Wed, 8 Mar 2023 at 16:57, Andrew Dunstan <andrew@dunslane.net> wrote:
> So if I understand this right, the way to use this would be something like:
>
>
> local $ENV{DESTDIR} = $installdir;
>
> run_log("meson compile -C $pgsql install-test-files");
>
>
> Is that right? I did that but it didn't work :-(
I think you shouldn't set DESTDIR to the $installdir. If DESTDIR is
set, it joins $DESTDIR and $install_dir(-Dprefix). So, when you run
local $ENV{DESTDIR} = $installdir;
run_log("meson compile -C $pgsql install-test-files");
it installs these files to the '$install_dir/$install_dir'.
Could you try only running 'run_log("meson compile -C $pgsql
install-test-files");' without setting DESTDIR, this could work.
Regards,
Nazir Bilal Yavuz
Microsoft
On 2023-03-08 We 08:57, Andrew Dunstan wrote:
On 2023-03-07 Tu 20:29, Andres Freund wrote:Hi, On 2023-03-07 15:47:54 -0500, Andrew Dunstan wrote:On 2023-03-07 Tu 14:37, Andres Freund wrote:The failures are like this: +ERROR: extension "dummy_index_am" is not available +DETAIL: Could not open extension control file "/home/bf/bf-build/piculet-meson/HEAD/inst/share/postgresql/extension/dummy_index_am.control": No such file or directory. +HINT: The extension must first be installed on the system where PostgreSQL is running. I assume this is in an interaction with b6a0d469cae. I think we need a install-test-modules or such that installs into the normal directory.Exactly.Here's a prototype for that. It adds an install-test-files target, Because we want to install into a normal directory, I removed the necessary munging of the target paths from meson.build and moved it into install-test-files. I also added DESTDIR support, so that installing can redirect the directory if desired. That's used for the tmp_install/ installation now. I didn't like the number of arguments necessary for install_test_files, so I changed it to use --install target list of files which makes it easier to use for further directories, if/when we need them.
So if I understand this right, the way to use this would be something like:
local $ENV{DESTDIR} = $installdir;
run_log("meson compile -C $pgsql install-test-files");
Is that right? I did that but it didn't work :-(
OK, tried without the `local` line and it worked, so let's push this.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi,
On 2023-03-08 09:41:57 -0500, Andrew Dunstan wrote:
> On 2023-03-08 We 08:57, Andrew Dunstan wrote:
> > On 2023-03-07 Tu 20:29, Andres Freund wrote:
> > > On 2023-03-07 15:47:54 -0500, Andrew Dunstan wrote:
> > > Here's a prototype for that.
> > >
> > > It adds an install-test-files target, Because we want to install into a normal
> > > directory, I removed the necessary munging of the target paths from
> > > meson.build and moved it into install-test-files. I also added DESTDIR
> > > support, so that installing can redirect the directory if desired. That's used
> > > for the tmp_install/ installation now.
> > >
> > > I didn't like the number of arguments necessary for install_test_files, so I
> > > changed it to use
> > >
> > > --install target list of files
> > >
> > > which makes it easier to use for further directories, if/when we need them.
> > >
> >
> > So if I understand this right, the way to use this would be something
> > like:
> >
> >
> > local $ENV{DESTDIR} = $installdir;
> >
> > run_log("meson compile -C $pgsql install-test-files");
> >
> >
> > Is that right? I did that but it didn't work :-(
Bilal's explanation of why that doesn't work was right. You'd only want to use
DESTDIR to install into somewhere other than the real install path.
> OK, tried without the `local` line and it worked, so let's push this.
Done. It's possible that we might some more refinement here, but I thought
it important to unblock the buildfarm work...
Greetings,
Andres Freund
Hi, On 2023-03-02 17:35:26 -0500, Andrew Dunstan wrote: > On 2023-03-02 Th 17:06, Andres Freund wrote: > > Hi > > > > On 2023-03-02 17:00:47 -0500, Andrew Dunstan wrote: > > > On 2023-03-01 We 16:32, Andres Freund wrote: > > > > > This is now working > > > > > on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. > > > > > I do get a whole lot of annoying messages like this: > > > > > > > > > > Unknown TAP version. The first line MUST be `TAP version <int>`. Assuming > > > > > version 12. > > > > The newest minor version has fixed that, it was a misunderstanding about / > > > > imprecision in the tap 14 specification. > > > > > > > Unfortunately, meson v 1.0.1 appears to be broken on Windows, I had to > > > downgrade back to 1.0.0. > > Is it possible that you're using a PG checkout from a few days ago? A > > hack I used was invalidated by 1.0.1, but I fixed that already. > > > > CI is running with 1.0.1: > > https://cirrus-ci.com/task/5806561726038016?logs=configure#L8 > > > > No, running against PG master tip. I'll get some details - it's not too hard > to switch back and forth. Any more details? Greetings, Andres Freund
On 2023-03-08 We 14:22, Andres Freund wrote:
Hi, On 2023-03-02 17:35:26 -0500, Andrew Dunstan wrote:On 2023-03-02 Th 17:06, Andres Freund wrote:Hi On 2023-03-02 17:00:47 -0500, Andrew Dunstan wrote:On 2023-03-01 We 16:32, Andres Freund wrote:This is now working on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. I do get a whole lot of annoying messages like this: Unknown TAP version. The first line MUST be `TAP version <int>`. Assuming version 12.The newest minor version has fixed that, it was a misunderstanding about / imprecision in the tap 14 specification.Unfortunately, meson v 1.0.1 appears to be broken on Windows, I had to downgrade back to 1.0.0.Is it possible that you're using a PG checkout from a few days ago? A hack I used was invalidated by 1.0.1, but I fixed that already. CI is running with 1.0.1: https://cirrus-ci.com/task/5806561726038016?logs=configure#L8No, running against PG master tip. I'll get some details - it's not too hard to switch back and forth.Any more details?
I was held up by difficulties even with meson 1.0.0 (the test modules stuff). Now I again have a clean build with meson 1.0.0 on Windows as a baseline I will get back to trying meson 1.0.1.
cheers
andrew
--Andrew Dunstan
EDB: https://www.enterprisedb.com
On 2023-03-08 We 17:23, Andrew Dunstan wrote:
On 2023-03-08 We 14:22, Andres Freund wrote:Hi, On 2023-03-02 17:35:26 -0500, Andrew Dunstan wrote:On 2023-03-02 Th 17:06, Andres Freund wrote:Hi On 2023-03-02 17:00:47 -0500, Andrew Dunstan wrote:On 2023-03-01 We 16:32, Andres Freund wrote:This is now working on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. I do get a whole lot of annoying messages like this: Unknown TAP version. The first line MUST be `TAP version <int>`. Assuming version 12.The newest minor version has fixed that, it was a misunderstanding about / imprecision in the tap 14 specification.Unfortunately, meson v 1.0.1 appears to be broken on Windows, I had to downgrade back to 1.0.0.Is it possible that you're using a PG checkout from a few days ago? A hack I used was invalidated by 1.0.1, but I fixed that already. CI is running with 1.0.1: https://cirrus-ci.com/task/5806561726038016?logs=configure#L8No, running against PG master tip. I'll get some details - it's not too hard to switch back and forth.Any more details?
I was held up by difficulties even with meson 1.0.0 (the test modules stuff). Now I again have a clean build with meson 1.0.0 on Windows as a baseline I will get back to trying meson 1.0.1.
OK, I have now got a clean run using meson 1.0.1 / MSVC. Not sure what made the difference. One change I did make was to stop using "--backend vs" and thus use the ninja backend even for MSVC. That proved necessary to run the new install-test-files target which failed miserably with "--backend vs". Not sure if we have it documented, but if not it should be that you need to use the ninja backend on all platforms.
At this stage I think I'm prepared to turn this loose on a couple of my buildfarm animals, and if nothing goes awry for the remainder of the month merge the dev/meson branch and push a new release.
There is still probably a little polishing to do, especially w.r.t. log file artefacts.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2023-03-09 Th 08:28, Andrew Dunstan wrote:
At this stage I think I'm prepared to turn this loose on a couple of my buildfarm animals, and if nothing goes awry for the remainder of the month merge the dev/meson branch and push a new release.
There is still probably a little polishing to do, especially w.r.t. log file artefacts.
A few things I've found:
. We don't appear to have an equivalent of the headerscheck and cpluspluscheck GNUmakefile targets
. I don't know how to build other docs targets (e.g. postgres-US.pdf)
. There appears to be some mismatch in database names (e.g. regression_dblink vs contrib_regression_dblink). That's going to cause some issues with the module that adjusts things for cross version upgrade.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi, On 2023-03-09 14:47:36 -0500, Andrew Dunstan wrote: > On 2023-03-09 Th 08:28, Andrew Dunstan wrote: > > At this stage I think I'm prepared to turn this loose on a couple of my > > buildfarm animals, and if nothing goes awry for the remainder of the > > month merge the dev/meson branch and push a new release. Cool! > > There is still probably a little polishing to do, especially w.r.t. log > > file artefacts. > A few things I've found: > > . We don't appear to have an equivalent of the headerscheck and > cpluspluscheck GNUmakefile targets Yes. I have a pending patch for it, but haven't yet cleaned it up sufficiently. The way headercheck/cpluspluscheck query information from Makefile.global is somewhat nasty. > . I don't know how to build other docs targets (e.g. postgres-US.pdf) There's an 'alldocs' target, or you can do ninja doc/src/sgml/postgres-US.pdf > . There appears to be some mismatch in database names (e.g. > regression_dblink vs contrib_regression_dblink). That's going to cause some > issues with the module that adjusts things for cross version upgrade. I guess we can try to do something about that, but the make situation is overly complicated. I don't really want to emulate having randomly differing database names just because a test is in contrib/ rather than src/. Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> On 2023-03-09 14:47:36 -0500, Andrew Dunstan wrote:
>> . There appears to be some mismatch in database names (e.g.
>> regression_dblink vs contrib_regression_dblink). That's going to cause some
>> issues with the module that adjusts things for cross version upgrade.
> I guess we can try to do something about that, but the make situation is
> overly complicated. I don't really want to emulate having randomly differing
> database names just because a test is in contrib/ rather than src/.
We could talk about adjusting the behavior on the make side instead,
perhaps, but something needs to be done there eventually.
Having said that, I'm not sure that the first meson-capable buildfarm
version needs to support cross-version-upgrade testing.
regards, tom lane
On 2023-03-09 Th 15:25, Tom Lane wrote:
Andres Freund <andres@anarazel.de> writes:On 2023-03-09 14:47:36 -0500, Andrew Dunstan wrote:. There appears to be some mismatch in database names (e.g. regression_dblink vs contrib_regression_dblink). That's going to cause some issues with the module that adjusts things for cross version upgrade.I guess we can try to do something about that, but the make situation is overly complicated. I don't really want to emulate having randomly differing database names just because a test is in contrib/ rather than src/.We could talk about adjusting the behavior on the make side instead, perhaps, but something needs to be done there eventually. Having said that, I'm not sure that the first meson-capable buildfarm version needs to support cross-version-upgrade testing.
Well, I want to store up as little future work as possible. This particular issue won't be much of a problem for several months until we branch the code, as we don't do database adjustments for a same version upgrade. At that stage I think a small modification to AdjustUpgrade.pm will do the trick. We just need to remember to do it.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2023-03-09 Th 14:47, Andrew Dunstan wrote:
On 2023-03-09 Th 08:28, Andrew Dunstan wrote:
At this stage I think I'm prepared to turn this loose on a couple of my buildfarm animals, and if nothing goes awry for the remainder of the month merge the dev/meson branch and push a new release.
There is still probably a little polishing to do, especially w.r.t. log file artefacts.
A few things I've found:
. We don't appear to have an equivalent of the headerscheck and cpluspluscheck GNUmakefile targets
. I don't know how to build other docs targets (e.g. postgres-US.pdf)
. There appears to be some mismatch in database names (e.g. regression_dblink vs contrib_regression_dblink). That's going to cause some issues with the module that adjusts things for cross version upgrade.
Another thing: the test for uuid.h is too strict. On Fedora 36 the OSSP header is in /usr/include, not /usr/include/ossp (I got around that for now by symlinking it, but obviously that's a nasty hack we can't ask people to do)
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi, On 2023-03-09 11:55:57 -0800, Andres Freund wrote: > On 2023-03-09 14:47:36 -0500, Andrew Dunstan wrote: > > On 2023-03-09 Th 08:28, Andrew Dunstan wrote: > > > At this stage I think I'm prepared to turn this loose on a couple of my > > > buildfarm animals, and if nothing goes awry for the remainder of the > > > month merge the dev/meson branch and push a new release. > > Cool! I moved a few of my animals to it to, so far no problems. The only other thing I noticed so far is that the status page doesn't yet know how to generate the right "flags", but that's fairly minor... Greetings, Andres Freund
Hi, On 2023-03-09 18:31:10 -0500, Andrew Dunstan wrote: > Another thing: the test for uuid.h is too strict. On Fedora 36 the OSSP > header is in /usr/include, not /usr/include/ossp (I got around that for now > by symlinking it, but obviously that's a nasty hack we can't ask people to > do) Yea, that was just wrong. It happened to work on debian and a few other OSs, but ossp's .pc puts whatever the right directory is into the include path. Pushed the fairly obvious fix. Greetings, Andres Freund
On 2023-03-10 Fr 18:05, Andres Freund wrote:
Hi, On 2023-03-09 11:55:57 -0800, Andres Freund wrote:On 2023-03-09 14:47:36 -0500, Andrew Dunstan wrote:On 2023-03-09 Th 08:28, Andrew Dunstan wrote:At this stage I think I'm prepared to turn this loose on a couple of my buildfarm animals, and if nothing goes awry for the remainder of the month merge the dev/meson branch and push a new release.Cool!I moved a few of my animals to it to, so far no problems. The only other thing I noticed so far is that the status page doesn't yet know how to generate the right "flags", but that's fairly minor...
The status page should be fixed now. Still a bit of work to do for the failures page.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2023-03-11 Sa 16:25, Andres Freund wrote:
Hi, On 2023-03-09 18:31:10 -0500, Andrew Dunstan wrote:Another thing: the test for uuid.h is too strict. On Fedora 36 the OSSP header is in /usr/include, not /usr/include/ossp (I got around that for now by symlinking it, but obviously that's a nasty hack we can't ask people to do)Yea, that was just wrong. It happened to work on debian and a few other OSs, but ossp's .pc puts whatever the right directory is into the include path. Pushed the fairly obvious fix.
Another issue: building plpython appears impossible on Windows because it's finding meson's own python:
Program python3 found: YES (C:\Program Files\Meson\meson.exe runpython)
Could not find Python3 library 'C:\\Program Files\\Meson\\libs\\python311.lib'
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi, On 2023-03-18 17:53:38 -0400, Andrew Dunstan wrote: > On 2023-03-11 Sa 16:25, Andres Freund wrote: > > Hi, > > > > On 2023-03-09 18:31:10 -0500, Andrew Dunstan wrote: > > > Another thing: the test for uuid.h is too strict. On Fedora 36 the OSSP > > > header is in /usr/include, not /usr/include/ossp (I got around that for now > > > by symlinking it, but obviously that's a nasty hack we can't ask people to > > > do) > > Yea, that was just wrong. It happened to work on debian and a few other OSs, > > but ossp's .pc puts whatever the right directory is into the include > > path. Pushed the fairly obvious fix. > > > Another issue: building plpython appears impossible on Windows because it's > finding meson's own python: > > > Program python3 found: YES (C:\Program Files\Meson\meson.exe runpython) > Could not find Python3 library 'C:\\Program > Files\\Meson\\libs\\python311.lib' Any more details - windows CI builds with python. What python do you want to use and where is it installed? Greetings, Andres Freund
On 2023-03-18 Sa 19:00, Andres Freund wrote:
Hi, On 2023-03-18 17:53:38 -0400, Andrew Dunstan wrote:On 2023-03-11 Sa 16:25, Andres Freund wrote:Hi, On 2023-03-09 18:31:10 -0500, Andrew Dunstan wrote:Another thing: the test for uuid.h is too strict. On Fedora 36 the OSSP header is in /usr/include, not /usr/include/ossp (I got around that for now by symlinking it, but obviously that's a nasty hack we can't ask people to do)Yea, that was just wrong. It happened to work on debian and a few other OSs, but ossp's .pc puts whatever the right directory is into the include path. Pushed the fairly obvious fix.Another issue: building plpython appears impossible on Windows because it's finding meson's own python: Program python3 found: YES (C:\Program Files\Meson\meson.exe runpython) Could not find Python3 library 'C:\\Program Files\\Meson\\libs\\python311.lib'Any more details - windows CI builds with python. What python do you want to use and where is it installed?
It's in c:/python37, which is at the front of the PATH. It fails as above if I add -Dplpython=enabled to the config.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2023-03-18 Sa 21:32, Andrew Dunstan wrote:
On 2023-03-18 Sa 19:00, Andres Freund wrote:Hi, On 2023-03-18 17:53:38 -0400, Andrew Dunstan wrote:On 2023-03-11 Sa 16:25, Andres Freund wrote:Hi, On 2023-03-09 18:31:10 -0500, Andrew Dunstan wrote:Another thing: the test for uuid.h is too strict. On Fedora 36 the OSSP header is in /usr/include, not /usr/include/ossp (I got around that for now by symlinking it, but obviously that's a nasty hack we can't ask people to do)Yea, that was just wrong. It happened to work on debian and a few other OSs, but ossp's .pc puts whatever the right directory is into the include path. Pushed the fairly obvious fix.Another issue: building plpython appears impossible on Windows because it's finding meson's own python: Program python3 found: YES (C:\Program Files\Meson\meson.exe runpython) Could not find Python3 library 'C:\\Program Files\\Meson\\libs\\python311.lib'Any more details - windows CI builds with python. What python do you want to use and where is it installed?
It's in c:/python37, which is at the front of the PATH. It fails as above if I add -Dplpython=enabled to the config.
Looks like the answer is not to install using the MSI installer, which provides its own Python, but to install meson and ninja into an existing python installation via pip. That's a bit sad, but manageable.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com