Обсуждение: controlling meson's parallelism (and some whining)
Hi, Twice now, I've had 'meson test' fail because it tried to start too many copies of the server at the same time. In the server log, I get the complaint about needing to raise SHMMNI. This is a macos machine, with kern.sysv.shmmni=32. The obvious fix to this is to just tell 'meson test' how many processes I'd like it to run. I thought maybe I could just do 'meson -j8 test' but that does not work, because the option is --num-processes and has no short version. Even typing -j8 every time would be kind of annoying; typing --num-processes 8 every time is ridiculously verbose. My next thought was that there ought to be some environmental variable that I could set to control this behavior. But I can't find a list of environment variables that affect meson behavior anywhere. I guess the authors don't believe in environment variable as a control mechanism. Or, at the risk of sounding a bit testy, maybe their documentation just isn't quite up to par. It's not that hard to find lists of options for particular subcommands, either from the tool itself or on the web site. But unlike git, where you can do something like 'man git-checkout' and actually get more information than the command help itself provides, there are no man pages for the main subcommands, and I can't really find any good documentation on the web site either. Knowing that a certain subcommand has a flag called --pkgconfig.relocatable or that some other command has a flag called --cross-file CROSS_FILE whose argument is, and I quote, a "File describing cross compilation environment," is not good enough. So my questions are: 1. Is there some better way to control testing parallelism than specifying --num-processes N every single time? 2. Is there better documentation somewhere? Thanks, -- Robert Haas EDB: http://www.enterprisedb.com
Hi, On 2023-10-19 13:44:20 -0400, Robert Haas wrote: > Twice now, I've had 'meson test' fail because it tried to start too > many copies of the server at the same time. In the server log, I get > the complaint about needing to raise SHMMNI. This is a macos machine, > with kern.sysv.shmmni=32. Hm. Did you not run into simmilar issues with make check-world? I found the concurrency of that to be even more variable over a run. But perhaps there's something else wrong here? Perhaps we should deal with this in Cluster.pm to some degree? Controlling this from the level of meson (or make/prove for that matter) doesn't really work well, because different tests start differently many postgres instances. How many cores does your machine have? I've run the tests in a loop on my m1 mac mini in the past without running into this issue. It has "only" 8 cores though, whereas I infer, from you mentioning -j8, that you have more cores? > The obvious fix to this is to just tell 'meson test' how many processes I'd > like it to run. I thought maybe I could just do 'meson -j8 test' but that > does not work, because the option is --num-processes and has no short > version. Even typing -j8 every time would be kind of annoying; typing > --num-processes 8 every time is ridiculously verbose. I've also wondered why there's no support for -j, maybe we should open an issue... > My next thought was that there ought to be some environmental variable > that I could set to control this behavior. But I can't find a list of > environment variables that affect meson behavior anywhere. I guess the > authors don't believe in environment variable as a control mechanism. They indeed do not like them - but there is one in this case: MESON_TESTTHREADS There's even documentation for it: https://mesonbuild.com/Unit-tests.html#parallelism > Or, at the risk of sounding a bit testy, maybe their documentation > just isn't quite up to par. It's not that hard to find lists of > options for particular subcommands, either from the tool itself or on > the web site. But unlike git, where you can do something like 'man > git-checkout' and actually get more information than the command help > itself provides, there are no man pages for the main subcommands, and > I can't really find any good documentation on the web site either. > Knowing that a certain subcommand has a flag called > --pkgconfig.relocatable or that some other command has a flag called > --cross-file CROSS_FILE whose argument is, and I quote, a "File > describing cross compilation environment," is not good enough. I agree that meson's documentation is of, let's say, varying quality. But https://mesonbuild.com/Commands.html#test does link to https://mesonbuild.com/Unit-tests.html which in turn has the bit about MESON_TESTTHREADS I do agree that it'd be nice if the online docs were converted to command specific manpages... Greetings, Andres Freund
On Thu Oct 19, 2023 at 12:44 PM CDT, Robert Haas wrote: > The obvious fix to this is to just tell 'meson test' how many > processes I'd like it to run. I thought maybe I could just do 'meson > -j8 test' but that does not work, because the option is > --num-processes and has no short version. Even typing -j8 every time > would be kind of annoying; typing --num-processes 8 every time is > ridiculously verbose. I submitted a patch[0] to Meson to add -j. [0]: https://github.com/mesonbuild/meson/pull/12403 -- Tristan Partin Neon (https://neon.tech)
On Fri Oct 20, 2023 at 11:22 AM CDT, Tristan Partin wrote: > On Thu Oct 19, 2023 at 12:44 PM CDT, Robert Haas wrote: > > The obvious fix to this is to just tell 'meson test' how many > > processes I'd like it to run. I thought maybe I could just do 'meson > > -j8 test' but that does not work, because the option is > > --num-processes and has no short version. Even typing -j8 every time > > would be kind of annoying; typing --num-processes 8 every time is > > ridiculously verbose. > > I submitted a patch[0] to Meson to add -j. > > [0]: https://github.com/mesonbuild/meson/pull/12403 You will see this in the 1.3.0 release which will be happening soon™️. -- Tristan Partin Neon (https://neon.tech)
On Thu, Oct 19, 2023 at 6:09 PM Andres Freund <andres@anarazel.de> wrote: > Hm. Did you not run into simmilar issues with make check-world? I found the > concurrency of that to be even more variable over a run. I did not, but I didn't generally run that in parallel, either, mostly for fear of being unable to see failures properly in the output. I used -j8 when building, though. > But perhaps there's something else wrong here? Perhaps we should deal with > this in Cluster.pm to some degree? Controlling this from the level of meson > (or make/prove for that matter) doesn't really work well, because different > tests start differently many postgres instances. I'm not sure, but I'm open to however anybody would like to improve things. > How many cores does your machine have? I've run the tests in a loop on my m1 > mac mini in the past without running into this issue. It has "only" 8 cores > though, whereas I infer, from you mentioning -j8, that you have more cores? System Information shows "Total Number of Cores: 8" but sysctl hw.ncpu returns 16. No real idea what is fastest, I haven't been real scientific about choosing values for -j. > > My next thought was that there ought to be some environmental variable > > that I could set to control this behavior. But I can't find a list of > > environment variables that affect meson behavior anywhere. I guess the > > authors don't believe in environment variable as a control mechanism. > > They indeed do not like them - but there is one in this > case: MESON_TESTTHREADS > > There's even documentation for it: https://mesonbuild.com/Unit-tests.html#parallelism I mean, I probably glanced at that page at some point, but it's hardly obvious that there's a mention of an environment variable buried somewhere in the middle of the page. Most of the code you see looking at the page is Python, and the other environment variables mentioned seem to be ones that it sets, rather than ones that you can set. They really ought to have a documentation page somewhere that lists all of the environment variables that the user can set, and maybe another one that lists all the ones that the tool itself sets before running subprocesses. You can't expect people to navigate through every page of the documentation and read every word on the page carefully to find stuff like this. -- Robert Haas EDB: http://www.enterprisedb.com
On Fri, Oct 20, 2023 at 1:08 PM Tristan Partin <tristan@neon.tech> wrote: > You will see this in the 1.3.0 release which will be happening soon™️. Cool, thanks! -- Robert Haas EDB: http://www.enterprisedb.com