Обсуждение: make install getting slower
Hi, best of three of time make -s install INSTALL=/bin/true 9.3: 1.53s 9.4: 1.55s 9.5: 1.90s 9.6: 2.10s 10: 2.45s 11: 2.81s master: 2.99s Obviously a part of that can be explained by the tree getting bigger, but that can't be all of it. It also gets a bit worse if LLVM is enabled (due to the bitcode files getting installed), but that's "just" 0.3s for me. We should optimize that for fewer install invocations / using cp, but that seems seperate. I think this also has gotten more pronounced with newer OS versions, as the overhead of the spectre/meltdown mitigations have increased the cost of syscalls and forks. Obviously this doesn't terribly matter for actual installations, but I do find it fairly annoying when running regression tests... Greetings, Andres Freund
On 12/4/18 12:31 PM, Andres Freund wrote: > Hi, > > best of three of > time make -s install INSTALL=/bin/true > > 9.3: 1.53s > 9.4: 1.55s > 9.5: 1.90s > 9.6: 2.10s > 10: 2.45s > 11: 2.81s > master: 2.99s > > Obviously a part of that can be explained by the tree getting bigger, > but that can't be all of it. > > It also gets a bit worse if LLVM is enabled (due to the bitcode files > getting installed), but that's "just" 0.3s for me. We should optimize > that for fewer install invocations / using cp, but that seems seperate. > > I think this also has gotten more pronounced with newer OS versions, as > the overhead of the spectre/meltdown mitigations have increased the cost > of syscalls and forks. > > Obviously this doesn't terribly matter for actual installations, but I > do find it fairly annoying when running regression tests... Wouldn't using -j option with make help this? JD > > Greetings, > > Andres Freund > -- Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc *** A fault and talent of mine is to tell it exactly how it is. *** PostgreSQL centered full stack support, consulting and development. Advocate: @amplifypostgres || Learn: https://postgresconf.org ***** Unless otherwise stated, opinions are my own. *****
Hi, On 2018-12-04 12:31:48 -0800, Andres Freund wrote: > best of three of > time make -s install INSTALL=/bin/true > > 9.3: 1.53s > 9.4: 1.55s > 9.5: 1.90s > 9.6: 2.10s > 10: 2.45s > 11: 2.81s > master: 2.99s > > Obviously a part of that can be explained by the tree getting bigger, > but that can't be all of it. > > It also gets a bit worse if LLVM is enabled (due to the bitcode files > getting installed), but that's "just" 0.3s for me. We should optimize > that for fewer install invocations / using cp, but that seems seperate. > > I think this also has gotten more pronounced with newer OS versions, as > the overhead of the spectre/meltdown mitigations have increased the cost > of syscalls and forks. > > Obviously this doesn't terribly matter for actual installations, but I > do find it fairly annoying when running regression tests... As a comparison, I'd a tree that had the cmake patchset applied around (~1.5 yo tree). Using the ninja generator gets a clean build to 0m0.073s, a first install to 0m0.201s and a repeat install to 0m0.170s. Greetings, Andres Freund
On 04/12/2018 21:31, Andres Freund wrote: > Obviously this doesn't terribly matter for actual installations, but I > do find it fairly annoying when running regression tests... One possible workaround: https://www.postgresql.org/message-id/5d14d20c-b1f6-69c8-7465-df8adb424aea@2ndquadrant.com -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 04/12/2018 21:56, Andres Freund wrote: > As a comparison, I'd a tree that had the cmake patchset applied around > (~1.5 yo tree). Using the ninja generator gets a clean build to > 0m0.073s, a first install to 0m0.201s and a repeat install to 0m0.170s. Yeah, I've been playing with meson+ninja, and those run times are amazing. I'm not confident whether a high-level build system generator like cmake or meson would ever give us the flexibility we need. I'm just thinking of the most recent episode where we had to override the timestamp handling of ranlib of macOS. There are many others like it. But I think that we realistically could generate ninja files using some Perl code not unlike the one that does the catalog .dat to .bki transformation. Just a thought at this point. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On December 7, 2018 8:04:10 AM PST, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote: >On 04/12/2018 21:56, Andres Freund wrote: >> As a comparison, I'd a tree that had the cmake patchset applied >around >> (~1.5 yo tree). Using the ninja generator gets a clean build to >> 0m0.073s, a first install to 0m0.201s and a repeat install to >0m0.170s. > >Yeah, I've been playing with meson+ninja, and those run times are >amazing. It's not just the the faster times due to better tools, a non recursive implementation also allows for much cleaner dependenciesthen we have right now. Which also increases speed... >I'm not confident whether a high-level build system generator like >cmake >or meson would ever give us the flexibility we need. I'm just thinking >of the most recent episode where we had to override the timestamp >handling of ranlib of macOS. There are many others like it. I don't know much about meson, but I don't see why cmake can't handle that kind of thing. >But I think that we realistically could generate ninja files using some >Perl code not unlike the one that does the catalog .dat to .bki >transformation. Just a thought at this point. That seems not great, because we would lose the ability to generate mscv files from the same source. Or whatever the eventualreplacement of ninja will be. Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On 07/12/2018 18:34, Andres Freund wrote: >> But I think that we realistically could generate ninja files using some >> Perl code not unlike the one that does the catalog .dat to .bki >> transformation. Just a thought at this point. > That seems not great, because we would lose the ability to generate mscv files from the same source. Or whatever the eventualreplacement of ninja will be. We already generate msvc files using custom Perl code. Right now we generated them by parsing makefiles in a hackish way. If we were to change that to parsing a well-structured ".dat" file, that seems entirely doable and wouldn't be worse than what we do now, I think. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services