Re: [HACKERS] TAP tests take a long time

On 04/11/2017 08:12 PM, Craig Ringer wrote:
>
>
> On 12 Apr. 2017 03:14, "Andrew Dunstan"
> <andrew.dunstan@2ndquadrant.com
> <mailto:andrew.dunstan@2ndquadrant.com>> wrote:
>
>
>
>     On 04/11/2017 12:08 PM, Tom Lane wrote:
>     > Robert Haas <robertmhaas@gmail.com
>     <mailto:robertmhaas@gmail.com>> writes:
>     >> On Tue, Apr 11, 2017 at 11:32 AM, Andrew Dunstan
>     >> <andrew.dunstan@2ndquadrant.com
>     <mailto:andrew.dunstan@2ndquadrant.com>> wrote:
>     >>> This buildfarm run as you can see takes 33m32s, and the Tap
>     tests take a
>     >>> combined 19m52s of that time.
>     >> I don't think it's quite fair to complain about the TAP tests
>     taking a
>     >> long time to run as a general matter.  Many people here have long
>     >> wanted a separate test-suite for long running tests that can be
>     run to
>     >> really check everything possible, and apparently, TAP tests are it.
>     >> What I think would be more useful is to break down where the
>     time is
>     >> getting spent.  It may be that some of those tests are not adding
>     >> value proportionate to their runtime.
>
>
>
>     Well, you can get a lot of information on timings in crake's latest
>     builds for example, or nightjar's.
>
>     Here's an interesting fact: almost all the time taken up in the
>     scripts
>     test set seems to be consumed in running initdb over and over.
>
>
>
> Is it worth adding an init(cache => 1) option to PostgresNode, where
> we stash an initdb'd db in tmp_check/  and just do a simple fs copy of
> it ? 
>
> Even default to caching and allow an option to disable the cached copy.
>
> We're going to need to initdb a lot in the TAP tests. We often need a
> clean state. Tests also get harder to debug the more you pack into a
> single test run and more difficult to run individually to test some
> specific failure. So IMO the best thing is to try to make that repeat
> initdb as fast as possible.
>
> It reduces our coverage of initdb only incredibly slightly - all that
> repeat runs will do is help find very uncommon intermittent failures.
> And we rerun the buildfarm all the time so it's not like there's a
> shortage of initdb runs anyway.
>
> We should also have a debug --no-fsync option for initdb, or maybe
> allow it to take -o options to pass to child postgres so we can pass
> fsync=off .


Absolutely worth it I should say.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services