Re: buildfarm does not test "make check"

On 08/13/2015 02:11 PM, Alvaro Herrera wrote:
> So I added the brin isolation test back.  Because it needs the
> pageinspect module, it can only work under "make check", not
> installcheck.  The problem is that this means buildfarm will not run it,
> because it only runs installcheck :-(
>
> I realized that the important modules that run under "make check"
> already have custom buildfarm modules, namely pg_upgrade and
> test_decoding.  I wonder if it would make more sense to have this be
> customizable from the buildfarm server side, without having to resort to
> writing a buildfarm module for each new thing.  I envision a file hosted
> in the buildfarm server, perhaps a JSON blob, that lists modules that
> need "make check" treatment.  Within the JSON object for each such
> module there could live the details such as what files need to be
> saved.  That way, we could add more things to test without requiring the
> buildfarm client to be upgraded each time.
>
> This approach seems to have worked very well to customize the branches
> that each animal builds, which is why I suggest that it be used here
> too.
>


The buildfarm server doesn't control anything the clients do except it 
provides them with a list of branches we are interested in. Even that is 
invisible to the main program, and only used by the wrapper 
run_branches.pl. The main program can run entirely offline and I'm going 
to resist moves to make it otherwise. (Yes it needs access to git, but 
even that can be finessed.)

If you want some way of specifying this, put it in the source, e.g. 
invent a directory src/test/buildfarm_support and put it there. That way 
committers have automatic access to it, which they certainly don't to 
the buildfarm server. I'd rather just make it a small piece of perl 
instead of JSON,though.

You're also going to have to handle the msvc side of things. That won't 
be trivial. See discussion elsewhere today about how we've got that 
wrong recently.

cheers

andrew