Обсуждение: pgsql: make dist uses git archive

make dist uses git archive

This changes "make dist" to directly use "git archive", rather than
the custom shell script it currently runs.

This is to make the creation of the distribution tarball more directly
traceable to the git repository.  That is why we removed the "make
distprep" step.

"make dist" continues to produce a .gz and a .bz2 tarball as before.

The archives produced this way are deterministic and reproducible,
meaning for a given commit the result file should always be
bit-for-bit identical.  The exception is that if you use a git version
older than 2.38.0, gzip records the platform in the archive, so you'd
get a different output on Windows vs. macOS vs. "UNIX" (everything
else).  In git 2.38.0, this was changed so that everything is recorded
as "UNIX" now.  This is just something to keep in mind.  This issue is
specific to the gzip format, it does not affect other compression
formats.

Meson has its own distribution building command (meson dist), but we
are not using that at this point.  The main problem is that, the way
they have implemented it, it is not deterministic in the above sense.
Also, we want a "make" version for the time being.  But the target
name "dist" in meson is reserved for that reason, so we call the
custom target "pgdist" (so call something like "meson compile -C build
pgdist").

Reviewed-by: Tristan Partin <tristan@neon.tech>
Discussion: https://www.postgresql.org/message-id/flat/40e80f77-a294-4f29-a16f-e21bc7bc75fc%40eisentraut.org

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/619bc23a1a2f3750ac3668fe5a7564bc51e01684

Modified Files
--------------
GNUmakefile.in | 32 +++++++++++++----------------
meson.build    | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 79 insertions(+), 18 deletions(-)

On 25.03.24 10:04, Magnus Hagander wrote:
> This seems to have broken buildfarm member guaibasaurus, which I believe
> is the only one that runs "make dist" as part of the buildfarm. And as a
> result,t hat breaks the snapshot build uploads.
>
> I don't know exactly how it does it, it appears to not run from within
> the git repository. It might be this needs an update to the buildfarm
> code itself to handle this path? I haven't dug into how that code does
> the make dist part (thus cc:ing in Andrew as well).

I don't see any code for this in the client-code git repository.  So
maybe it is a local custom module?

On Mon, Mar 25, 2024 at 8:30 PM Peter Eisentraut <peter@eisentraut.org> wrote:

On 25.03.24 10:04, Magnus Hagander wrote:
> This seems to have broken buildfarm member guaibasaurus, which I believe
> is the only one that runs "make dist" as part of the buildfarm. And as a
> result,t hat breaks the snapshot build uploads.
>
> I don't know exactly how it does it, it appears to not run from within
> the git repository. It might be this needs an update to the buildfarm
> code itself to handle this path? I haven't dug into how that code does
> the make dist part (thus cc:ing in Andrew as well).

I don't see any code for this in the client-code git repository.  So
maybe it is a local custom module?

Huh, it seems you're right. I didn't realize that :) We received that module from Andrew back in Nov 2011, but I guess it was never committed to the repo (assuming that was the intention) :) But it has served us well so far...

PFA the module. It's pretty simple, but I don't know enough (anything, really) about the bf client code to say how to fix it...

'use_vpath' => undef,

On 25.03.24 23:20, Andrew Dunstan wrote:
> The problem occurs because except in vpath mode the buildfarm operates
> on a copy of the code, and for efficiency reasons we don't copy the ,git
> directory.
>
> The simplest fix is probably to change the machine's config to use a
> vpath build.
>
> In the config file change the undef here to a 1:
>
> 'use_vpath' => undef,
>
> That worked for me in a quick test.

Note that "make dist" doesn't work with vpath in backbranches. :-/

On Tue, Mar 26, 2024 at 3:20 AM Peter Eisentraut <peter@eisentraut.org> wrote:

On 25.03.24 23:20, Andrew Dunstan wrote:
> The problem occurs because except in vpath mode the buildfarm operates
> on a copy of the code, and for efficiency reasons we don't copy the ,git
> directory.
>
> The simplest fix is probably to change the machine's config to use a
> vpath build.
>
> In the config file change the undef here to a 1:
>
> 'use_vpath' => undef,
>
> That worked for me in a quick test.

Note that "make dist" doesn't work with vpath in backbranches. :-/

Ugh.

OK, plan B :

In the Module's setup code, right after the "die", put

symlink("$buildroot/$branch/pgsql/.git","$pgsql/.git");

On Tue, Mar 26, 2024 at 9:13 AM Andrew Dunstan <andrew@dunslane.net> wrote:

On Tue, Mar 26, 2024 at 3:20 AM Peter Eisentraut <peter@eisentraut.org> wrote:

On 25.03.24 23:20, Andrew Dunstan wrote:
> The problem occurs because except in vpath mode the buildfarm operates
> on a copy of the code, and for efficiency reasons we don't copy the ,git
> directory.
>
> The simplest fix is probably to change the machine's config to use a
> vpath build.
>
> In the config file change the undef here to a 1:
>
> 'use_vpath' => undef,
>
> That worked for me in a quick test.

Note that "make dist" doesn't work with vpath in backbranches. :-/

Ugh.

OK, plan B :

In the Module's setup code, right after the "die", put

symlink("$buildroot/$branch/pgsql/.git","$pgsql/.git");

If I understand you right, you meant:

        die "can't run this module with vpath builds"
          if $conf->{vpath};
+       symlink("$buildroot/$branch/pgsql/.git","$pgsql/.git");

        my $self  = {buildroot => $buildroot, pgbranch=> $branch, bfconf => $conf, pgsql => $pgsql};
        bless ($self, $class);

If so, I tried that, but it didn't work. That's this run here: https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=guaibasaurus&dt=2024-03-26%2009%3A02%3A51&stg=make-dist

On Tue, Mar 26, 2024 at 5:15 AM Magnus Hagander <magnus@hagander.net> wrote:

On Tue, Mar 26, 2024 at 9:13 AM Andrew Dunstan <andrew@dunslane.net> wrote:

On Tue, Mar 26, 2024 at 3:20 AM Peter Eisentraut <peter@eisentraut.org> wrote:

On 25.03.24 23:20, Andrew Dunstan wrote:
> The problem occurs because except in vpath mode the buildfarm operates
> on a copy of the code, and for efficiency reasons we don't copy the ,git
> directory.
>
> The simplest fix is probably to change the machine's config to use a
> vpath build.
>
> In the config file change the undef here to a 1:
>
> 'use_vpath' => undef,
>
> That worked for me in a quick test.

Note that "make dist" doesn't work with vpath in backbranches. :-/

Ugh.

OK, plan B :

In the Module's setup code, right after the "die", put

symlink("$buildroot/$branch/pgsql/.git","$pgsql/.git");

If I understand you right, you meant:

        die "can't run this module with vpath builds"
          if $conf->{vpath};
+       symlink("$buildroot/$branch/pgsql/.git","$pgsql/.git");

        my $self  = {buildroot => $buildroot, pgbranch=> $branch, bfconf => $conf, pgsql => $pgsql};
        bless ($self, $class);

If so, I tried that, but it didn't work. That's this run here: https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=guaibasaurus&dt=2024-03-26%2009%3A02%3A51&stg=make-dist

*sigh*

OK, instead of speculating I'll experiment and come up with an answer.

On Tue, Mar 26, 2024 at 6:10 AM Andrew Dunstan <andrew@dunslane.net> wrote:

On Tue, Mar 26, 2024 at 5:15 AM Magnus Hagander <magnus@hagander.net> wrote:

On Tue, Mar 26, 2024 at 9:13 AM Andrew Dunstan <andrew@dunslane.net> wrote:

On Tue, Mar 26, 2024 at 3:20 AM Peter Eisentraut <peter@eisentraut.org> wrote:

On 25.03.24 23:20, Andrew Dunstan wrote:
> The problem occurs because except in vpath mode the buildfarm operates
> on a copy of the code, and for efficiency reasons we don't copy the ,git
> directory.
>
> The simplest fix is probably to change the machine's config to use a
> vpath build.
>
> In the config file change the undef here to a 1:
>
> 'use_vpath' => undef,
>
> That worked for me in a quick test.

Note that "make dist" doesn't work with vpath in backbranches. :-/

Ugh.

OK, plan B :

In the Module's setup code, right after the "die", put

symlink("$buildroot/$branch/pgsql/.git","$pgsql/.git");

If I understand you right, you meant:

        die "can't run this module with vpath builds"
          if $conf->{vpath};
+       symlink("$buildroot/$branch/pgsql/.git","$pgsql/.git");

        my $self  = {buildroot => $buildroot, pgbranch=> $branch, bfconf => $conf, pgsql => $pgsql};
        bless ($self, $class);

If so, I tried that, but it didn't work. That's this run here: https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=guaibasaurus&dt=2024-03-26%2009%3A02%3A51&stg=make-dist

*sigh*

OK, instead of speculating I'll experiment and come up with an answer.

OK, this should work. Put these lines after the first print statement of the module's build() routine - the setup() routine runs too early:

my $src = "$self->{buildroot}/$self->{pgbranch}/pgsql";
symlink("$src/.git", "$self->{pgsql}/.git");