Обсуждение: upload of rebuilt packages to the repository

Hi all,


I've noticed that from time to time PostgreSQL pushes rebuilds of
already existing packages[1] – i.e. same %{name}, %{epoch}, %{version},
%{release} and same filename but different content – into the yum
repository.

Please don't to that.

If the existing package is buggy or corrupt, just increase %{release}
and push the new package to the repo.
Otherwise yum and mirrors might not handle it correctly. With cached
metadata, yum will complain about checksum mismatches. Mirrors that are
using e.g. reposync will create a corrupted file if the former package
was smaller than the updated one by appending additional bytes to the
existing file. If the former package was larger than the updated one,
your the servers will respond with a "416 range not satisfiable".


Best,

Philippe

[1] for example amcheck_next10-1.3-1.rhel7.x86_64 on march 1st

Hi,

On Tue, 2018-03-06 at 13:36 +0100, Philippe Kueck wrote:

> I've noticed that from time to time PostgreSQL pushes rebuilds of
> already existing packages[1] – i.e. same %{name}, %{epoch}, %{version},
> %{release} and same filename but different content – into the yum
> repository.
>
> Please don't to that.

Err, we *never* do so. Can you please give me an example?

> If the existing package is buggy or corrupt, just increase %{release}
> and push the new package to the repo.
> Otherwise yum and mirrors might not handle it correctly. With cached
> metadata, yum will complain about checksum mismatches.

When we add/update a new package, the metadata is updated, so if the cached
version (on OS side) does not match, you get this message. That is the only
reason for this issue (and it is not an issue, IMHO)

Regards,
--
Devrim Gündüz
EnterpriseDB: https://www.enterprisedb.com
PostgreSQL Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

Hi,


On 06.03.2018 15:33, Devrim Gündüz wrote:
> Err, we *never* do so. Can you please give me an example?

I didn't check the previous package of amcheck_next10-1.3-1.rhel7.x86_64
of before march 1st, sorry for that.

But a while ago I filed a bug report for yum and used one of your
packages (libgeotiff) as example:
https://bugzilla.redhat.com/show_bug.cgi?id=1527417

> When we add/update a new package, the metadata is updated, so if the cached
> version (on OS side) does not match, you get this message. That is the only
> reason for this issue (and it is not an issue, IMHO)

If we replace an existing package we get this message. Not by adding a
newer version of it.

Below is a little poc. I'm using dnf (Fedora) but its behaviour doesn't
differ from yum in this case.


Kind regards,

Philippe

--%snip%--
# prepare a dummy repo
$ mkdir -p /tmp/myrepo

# add this dummy repo to yum/dnf
$ cat > /etc/yum.repos.d/test.repo <<EOF
[test]
name=testrepo
baseurl=file:///tmp/myrepo
enabled=0
gpgcheck=0
EOF

# create a dummy package
$ cat > testrpm.spec <<EOF
Name: testrpm
Version: 0.1
Release: 1%{?dist}
Summary: this is a test package
Group: Other
License: GPLv3

%description
Bogus.

%build
date +%s > dummyfile

%install
install -D -p -m644 dummyfile %{buildroot}/%{_sysconfdir}/dummyfile

%files
%{_sysconfdir}/dummyfile
EOF

# build it
$ rpmbuild -bb testrpm.spec

# and add it to the dummy repository
$ mv $(rpm -E %{_rpmdir}/%{_arch})/testrpm*.x86_64.rpm /tmp/myrepo/

# create the dummy repo
$ createrepo /tmp/myrepo/

# build cache
$ dnf --disablerepo='*' --enablerepo=test makecache

# rebuild dummy package
$ rpmbuild -bb testrpm.spec

# replace dummy package
$ mv -f $(rpm -E %{_rpmdir}/%{_arch})/testrpm*.x86_64.rpm /tmp/myrepo/

# and recreate/update the dummy repo
$ createrepo /tmp/myrepo/

# try to install the dummy package
$ dnf --disablerepo='*' --enablerepo=test -y install testrpm
Last metadata expiration check: 0:01:17 ago on Tue 06 Mar 2018 03:53:01
PM CET.
Dependencies resolved.
============================================================================================
 Package              Arch                Version
Repository         Size
============================================================================================
Installing:
 testrpm              x86_64              0.1-1.fc27                test
             6.2 k

Transaction Summary
============================================================================================
Install  1 Package

Total download size: 6.2 k
Installed size: 11
Downloading Packages:
Package "testrpm-0.1-1.fc27.x86_64" from local repository "test" has
incorrect checksum
Error: Some packages from local repository have incorrect checksum

# rebuild cache (which has not expired yet)
$ dnf --disablerepo='*' --enablerepo=test makecache
Last metadata expiration check: 0:02:18 ago on Tue 06 Mar 2018 03:53:01
PM CET.
Metadata cache created.

# try again
$ dnf --disablerepo='*' --enablerepo=test -y install testrpm
Last metadata expiration check: 0:02:21 ago on Tue 06 Mar 2018 03:53:01
PM CET.
Dependencies resolved.
=======================================================================================================
 Package                 Arch                   Version
   Repository            Size
=======================================================================================================
Installing:
 testrpm                 x86_64                 0.1-1.fc27
   test                 6.2 k

Transaction Summary
=======================================================================================================
Install  1 Package

Total download size: 6.2 k
Installed size: 11
Downloading Packages:
Package "testrpm-0.1-1.fc27.x86_64" from local repository "test" has
incorrect checksum
Error: Some packages from local repository have incorrect checksum

# this fixes it
$ dnf --disablerepo='*' --enablerepo=test clean metadata

# so let's start over with a new release

# build and add it
$ sed -i -r -e 's/(Release: )([0-9])/echo \1$((\2+1))/e' testrpm.spec
$ rpmbuild -bb testrpm.spec
$ mv -f $(rpm -E %{_rpmdir}/%{_arch})/testrpm*.x86_64.rpm /tmp/myrepo/
$ createrepo /tmp/myrepo/

# it doesn't install the latest release because of non-expired metadata
# but at least it doesn't show checksum errors
$ yes n | dnf --disablerepo='*' --enablerepo=test install testrpm
Last metadata expiration check: 0:03:31 ago on Tue 06 Mar 2018 04:07:07
PM CET.
Dependencies resolved.
=======================================================================================================
 Package                 Arch                   Version
   Repository            Size
=======================================================================================================
Installing:
 testrpm                 x86_64                 0.1-1.fc27
   test                 6.2 k

Transaction Summary
=======================================================================================================
Install  1 Package

Total size: 6.2 k
Installed size: 11
Is this ok [y/N]: Operation aborted.
--%snip%--

Hi all,

it happened again. Altered rpm packages were uploaded to the repository
without bumping the version string. Why do you do this? Mirrors haven
been shattered, breaking our own repositories and yum caches.

If you need to add a gpg signature to your packages, increase the
release number and delete the unsigned package.
If you released a faulty/corrupt package, increase the release number
and delete the previous package.
If after uploading your package to the repository you find out your
package is missing a bugfix, increase the release number.
If you need to rebuild your package against a different library set,
increase the release number.
If you need to rebuild your package with different compiler flags,
increase the release number.
Even if hell freezes over, increase the release number.

That's what I found so far:
> 0_existing/osm2pgrouting_10-2.3.3-1.rhel7.x86_64.rpm
> # osm2pgrouting_10-2.3.3-1.rhel7.x86_64
>   built: Tue Dec 19 22:56:36 2017
>   signature: (none)
>   md5: 3eeff0d2547711082c465880ba710cfc
>   size: 137028
> 
> 1_online/osm2pgrouting_10-2.3.3-1.rhel7.x86_64.rpm
> # osm2pgrouting_10-2.3.3-1.rhel7.x86_64
>   built: Tue Dec 19 22:56:36 2017
>   signature: DSA/SHA1, Mon Jun 18 15:17:19 2018, Key ID 1f16d2e1442df0f8
>   md5: cad0d9aedf3608e50f6cfb0221b3119b
>   size: 137028
> 
> 0_existing/osm2pgrouting_10-debuginfo-2.3.3-1.rhel7.x86_64.rpm
> # osm2pgrouting_10-debuginfo-2.3.3-1.rhel7.x86_64
>   built: Tue Dec 19 22:56:36 2017
>   signature: (none)
>   md5: a410f47dab9a087c99decef93e282eb7
>   size: 17680
> 
> 1_online/osm2pgrouting_10-debuginfo-2.3.3-1.rhel7.x86_64.rpm
> # osm2pgrouting_10-debuginfo-2.3.3-1.rhel7.x86_64
>   built: Tue Dec 19 22:56:36 2017
>   signature: DSA/SHA1, Mon Jun 18 15:17:19 2018, Key ID 1f16d2e1442df0f8
>   md5: 0d8467f42671e341fe2f7d17111dffc7
>   size: 17680
> 
> 0_existing/pgadmin4-python-pbr-3.1.1-1.rhel7.noarch.rpm
> # pgadmin4-python-pbr-3.1.1-1.rhel7.noarch
>   built: Wed Apr 11 02:44:54 2018
>   signature: DSA/SHA1, Wed Apr 11 02:44:55 2018, Key ID 1f16d2e1442df0f8
>   md5: bccdf4366e5cf371312741509024eef2
>   size: 77368
> 
> 1_online/pgadmin4-python-pbr-3.1.1-1.rhel7.noarch.rpm
> # pgadmin4-python-pbr-3.1.1-1.rhel7.noarch
>   built: Wed Apr 11 02:45:00 2018
>   signature: DSA/SHA1, Wed Apr 11 02:45:01 2018, Key ID 1f16d2e1442df0f8
>   md5: 4a75384dd952d7654f5c2597baf80234
>   size: 77364
> 
> 0_existing/pgadmin4-python-simplejson-3.13.2-1.rhel7.x86_64.rpm
> # pgadmin4-python-simplejson-3.13.2-1.rhel7.x86_64
>   built: Wed Apr 11 02:49:44 2018
>   signature: DSA/SHA1, Wed Apr 11 02:49:47 2018, Key ID 1f16d2e1442df0f8
>   md5: 364341231594bc82aa203315d8fe8db6
>   size: 188664
> 
> 1_online/pgadmin4-python-simplejson-3.13.2-1.rhel7.x86_64.rpm
> # pgadmin4-python-simplejson-3.13.2-1.rhel7.x86_64
>   built: Wed Apr 11 02:49:54 2018
>   signature: DSA/SHA1, Wed Apr 11 02:49:56 2018, Key ID 1f16d2e1442df0f8
>   md5: 517f5e3bcf2086be16e04e0b8a736ebd
>   size: 188664
> 
> 0_existing/pgadmin4-python-simplejson-debuginfo-3.13.2-1.rhel7.x86_64.rpm
> # pgadmin4-python-simplejson-debuginfo-3.13.2-1.rhel7.x86_64
>   built: Wed Apr 11 02:49:44 2018
>   signature: DSA/SHA1, Wed Apr 11 02:49:47 2018, Key ID 1f16d2e1442df0f8
>   md5: ec13ec30d9958ac87d8a0410019734b1
>   size: 56512
> 
> 1_online/pgadmin4-python-simplejson-debuginfo-3.13.2-1.rhel7.x86_64.rpm
> # pgadmin4-python-simplejson-debuginfo-3.13.2-1.rhel7.x86_64
>   built: Wed Apr 11 02:49:54 2018
>   signature: DSA/SHA1, Wed Apr 11 02:49:57 2018, Key ID 1f16d2e1442df0f8
>   md5: 37e02e71c1b3231d7028c9fea8a56f24
>   size: 56488
> 
> 0_existing/pgadmin4-python-sshtunnel-0.1.3-1.rhel7.noarch.rpm
> # pgadmin4-python-sshtunnel-0.1.3-1.rhel7.noarch
>   built: Wed Jun 27 01:04:30 2018
>   signature: DSA/SHA1, Wed Jun 27 01:04:30 2018, Key ID 1f16d2e1442df0f8
>   md5: 27bd4760e4424a02d2d24f2d92a2d411
>   size: 37460
> 
> 1_online/pgadmin4-python-sshtunnel-0.1.3-1.rhel7.noarch.rpm
> # pgadmin4-python-sshtunnel-0.1.3-1.rhel7.noarch
>   built: Wed Jun 27 01:04:34 2018
>   signature: DSA/SHA1, Wed Jun 27 01:04:34 2018, Key ID 1f16d2e1442df0f8
>   md5: 645328bcb6a353ce832f940643625628
>   size: 37456

Kind regards

Philippe



On 06.03.2018 13:36, Philippe Kueck wrote:
> Hi all,
> 
> 
> I've noticed that from time to time PostgreSQL pushes rebuilds of
> already existing packages[1] – i.e. same %{name}, %{epoch}, %{version},
> %{release} and same filename but different content – into the yum
> repository.
> 
> Please don't to that.
> 
> If the existing package is buggy or corrupt, just increase %{release}
> and push the new package to the repo.
> Otherwise yum and mirrors might not handle it correctly. With cached
> metadata, yum will complain about checksum mismatches. Mirrors that are
> using e.g. reposync will create a corrupted file if the former package
> was smaller than the updated one by appending additional bytes to the
> existing file. If the former package was larger than the updated one,
> your the servers will respond with a "416 range not satisfiable".
> 
> 
> Best,
> 
> Philippe
> 
> [1] for example amcheck_next10-1.3-1.rhel7.x86_64 on march 1st
> 

Hi,

On Fri, 2018-06-29 at 14:54 +0200, Philippe Kueck wrote:
> it happened again. Altered rpm packages were uploaded to the repository
> without bumping the version string.

I don't *think* this is the case here. Let me explain:

I pushed a lot of packages during the weekend, mostly to 11 tree. The process
is:

1) Build the package
2) Push to staging repo (to yum.postgresql.org)
3) rsync to main repo from staging repo (to download.postgresql.org)

rsync is done once an hour, automagically. I don't have control on that, but I
control the first two steps. If I push packages around rsync time, rsync fails
to sync all files correctly, and this is resolved in the next rsync, causing
partial updates in the repo.

I tried to minimize this before, but I was very tired, and forgot to take a
look at the clock. I had this issue on my Fedora 28 box as well, a cache
refresh fixed my  problem.

Regards,
--
Devrim Gündüz
EnterpriseDB: https://www.enterprisedb.com
PostgreSQL Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

Hi,


thanks for your reply. This kind of issue doesn't come up with
incomplete rsyncs. The package files weren't corrupt, invalid or only
partially available (which might be the case when using inplace rsyncs).
The thing is that at some point (a) before the 19th of june these
packages existed in your repository:

(1) osm2pgrouting_10-2.3.3-1.rhel7.x86_64.rpm
(2) osm2pgrouting_10-debuginfo-2.3.3-1.rhel7.x86_64.rpm

with a checksum of C(a), a filesize of S(a), a gpg signature of G(a) (or
no signature at all) and a built date of B(a).

Now eventually (b) around the 19th of june the existing packages were
overwritten with packages of the *same name* but with a different
checksum and either

- a different filesize S(b) != S(a) and/or
- a different gpg signature G(b) != G(a) and/or
- a different built date B(b) != B(a)

(for details see my previous email).

These packages changed between the 28th and the 29th of june:

(3) pgadmin4-python-pbr-3.1.1-1.rhel7.noarch.rpm
(4) pgadmin4-python-simplejson-3.13.2-1.rhel7.x86_64.rpm
(5) pgadmin4-python-simplejson-debuginfo-3.13.2-1.rhel7.x86_64.rpm
(6) pgadmin4-python-sshtunnel-0.1.3-1.rhel7.noarch.rpm

in the same way as above.

Yum/dnf has two caches. One is the /var/cache/{dnf,yum} directory, which
can be cleared easily when dealing with such repositories, the other one
is the http_caching which needs to be dis- and re-enabled to resolve
that kind of issues. But unattended updates or downloads won't do this
automatically and will require manual intervention.

Let's assume you're mirroring a repository using the yum-utils' provided
'reposync'. This utility, which relies on yum's libraries, has a simple
but widespread mechanismn for downloading files: if a file already
exists locally, it tries to download it using a "Range:"-header in order
to resume possibly aborted downloads. So when the remote package is
bigger than the existing one – which is the case for (3) – it appends
the differing four bytes to the existing file, corrupting it (same for
e.g. wget -c et al). On the other hand, if the remote file is smaller
than the local file, a "Range:"-request results in a "416 range not
satisfiable".

Please examine the attached files if there's still something unclear:

> rpm -qp --qf '%{name}-%{version}-%{release}: %{buildtime:date}; %{siggpg:pgpsig}\n' *.rpm.{old,new}
> stat -c '%n %s' *.rpm.{old,new}
> md5sum *.rpm.{old,new}


Best,

Philippe