On Fri, Mar 11, 2022 at 8:52 PM Andres Freund <andres@anarazel.de> wrote:
> You could also just append a manifest as a compresed tar to the compressed tar
> stream. Unfortunately GNU tar requires -i to read concated compressed
> archives, so perhaps that's not quite an alternative.
s/Unfortunately/Fortunately/ :-p
I think we've already gone way too far in the direction of making this
stuff rely on specific details of the tar format. What if someday we
wanted to switch to pax, cpio, zip, 7zip, whatever, or even just have
one of those things as an option? It's not that I'm dying to have
PostgreSQL produce rar or arj files, but I think we box ourselves into
a corner when we just assume tar everywhere. As an example of a
similar issue with real consequences, consider the recent discovery
that we can't easily add support for LZ4 or ZSTD compression of
pg_wal.tar. The problem is that the existing code tells the gzip
library to emit the tar header as part of the compressed stream
without actually compressing it, and then it goes back and overwrites
that data later! Unsurprisingly, that's not a feature every
compression library offers.
--
Robert Haas
EDB: http://www.enterprisedb.com