Re: parallelizing the archiver

> 30 сент. 2021 г., в 09:47, Bossart, Nathan <bossartn@amazon.com> написал(а):
>
> The attached patch is a first try at adding alternatives for
> archive_command
Looks like an interesting alternative design.

> I tested the sample archive_command in the docs against the sample
> archive_library implementation in the patch, and I saw about a 50%
> speedup.  (The archive_library actually syncs the files to disk, too.)
> This is similar to the improvement from batching.
Why test sample agains sample? I think if one tests this agains real archive tool doing archive_status lookup and
ready->donerenaming results will be much different. 

> Of course, there are drawbacks to using an extension.  Besides the
> obvious added complexity of building an extension in C versus writing
> a shell command, the patch disallows changing the libraries without
> restarting the server.  Also, the patch makes no effort to simplify
> error handling, memory management, etc.  This is left as an exercise
> for the extension author.
I think the real problem with extension is quite different than mentioned above.
There are many archive tools that already feature parallel archiving. PgBackrest, wal-e, wal-g, pg_probackup, pghoard,
pgbarmanand others. These tools by far outweight tools that don't look into archive_status to parallelize archival. 
And we are going to ask them: add also a C extension without any feasible benefit to the user. You only get some
restrictionslike system restart to enable shared library. 

I think we need a design that legalises already existing de-facto standard features in archive tools. Or event better -
enablesthese tools to be more efficient, reliable etc. Either way we will create legacy code from the scratch. 

Thanks!

Best regards, Andrey Borodin.