Re: Parallel WAL Archival Options
| От | Nikhil Shetty |
|---|---|
| Тема | Re: Parallel WAL Archival Options |
| Дата | |
| Msg-id | CAFpL5VyOqq8W68_ZG_fhxGiGXH=v-G1gJc54=8GKyDT-BQo5fQ@mail.gmail.com обсуждение исходный текст |
| Ответ на | Re: Parallel WAL Archival Options (Ron <ronljohnsonjr@gmail.com>) |
| Список | pgsql-admin |
Hi Ron,
Uploads to a remote server?
— Yes, to S3
— Yes, to S3
Does wal-g compress files before sending them across the wire? By how much? Are you CPU or IO bound by having to compress that much data?
— Yes, we use default compression i.e lz4. There is no pressure on resources but the number of wal files that are processed in parallel, if we increase the streams it uses up a lot of memory.
Thanks,
Nikhil
On Sun, 6 Aug 2023 at 13:51, Ron <ronljohnsonjr@gmail.com> wrote:
On 8/6/23 02:43, Nikhil Shetty wrote:Hi Team,I would like to know which backup&restore tools will be better for scenarios where the database is generating around 400 WALs per minute.
If my math is correct, 400x 16MB WAL files per minute is 400*(16*2^20)/60*8 / 10^6 = 895 MBits per second. Plus overhead.
That's about 1Gbit/second. Definitely nothing to sneeze at.We are using wal-g but it is not able to keep pace with the wal generation. We increased the upload streams to 256 but no luck
Uploads to a remote server?
Does wal-g compress files before sending them across the wire? By how much? Are you CPU or IO bound by having to compress that much data?--
Born in Arizona, moved to Babylonia.
В списке pgsql-admin по дате отправления: