Обсуждение: Re: Compression of bigger WAL records
On 2025/01/22 3:24, Andrey M. Borodin wrote: > > >> On 12 Jan 2025, at 17:43, Andrey M. Borodin <x4mmm@yandex-team.ru> wrote: >> >> I attach a prototype patch. > > Here's v2, now it passes all the tests with wal_debug. I like the idea of WAL compression more. With the current approach, each backend needs to allocate memory twice the size of the total WAL record. Right? One area is for the gathered WAL record data (from rdt and registered_buffers), and the other is for storing the compressed data. Could this lead to potential memory usage concerns? Perhaps we should consider setting a limit on the maximum memory each backend can use for WAL compression? Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION
Hi Andery
I have a question ,If wal_compression_threshold is set to more than the block size of the wal log, then the FPI is not compressed, and if so, it might make sense to have a maximum value of this parameter that does not exceed the block size of the wal log?
Best regards
On Thu, Jan 30, 2025 at 9:26 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> On 23 Jan 2025, at 20:13, Japin Li <japinli@hotmail.com> wrote:
>
>
> I find this feature interesting;
Thank you for your interest in the patch!
> however, it cannot be applied to the current
> master (b35434b134b) due to commit 32a18cc0a73.
PFA a rebased version.
>
> I see the patch compresses the WAL record according to the wal_compression,
> IIRC the wal_compression is only used for FPI, right? Maybe we should update
> the description of this parameter.
Yes, I'll udpate documentation in future versions too.
> I see that the wal_compression_threshold defaults to 512. I wonder if you
> chose this value based on testing or randomly.
Voices in my head told me it's a good number.
> On 28 Jan 2025, at 22:10, Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
>
> I like the idea of WAL compression more.
Thank you!
> With the current approach, each backend needs to allocate memory twice
> the size of the total WAL record. Right? One area is for the gathered
> WAL record data (from rdt and registered_buffers), and the other is for
> storing the compressed data.
Yes, exactly. And also a decompression buffer for each WAL reader.
> Could this lead to potential memory usage
> concerns? Perhaps we should consider setting a limit on the maximum
> memory each backend can use for WAL compression?
Yes, the limit makes sense.
Also, we can reduce memory consumption by employing a streaming compression. Currently, I'm working on a prototype of such technology, because it would allow wholesale WAL compression. The idea is to reuse compression context from previous records to better compress new records. This would allow efficient compression of even very small records. However, there is exactly 0 chance to get it done in a decent shape before feature freeze.
The chances of getting currently proposed approach to v18 seems slim either... I'm hesitating to register this patch on the CF. What do you think?
Best regards, Andrey Borodin.
> On 31 Jan 2025, at 08:37, wenhui qiu <qiuwenhuifx@gmail.com> wrote: > > Hi Andery > I have a question ,If wal_compression_threshold is set to more than the block size of the wal log, then the FPI isnot compressed, and if so, it might make sense to have a maximum value of this parameter that does not exceed the blocksize of the wal log? Oops, looks like I missed your question. Sorry for so long delay. User might want to compress only megabyte+ records, there's nothing wrong with it. WAL record itself is capped by 1Gb (XLogRecordMaxSize),I do not see a reason to restrict wal_compression_threshold by lower value. PFA rebased version. Best regards, Andrey Borodin.