Обсуждение: Re: Compression of bigger WAL records

Поиск
Список
Период
Сортировка

Re: Compression of bigger WAL records

От
Fujii Masao
Дата:

On 2025/01/22 3:24, Andrey M. Borodin wrote:
> 
> 
>> On 12 Jan 2025, at 17:43, Andrey M. Borodin <x4mmm@yandex-team.ru> wrote:
>>
>> I attach a prototype patch.
> 
> Here's v2, now it passes all the tests with wal_debug.

I like the idea of WAL compression more.

With the current approach, each backend needs to allocate memory twice
the size of the total WAL record. Right? One area is for the gathered
WAL record data (from rdt and registered_buffers), and the other is for
storing the compressed data. Could this lead to potential memory usage
concerns? Perhaps we should consider setting a limit on the maximum
memory each backend can use for WAL compression?

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION




Re: Compression of bigger WAL records

От
wenhui qiu
Дата:
Hi Andery 
     I have a question ,If wal_compression_threshold is set to more than the block size of the wal log, then the FPI is not compressed, and if so, it might make sense to have a maximum value of this parameter that does not exceed the block size of the wal log?

Best regards


On Thu, Jan 30, 2025 at 9:26 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:


> On 23 Jan 2025, at 20:13, Japin Li <japinli@hotmail.com> wrote:
>
>
> I find this feature interesting;

Thank you for your interest in the patch!

> however, it cannot be applied to the current
> master (b35434b134b) due to commit 32a18cc0a73.

PFA a rebased version.

>
> I see the patch compresses the WAL record according to the wal_compression,
> IIRC the wal_compression is only used for FPI, right?  Maybe we should update
> the description of this parameter.

Yes, I'll udpate documentation in future versions too.

> I see that the wal_compression_threshold defaults to 512. I wonder if you
> chose this value based on testing or randomly.

Voices in my head told me it's a good number.


> On 28 Jan 2025, at 22:10, Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
>
> I like the idea of WAL compression more.

Thank you!

> With the current approach, each backend needs to allocate memory twice
> the size of the total WAL record. Right? One area is for the gathered
> WAL record data (from rdt and registered_buffers), and the other is for
> storing the compressed data.

Yes, exactly. And also a decompression buffer for each WAL reader.

> Could this lead to potential memory usage
> concerns? Perhaps we should consider setting a limit on the maximum
> memory each backend can use for WAL compression?

Yes, the limit makes sense.

Also, we can reduce memory consumption by employing a streaming compression. Currently, I'm working on a prototype of such technology, because it would allow wholesale WAL compression. The idea is to reuse compression context from previous records to better compress new records. This would allow efficient compression of even very small records. However, there is exactly 0 chance to get it done in a decent shape before feature freeze.

The chances of getting currently proposed approach to v18 seems slim either... I'm hesitating to register this patch on the CF. What do you think?


Best regards, Andrey Borodin.

Re: Compression of bigger WAL records

От
Andrey Borodin
Дата:

> On 31 Jan 2025, at 08:37, wenhui qiu <qiuwenhuifx@gmail.com> wrote:
>
> Hi Andery
>      I have a question ,If wal_compression_threshold is set to more than the block size of the wal log, then the FPI
isnot compressed, and if so, it might make sense to have a maximum value of this parameter that does not exceed the
blocksize of the wal log? 

Oops, looks like I missed your question. Sorry for so long delay.

User might want to compress only megabyte+ records, there's nothing wrong with it. WAL record itself is capped by 1Gb
(XLogRecordMaxSize),I do not see a reason to restrict wal_compression_threshold by lower value. 

PFA rebased version.


Best regards, Andrey Borodin.


Вложения