Re: [proposal] de-TOAST'ing using a iterator

On Thu, Jun 20, 2019 at 1:51 AM Binguo Bao <djydewang@gmail.com> wrote:
> Hi hackers!
> This proposal aims to provide the ability to de-TOAST a fully TOAST'd and compressed field using an iterator and then
updatethe appropriate parts of the code to use the iterator where possible instead of de-TOAST'ing and de-compressing
theentire value. Examples where this can be helpful include using position() from the beginning of the value, or doing
apattern or substring match. 
>
> de-TOAST iterator overview:
> 1. The caller requests the slice of the attribute value from the de-TOAST iterator.
> 2. The de-TOAST iterator checks if there is a slice available in the output buffer, if there is, return the result
directly,
>     otherwise goto the step3.
> 3. The de-TOAST iterator checks if there is the slice available in the input buffer, if there is, goto step44.
Otherwise,
>     call fetch_datum_iterator to fetch datums from disk to input buffer.
> 4. If the data in the input buffer is compressed, extract some data from the input buffer to the output buffer until
thecaller's 
>     needs are met.
>
> I've implemented the prototype and apply it to the position() function to test performance.

Hi Binguo,

Interesting work, and nice performance improvements so far.  Just by
the way, the patch currently generates warnings:

https://travis-ci.org/postgresql-cfbot/postgresql/builds/554345719

--
Thomas Munro
https://enterprisedb.com