Re: Compressed TOAST Slicing

On Wed, Feb 20, 2019 at 1:45 PM Paul Ramsey <pramsey@cleverelephant.ca> wrote:
> What this does not support: any function that probably wants less-than-everything, but doesn’t know how big a slice
tolook for. Stephen thinks I should put an iterator on decompression, which would be an interesting piece of work.
Havinglooked at the json code a little doing partial searches would require a lot of re-work that is above my paygrade,
butif there was an iterator in place, at least that next stop would then be open. 
>
> Note that adding an iterator isn’t adding two ways to do the same thing, since the iterator would slot nicely
underneaththe existing slicing API, and just iterate to the requested slice size. So this is easily just “another step”
alongthe train line to providing streaming access to compressed and TOASTed data. 

Yeah.  Plus, I'm not sure the iterator thing is even the right design
for the JSONB case.  It might be better to think, for that case, about
whether there's someway to operate directly on the compressed data.
If you could somehow jigger the format and the chunking so that you
could jump directly to the right chunk and decompress from there,
rather than having to walk over all of the earlier chunks to figure
out where the data you want is, you could probably obtain a large
performance benefit.  But figuring out how to design such a scheme
seems pretty far afield from the topic at hand.

I'd actually be inclined not to add an iterator until we have a real
user for it, for exactly the reason that we don't know that it is the
right thing.  But there is certain value in decompressing partially,
to a known byte position, as your patch does, no matter what we decide
to do about that stuff.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company