Re: Proposal: custom compression methods

On 12/14/2015 07:28 PM, Jim Nasby wrote:
> On 12/14/15 12:50 AM, Craig Ringer wrote:
>> The issue with per-Datum is that TOAST claims two bits of a varlena
>> header, which already limits us to 1 GiB varlena values, something
>> people are starting to find to be a problem. There's no wiggle room to
>> steal more bits. If you want pluggable compression you need a way to
>> store knowledge of how a given datum is compressed with the datum or
>> have a fast, efficient way to check.
>
> ... unless we allowed for 8 byte varlena headers. Compression changes
> themselves certainly don't warrant that, but if people are already
> unhappy with 1GB TOAST then maybe that's enough.
>
> The other thing this might buy us are a few bits that could be used to
> support Datum versioning for other purposes, such as when the binary
> format of something changes. I would think that at some point we'll need
> that for pg_upgrade.

While versioning or increasing the 1GB limit are interesting, they have 
nothing to do with this particular patch. (BTW I don't see how the 
versioning would work at varlena level - that's something that needs to 
be handled at data type level).

I think the only question we need to ask here is whether we want to 
allow mixed compression for a column. If no, we're OK with the current 
header. This is what the patch does, as it requires a rewrite after 
changing the compression method.

And we're not painting ourselves in the corner - if we decide to 
increase the varlena header size in the future, this patch does not make 
it any more complicated.

regards

--
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services