Re: Variable length varlena headers redux

Bruce Momjian wrote:
> Tom Lane wrote:
> > Gregory Stark <stark@enterprisedb.com> writes:
> > > That seems like an awful lot of copying and pallocs that aren't there
> > > currently though. And it'll make us reluctant to change over frequently used
> > > data types like text -- which are precisely the ones that would gain us the
> > > most.
> > 
> > > It seems to me that it might be better to change to storing varlena lengths in
> > > network byte order instead. That way we can dedicate the leading bits to toast
> > > flags and read more bytes as necessary.
> > 
> > This'll add its own overhead ... but probably less than pallocs and
> > data-copying would.  And I agree we can find (pretty much) all the
> > places that need changing by the expedient of deliberately renaming
> > the macros and struct fields.
> 
> I think we should go with the pallocs and see how it performs.  That is
> certainly going to be easier to do, and we can test it pretty easily.
> 
> One palloc optimization idea would be to split out the representation so
> the length is stored seprately from the data in memory, and we could use
> an int32 for the length, and point to the shared buffer for the data. 
> However I don't think our macros can handle that so it might be a
> non-starter.
> 
> However, I think we should find out of the palloc is a problem before
> avoiding it.

Another idea about reducing palloc is that we know every short column is
at most 128 + 4 = 132 bytes, so we could allocate a 132-byte buffer for
every short column in the scan, and just re-use the buffer for every
row.

--  Bruce Momjian  <bruce@momjian.us>          http://momjian.us EnterpriseDB
http://www.enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +