Re: pg_dump: largeobject behavior issues (possible bug)

On 04/23/2015 04:04 PM, Andrew Gierth wrote:
>>>>>> "Joshua" == Joshua D Drake <jd@commandprompt.com> writes:
>   Joshua> The database dumps fine as long as we don't dump large
>   Joshua> objects. However, if we try to dump the large objects, FreeBSD
>   Joshua> will kill pg_dump as it will consume all free memory and
>   Joshua> swap. With Andrew's help we were able to determine the
>   Joshua> following:
>
>   Joshua> There is a memory cost of about 160 bytes per largeobject.
>
> I may have the exact number here wrong, it was just a quick eyeball of
> the data structures (and depends on malloc overheads anyway).
>
> The relevant code is getBlobs in pg_dump.c, which queries the whole of
> pg_largeobject_metadata without using a cursor (so the PGresult is
> already huge thanks to having >100 million rows), and then mallocs a
> BlobInfo array and populates it from the PGresult, also using pg_strdup
> for the oid string, owner name, and ACL if any.
>


I'm surprised this hasn't come up before. I have a client that I 
persuaded to convert all their LOs to bytea fields because of problems 
with pg_dump handling millions of LOs, and kept them on an older 
postgres version until they made that change.

cheers

andrew