pg_dump: largeobject behavior issues (possible bug)

Hello,

I have been working a problem with Andrew Gierth (sp?) in regards to 
pg_dump. Here is the basic breakdown:

FreeBSD 10.1
PostgreSQL 9.3.6
64GB ~ memory
500GB database
228G of largeobjects (106M objects)

The database dumps fine as long as we don't dump large objects. However, 
if we try to dump the large objects, FreeBSD will kill pg_dump as it 
will consume all free memory and swap. With Andrew's help we were able 
to determine the following:

There is a memory cost of about 160 bytes per largeobject. Based on the 
number of largeobjects we have that would be about 16GB of memory. Also 
when pg_dump is reading in the largobject list there is a point where 
pg_dump has a PGresult containing the entire contents of 
pg_largeobject_metadata and a malloc of an array where it is going to 
copy the data to. That could easily get above the 40G thus causeFreeBSD 
to kill the process.

tl;dr

The memory issue comes down to the fact that in the prep stage, pg_dump 
creates a TOC entry for every individual large object.

It seems that pg_dump should be much more efficient about dumping these
objects.

Sincerely,

JD


-- 
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Announcing "I'm offended" is basically telling the world you can't
control your own emotions, so everyone else should do it for you.