Re: Large objects and out-of-memory

On 21.12.2020 21:27, Tom Lane wrote:
> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
>> The following sequence of command cause backend's memory to exceed 10Gb:
>> INSERT INTO image1 SELECT lo_creat(-1) FROM generate_series(1,10000000);
>> REASSIGN OWNED BY alice TO testlo;
> [ shrug... ]  You're asking to change the ownership of 10000000 objects.
> This is not going to be a cheap operation.  AFAIK it's not going to be
> any more expensive than changing the ownership of 10000000 tables, or
> any other kind of object.
>
> The argument for allowing large objects to have per-object ownership and
> permissions in the first place was that useful scenarios wouldn't have a
> huge number of them (else you'd run out of disk space, if they're actually
> "large"), so we needn't worry too much about the overhead.
>
> We could possibly bound the amount of space used in the inval queue by
> switching to an "invalidate all" approach once we got to an unreasonable
> amount of space.  But this will do nothing for the other costs involved,
> and I'm not really sure it's worth adding complexity for.
>
>             regards, tom lane

It seems to me that several millions large objects (documents, images, 
... any other blobs) is something less exotic
than several millions tables.
In any case I think that such command REASSIGN should not cause crash of 
the server.

I attached small patch based on your idea: replace individual 
invalidation messages with invalidate-all messages.
If GUC variable invalidate_all_threshold is set to non zero value, then 
there is no memory overflow in this scenario.

I am not sure if the proposed approach to "collapse" invalidation 
message is really good.
Especially I do not like that there is no stored counter of invalidation 
message and I have to traverse all chunk list to calculate it.

Also I noticed small inconsistency in inval.c:

     else if (msg->id == SHAREDINVALSNAPSHOT_ID)
     {
         /* We only care about our own database and shared catalogs */
         if (msg->rm.dbId == InvalidOid)
             InvalidateCatalogSnapshot();
         else if (msg->rm.dbId == MyDatabaseId)
             InvalidateCatalogSnapshot();
     }

Here we are processing snapshot invalidation message but using structure 
for relation map invalidation.
It doesn't cause problems because only dbId field is used and it has the 
same offset
in SharedInvalRelmapMsg and SharedInvalSnapshotMsg structures.
But it should be fixed.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company