Обсуждение: Support allocating memory for large strings

PostgreSQL's MaxAllocSize limit prevents storing individual variable-length character strings exceeding ~1GB, causing
"invalidmemory alloc request size" errors during INSERT operations on tables with large text columns. Example
reproductionincluded in artifacts.md. 

This limitation also affects pg_dump when exporting a PostgreSQL database with such data. The attached patches
demonstratesa proof of concept using palloc_extended with MCXT_ALLOC_HUGE in the write path. For the read path, there
area couple of possible approaches: extending existing functions to handle huge allocations, or implementing a chunked
storagemechanism that avoids single large allocations. 

Thoughts?

Maxim

Maxim Zibitsker <max.zibitsker@gmail.com> writes:
> PostgreSQL's MaxAllocSize limit prevents storing individual variable-length character strings exceeding ~1GB, causing
"invalidmemory alloc request size" errors during INSERT operations on tables with large text columns. 

This is news to no one.  We are not especially interested in trying to
relax that limit, because doing so would bleed over into approximately
everything in the backend, and create opportunities for
integer-overflow bugs in many places that are perfectly okay today.
The cost-benefit ratio for changing this decision is horrible.

> The attached patches demonstrates a proof of concept using
> palloc_extended with MCXT_ALLOC_HUGE in the write path.

"Proof of concept"?  This can't possibly fix your problem, because it
does nothing for the fact that tuple size fields are still limited
to 1GB, as are varlena headers for individual fields.  A serious
attack on this limitation, at a guess, would require a patch on the
order of 100K lines, and that might be an underestimate.

            regards, tom lane

On 8/11/25 3:15, Maxim Zibitsker wrote:
> PostgreSQL's MaxAllocSize limit prevents storing individual variable-length character strings exceeding ~1GB, causing
"invalidmemory alloc request size" errors during INSERT operations on tables with large text columns. Example
reproductionincluded in artifacts.md.
 

Tom Lane's very appropriate response not withstanding....

a) Why is this a problem? (Please share a bit more about your intended 
use case)

b) Why would someone need to store >1GB worth of TEXT (in a single 
string, no less!) in a column in an (albeit very flexible) Relational 
Database ?

     (I'm assuming no internal structure that would allow such amount of 
text to be split/spread over multiple records)

c) There exists LObs (Large OBjects) intended for this use, precisely... 
why is this mechanism not a good solution to your need?

d) Wouldn't a (journalling) File System (with a slim abstraction layer 
on top for directory hashing/indexing) not be a better solution for this 
particular application?

     Full Text Search on the stored data doesn't look like it would ever 
be performant... there exist specialized tools for that


And... how did you get "invalid" data in the database, that pg_dump 
wouldn't process, in the first place? (maybe just speculating/projecting 
and I didn't pick up the nuance properly)


Mostly curious about the problem / intended use case.... when we 
explored limits and limitations in Postgres almost 15 years ago, we 
never considered this even :o



Thanks,

-- 
Parkinson's Law: Work expands to fill the time alloted to it.

On Fri, Nov 07, 2025 at 09:32:45PM -0500, Tom Lane wrote:
> Maxim Zibitsker <max.zibitsker@gmail.com> writes:
>> PostgreSQL's MaxAllocSize limit prevents storing individual
>> variable-length character strings exceeding ~1GB, causing "invalid
>> memory alloc request size" errors during INSERT operations on tables
>> with large text columns.
> 
> This is news to no one.  We are not especially interested in trying to
> relax that limit, because doing so would bleed over into approximately
> everything in the backend, and create opportunities for
> integer-overflow bugs in many places that are perfectly okay today.
> The cost-benefit ratio for changing this decision is horrible.

FWIW something I am hearing about more often these days, and what I believe
Maxim's patch is actually after, is the 1GB limit on row size.  Even if
each field doesn't exceed 1GB (which is what artifacts.md seems to
demonstrate), heap_form_tuple() and friends can fail to construct the whole
tuple.  This doesn't seem to be covered in the existing documentation about
limits [0].

[0] https://www.postgresql.org/docs/devel/limits.html

-- 
nathan

Nathan Bossart <nathandbossart@gmail.com> writes:
> FWIW something I am hearing about more often these days, and what I believe
> Maxim's patch is actually after, is the 1GB limit on row size.  Even if
> each field doesn't exceed 1GB (which is what artifacts.md seems to
> demonstrate), heap_form_tuple() and friends can fail to construct the whole
> tuple.  This doesn't seem to be covered in the existing documentation about
> limits [0].

Yeah.  I think our hopes of relaxing the 1GB limit on individual
field values are about zero, but maybe there is some chance of
allowing tuples that are wider than that.  The notion that it's
a one-line fix is still ludicrous though :-(

One big problem with a scheme like that is "what happens when
I try to make a bigger-than-1GB tuple into a composite datum?".

Another issue is what happens when a wider-than-1GB tuple needs
to be sent to or from clients.  I think there are assumptions
in the wire protocol about message lengths fitting in an int,
for example.  Even if the protocol were okay with it, I wouldn't
count on client libraries not to fall over.

On the whole, it's a nasty can of worms, and I stand by the
opinion that the cost-benefit ratio of removing the limit is
pretty awful.

            regards, tom lane