Re: pgsql: Permit super-MaxAllocSize allocations with MemoryContextAllocHug

On Tue, Jul 02, 2013 at 10:33:38AM -0400, Noah Misch wrote:
> On Tue, Jul 02, 2013 at 12:08:42PM +0200, Andres Freund wrote:
> > On 2013-06-27 19:05:22 +0000, Noah Misch wrote:
> > > Permit super-MaxAllocSize allocations with MemoryContextAllocHuge().
> > >
> > > The MaxAllocSize guard is convenient for most callers, because it
> > > reduces the need for careful attention to overflow, data type selection,
> > > and the SET_VARSIZE() limit.  A handful of callers are happy to navigate
> > > those hazards in exchange for the ability to allocate a larger chunk.
> > > Introduce MemoryContextAllocHuge() and repalloc_huge().  Use this in
> > > tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX
> > > tuples, a factor-of-48 increase.  In particular, B-tree index builds can
> > > now benefit from much-larger maintenance_work_mem settings.
> >
> > This commit causes a bunch of warnings like:
> >
> > src/backend/utils/sort/tuplesort.c: In function ???tuplesort_begin_common???:
> > src/backend/utils/sort/tuplesort.c:399:33: warning: comparison of unsigned expression < 0 is always false
> > [-Wtype-limits]
> >  #define LACKMEM(state)  ((state)->availMem < 0)
> >
> > to be thrown during compilation. And I think it is spot on. Unless you
> > overhaul a good bit of the respective logic making availMem unsigned
> > isn't going to fly.
>
> True.  Will look into it; thanks.

In retrospect, I did not need the "long" -> "Size" changes at all.  I thought
they were necessary to prevent new overflow possibilities on 64-bit Windows,
but we already limited work_mem to 2 GiB on that platform to account for its
32-bit "long".  That's a limit to be removed rather than perpetuated.
Therefore, rather than change back to "long", I've changed to "int64".  That's
simple to reason about, optimal on 64-bit, and no big loss on 32-bit.

Thanks,
nm

--
Noah Misch
EnterpriseDB                                 http://www.enterprisedb.com