Re: Question about lazy_space_alloc() / linux over-commit

On Thu, Mar 05, 2015 at 03:28:12PM -0600, Jim Nasby wrote:
> On 3/4/15 9:10 AM, Robert Haas wrote:
> >On Wed, Feb 25, 2015 at 5:06 PM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:
> >>Could the large allocation[2] for the dead tuple array in lazy_space_alloc
> >>cause problems with linux OOM? [1] and some other things I've read indicate
> >>that a large mmap will count towards total system memory, including
> >>producing a failure if overcommit is disabled.
> >
> >I believe that this is possible.

I have seen that in the field, albeit on a server with a 10 GiB allocation
limit, years ago.

> >>Would it be worth avoiding the full size allocation when we can?
> >
> >Maybe.  I'm not aware of any evidence that this is an actual problem
> >as opposed to a theoretical one.  vacrelstats->dead_tuples is limited
> >to a 1GB allocation, which is not a trivial amount of memory, but it's
> >not huge, either.  But we could consider changing the representation
> >from a single flat array to a list of chunks, with each chunk capped
> >at say 64MB.  That would not only reduce the amount of memory that we
> 
> I was thinking the simpler route of just repalloc'ing... the memcpy would
> suck, but much less so than the extra index pass. 64M gets us 11M tuples,
> which probably isn't very common.

+1.  Start far below 64 MiB; grow geometrically using repalloc_huge(); cap
growth at vac_work_mem.