Re: [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

On Thu, Jan 9, 2014 at 2:22 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> It would be nice to have better operating system support for this.
> For example, IIUC, 64-bit Linux has 128TB of address space available
> for user processes.  When you clone(), it can either share the entire
> address space (i.e. it's a thread) or none of it (i.e. it's a
> process).  There's no option to, say, share 64TB and not the other
> 64TB, which would be ideal for us.  We could then map dynamic shared
> memory segments into the shared portion of the address space and do
> backend-private allocations in the unshared part.  Of course, even if
> we had that, it wouldn't be portable, so who knows how much good it
> would do.  But it would be awfully nice to have the option.

You can map a segment at fork time, and unmap it after forking. That
doesn't really use RAM, since it's supposed to be lazily allocated (it
can be forced to be so, I believe, with PROT_NONE and MAP_NORESERVE,
but I don't think that's portable).

That guarantees it's free.

Next, you can map shared memory at explicit addresses (linux's mmap
has support for that, and I seem to recall Windows did too).

All you have to do, is some book-keeping in shared memory (so all
processes can coordinate new mappings).