Re: [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

On 01/09/2014 11:48 PM, Claudio Freire wrote:
> On Thu, Jan 9, 2014 at 4:39 PM, knizhnik <knizhnik@garret.ru> wrote:
>>> At fork time I only wrote about reserving the address space. After
>>> reserving it, all you have to do is implement an allocator that works
>>> in shared memory (protected by a lwlock of course).
>>>
>>> In essence, a hypothetical pg_dsm_alloc(region_name) would use regular
>>> shared memory to coordinate returning an already mapped region (same
>>> address which is guaranteed to work since we reserved that region), or
>>> allocate one (within the reserved address space).
>> Why do we need named segments? There is ShmemAlloc function in PostgreSQL
>> API.
>> If RequestAddinShmemSpace can be used without requirement to place module in
>> preloaded list, then isn't it enough for most extensions?
>> And ShmemInitHash can be used to maintain named regions if it is needed...
> If you want to dynamically create the segments, you need some way to
> identify them. That is, the name. Otherwise, RequestWhateverShmemSpace
> won't know when to return an already-mapped region or not.
>
> Mind you, the name can be a number. No need to make it a string.
>
>> So if we have some reserved address space, do we actually need some special
>> allocator for this space to allocate new segments in it?
>> Why existed API to shared memory is not enough?
> I don't know this existing API you mention. But I think this is quite
> a specific case very unlikely to be serviced from existing APIs. You
> need a data structure that can map names to regions, any hash map will
> do, or even an array since one wouldn't expect it to be too big, or
> require it to be too fast, and then you need to unmap the "reserve"
> mapping and put a shared region there instead, before returning the
> pointer to this shared region.
>
> So, the special thing is, the book-keeping region sits in regular
> shared memory, whereas the allocated regions sit in newly-created
> segments. And segments are referenced by pointers (since the address
> space is fixed and shared). Is there something like that already?
By existed API I mostly mean 6 functions:

RequestAddinShmemSpace()
RequestAddinLWLocks()
ShmemInitStruct()
LWLockAssign()
ShmemAlloc()
ShmemInitHash()

If it will be possible to use this function without requirement for 
module to be included in "shared_preload_libraries" list, then do we 
really need DSM?
And it can be achieved by
1. Preserving address space (as you suggested)
2. Preserving some fixed number of free LWLocks (not very large < 100).

I do not have something against creation of own allocator of named 
shared memory segments within preserved address space.
I just not sure if it is actually needed. In some sense 
RequestAddinShmemSpace() can be such allocator.