Re: dynamically allocating chunks from shared memory

Hi,

On 08/09/2010 05:02 PM, Bruce Momjian wrote:
> [ Sorry to be jumping into this thread late.]

No problem at all.

> I am not sure threads would greatly help us.

Note that I'm absolutely, certainly not advocating the use of threads 
for Postgres.

> The major problem is that
> all of our our structures are currently contiguous in memory for quick
> access.  I don't see how threading would help with that.  We could use
> realloc(), but we can do the same in shared memory if we had a chunk
> infrastructure, though concurrent access to that memory would hurt us in
> either threads or shared memory.

I don't quite follow what you are trying to say here. Whether or not 
structures are contiguous in memory might affect performance, but I 
don't see the relation to programmer's habits and/or knowledge.

With our process-based design, the default is private memory (i.e. not 
shared). If you need shared memory, you must specify a certain amount in 
advance. That chunk of shared memory then is reserved and can't ever be 
used by another subsystem. Even if you barely ever need that much shared 
memory for the subsystem in question.

That's opposed to what lots of people are used to with the threaded 
approach, where shared memory is the default. And where you can easily 
and dynamically allocate *shared* memory. Whatever chunk of shared 
memory one subsystem doesn't need is available to another one (modulo 
fragmentation of the dynamic allocator, perhaps, but..)

> Fundamentally, recreating the libc memory allocation routines is not
> that hard.

Uh.. well, writing a good, scalable, dynamic allocator certainly poses 
some very interesting problems. Writing one that doesn't violate any 
patent or other IP as an additional requirement seems like a pretty 
tough problem to me.

> (Everyone has to detach from the shared memory segment, but
> they have to stop using it too, so it doesn't seem that hard.)

So far, I only considered dynamically allocating from a pool of shared 
memory that's initially fixed in size. So as to be able to make better 
use of shared memory.

Resizing the overall pool the easy way, requiring every backend to 
detach would cost a lot of performance. So that's certainly not 
something you want to do often.

The purpose of such a dynamic allocator as I see it rather is to be able 
to re-allocate unused memory of one subsystem to another one *on the 
fly*. Not just for performance, but also for ease of use for the admin 
and the developer, IMO.

Regards

Markus Wanner