Re: dynamically allocating chunks from shared memory

Hi,

On 07/26/2010 06:33 PM, Robert Haas wrote:
> It would be nice to think, for example, that this could be used as
> infrastructure for parallel query to stream results back from worker
> processes to the backend connected to the user.  If you're using 16
> processors to concurrently scan 16 partitions of an appendrel and
> stream those results back to the master

Now, *that* sounds like music to my ears ;-)

Or put another way: yes, I think imessages and the bgworker 
infrastructure stuff could enable or at least help that goal.

> Dynamically allocating out of a 2MB
> segment gives up most of that flexibility.

Absolutely, that's why I'd like to see other modules that use the 
dynamic allocator. The more the better.

> What I think will end up happening here is that you'll always have to
> size the segment used by the dynamic allocator considerably larger
> than the amount of memory you expect to actually be used, so that
> performance doesn't go into the toilet when it fills up.  As Markus
> pointed out upthread, you'll always need some hard limit on the amount
> of space that imessages can use, but you can make that limit much
> larger if it's not reserved for a single purpose.  If you use the
> "temporarily allocated shared buffers" method, then you could set the
> default limit to something like "64MB, but not more than 1/8th of
> shared buffers".

I've been thinking about such rules as well. They quickly get more 
complex if you begin to take OOM situations and their counter-measures 
into account.

In a way, fixing every separate pool to its specific size just is the 
very simples rule-set I can think of. The dynamic allocator buys you 
more flexibility, but choosing good limits and rules between the 
sub-systems is another issue.

Regards

Markus Wanner