Обсуждение: calculating an aspect of shared buffer state from a background worker
Dear Hackers --
I'm looking at doing a calculation to determine the number of free buffers available. A n example ratio that is based on some data structures in freelist.c as follows:
(StrategyControl->lastFreeBuffer - StrategyControl->firstFreeBuffer) / (double) NBuffers
Is there a way to get access to the StrategyControl pointer in the context of a background worker?
Thanks,
Robert
On Mon, Mar 10, 2014 at 2:09 PM, Robert Berry <berrydigital@gmail.com> wrote: > Is there a way to get access to the StrategyControl pointer in the context > of a background worker? StrategyControl is inherent to freelist.c and has no external declaration so you could not have it even if you the BGWORKER_SHMEM_ACCESS flag. In order to calculate that, an idea could be to go through the array of BufferDescriptors and then get the information necessary. Locks are necessary when doing that if you want to get a consistent picture of the buffers. Perhaps more experienced people have better ideas though... Regards, -- Michael
Robert Berry <berrydigital@gmail.com> writes:
> I'm looking at doing a calculation to determine the number of free buffers
> available. A n example ratio that is based on some data structures in
> freelist.c as follows:
> (StrategyControl->lastFreeBuffer - StrategyControl->firstFreeBuffer) /
> (double) NBuffers
> Is there a way to get access to the StrategyControl pointer in the context
> of a background worker?
The BufferStrategyControl struct is in shared memory, so you can certainly
get at it. One way would be to modify freelist.c to export its static
pointer variable. Alternatively, you could call ShmemInitStruct an extra
time to look up the struct for yourself, and then save it in your own
static variable.
Having said that, though, I'm pretty dubious of the premise. I trust you
realize that the above calculation is entirely wrong; firstFreeBuffer and
lastFreeBuffer are list head and tail pointers, and have no numerical
relation to the list length. The only way to determine the list length
accurately would be to chase down the whole list, which you'd have to hold
the BufFreelistLock while doing, which'd be disastrous for performance if
the list was long. (If you're okay with modifying the backend code you
could dodge this by teaching freelist.c to maintain a counter, I guess.)
An even bigger issue is that it's not clear that the length of the free
list is actually a useful number to have; in steady-state usage it
frequently is always zero. Buffers only get put back on the freelist if
they're invalidated, eg by dropping the relation they belonged to. Normal
usage tends to allocate buffers by reclaiming ones whose usage_count has
reached zero in the clock sweep algorithm. So a better picture of the
availability of buffers would require scanning the buffer pool to see how
many there are of each usage_count level.
regards, tom lane
Thank you both for the thoughtful and helpful responses.
The utility of the length of the free list is somewhat dubious. I imagine it could be useful to answer the question of "is there a chance that increasing shared buffers would be useless?" in an optimization context. Agreed it's not useful in most steady state scenarios.
I saw the approach in the pg_buffercache contrib module and am looking for lockless alternatives for at least estimating the size of free buffers.
I'm a relatively inexperienced so I'd be curious to know whether there is a danger beyond an inconsistent result in traversing / sampling the BufferDescriptors without a lock?
Also I got the impression that there is a ring approach to freeing buffers, and that assuming the descriptors are allocated in sequential addresses, taking the difference in the first and last could be used to get a rough estimate accounting for sizes or other shenanigans?
Thank you again, the clues to look at buffer descriptors and ShmemInitStruct are very helpful.
Best Regards,
Robert
On Mon, Mar 10, 2014 at 7:33 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
The BufferStrategyControl struct is in shared memory, so you can certainlyRobert Berry <berrydigital@gmail.com> writes:
> I'm looking at doing a calculation to determine the number of free buffers
> available. A n example ratio that is based on some data structures in
> freelist.c as follows:
> (StrategyControl->lastFreeBuffer - StrategyControl->firstFreeBuffer) /
> (double) NBuffers
> Is there a way to get access to the StrategyControl pointer in the context
> of a background worker?
get at it. One way would be to modify freelist.c to export its static
pointer variable. Alternatively, you could call ShmemInitStruct an extra
time to look up the struct for yourself, and then save it in your own
static variable.
Having said that, though, I'm pretty dubious of the premise. I trust you
realize that the above calculation is entirely wrong; firstFreeBuffer and
lastFreeBuffer are list head and tail pointers, and have no numerical
relation to the list length. The only way to determine the list length
accurately would be to chase down the whole list, which you'd have to hold
the BufFreelistLock while doing, which'd be disastrous for performance if
the list was long. (If you're okay with modifying the backend code you
could dodge this by teaching freelist.c to maintain a counter, I guess.)
An even bigger issue is that it's not clear that the length of the free
list is actually a useful number to have; in steady-state usage it
frequently is always zero. Buffers only get put back on the freelist if
they're invalidated, eg by dropping the relation they belonged to. Normal
usage tends to allocate buffers by reclaiming ones whose usage_count has
reached zero in the clock sweep algorithm. So a better picture of the
availability of buffers would require scanning the buffer pool to see how
many there are of each usage_count level.
regards, tom lane