Re: Experimental dynamic memory allocation of postgresql shared memory

On Sat, Jun 18, 2016 at 12:45 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Aleksey Demakov <ademakov@gmail.com> writes:
>> On Fri, Jun 17, 2016 at 10:54 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> In my opinion, that's not going to fly.  If I thought otherwise, I
>>> would not have developed the DSM facility in the first place.
>
>> Essentially this is pessimizing for the lowest common denominator
>> among OSes.
>
> You're right, but that doesn't mean that the community is going to take
> much interest in an unportable replacement for code that already exists.

Excuse me, what code already exists? As far as I understand, we
compare the approach taken in my code against Robert's code that
is not yet available to the community.

Discussing DSM is beyond the point.

My code might be smoothly hooked into the existing system from an
extension module just with a couple of calls:

RequestAddinShmemSpace() and ShmemInitStruct().

After that this extension might use my concurrent memory allocator
and safe memory reclamation for implementing highly optimized
concurrent data structures of their choice. E.g. concurrent data
structures that I am going to add to the package in the future.

All in all, currently this is not a replacement for anything. This is an
experimental add-on and a food for thought for interested people.

Integrating my code right into the core to replace anything there is
a very remote possibility. I understand if it ever happens it would
take very serious work and multiple iterations.

> Especially not an unportable replacement that also needs sweeping
> assumptions like "disciplined use of mmap in postgresql core and
> extensions".  You don't have to look further than the availability of
> mmap to plperlu programmers to realize that that won't fly.  (Even if
> we threw all the untrusted PLs overboard, I believe plain old stdio
> is willing to use mmap in many versions of libc.)
>

Sorry. I made a sloppy statement about mmap/munmap use. As
correctly pointed out by Andres Freund, it is problematic. So the
whole line about "disciplined use of mmap in postgresql core and
extensions" goes away. Forget it.

But the other techniques that I mentioned do not take such a
special discipline.

The corrected statement is that a single contiguous shared space
is practically doable on many platforms with some effort. And this
approach would make implementation of many shared data
structures more efficient.

Furthermore, I'd guess there is no much point to enable parallel
query execution on a macbook. Or at least one wouldn't expect
superb results from this anyway.

I'd make a wild claim that users who would benefit from parallel
queries or my concurrency work most of the time are the same
users who run platforms that can support single address space.

Thus if there is a solution that benefits e.g. 95% of target users
then why refrain from it in the name of the other 5%? Should not
the support of those 5% be treated as a lower-priority fallback,
while the main effort be put on optimizing for 95-percenters?

Regards,
Aleksey