Re: Streaming base backups

2011/1/10 Magnus Hagander <magnus@hagander.net>:
> On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
> <cedric.villemain.debian@gmail.com> wrote:
>> 2011/1/7 Magnus Hagander <magnus@hagander.net>:
>>> On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
>>> <cedric.villemain.debian@gmail.com> wrote:
>>>> 2011/1/5 Magnus Hagander <magnus@hagander.net>:
>>>>> On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
>>>>>> Magnus Hagander <magnus@hagander.net> writes:
>>>>>>> * Stefan mentiond it might be useful to put some
>>>>>>> posix_fadvise(POSIX_FADV_DONTNEED)
>>>>>>>   in the process that streams all the files out. Seems useful, as long as that
>>>>>>>   doesn't kick them out of the cache *completely*, for other backends as well.
>>>>>>>   Do we know if that is the case?
>>>>>>
>>>>>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
>>>>>> not already in SHM?
>>>>>
>>>>> I think that's way more complex than we want to go here.
>>>>>
>>>>
>>>> DONTNEED will remove the block from OS buffer everytime.
>>>
>>> Then we definitely don't want to use it - because some other backend
>>> might well want the file. Better leave it up to the standard logic in
>>> the kernel.
>>
>> Looking at the patch, it is (very) easy to add the support for that in
>> basebackup.c
>> That supposed allowing mincore(), so mmap(), and so probably switch
>> the fopen() to an open() (or add an open() just for mmap
>> requirement...)
>>
>> Let's go ?
>
> Per above, I still don't think we *should* do this. We don't want to
> kick things out of the cache underneath other backends, and since we

we are dropping stuff underneath other backends  anyway but I
understand your point.

> can't control that. Either way, it shouldn't happen in the beginning,
> and if it does, should be backed with proper benchmarks.

I agree.

>
> I've committed the backend side of this, without that. Still working
> on the client, and on cleaning up Heikki's patch for grammar/parser
> support.

--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support