Re: pg_basebackup blocking all queries with horrible performance

On Thu, Jun 7, 2012 at 5:07 PM, Jerry Sievers <gsievers19@comcast.net> wrote:
> Lonni J Friedman <netllama@gmail.com> writes:
>
>> On Thu, Jun 7, 2012 at 12:40 PM, Magnus Hagander <magnus@hagander.net> wrote:
>>
>>> On Thu, Jun 7, 2012 at 8:04 PM, Lonni J Friedman <netllama@gmail.com> wrote:
>>>> On Thu, Jun 7, 2012 at 10:41 AM, Lonni J Friedman <netllama@gmail.com> wrote:
>>>>> Greetings,
>>>>> I have a 4 server postgresql-9.1.3 cluster (one master doing streaming
>>>>> replication to 3 hot standby servers).  All of them are running
>>>>> Fedora-16-x86_64.
>>>>>
>>>>> http://wiki.postgresql.org/wiki/Lock_Monitoring
>>>>
>>>> err, i included that URL but neglected to explain why.  On a different
>>>> list someone suggested that I verify that there were no locks that
>>>> were blocking things, and I did so, and found no locks.
>>>>
>>>> So I'm still at a loss why pg_basebackup is killing perf, and would
>>>> appreciate pointers on how to debug it or at least reduce its impact
>>>> on performance if that is possible.
>>>>
>>>
>>> My guess would be that you are overloading your I/O system. You should
>>> look at values from iostat and vmstat from when the system works fine
>>> and when you run pg_basebackup, that should give you a hint in the
>>> right direction.
>>
>> ok, thanks.  i'll take a look at that.  If this turns out to be the
>> issue, is there some way to get pg_basebackup to run more slowly, so
>> that it has less impact?  Or could I do this with ionice on the
>> pg_basebackup process?
>
> You might try stopping pg_basebackup in place with SIGSTOP and check
> if problem goes away.  SIGCONT and you should  start having
> sluggishness again.
>
> If verified, then any sort of throttling mechanism should work.

I'm certain that the problem is triggered only when pg_basebackup is
running.  Its very predictable, and goes away as soon as pg_basebackup
finishes running.  What do you mean by a throttling mechanism?