Обсуждение: pg_standby and build farm

Поиск
Список
Период
Сортировка

pg_standby and build farm

От
"Simon Riggs"
Дата:
I've written up a Warm Standby script as a .c program rather than
scripts, to allow it to be portable and potentially shipped as part of
PostgreSQL core.

pg_standby is designed to be a wait-for restore_command, required to
turn a normal archive recovery into a Warm Standby. Within the
restore_command of the recovery.conf you could configure pg_standby in
the following way:
   restore_command = 'pg_standby archiveDir %f %p'

It has a number of other useful options and works with Win32 also.

So far I've written this as a contrib module. Would people like to see
this submitted to -patches, or should I make it available via PgFoundry
HAPITR?

Should this be reworked as an src/bin program? Or is the contrib module
the best form for this in 8.3?

I've also written a test program that uses this. I'm using this on a
server at EDB to run a continuous Warm Standby test.

Would anybody consider this test script worth including somewhere in
core? and/or: Might it become part of the build farm?

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com




Re: pg_standby and build farm

От
Tatsuo Ishii
Дата:
I'd love to see your program. Could you please submit to -pacthes?
--
Tatsuo Ishii
SRA OSS, Inc. Japan

> I've written up a Warm Standby script as a .c program rather than
> scripts, to allow it to be portable and potentially shipped as part of
> PostgreSQL core.
> 
> pg_standby is designed to be a wait-for restore_command, required to
> turn a normal archive recovery into a Warm Standby. Within the
> restore_command of the recovery.conf you could configure pg_standby in
> the following way:
> 
>     restore_command = 'pg_standby archiveDir %f %p'
> 
> It has a number of other useful options and works with Win32 also.
> 
> So far I've written this as a contrib module. Would people like to see
> this submitted to -patches, or should I make it available via PgFoundry
> HAPITR?
> 
> Should this be reworked as an src/bin program? Or is the contrib module
> the best form for this in 8.3?
> 
> I've also written a test program that uses this. I'm using this on a
> server at EDB to run a continuous Warm Standby test.
> 
> Would anybody consider this test script worth including somewhere in
> core? and/or: Might it become part of the build farm?
> 
> -- 
>   Simon Riggs             
>   EnterpriseDB   http://www.enterprisedb.com
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
>        choose an index scan if your joining column's datatypes do not
>        match
> 


Re: pg_standby and build farm

От
Euler Taveira de Oliveira
Дата:
Simon Riggs wrote:

> I've written up a Warm Standby script as a .c program rather than
> scripts, to allow it to be portable and potentially shipped as part of
> PostgreSQL core.
> 
Great! It'll make the DBA's work much easier, could help the adoption of
PITR (nowadays people thinks it's to complicate to code a script) and
help Win32 users.

> So far I've written this as a contrib module. Would people like to see
> this submitted to -patches, or should I make it available via PgFoundry
> HAPITR?
> 
> Should this be reworked as an src/bin program? Or is the contrib module
> the best form for this in 8.3?
> 
Please submit it to -patches. IMHO it should stay in src/bin because
it'll be part of a solution that is tightly integrated to core (PITR).


--  Euler Taveira de Oliveira http://www.timbira.com/



Re: pg_standby and build farm

От
Tom Lane
Дата:
Euler Taveira de Oliveira <euler@timbira.com> writes:
> Simon Riggs wrote:
>> Should this be reworked as an src/bin program? Or is the contrib module
>> the best form for this in 8.3?
>> 
> Please submit it to -patches. IMHO it should stay in src/bin because
> it'll be part of a solution that is tightly integrated to core (PITR).

I think src/bin would be a seriously bad place for it, because that
carries the presumption that "there can be only one", whereas in reality
this will be more like an example of one way to do it.  contrib seems OK.

(And no, it should NOT be perceived as "tightly integrated to core".
The entire point of the design is to keep that behavior separate.)
        regards, tom lane


Re: pg_standby and build farm

От
"Andrew Dunstan"
Дата:
Simon Riggs wrote:

>
> I've also written a test program that uses this. I'm using this on a
> server at EDB to run a continuous Warm Standby test.
>
> Would anybody consider this test script worth including somewhere in
> core? and/or: Might it become part of the build farm?
>


Buildfarm automatically runs all regression tests in core, including in
contrib. Running some other test where the code is in core could be
trivial, as long as there was a boolean success criterion. But of course
it has to run on a single machine.

Send me details and I can take a look.

cheers

andrew



Re: pg_standby and build farm

От
"Simon Riggs"
Дата:
On Wed, 2006-12-13 at 19:43 -0500, Tom Lane wrote:
> Euler Taveira de Oliveira <euler@timbira.com> writes:
> > Simon Riggs wrote:
> >> Should this be reworked as an src/bin program? Or is the contrib module
> >> the best form for this in 8.3?
> >> 
> > Please submit it to -patches. IMHO it should stay in src/bin because
> > it'll be part of a solution that is tightly integrated to core (PITR).

Posted to pgsql-patches

> I think src/bin would be a seriously bad place for it, because that
> carries the presumption that "there can be only one", whereas in reality
> this will be more like an example of one way to do it.  contrib seems OK.

Agreed.

I will be updating it each time we have feedback on integration with
various BAR systems to provide more options as standard.

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com




Re: pg_standby and build farm

От
"Simon Riggs"
Дата:
On Thu, 2006-12-14 at 07:35 +0900, Tatsuo Ishii wrote:
> Could you please submit to -pacthes?

Patch submitted to -patches as requested last Thursday.

Any feedback would be gratefully received.

Thanks,

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com




Re: pg_standby and build farm

От
Doug Knight
Дата:
Hi all,<br /> I'm new to the forums, so bear with me on my questions. I've set up an auto-archive and auto-recover
pairof databases using pg_standby, which I'm prototyping various products for high availability. I've noticed that when
Iattempt to fail from the primary archiver to the secondary recovery db using the pg_standby trigger file, the
secondarydetects the trigger file, flags that it couldn't read the current WAL file pg_standby was waiting on, then
attemptsto read in the previous WAL file. I use the -m option in pg_standby, so the previous WAL file no longer exists,
whichcauses the secondary postgres to "panic" on not being able to open the previous WAL and terminates. Is there a way
toprevent the looking for the previous, or preserving the previous WAL file so that when the trigger file is detected,
thesecondary will come all the way up, completely its recovery mode? <br /><br /> Thanks,<br /> Doug Knight  

Re: pg_standby and build farm

От
"Simon Riggs"
Дата:
On Tue, 2006-12-26 at 10:34 -0500, Doug Knight wrote:

> I'm new to the forums, so bear with me on my questions. I've set up an
> auto-archive and auto-recover pair of databases using pg_standby,
> which I'm prototyping various products for high availability. 

Thanks for the feedback. pg_standby was written as a by-product of some
WAL recovery testing, so its in use, but only by me so far.

> I've noticed that when I attempt to fail from the primary archiver to
> the secondary recovery db using the pg_standby trigger file, the
> secondary detects the trigger file, flags that it couldn't read the
> current WAL file pg_standby was waiting on, then attempts to read in
> the previous WAL file. I use the -m option in pg_standby, so the
> previous WAL file no longer exists, which causes the secondary
> postgres to "panic" on not being able to open the previous WAL and
> terminates. Is there a way to prevent the looking for the previous, or
> preserving the previous WAL file so that when the trigger file is
> detected, the secondary will come all the way up, completely its
> recovery mode? 

Well, on a separate point I realised today that the -m option is faster
but just doesn't work perfectly with restartable recovery, so I had
already decided before reading this post that the move option needs to
be removed. Doubly so, now. Apologies for my slip.

Use the default copy mode and it should work fine.

The reason for the -m option was performance. Recovery is I/O-bound,
with 50% of the CPU it does use coming from IsRecordValid() - which is
where the CRC checking takes place. (I can add an option to recover
without CRC checks, if anyone wants it, once I've rejigged the option
parsing for recovery.conf.)

Should be able to use hard links, i.e. ln -f -s /archivepath/%f %p
instead. I'll test that tomorrow then issue a new version.

I'll also add a signal handler for SIGINT and SIGQUIT.

Comments?

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com




Re: pg_standby and build farm

От
"Simon Riggs"
Дата:
On Wed, 2006-12-27 at 20:09 +0000, Simon Riggs wrote:

> The reason for the -m option was performance. Recovery is I/O-bound,
> with 50% of the CPU it does use coming from IsRecordValid() - which is
> where the CRC checking takes place. (I can add an option to recover
> without CRC checks, if anyone wants it, once I've rejigged the option
> parsing for recovery.conf.)

Make that 70% of the CPU, for long running recoveries, but the CPU only
gets as high as 20% on my tests, so still I/O bound.

> Should be able to use hard links, i.e. ln -f -s /archivepath/%f %p
> instead. I'll test that tomorrow then issue a new version.

The ln works, and helps, but not that much. I'll remove the -m option
and replace it with an -l option. Must be careful to use the -f option.

The majority of the I/O comes from writing dirty buffers, so enabling
the bgwriter during recovery would probably be more helpful.

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com




Re: pg_standby and build farm

От
Doug Knight
Дата:
Thanks for the response, Simon. I'm continuing to use your script, so if there's anything I can help you with
(testing,ideas, etc), just let me know.<br /><br /> Doug<br /><br /> On Thu, 2006-12-28 at 13:40 +0000, Simon Riggs
wrote:<blockquote type="CITE"><pre>
 
<font color="#000000">On Wed, 2006-12-27 at 20:09 +0000, Simon Riggs wrote:</font>

<font color="#000000">> The reason for the -m option was performance. Recovery is I/O-bound,</font>
<font color="#000000">> with 50% of the CPU it does use coming from IsRecordValid() - which is</font>
<font color="#000000">> where the CRC checking takes place. (I can add an option to recover</font>
<font color="#000000">> without CRC checks, if anyone wants it, once I've rejigged the option</font>
<font color="#000000">> parsing for recovery.conf.)</font>

<font color="#000000">Make that 70% of the CPU, for long running recoveries, but the CPU only</font>
<font color="#000000">gets as high as 20% on my tests, so still I/O bound.</font>

<font color="#000000">> Should be able to use hard links, i.e. ln -f -s /archivepath/%f %p</font>
<font color="#000000">> instead. I'll test that tomorrow then issue a new version.</font>

<font color="#000000">The ln works, and helps, but not that much. I'll remove the -m option</font>
<font color="#000000">and replace it with an -l option. Must be careful to use the -f option.</font>

<font color="#000000">The majority of the I/O comes from writing dirty buffers, so enabling</font>
<font color="#000000">the bgwriter during recovery would probably be more helpful.</font>

</pre></blockquote>

Re: pg_standby and build farm

От
"Simon Riggs"
Дата:
On Thu, 2006-12-28 at 08:45 -0500, Doug Knight wrote:

> Thanks for the response, Simon. I'm continuing to use your script, so
> if there's anything I can help you with (testing, ideas, etc), just
> let me know.

My earlier comments about mv were not correct; when fully retesting
things, I noted another error instead and have corrected that.

pg_standby now supports cp, mv and ln - all 3 now tested on Linux.

Posting new version to patches; no signal handling (yet).

Another other comments gratefully received.

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com