Re: Changeset Extraction v7.3

On Tue, Jan 28, 2014 at 11:53 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> I've rebased it here and am hacking on it still.

OK.  The attached patch is a combination of Andres' first two patches
with lots more changes from both Andres and myself.   At this point,
I'm pretty happy with this, and propose to commit the attached
version, absent objection.  All by itself, it provides a useful new
option for users, and it sets the stage for the subsequent logical
decoding patches as well.  For those not following along closely,
here's the short version: if you choose to create a replication slot,
you can make the master retain the exact amount of WAL that the
standby still needs, rather than guessing what value to set for
wal_keep_segments; also, you can avoid hot standby conflicts even when
the connection between master and slave is interrupted (but the master
will bloat if the interruption is long, so watch out).  For logical
decoding, this functionality is essential rather than nice-to-have.

Here's a summary of what we've changed since the last version Andres posted:

- Fairly extensive revisions to slot error handling, eliminating the
PG_TRY/PG_CATCH blocks that were present before (which I didn't
believe would work as designed) and cleaning up some corner cases to
eliminate unnecessary failures.
- Rigid enforcement of existing PG practices around spinlocks,
volatile pointers, and memory barriers.
- Modification of the on-disk format for slots so that we don't
serialize junk that properly only lives in memory.
- Removal of various references to decoding and logical replication
that properly belong in subsequent patches.
- Support for using slots via a new recovery.conf parameter,
primary_slotname, and a new pg_receivexlog option, --slot.  I felt
this was important because, without this, you couldn't test that it
actually works without applying the remainder of Andres's patch set,
and even then you could mostly only test logical replication.  With
it, this is an independently useful and testable feature.
- Exclude pg_replslot from base backups.  This might need more thought
and documentation; people who use the filesystem method to perform
backups might need to be advised to remove this directory in some
cases also, or people who use pg_basebackup might want to keep it in
some cases (not sure).
- Lots of renaming to make the names more clear and consistent.
- Lots of bug fixes, minor tinkering, comment changes, and cleanups.
- Documentation.

For those wishing to see the blow-by-blow:

http://git.postgresql.org/gitweb/?p=users/rhaas/postgres.git;a=shortlog;h=refs/heads/slot2

In the future (i.e. post-9.4), I think we'll likely want to extend
this in a bunch of interesting ways.  I strongly suspect people are
going to want to have an option for slots that pin the LSN but not
xmin, and I also think they're going to want slots that hold LSN or
xmin for a certain amount of time after a disconnect but then give up,
or maybe a certain number of segments/transaction IDs and then give
up.  Nonwithstanding those important improvements, I think this is a
very credible v1 of this functionality.

Thanks,

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company