Re: [BUGS] server crash in very big transaction [postgresql

On Tue, 24 Aug 2004, Tom Lane wrote:

> I wrote:
> > What is happening of course is that more than 16K subtransaction IDs
> > won't fit in a commit record (since XLOG records have a 16-bit length
> > field).  We're gonna have to rethink the representation of subxact
> > commit in XLOG.
>
> After some further thought, I think there are basically two ways to
> attack this:
>
> 1. Allow XLOG records to be larger than 64K.
>
> 2. Split transaction commit into multiple XLOG records when there are
>    many subtransactions.
>

[snip]

> I'm inclined to go with #1.  There are various ways we could do it
> but the most straightforward would be to just widen the xl_len field
> to 32 bits.  This would cost either 4 or 8 bytes per XLOG record
> (because of MAXALIGN restrictions) but we could more than buy that back
> by eliminating the xl_prev and/or xl_xact_prev fields, which have no use
> in the current system.  (They were intended to support UNDO but it seems
> clear that we will never do that.)

If we have to do an initdb for a subsequent beta, could we just remove
these anyway? By my count, we've got at least 16 bytes there.

As for extending the length of xl_len, what happens if someone now has
2^30 subtransaction IDs (as unlikely as that sounds)? What I mean is, it
would be good if we could detect this at a point when we can issue an
ERROR. If we go down this path, we should also document the maximum number
of sub transaction IDs which can be used within a single block so that
if/when people look at doing stuff on that scale are aware of the
limitations.

>
> Or we could assign an rmgr value to represent an "extension" record that
> is to be merged with a following "normal" record.  This is kinda klugy
> but would avoid wasting bits on xl_len in the vast majority of records.
> Also we'd not have to force an initdb since the file format would
> remain upward-compatible.

This is a better idea, I think, as it avoids the problems above and, as
you say, will be binary compatible.

Gavin