pg_upgrade: How to deal with toast

We are now in discussion about toast table upgrading. I try to collect ideas and
figure out how it should work and where are problems.

Overview:
---------

A few weeks ago we made a decision to use convert on read. We already made a 
decision how to solve problem with overflow data after conversion. Now we need 
to make decision how to deal with toast table.

Toasted data are split into chunks and these chunks are stored into toasted 
table as a record with following structure:

(valueid oid, residx int32, chunk varlena)

Chunk size is defined by TOAST_MAX_CHUNK_SIZE.

Toast table could contain different datatypes and one page can contain different 
type chunks.

toast_fetch_datum, toast_fetch_datum_slice and functions are low level function 
which do a main job.

How to upgrade it:
------------------

Toasted values is processed on demand when some part of postgresql needs to have 
detosted value (see for example pg_detoast_datum()). The toast_fetch_datum 
function starts index scan where valueid is a search key. This scan invokes a 
toast index and toasttable page upgrade provided by hook in ReadBuffer and 
toast_fetch_datum gets already converted tuple, but chunked data will stay 
untouched.

Toasted datum can be converted only when is completely connected together. It of 
course invokes lot of page conversions.

The idea is to read toasted datum, convert it, store it back and old chunks mark 
as deleted. Same method will be use for slice access, because we cannot access 
selected slice until the toasted datum is not converted.

the implementation will add hook in toast_fetch_datum, toast_fetch_datum_slice 
functions which handle conversion (similar to hook in ReadBuffer).

Issues:
-------

1) different chunk size in old and new format.

It is not issue because, old format is read and connected in the conversion 
function and this function can accept different chunk size. (originally I 
supposed to replace residx with offset, but I think it is not necessary now for 
upgrade)

2) data type is unknown

Unfortunately, in low function is no clue what data type is really stored in a 
chunks. One idea how to solve it is to add attno to chunk record structure. By 
my opinion, It is best solution, but it disallow to upgrade from 8.3->8.4!

3) How to detect which toasted datum need conversion

One idea is to make magic with XMIN. Everything older then "upgraded" 
transaction needs conversion. But vacuum probably can freeze some old or new 
tuples and after that we lost information. Another possible solution is probably 
mark tuples on converted page in info mask.

    Ideas, comments?
            Zdenek