Re: Load Distributed Checkpoints, take 3

Tom Lane wrote:
> Heikki Linnakangas <heikki@enterprisedb.com> writes:
>> Tom Lane wrote:
>>> Who's "we"?  AFAICS, CVS HEAD will treat a large copy the same as any
>>> other large heapscan.
>
>> Umm, I'm talking about populating a table with COPY *FROM*. That's not a
>> heap scan at all.
>
> No wonder we're failing to communicate.  I assumed you were talking
> about copy-to-file.  Copy-from-file is going to be creating WAL entries
> hence the no-checkpoint case doesn't apply anyway, no?

We are indeed having trouble to communicate :(.

No, I'm not talking about the new non-WAL-logged COPY optimization. COPY
FROM *would* create WAL entries, and the next checkpoint would clean
them. So far so good. But then you run VACUUM, as you often do after
loading a table, to set all hint bits. That will *not* generate WAL, and
next checkpoint is skipped.

To recap, the sequence is:

1. COPY FROM
2. checkpoint
3. VACUUM

Now you have buffer cache full of dirty buffers with usage_count=1, and
no WAL activity since last checkpoint. They will not be flushed until:
a) WAL activity happens and next checkpoint comes
b) database is shut down, or manual CHECKPOINT is issued
c) clock hand advances and decrements the usage_counts

It's a corner case. Probably not a problem in practice; you usually run
CREATE INDEX after loading a table, for example. But it exists.

--
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com