Re: Load Distributed Checkpoints, take 3

Heikki Linnakangas <heikki@enterprisedb.com> writes:
> Tom Lane wrote:
>> Heikki Linnakangas <heikki@enterprisedb.com> writes:
>>> If you have a system with a very bursty transaction rate, it's possible
>>> that when it's time for a checkpoint, there hasn't been any WAL logged
>>> activity since last checkpoint, so we skip it. When that happens, the
>>> buffer cache might still be full of dirty pages, because of hint bit
>>> updates. That still isn't a problem on it's own, but if you then do a
>>> huge batch update, you have to flush those dirty pages at that point. It
>>> would be better to trickle flush those dirty pages during the idle period.
>>
>> But wouldn't the LRU-based scan accomplish that?

> It only scans bgwriter_lru_percent buffers ahead of the clock hand. If
> the hand isn't moving, it keeps scanning the same buffers over and over
> again. You can crank it all the way up to 100%, though, in which case it
> would work, but that starts to get expensive CPU-wise.

Hmm.  But if we're going to do that, we might as well have a checkpoint
for our troubles, no?  The reason for the current design is the
assumption that a bgwriter_all scan is less burdensome than a
checkpoint, but that is no longer true given this rewrite.  AFAICS all
the bgwriter_all scan will accomplish is induce extra I/O in most
scenarios.  And it's certainly extra code in an area that's already been
rendered pretty darn baroque by this patch.

Also, the design assumption here is that the bgwriter_lru scan is
supposed to keep enough buffers clean to satisfy the system's need for
new buffers.  If it's not able to do so, it's not apparent to me that
the bgwriter_all scan helps --- the latter is most likely flushing the
wrong buffers, ie, those not close in front of the clock sweep.  You'd
be well advised to increase lru_percent anyway to keep more buffers
clean, if this scenario is worth optimizing rather than others for your
usage.

            regards, tom lane