Re: pg_combinebackup does not detect missing files

On 4/18/24 01:03, Robert Haas wrote:
> On Tue, Apr 16, 2024 at 7:25 PM David Steele <david@pgmasters.net> wrote:
> 
> But it will not go out of its way to perform checks that are unrelated
> to its documented purpose. And I don't think it should, especially if
> we have another tool that already does that.
> 
>>> I'm not averse to having some kind of statement in the documentation
>>> along the lines of "Note that pg_combinebackup does not attempt to
>>> verify that the individual backups are intact; for that, use
>>> pg_verifybackup."
>>
>> I think we should do this at a minimum.
> 
> Here is a patch to do that.

I think here:

+   <application>pg_basebackup</application> only attempts to verify

you mean:

+   <application>pg_combinebackup</application> only attempts to verify

Otherwise this looks good to me.

>> Especially given how pg_combinebackup works, backups are going to
>> undergo a lot of user manipulation (pushing to and pull from storage,
>> decompressing, untaring, etc.) and I think that means we should take
>> extra care.
> 
> We are in agreement on that point, if nothing else. I am terrified of
> users having problems with pg_combinebackup and me not being able to
> tell whether those problems are due to user error, Robert error, or
> something else. I put a lot of effort into detecting dumb things that
> I thought a user might do, and a lot of effort into debugging output
> so that when things do go wrong anyway, we have a reasonable chance of
> figuring out exactly where they went wrong. We do seem to have a
> philosophical difference about what the scope of those checks ought to
> be, and I don't really expect what I wrote above to convince you that
> my position is correct, but perhaps it will convince you that I have a
> thoughtful position, as opposed to just having done stuff at random.

Fair enough. I accept that your reasoning is not random, but I'm still 
not very satisfied that the user needs to run a separate and rather 
expensive process to do the verification when pg_combinebackup already 
has the necessary information at hand. My guess is that most users will 
elect to skip verification.

At least now they'll have the information they need to make an informed 
choice.

Regards,
-David