Re: Report bytes and transactions actually sent downtream

Hi,

On Wed, Sep 24, 2025 at 12:51:29PM +0530, Ashutosh Bapat wrote:
> On Wed, Sep 24, 2025 at 12:32 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> > > > - create a table and use pg_logical_slot_get_changes with ('skip-empty-xacts', '1')
> > > > then I don't see plugin_sent_bytes increasing (which makes sense) but I also don't
> > > > see plugin_filtered_bytes increasing. I think that would make sense to also increase
> > > > plugin_filtered_bytes in this case (and for the other options that would skip
> > > > sending data). Thoughts?
> > >
> > > Thanks for bringing this up. I don't think we discussed this
> > > explicitly in the thread. The changes which are filtered out by the
> > > core itself e.g. changes to the catalogs or changes to other databases
> > > or changes from undesired origins are not added to the reorder buffer.
> > > They are not counted in total_bytes. The transactions containing only
> > > such changes are not added to reorder buffer, so even total_txns does
> > > not count such empty transactions. If we count these changes and
> > > transactions in plugin_filtered_bytes, and plugin_filtered_txns, that
> > > would create an anomaly - filtered counts being higher than total
> > > counts. Further since core does not add these changes and transactions
> > > to the reorder buffer, there is no way for a plugin to know about
> > > their existence and hence count them. Does that make sense?
> >
> > Yes. Do you think that the doc in the patch is clear enough regarding this point?
> > I mean the doc looks correct (mentioning the output plugin) but would that make
> > sense to insist that core filtering is not taken into account?
> 
> Do you mean, should we mention in the docs that core filtering is not
> taken into account?
> I would question whether that's called filtering
> at all, in the context of logical decoding. The view should be read in
> the context of logical decoding. For example, we aren't mentioning
> that total_bytes does not include changes from other database.

Right. But, in the example above, do you consider "skip-empty-xacts" as "core"
or "plugin" filtering?

It's an option part of the "test_decoding" plugin, so it's the plugin choice to
not display empty xacts (should the option be set accordingly). Then should it
be reported in plugin_filtered_bytes? (one could write a plugin, decide to
skip/filter empty xacts or whatever in the plugin callbacks: should that be
reported as plugin_filtered_bytes?)

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com