Re: PATCH: logical_work_mem and logical streaming of largein-progress transactions

On Tue, Jan 28, 2020 at 11:58 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > > It seems to me that we need to add all of this new handling because
> > > > while taking the decision whether to stream or not we don't know
> > > > whether the txn has changes that can't be streamed.  One idea to make
> > > > it work is that we identify it while decoding the WAL.  I think we
> > > > need to set a bit in the insert/delete WAL record to identify if the
> > > > tuple belongs to a toast relation.  This won't add any additional
> > > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > > and also decoding will be efficient.  If this is feasible, then we can
> > > > do the same for speculative insertions.
> > > The Idea looks good to me.  I will work on this.
> > >
> >
> > One more thing we can do is to identify whether the tuple belongs to
> > toast relation while decoding it.  However, I think to do that we need
> > to have access to relcache at that time and that might add some
> > overhead as we need to do that for each tuple. Can we investigate
> > what it will take to do that and if it is better than setting a bit
> > during WAL logging.
>
> IMHO, for the catalog scan, we will have to start/stop the transaction
> for each change.  So do you want that we should evaluate its
> performance?
>

No, I was not thinking about each change, but at the level of ReorderBufferTXN.

>  Also, during we get the change we might not have the
> complete historic snapshot ready to fetch the rel cache entry.
>

Before decoding each change (say DecodeInsert), we call
SnapBuildProcessChange.  Isn't that sufficient?

Even, if the above is possible, I am not sure how good is it for each
change we fetch rel cache entry, that is the point I was worried.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com