Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Hi Melih,

Last week we revisited your implementation of design#2. Vignesh rebased it, and then made a few other changes.

PSA v28*

The patch changes include:
* changed the logic slightly by setting recv_immediately(new variable), if this variable is set the main apply worker loop will not wait in this case.
* setting the relation state to ready immediately if there are no more incremental changes to be synced.
* receive the incremental changes if applicable and set the relation state to ready without waiting.
* reuse the worker if the worker is free before trying to start a new table sync worker
* restarting the tablesync worker only after wal_retrieve_retry_interval

~

FWIW, we just wanted to share with you the performance measurements seen using this design#2 patch set:

======

RESULTS (not busy tests)

------
10 empty tables
                2w      4w      8w      16w
HEAD:           125     119     140     133
HEAD+v28*:      92      93      123     134
%improvement:   27%     22%     12%     -1%
------
100 empty tables
                2w      4w      8w      16w
HEAD:           1037    843     1109    1155
HEAD+v28*:      591     625     2616    2569
%improvement:   43%     26%     -136%   -122%
------
1000 empty tables
                2w      4w      8w      16w
HEAD:           15874   10047   9919    10338
HEAD+v28*:      33673   12199   9094    9896
%improvement:   -112%   -21%    8%      4%
------
2000 empty tables
                2w      4w      8w      16w
HEAD:           45266   24216   19395   19820
HEAD+v28*:      88043   21550   21668   22607
%improvement:  -95%     11%     -12%    -14%

~~~

Note - the results were varying quite a lot in comparison to the HEAD
e.g. HEAD results are very consistent, but the v28* results observed are not
HEAD 1000 (2w): 15861, 15777, 16007, 15950, 15886, 15740, 15846, 15740, 15908, 15940
v28* 1000 (2w):  34214, 13679, 8792, 33289, 31976, 56071, 57042, 56163, 34058, 11969

------
Kind Regards,
Peter Smith.
Fujitsu Australia