Lock timeouts and unusual spikes in replication lag with logical parallel transaction streaming

Hi all,

In testing out the subscription parameter "streaming" we've noticed some unusual spikes in replay_lsn lag (and also confirmed_flush_lsn lag) when using "streaming = parallel".
During the periods of unusual lag (when compared to a separate "streaming = off" subscriber) we also see some additional logging from the (replica) postgres server about waiting for / acquiring locks, as well as error messages about logical apply workers exiting.

Then, today we found the logical replica stuck with a lag of about 30GB. The logs revealed a looping lock timeout with the parallel apply workers, the relevant section of which is attached.

While these workers were stuck, we queried the pg_locks catalog table in an attempt to collect more information. The query itself, and the resulting data (run with "\watch 1") are attached.

What I see in the output is that the lead apply worker (pid 2857187) is blocking the parallel worker (pid 2857523), then later another parallel worker (pid 2861846) is also blocking the lead apply worker (pid 2857187).
What I can't see, is anything blocking pid 2861846, and the logs don't show much about this worker (only that it starts and then is terminated). It's not clear to me whether this is a deadlock, or a case where we need to increase the lock_timeout (we didn't try).

At this point, I don't have a description of the upstream load that leads to this issue, and I don't have a reproducer script

One thing I should mention is that the schema of the publisher and subscriber are completely identical; the subscriber was created using pg_createsubscriber, and has not had any DDL applied (except for temp tables) since. I know that there is a risk of deadlocks in the case of schema differences, but there certainly isn't an extra index, constraint, or anything like that on the subscriber.

Given the poor performance and issues we are having with the "streaming = parallel" option, we will likely be sticking with the "streaming = on" option, which successfully worked to reduce our logical replication lag.
I noticed that the "parallel" option is slated to become the default for new subscriptions in PostgreSQL 18, and I wonder whether this is wise, given the potential for locking issues; in our replication cluster, several distinct scheduled jobs lead to the parallel streaming logical replication falling far behind the non-streaming version.

PostgreSQL version and some relevant configuration information:

                                                 version
----------------------------------------------------------------------------------------------------------
PostgreSQL 17.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5), 64-bit

shared_buffers
----------------
98172MB

work_mem
----------
6GB

max_worker_processes
----------------------
36

max_logical_replication_workers
---------------------------------
8

max_parallel_apply_workers_per_subscription
---------------------------------------------
8

Thanks,
Zane