Re: Flushing large data immediately in pqcomm

Hi hackers,

I did some experiments with this patch, after previous discussions. This probably does not answer all questions, but would be happy to do more if needed.

First, I updated the patch according to what suggested here [1]. PSA  v2.

I tweaked the master branch a bit to not allow any buffering. I compared HEAD, this patch and no buffering at all.

I also added a simple GUC to control PqSendBufferSize, this change only allows to modify the buffer size and should not have any impact on performance.

I again ran the COPY TO STDOUT command and timed it. AFAIU COPY sends data row by row, and I tried running the command under different scenarios with different # of rows and row sizes. You can find the test script attached (see test.sh).

All timings are in ms.

1- row size = 100 bytes, # of rows = 1000000

Some quick observations:

1- Even though I expect both the patch and HEAD behave similarly in case of small data (case 1: 100 bytes), the patch runs slightly slower than HEAD.

2- In cases where the data does not fit into the buffer, the patch starts performing better than HEAD. For example, in case 2, patch seems faster until the buffer size exceeds the data length. When the buffer size is set to something larger than 10KB (16KB/32KB in this case), there is again a slight performance loss with the patch as in case 1.

3- With large row sizes (i.e. sizes that do not fit into the buffer) not buffering at all starts performing better than HEAD. Similarly the patch performs better too as it stops buffering if data does not fit into the buffer.

[1]

Thanks,