Обсуждение: pgsql: Implement pipeline mode in libpq

Implement pipeline mode in libpq

Pipeline mode in libpq lets an application avoid the Sync messages in
the FE/BE protocol that are implicit in the old libpq API after each
query.  The application can then insert Sync at its leisure with a new
libpq function PQpipelineSync.  This can lead to substantial reductions
in query latency.

Co-authored-by: Craig Ringer <craig.ringer@enterprisedb.com>
Co-authored-by: Matthieu Garrigues <matthieu.garrigues@gmail.com>
Co-authored-by: Álvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Aya Iwata <iwata.aya@jp.fujitsu.com>
Reviewed-by: Daniel Vérité <daniel@manitou-mail.org>
Reviewed-by: David G. Johnston <david.g.johnston@gmail.com>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Kirk Jamison <k.jamison@fujitsu.com>
Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
Reviewed-by: Nikhil Sontakke <nikhils@2ndquadrant.com>
Reviewed-by: Vaishnavi Prabakaran <VaishnaviP@fast.au.fujitsu.com>
Reviewed-by: Zhihong Yu <zyu@yugabyte.com>

Discussion: https://postgr.es/m/CAMsr+YFUjJytRyV4J-16bEoiZyH=4nj+sQ7JP9ajwz=B4dMMZw@mail.gmail.com
Discussion: https://postgr.es/m/CAJkzx4T5E-2cQe3dtv2R78dYFvz+in8PY7A8MArvLhs_pg75gg@mail.gmail.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/acb7e4eb6b1c614c68a62fb3a6a5bba1af0a2659

Modified Files
--------------
doc/src/sgml/libpq.sgml                            |  522 +++++++-
doc/src/sgml/lobj.sgml                             |    4 +
.../libpqwalreceiver/libpqwalreceiver.c            |    6 +
src/bin/pg_amcheck/pg_amcheck.c                    |    2 +
src/interfaces/libpq/exports.txt                   |    4 +
src/interfaces/libpq/fe-connect.c                  |   37 +-
src/interfaces/libpq/fe-exec.c                     |  717 +++++++++--
src/interfaces/libpq/fe-protocol3.c                |   77 +-
src/interfaces/libpq/libpq-fe.h                    |   21 +-
src/interfaces/libpq/libpq-int.h                   |   60 +-
src/test/modules/Makefile                          |    1 +
src/test/modules/libpq_pipeline/.gitignore         |    5 +
src/test/modules/libpq_pipeline/Makefile           |   20 +
src/test/modules/libpq_pipeline/README             |    1 +
src/test/modules/libpq_pipeline/libpq_pipeline.c   | 1303 ++++++++++++++++++++
.../modules/libpq_pipeline/t/001_libpq_pipeline.pl |   28 +
src/tools/msvc/Mkvcbuild.pm                        |    9 +-
src/tools/pgindent/typedefs.list                   |    2 +
18 files changed, 2706 insertions(+), 113 deletions(-)

Hi Alvaro,

On Tue, 16 Mar 2021 at 10:20, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> Implement pipeline mode in libpq

> src/test/modules/libpq_pipeline/libpq_pipeline.c   | 1303 ++++++++++++++++++++

I'm wondering if you meant to leave the "#define DEBUG" line at line
34 in the above file?

It seems pretty strange to do:

#define DEBUG
#ifdef DEBUG
#define pg_debug(...)  do { fprintf(stderr, __VA_ARGS__); } while (0)
#else
#define pg_debug(...)
#endif

pg_debug will never be an empty macro.

I noticed this when testing compiling postgres with MSVC. The compiler
is kicking out a warning: 'DEBUG': macro redefinition, which will be
because that compiler defines DEBUG when doing non-production builds.

David

On Tue, 30 Mar 2021 at 20:19, David Rowley <dgrowleyml@gmail.com> wrote:
> On Tue, 16 Mar 2021 at 10:20, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> > Implement pipeline mode in libpq
>
> > src/test/modules/libpq_pipeline/libpq_pipeline.c   | 1303 ++++++++++++++++++++
>
> I'm wondering if you meant to leave the "#define DEBUG" line at line
> 34 in the above file?
>
> It seems pretty strange to do:
>
> #define DEBUG
> #ifdef DEBUG
> #define pg_debug(...)  do { fprintf(stderr, __VA_ARGS__); } while (0)
> #else
> #define pg_debug(...)
> #endif
>
> pg_debug will never be an empty macro.
>
> I noticed this when testing compiling postgres with MSVC. The compiler
> is kicking out a warning: 'DEBUG': macro redefinition, which will be
> because that compiler defines DEBUG when doing non-production builds.

I'm still not really sure why this code does #define directly followed
by an #ifdef for that define.  However, just to clear up the final
compiler warning that's currently being produced on MSVC, I've
attached a proposed patch to fix it.

None of the buildfarm animals are producing this warning as they all
seem to be building as "Release" builds.

libpq_pipeline.c on my machine is compiled with the following:

         C:\Program Files (x86)\Microsoft Visual
Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\CL.exe
/c /Isrc/include /Isrc/include/port/win32
/Isrc/include/port/win32_msvc /Isrc\interfaces\libpq /Zi /nologo /W3
/WX- /diagnostics:classic /Od /D WIN32 /D _WINDOWS /D __WINDOWS__ /D
__WIN32__ /D WIN32_STACK_RLIMIT=4194304 /D _CRT_SECURE_NO_DEPRECATE /D
_CRT_NONSTDC_NO_DEPRECATE /D _DEBUG /D DEBUG=1_MBCS /GF- /Gm- /EHsc
/MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline
/Fo".\Debug\libpq_pipeline\\" /Fd".\Debug\libpq_pipeline\vc141.pdb"
/Gd /TC /wd4018 /wd4244 /wd4273 /wd4102 /wd4090 /wd4267 /FC
/errorReport:queue /MP
src/test/modules/libpq_pipeline/libpq_pipeline.c
         libpq_pipeline.c

Note: /D DEBUG=1_MBCS

This causes:

Build succeeded.

       "L:\Projects\Postgres\e\pgsql.sln" (default target) (1) ->
       "L:\Projects\Postgres\e\libpq_pipeline.vcxproj" (default target) (88) ->
       (ClCompile target) ->
         l:\projects\postgres\e\src\test\modules\libpq_pipeline\libpq_pipeline.c(38):
warning C4005: 'DEBUG': macro redefinition
[L:\Projects\Postgres\e\libpq_pipeline.vcxproj]

    1 Warning(s)
    0 Error(s)

Whereas on hamerkop, they are [1]:

  C:\\Program Files (x86)\\Microsoft Visual
Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.15.26726\\bin\\HostX86\\x64\\CL.exe
/c /Isrc/include /Isrc/include/port/win32
/Isrc/include/port/win32_msvc /I"c:\\pkg\\zlib-dist\\include"
/I"c:\\OpenSSL-Win64\\include" /Isrc\\interfaces\\libpq /Zi /nologo
/W3 /WX- /diagnostics:classic /Ox /D WIN32 /D _WINDOWS /D __WINDOWS__
/D __WIN32__ /D WIN32_STACK_RLIMIT=4194304 /D _CRT_SECURE_NO_DEPRECATE
/D _CRT_NONSTDC_NO_DEPRECATE /D _MBCS /GF /Gm- /EHsc /MD /GS
/fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline
/Fo".\\Release\\libpq_pipeline\\\\"
/Fd".\\Release\\libpq_pipeline\\vc141.pdb" /Gd /TC /wd4018 /wd4244
/wd4273 /wd4102 /wd4090 /wd4267 /FC /errorReport:queue /MP
src/test/modules/libpq_pipeline/libpq_pipeline.c
  libpq_pipeline.c

Note the mention of "Release", indicating the Release rather than Debug build.

Does anyone want to comment on the #define just before the #ifdef?

David

[1] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=hamerkop&dt=2021-04-02%2010%3A18%3A13&stg=make