BUG #15693: Suggestion on libpq memory management
От | PG Bug reporting form |
---|---|
Тема | BUG #15693: Suggestion on libpq memory management |
Дата | |
Msg-id | 15693-0df90da151425ff5@postgresql.org обсуждение исходный текст |
Список | pgsql-bugs |
The following bug has been logged on the website: Bug reference: 15693 Logged by: ilya b. Email address: partorg.dunaev@gmail.com PostgreSQL version: 10.7 Operating system: OS X, Ubuntu linux Description: Hi everyone! I spent some time investigating an occurence of a supposed memory leak in our app. The application is built using libpq and it's main purpose is to listen on the network and put some of the received data to the database. To do that we are calling stored functions with string parameters. Most requests are small (under 1K), but some are quite large, sometimes up to hundred Ks. What I noticed is that some of those rare large requests made libpq allocate and retain memory. Allocation happened at the time of stored function call sending and it wasn't released until DB connection was done. Please see Valgring report below: ==2418== 524,288 bytes in 1 blocks are still reachable in loss record 633 of 635 ==2418== at 0x4C31D2F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==2418== by 0x9651E23: pqCheckOutBufferSpace (fe-misc.c:369) ==2418== by 0x9651E74: pqPutMsgBytes (fe-misc.c:569) ==2418== by 0x9651F95: pqPutnchar (fe-misc.c:250) ==2418== by 0x964F556: PQsendQueryGuts (fe-exec.c:1559) ==2418== by 0x964F92F: PQsendQueryPrepared (fe-exec.c:1401) ==2418== by 0x96509A1: PQexecPrepared (fe-exec.c:1968) The function responsible for memory allocation was indeed pqCheckOutBufferSpace. Once it encountered a chunk that was larger than available buffer size, it reallocated buffer to fit the chunk. The problem here is that buffers stay in place until PGconn is released. Since our app has a connection pool and we prefer to keep DB connections alive as long as we can, idle memory usage becomes a problem. After some time every connection aquires a good chunk of memory (somewhat proportional to the largest call it made). My suggestion would be to add PGreclaimBuffers(PQconn *) function that would realloc unused buffers back to their initial size. So in our case the usage might be as follows: conn = PQconnectdb(...) for(data) { res = PQexecPrepared(conn, data) // data processing PQclear(res) if (data.is_large) { PGreclaimBuffers(conn) } } My solution was to patch sources and rebuild libpq.so we use to link our app with. But hopefully libpq API could be extended to include buffer reclaiming specifically for the case of long lived connections and/or large requests.
В списке pgsql-bugs по дате отправления: