Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS
От | Tom Lane |
---|---|
Тема | Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS |
Дата | |
Msg-id | 23331.1370711218@sss.pgh.pa.us обсуждение исходный текст |
Ответы |
Re: Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS
Re: Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS |
Список | pgsql-hackers |
Recently we had a gripe about how you can't read very many files concurrently with contrib/file_fdw: http://www.postgresql.org/message-id/OF419B5767.8A3C9ADB-ON85257B79.005491E9-85257B79.0054F6F1@isn.rtss.qc.ca The reason for this is that file_fdw goes through the COPY code, which uses AllocateFile(), which has a wired-in assumption that not very many files need to be open concurrently. I thought for a bit about trying to get COPY to use a "virtual" file descriptor such as is provided by the rest of fd.c, but that didn't look too easy, and in any case probably nobody would be excited about adding additional overhead to the COPY code path. What seems more practical is to relax the hard-coded limit on the number of files concurrently open through AllocateFile(). Now, we could certainly turn that array into some open-ended list structure, but that still wouldn't let us have an arbitrary number of files open, because at some point we'd run into platform-specific EMFILES or ENFILES limits on the number of open file descriptors. In practice we want to keep it under the max_safe_fds limit that fd.c goes to a lot of trouble to determine. So it seems like the most useful compromise is to keep the allocatedDescs[] array data structure as-is, but allow its size to depend on max_safe_fds. In the attached proposed patch I limit it to max_safe_fds / 2, so that there's still a reasonable number of FDs available for fd.c's main pool of virtual FDs. On typical modern platforms that should be at least a couple of hundred, thus making the effective limit about an order of magnitude more than it is currently (32). Barring objections or better ideas, I'd like to back-patch this as far as 9.1 where file_fdw was introduced. regards, tom lane
Вложения
В списке pgsql-hackers по дате отправления: