Recently we had a gripe about how you can't read very many files
concurrently with contrib/file_fdw:
http://www.postgresql.org/message-id/OF419B5767.8A3C9ADB-ON85257B79.005491E9-85257B79.0054F6F1@isn.rtss.qc.ca
The reason for this is that file_fdw goes through the COPY code, which
uses AllocateFile(), which has a wired-in assumption that not very many
files need to be open concurrently. I thought for a bit about trying to
get COPY to use a "virtual" file descriptor such as is provided by the
rest of fd.c, but that didn't look too easy, and in any case probably
nobody would be excited about adding additional overhead to the COPY
code path. What seems more practical is to relax the hard-coded limit
on the number of files concurrently open through AllocateFile(). Now,
we could certainly turn that array into some open-ended list structure,
but that still wouldn't let us have an arbitrary number of files open,
because at some point we'd run into platform-specific EMFILES or ENFILES
limits on the number of open file descriptors. In practice we want to
keep it under the max_safe_fds limit that fd.c goes to a lot of trouble
to determine. So it seems like the most useful compromise is to keep
the allocatedDescs[] array data structure as-is, but allow its size to
depend on max_safe_fds. In the attached proposed patch I limit it to
max_safe_fds / 2, so that there's still a reasonable number of FDs
available for fd.c's main pool of virtual FDs. On typical modern
platforms that should be at least a couple of hundred, thus making the
effective limit about an order of magnitude more than it is currently
(32).
Barring objections or better ideas, I'd like to back-patch this as far
as 9.1 where file_fdw was introduced.
regards, tom lane