Обсуждение: BUG #5166: readdir() failure on Mac OS-X is HFS "feature"
The following bug has been logged online: Bug reference: 5166 Logged by: Stephen Tyler Email address: stephen@stephen-tyler.com PostgreSQL version: 8.4.1 Operating system: Snow Leopard OS-X 10.6.1 (64 bit) Description: readdir() failure on Mac OS-X is HFS "feature" Details: I'm frequently getting these errors in my console: 4/11/09 2:25:04 PM org.postgresql.postgres[192] ERROR: could not read directory "pg_xlog": Invalid argument 4/11/09 2:25:56 PM org.postgresql.postgres[192] ERROR: could not read directory "pg_xlog": Invalid argument 4/11/09 2:36:03 PM org.postgresql.postgres[192] ERROR: could not read directory "pg_xlog": Invalid argument and rarely: 3/11/09 10:32:31 PM org.postgresql.postgres[217] ERROR: could not read directory "pg_clog": Invalid argument The pg_xlog errors occur periodically, or can be induced by any large change (eg deleting 1 million rows) System: Mac Pro Quad Nahelem 2.93GHz, 16GB RAM running Snow Leopard OS X 10.6.1 in 64bit mode Postgres 8.4.1 (Intel 64 bit) from http://www.kyngchaos.com/software:postgres running in 64 bit mode Database is on an SSD Raid 0 array The error appears to be generated in src/port/dirmod.c: pgfnames(const char *path) { .... while ((file = readdir(dir)) != NULL) { .... errno = 0; } .... if (errno) { .... fprintf(stderr, _("could not read directory \"%s\": %s\n"), path, strerror(errno)); .... } I previously posted this to pgsql-general "Could not read directory \"pg_xlog\": Invalid argument (on SSD Raid)" on Nov 4 2009. Tom Lane replied: This is a known bug in Snow Leopard --- readdir() calls fail after having deleted a file in the directory. We are hoping that Apple fixes it in 10.6.2, because trying to kluge around it seems like a mess. I am submitting it as a bug because: 1) It occurs in a different place than previously reported readdir() related errors 2) It appears to be a "feature" of HFS on OS-X, rather than a "bug": http://support.apple.com/kb/TA21420?viewlocale=en_US <quote> Some implementations of the "rm" command line tool (and other tools that support the recursive removal of files and directories) depend on an unspecified behavior outlined below. opendir(); while (readdir()) unlink(); closedir(); The unspecified behavior is what readdir() should return after the directory has been modified. Many file systems have been implemented such that subsequent readdir() calls will return the next directory entry. The implementation of the HFS file system cannot guarantee that all enclosed files or directories will be removed using the above method. The Mac OS X "rm" command does not exhibit this behavior because it uses the fts(3) library functions to traverse the directory hierarchy, which causes the entire contents of the directory to be read before modifying it. Solution The readdir()/unlink() loop needs to call rewinddir() either after each unlink() call, or whenever readdir returns NULL immediately after files have been unlinked. For example: "The following pseudocode describes this solution:" opendir(); do { unlinkedfiles = 0; while(readdir()) { unlink(); unlinkedfiles = 1; } if (unlinkedfiles) rewinddir(); } while (unlinkedfiles); closedir(); </quote> My reading of the above support article from Apple leads me to believe that Apple has no plans in the near future to alter/fix the behaviour of readdir() in the OS. The previously suggested hack/patch to fix DROP TABLESPACE is insufficient: pgsql/src/backend/commands: tablespace.c (r1.61 -> r1.62) (http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/commands/tablesp ace.c?r1=1.61&r2=1.62) I have found readdir() calls in: src/port/dirent.c src/port/dirmod.c src/bin/initdb/initdb.c src/bin/pg_resetxlog/pg_resetxlog.c src/bin/psql/create_help.pl src/tools/msvc/Install.pm src/tools/msvc/Mkvcbuild.pm src/backend/storage/file/fd.c contrib/pg_standby/pg_standby.c I don't know which of the above depend on the "unspecified" readdir behaviour (beyond dirmod.c), and are thus prone to failure on Mac OS X.
Stephen Tyler wrote: > I have found readdir() calls in: > src/port/dirent.c > src/port/dirmod.c > src/bin/initdb/initdb.c > src/bin/pg_resetxlog/pg_resetxlog.c > src/bin/psql/create_help.pl > src/tools/msvc/Install.pm > src/tools/msvc/Mkvcbuild.pm > src/backend/storage/file/fd.c > contrib/pg_standby/pg_standby.c > > I don't know which of the above depend on the "unspecified" readdir > behaviour (beyond dirmod.c), and are thus prone to failure on Mac OS X. Probably the most relevant one is in storage/file/fd.c, in AllocateDir, which is used in a lot of places, some of which do delete files: for example the one in RemoveOldXlogFiles probably explains the pg_xlog failures, and SlruScanDirectory explains the pg_clog failures (along with others). I wouldn't be surprised if others like the ones in RelationCacheInitFileRemove and RelationCacheInitFileRemoveInDir need patching as well. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
"Stephen Tyler" <stephen@stephen-tyler.com> writes: > Tom Lane replied: > This is a known bug in Snow Leopard --- readdir() calls fail after having > deleted a file in the directory. We are hoping that Apple fixes it in > 10.6.2, because trying to kluge around it seems like a mess. > 2) It appears to be a "feature" of HFS on OS-X, rather than a "bug": > http://support.apple.com/kb/TA21420?viewlocale=en_US Please notice that that document refers to 10.0. If it's not a Snow Leopard bug, why has the behavior changed from every prior OSX release? regards, tom lane
On Thu, Nov 5, 2009 at 10:10 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > "Stephen Tyler" <stephen@stephen-tyler.com> writes: >> Tom Lane replied: >> This is a known bug in Snow Leopard --- readdir() calls fail after having >> deleted a file in the directory. =A0We are hoping that Apple fixes it in >> 10.6.2, because trying to kluge around it seems like a mess. > >> 2) It appears to be a "feature" of HFS on OS-X, rather than a "bug": >> http://support.apple.com/kb/TA21420?viewlocale=3Den_US > > Please notice that that document refers to 10.0. =A0If it's not a Snow > Leopard bug, why has the behavior changed from every prior OSX release? I don't think there is much doubt that this is a bug, however much Apple might like to rebrand it as a feature. The interesting question is whether they're likely to fix it, or just leave everyone in the entire universe to patch around a stupid behavior that is present in their, and only their, implementation. If it's the latter, there's no reason for us to be the last ones on board. ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Nov 5, 2009 at 10:10 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Please notice that that document refers to 10.0. If it's not a Snow
>> Leopard bug, why has the behavior changed from every prior OSX release?
> I don't think there is much doubt that this is a bug, however much
> Apple might like to rebrand it as a feature.
More to the point, 10.0 was a LONG time ago. Is this document still
current, or did Apple fix their problem since then?
regards, tom lane
Some more information on the readdir() failure. I investigated the way GNU handled the problem with rm for Darwin. http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/remove.c [Note the current version of this file no longer uses readdir() - more below] It appears that readdir() can fail: 1) On all Darwin/Mac OS-X for HFS+ and NFS 2) On SunOS on UFS It appears to be triggered be a combination of the number of changes and the total filename length of those changes forcing a new hash table to be created. It appears that you are unlikely to observe this bug if the number of files in the directory is no more than 180 or so. But I think they found pathological cases as low as 16. My pg_xlog currently has 259 files :-( Looking at coreutils/src/remove.c as at 2006-09-29: Changelog: +2006-09-29 Jim Meyering <jim@meyering.net> + + Work around a readdir bug in Darwin 7.9.0 (MacOS X 10.3.9) on HFS+ + and NFS, whereby rm would not remove all files in a directory. + * src/remove.c (CONSECUTIVE_READDIR_UNLINK_THRESHOLD): Reduce to 10. + (NEED_REWIND): New macro, so that we incur the cost of the work-around + rewinddir only on afflicted systems. + * NEWS: Clarify and correct. + * tests/rm/readdir-bug: New file. Test for the above fix. + * tests/rm/Makefile.am (TESTS): Add it. + Prompted by testing and analysis from Bruno Haible: + http://lists.gnu.org/archive/html/bug-coreutils/2006-09/msg00326.html remove.c: /* This is the maximum number of consecutive readdir/unlink calls that can be made (with no intervening rewinddir or closedir/opendir) before triggering a bug that makes readdir return NULL even though some directory entries have not been processed. The bug afflicts SunOS's readdir when applied to ufs file systems and Darwin 6.5's (and OSX v.10.3.8's) HFS+. This maximum is conservative in that demonstrating the problem requires a directory containing at least 16 deletable entries (which doesn't count . and ..). This problem also affects Darwin 7.9.0 (aka MacOS X 10.3.9) on HFS+ and NFS-mounted file systems, but not vfat ones. */ enum { CONSECUTIVE_READDIR_UNLINK_THRESHOLD = 10 }; #ifdef HAVE_WORKING_READDIR # define NEED_REWIND(readdir_unlink_count) 0 #else # define NEED_REWIND(readdir_unlink_count) \ (CONSECUTIVE_READDIR_UNLINK_THRESHOLD <= (readdir_unlink_count)) #endif #### But just recently, on 2009-09-11, all those hacks were replaced. From the changelog: +** Improvements + + rm: rewrite to use gnulib's fts + This makes rm -rf significantly faster (400-500%) in some pathological + cases, and slightly slower (20%) in at least one pathological case. A comment that was removed: -/* FIXME: in 2009, or whenever Darwin 7.9.0 (aka MacOS X 10.3.9) is no - longer relevant, remove this work-around code. Then, there will be - no need to perform the extra rewinddir call, ever. */ -#define NEED_REWIND(readdir_unlink_count) \ - (CONSECUTIVE_READDIR_UNLINK_THRESHOLD <= (readdir_unlink_count)) So apparently GNU thought that Apple would eventually fix readdir(). But I haven't been able to track down the documentation behind this FIXME. I can't see any specific Darwin code remaining in remove.c as of today. An example of the 2006 discussion of the readdir problem is here: http://lists.gnu.org/archive/html/bug-coreutils/2006-09/threads.html#00359 And the first annoucement of the complete rewrite of remove.c is here: http://lists.gnu.org/archive/html/bug-coreutils/2009-08/msg00295.html Stephen Tyler