another look at macOS SIP

I have figured out another solution to the problem that macOS SIP
defeats the use of DYLD_LIBRARY_PATH for running the temp-install
regression tests.  It's not without problems either, but it might show a
path forward.

First of all, I think I now know the exact mechanism by which this
breakage happens.

The precise issue is that /bin/sh filters out DYLD_* environment
variables (presumably all, but at least the ones we care about) when it
starts.  If you use a shell other than /bin/sh (say, a Homebrew
installation of bash or dash), there is no problem.

But /bin/sh is hardcoded in the system() library call, so in order to
fix that, you need to override that library call.  Attached is a patch
that shows how this could be done.  It uses the DYLD_INSERT_LIBRARIES
environment variable (equivalent to LD_PRELOAD) to substitute another
version of system(), which I hacked to allow overriding /bin/sh with
another shell using the environment variable PG_REGRESS_SHELL.  That works.

There are also some other places where PostgreSQL code itself hardcodes
/bin/sh as part of system()-like functionality.  These have to be fixed
up similarly, but that's easier.

The problem now is that DYLD_INSERT_LIBRARIES requires the "flat
namespace", which isn't the default.  You can either build PostgreSQL
with -Wl,-flat_namespace, which works, but it's probably weird as a
proper solution, or you can set the environment variable
DYLD_FORCE_FLAT_NAMESPACE at run time, which also works but makes
everything brutally slow.

I think the way forward here is to get rid of all uses of system() for
calling between PostgreSQL programs.  There are only a handful of those,
and we already have well-tested replacement code like spawn_process() in
pg_regress.c that could be used.  (Perhaps we could also use that
opportunity to get rid of the need for shell quoting?)

There is a minor second issue, namely that /usr/bin/perl also filters
out DYLD_* environment variables.  This can be worked around again by
using a third-party installation of Perl.  You just need to make sure
that the "prove" program calls that installation instead of the system
one.  (I just manually edited the shebang line.  There is probably a
proper way to do it.)

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services