I wrote:
> I think this is the issue Peter mentioned about needing to distinguish
> between empty strings and NULL strings. We're going to need to rethink
> the behavior of pg_strtok() a bit to fix that.
After staring at the code a bit, I think we don't need to touch
pg_strtok() per se. I propose that this can be resolved with changes
at the next higher level. Let's make outToken print NULL as <> as
it always has, but print an empty string as "" (two double quotes).
If the raw input string is two double quotes, print it as \"" to
disambiguate. This'd require a catversion bump when committed,
but I don't think there are any showstopper problems otherwise.
I'll work on fleshing that idea out.
regards, tom lane