Обсуждение: [PATCH] Fix severe performance regression with gettext 0.20+ on Windows

Hello hackers,

I've been investigating a performance issue on Windows with recent
gettext versions (0.20.1 and later) that causes exception-heavy
workloads to run significantly slower than with gettext 0.19.8.

Starting with gettext 0.20.1, the library changed its Windows locale
handling in a way that conflicts with how PostgreSQL sets LC_MESSAGES.
The performance regression manifests when raising many exceptions:

  - gettext 0.19.8: ~32 seconds for 1M exceptions
  - gettext 0.20.1+: ~180 seconds for 1M exceptions
  - gettext 0.2x.y+: ~39 seconds for 1M exceptions

The root cause is a combination of three issues:

1. Locale format mismatch
   gettext 0.20.1+ introduced a get_lcid() function that expects Windows
locale format ("English_United States.1252") rather than POSIX format
("en_US"). This function enumerates all Windows locales (~259) until a
match is found, then uses the resulting LCID to determine the catalog path.

   PostgreSQL, however, has always used IsoLocaleName() to convert
Windows locales to POSIX format before setting LC_MESSAGES. This means
we're passing "en_US" to a function expecting "English_United States.1252".

   The enumeration doesn't find "en_US" among Windows locale names,
returns 0, and gettext falls back to its internal locale resolution
(which still works correctly - translations are not broken, just slow).

2. Missing cache on failure
   The get_lcid() function has a cache, but it only updates the cache
when found_lcid > 0 (successful lookup). Failed lookups don't update the
cache, causing the 259-locale enumeration to repeat on every gettext() call.

   This is the actual performance bug in gettext - even if we passed a
valid Windows locale format, setting lc_messages to 'C' or 'POSIX'
(common in scripts and automation) would trigger the same issue since
these aren't Windows locale names.  Please see the bug I opened with the
gettext project [1].

3. Empty string bug in early 0.2x.y
   gettext 0.20.1 introduced a setlocale_null() wrapper that returns ""
instead of NULL when setlocale() fails. This causes get_lcid("") to be
called, triggering the enumeration bug even when LC_MESSAGES is unset.


The attached patch takes a pragmatic approach: for gettext 0.20.1+, we
avoid triggering the bug by using Windows locale format instead of
calling IsoLocaleName(). This works because gettext 0.20.1+ internally
converts the Windows format back to POSIX for catalog lookups, whereas
0.19.8 and earlier need POSIX format directly.

The patch uses LIBINTL_VERSION to detect the gettext version at compile
time and adjusts behavior accordingly. When locale is NULL, empty, or
set to 'C'/'POSIX', we fall back to using the LC_CTYPE value (which is
already in Windows format and always set).

For gettext 0.19.8 and earlier, the existing IsoLocaleName() path is
retained to maintain compatibility.

I don't have automated tests for this since we'd need to test against
multiple versions of a third-party library. I'm open to suggestions if
folks think we should add something to the buildfarm or CI.

Manual testing can be done with this test case:

-- Create test table
CREATE TABLE sampletest (
    a VARCHAR,
    b VARCHAR
);

-- Insert 1 million rows with random data
INSERT INTO sampletest (a, b)
SELECT
    substr(md5(random()::text), 0, 15),
    (100000000 * random())::integer::varchar
FROM generate_series(1, 1000000);

-- Create function that converts string to float with exception handling
CREATE OR REPLACE FUNCTION toFloat(str VARCHAR, val REAL)
RETURNS REAL AS $$
BEGIN
    RETURN CASE
        WHEN str IS NULL THEN val
        ELSE str::REAL
    END;
EXCEPTION
    WHEN OTHERS THEN
        RETURN val;
END;
$$ LANGUAGE plpgsql
   COST 1
   IMMUTABLE;

-- Test query to trigger 1M exceptions
-- (all conversions will fail since we inserted random MD5 strings)
\timing on
SELECT MAX(toFloat(a, NULL)) FROM sampletest;

The ~8 second difference is due to the initial enumeration and other
coding changes that were made by gettext.  Keep in mind that for 1M
exceptions we are probably calling gettext 2-3 million times.

-- 
Bryan Green
EDB: https://www.enterprisedb.com

[1] https://savannah.gnu.org/bugs/?67781

On 10.12.25 01:45, Bryan Green wrote:
> The attached patch takes a pragmatic approach: for gettext 0.20.1+, we
> avoid triggering the bug by using Windows locale format instead of
> calling IsoLocaleName(). This works because gettext 0.20.1+ internally
> converts the Windows format back to POSIX for catalog lookups, whereas
> 0.19.8 and earlier need POSIX format directly.

I can confirm that this patch fixes the performance deviation from 
activating --enable-nls on Windows (tested with MSYS2/UCRT64).

I wonder, this change that gettext did with the locale naming, does that 
also affect what guidance we need to provide to users about how to 
configure locale names?  For example, on a Unix-ish system, a user can 
do something like initdb ... --lc-messages=de_DE.  What locale name 
format do you need to use on Windows to get the translations to 
activate?  Does this also depend on the gettext version?

Hi,

On 2025-12-11 15:43:36 +0100, Peter Eisentraut wrote:
> On 10.12.25 01:45, Bryan Green wrote:
> > The attached patch takes a pragmatic approach: for gettext 0.20.1+, we
> > avoid triggering the bug by using Windows locale format instead of
> > calling IsoLocaleName(). This works because gettext 0.20.1+ internally
> > converts the Windows format back to POSIX for catalog lookups, whereas
> > 0.19.8 and earlier need POSIX format directly.
> 
> I can confirm that this patch fixes the performance deviation from
> activating --enable-nls on Windows (tested with MSYS2/UCRT64).

FWIW, Bilal and I had, IIRC, explicitly not enabled on windows CI because it
made the build process even slower. But perhaps we should re-measure the
difference and re-consider?

Greetings,

Andres Freund

On 12/11/2025 8:43 AM, Peter Eisentraut wrote:
> On 10.12.25 01:45, Bryan Green wrote:
>> The attached patch takes a pragmatic approach: for gettext 0.20.1+, we
>> avoid triggering the bug by using Windows locale format instead of
>> calling IsoLocaleName(). This works because gettext 0.20.1+ internally
>> converts the Windows format back to POSIX for catalog lookups, whereas
>> 0.19.8 and earlier need POSIX format directly.
> 
> I can confirm that this patch fixes the performance deviation from
> activating --enable-nls on Windows (tested with MSYS2/UCRT64).
> 
> I wonder, this change that gettext did with the locale naming, does that
> also affect what guidance we need to provide to users about how to
> configure locale names?  For example, on a Unix-ish system, a user can
> do something like initdb ... --lc-messages=de_DE.  What locale name
> format do you need to use on Windows to get the translations to
> activate?  Does this also depend on the gettext version?
> 
If the language catalogue is installed then they will get translated
messages as expected.  The downside is that because they are passing a
posix locale name then gettext will still do the enumeration everytime.
This will have the negative performance impact.  The good news is that
gettext has accepted my cache patch for their next release.  If a
Windows system is configured with lc_messages="de_DE", but has the next
release of gettext-- they should be fine.  If they don't have the next
release of gettext-- they will notice the performance issue, but that
can be fixed by just changing to from "de_DE" to the correct Windows
locale name.


Walk-through:


1. LCID Lookup: get_lcid("de_DE")
   - Enumerates Windows locales looking for "de_DE"
   - Fails: Windows locales are named "German_Germany", not "de_DE"
   - Returns: 0
   - BUG: Doesn't cache the failure, repeats on every call (patched on
next gettext release)

2. Catalog Search: _nl_make_l10nflist()
   - Tries: locale/de_DE/LC_MESSAGES/postgres-19.mo (not found)
   - Tries: locale/de/LC_MESSAGES/postgres-19.mo (found!)
   - Loads German translations
   - Success!

So, the user gets German messages (catalog fallback works) but
performance is poor (LCID lookup repeats every time) because we don't
cache the failed locale search.



More detailed information for the curious:

Even though get_lcid() returned 0, gettext continues with catalog lookup:

  Function: _nl_find_domain() and _nl_make_l10nflist()
  Location: gettext-runtime/intl/dcigettext.c and l10nflist.c

  Process:
    1. Parse "de_DE" into components:
       language = "de"
       territory = "DE"
       codeset = NULL
       modifier = NULL

    2. Try catalog paths in order (most specific to least specific):

       Try #1: language + territory + codeset + modifier
         Path: /share/locale/de_DE.UTF-8@euro/LC_MESSAGES/postgres-19.mo
         stat(): File not found

       Try #2: language + territory + codeset
         Path: /share/locale/de_DE.UTF-8/LC_MESSAGES/postgres-19.mo
         stat(): File not found

       Try #3: language + territory
         Path: /share/locale/de_DE/LC_MESSAGES/postgres-19.mo
         stat(): File not found (PostgreSQL doesn't ship de_DE)

       Try #4: language + codeset
         Path: /share/locale/de.UTF-8/LC_MESSAGES/postgres-19.mo
         stat(): File not found

       Try #5: language only
         Path: /share/locale/de/LC_MESSAGES/postgres-19.mo
         stat(): SUCCESS! File exists ✓

    3. Load catalog: _nl_load_domain()
       Parse .mo file, load German translations

    4. Look up message: _nl_find_msg()
       Binary search for "division by zero"
       Find translation: "Teilung durch Null"

    5. Return translated message


You might be wondering what happens if the "de" catalog doesn't exist?
It depends on whether the user has set the environment variable LANGUAGE
for their preferred ordered list of languages.  On Windows you can also
set this in the registry. Gettext figures this out.  If LANGUAGE is not
set on Windows then Gettext uses GetUserDefaultUILanguage() to determine
what locale to use. If everything fails, you would get back the msgid
you sent in to start with...so, untranslated.

-- 
Bryan Green
EDB: https://www.enterprisedb.com

On 12/11/2025 10:05 AM, Andres Freund wrote:
> Hi,
> 
> On 2025-12-11 15:43:36 +0100, Peter Eisentraut wrote:
>> On 10.12.25 01:45, Bryan Green wrote:
>>> The attached patch takes a pragmatic approach: for gettext 0.20.1+, we
>>> avoid triggering the bug by using Windows locale format instead of
>>> calling IsoLocaleName(). This works because gettext 0.20.1+ internally
>>> converts the Windows format back to POSIX for catalog lookups, whereas
>>> 0.19.8 and earlier need POSIX format directly.
>>
>> I can confirm that this patch fixes the performance deviation from
>> activating --enable-nls on Windows (tested with MSYS2/UCRT64).
> 
> FWIW, Bilal and I had, IIRC, explicitly not enabled on windows CI because it
> made the build process even slower. But perhaps we should re-measure the
> difference and re-consider?
> 
> Greetings,
> 
> Andres Freund
As long as you use Windows locale names once this patch is in place.
Posix locale names will still incur the performance hit until the next
gettext release. Once using the next gettext release there will not be a
performance penalty for using an invalid locale on Windows.


-- 
Bryan Green
EDB: https://www.enterprisedb.com

Hi,

On 2025-12-11 11:45:01 -0600, Bryan Green wrote:
> On 12/11/2025 10:05 AM, Andres Freund wrote:
> > On 2025-12-11 15:43:36 +0100, Peter Eisentraut wrote:
> >> On 10.12.25 01:45, Bryan Green wrote:
> >>> The attached patch takes a pragmatic approach: for gettext 0.20.1+, we
> >>> avoid triggering the bug by using Windows locale format instead of
> >>> calling IsoLocaleName(). This works because gettext 0.20.1+ internally
> >>> converts the Windows format back to POSIX for catalog lookups, whereas
> >>> 0.19.8 and earlier need POSIX format directly.
> >>
> >> I can confirm that this patch fixes the performance deviation from
> >> activating --enable-nls on Windows (tested with MSYS2/UCRT64).
> > 
> > FWIW, Bilal and I had, IIRC, explicitly not enabled on windows CI because it
> > made the build process even slower. But perhaps we should re-measure the
> > difference and re-consider?
> > 
> > Greetings,
> > 
> > Andres Freund
> As long as you use Windows locale names once this patch is in place.
> Posix locale names will still incur the performance hit until the next
> gettext release. Once using the next gettext release there will not be a
> performance penalty for using an invalid locale on Windows.

What I was referring to was that *building* with NLS support is slower than
building without, which is the reason why CI currently isn't testing NLS in
the "Windows - Server 2022, MinGW64 - Meson" task. Even with ccache, the CI
builds with mingw are pretty darn slow, adding the overhead of creating a good
number of additional files is (or was, haven't retested this recently) making
it even slower.

Greetings,

Andres Freund

Hi,

On Thu, 11 Dec 2025 at 20:49, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2025-12-11 11:45:01 -0600, Bryan Green wrote:
> > On 12/11/2025 10:05 AM, Andres Freund wrote:
> > > On 2025-12-11 15:43:36 +0100, Peter Eisentraut wrote:
> > >> On 10.12.25 01:45, Bryan Green wrote:
> > >>> The attached patch takes a pragmatic approach: for gettext 0.20.1+, we
> > >>> avoid triggering the bug by using Windows locale format instead of
> > >>> calling IsoLocaleName(). This works because gettext 0.20.1+ internally
> > >>> converts the Windows format back to POSIX for catalog lookups, whereas
> > >>> 0.19.8 and earlier need POSIX format directly.
> > >>
> > >> I can confirm that this patch fixes the performance deviation from
> > >> activating --enable-nls on Windows (tested with MSYS2/UCRT64).
> > >
> > > FWIW, Bilal and I had, IIRC, explicitly not enabled on windows CI because it
> > > made the build process even slower. But perhaps we should re-measure the
> > > difference and re-consider?
> > >
> > > Greetings,
> > >
> > > Andres Freund
> > As long as you use Windows locale names once this patch is in place.
> > Posix locale names will still incur the performance hit until the next
> > gettext release. Once using the next gettext release there will not be a
> > performance penalty for using an invalid locale on Windows.
>
> What I was referring to was that *building* with NLS support is slower than
> building without, which is the reason why CI currently isn't testing NLS in
> the "Windows - Server 2022, MinGW64 - Meson" task. Even with ccache, the CI
> builds with mingw are pretty darn slow, adding the overhead of creating a good
> number of additional files is (or was, haven't retested this recently) making
> it even slower.

I tested this and the timings (minute:seconds) of running tests:

MinGW + NLS [1]: 16:01
MinGW + Patch + NLS [2]: 13:57

I ran the CI again to make sure and the speed up was similar.

[1] https://cirrus-ci.com/task/5944143274311680
[2] https://cirrus-ci.com/task/5477244862201856

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Bryan Green <dbryan.green@gmail.com> writes:
> On 12/11/2025 8:43 AM, Peter Eisentraut wrote:
>> I wonder, this change that gettext did with the locale naming, does that
>> also affect what guidance we need to provide to users about how to
>> configure locale names?  For example, on a Unix-ish system, a user can
>> do something like initdb ... --lc-messages=de_DE.  What locale name
>> format do you need to use on Windows to get the translations to
>> activate?  Does this also depend on the gettext version?

> If the language catalogue is installed then they will get translated
> messages as expected.  The downside is that because they are passing a
> posix locale name then gettext will still do the enumeration everytime.
> This will have the negative performance impact.  The good news is that
> gettext has accepted my cache patch for their next release.  If a
> Windows system is configured with lc_messages="de_DE", but has the next
> release of gettext-- they should be fine.  If they don't have the next
> release of gettext-- they will notice the performance issue, but that
> can be fixed by just changing to from "de_DE" to the correct Windows
> locale name.

So IIUC, POSIX-style lc_messages settings do still work on Windows and
will continue to do so, they just incur some extra overhead with
current gettext versions?

If that's the case, I'm inclined to leave my NLS-testing patch [1] as-is,
unconditionally setting a POSIX lc_messages value.  I had anticipated
adding some logic to it to select a Windows locale name when on
Windows, but that seems rather messy, and it's not even clear that the
test would net out faster.  It only needs to do a dozen or so message
lookups, which has to be set against the time needed to identify what
platform we are running on.

            regards, tom lane

[1] https://www.postgresql.org/message-id/1038674.1765568967%40sss.pgh.pa.us

On 12/12/2025 2:01 PM, Tom Lane wrote:
> Bryan Green <dbryan.green@gmail.com> writes:
>> On 12/11/2025 8:43 AM, Peter Eisentraut wrote:
>>> I wonder, this change that gettext did with the locale naming, does that
>>> also affect what guidance we need to provide to users about how to
>>> configure locale names?  For example, on a Unix-ish system, a user can
>>> do something like initdb ... --lc-messages=de_DE.  What locale name
>>> format do you need to use on Windows to get the translations to
>>> activate?  Does this also depend on the gettext version?
> 
>> If the language catalogue is installed then they will get translated
>> messages as expected.  The downside is that because they are passing a
>> posix locale name then gettext will still do the enumeration everytime.
>> This will have the negative performance impact.  The good news is that
>> gettext has accepted my cache patch for their next release.  If a
>> Windows system is configured with lc_messages="de_DE", but has the next
>> release of gettext-- they should be fine.  If they don't have the next
>> release of gettext-- they will notice the performance issue, but that
>> can be fixed by just changing to from "de_DE" to the correct Windows
>> locale name.
> 
> So IIUC, POSIX-style lc_messages settings do still work on Windows and
> will continue to do so, they just incur some extra overhead with
> current gettext versions?
> 
> If that's the case, I'm inclined to leave my NLS-testing patch [1] as-is,
> unconditionally setting a POSIX lc_messages value.  I had anticipated
> adding some logic to it to select a Windows locale name when on
> Windows, but that seems rather messy, and it's not even clear that the
> test would net out faster.  It only needs to do a dozen or so message
> lookups, which has to be set against the time needed to identify what
> platform we are running on.
> 
>             regards, tom lane
> 
> [1] https://www.postgresql.org/message-id/1038674.1765568967%40sss.pgh.pa.us
Correct.  The translation done for 1M exceptions is what was used as a
benchmark.  That turns into a few million calls to gettext.  Your case
seems like it would not really notice the difference.

-- 
Bryan Green
EDB: https://www.enterprisedb.com

On Wed, Dec 10, 2025 at 1:45 PM Bryan Green <dbryan.green@gmail.com> wrote:
> 1. Locale format mismatch
>    gettext 0.20.1+ introduced a get_lcid() function that expects Windows
> locale format ("English_United States.1252") rather than POSIX format
> ("en_US"). This function enumerates all Windows locales (~259) until a
> match is found, then uses the resulting LCID to determine the catalog path.

I'm surprised there isn't a fast path that skips all that deprecated
(?) stuff when you use BCP 47 language codes...

On 12.12.25 10:18, Nazir Bilal Yavuz wrote:
>>>> FWIW, Bilal and I had, IIRC, explicitly not enabled on windows CI because it
>>>> made the build process even slower. But perhaps we should re-measure the
>>>> difference and re-consider?
>>>>
>>>> Greetings,
>>>>
>>>> Andres Freund
>>> As long as you use Windows locale names once this patch is in place.
>>> Posix locale names will still incur the performance hit until the next
>>> gettext release. Once using the next gettext release there will not be a
>>> performance penalty for using an invalid locale on Windows.
>>
>> What I was referring to was that *building* with NLS support is slower than
>> building without, which is the reason why CI currently isn't testing NLS in
>> the "Windows - Server 2022, MinGW64 - Meson" task. Even with ccache, the CI
>> builds with mingw are pretty darn slow, adding the overhead of creating a good
>> number of additional files is (or was, haven't retested this recently) making
>> it even slower.
> 
> I tested this and the timings (minute:seconds) of running tests:
> 
> MinGW + NLS [1]: 16:01
> MinGW + Patch + NLS [2]: 13:57
> 
> I ran the CI again to make sure and the speed up was similar.

Andres was asking about the build time with and without NLS.

There is a note in .cirrus.tasks.yml:

     # Keep -Dnls explicitly disabled, as the number of files it creates 
causes a
     # noticeable slowdown.


I have been testing this a bit.  Locally, using MinGW, I was not able to 
detect any significant difference.  On CI runs, the numbers were to 
erratic to get any consistent sense.  The CI image for Visual Studio 
does not appear to have gettext tools installed, so I couldn't activate 
it there, and I don't have Visual Studio set up locally, so I haven't 
been able to test that.

Hi,

On Mon, 15 Dec 2025 at 22:26, Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 12.12.25 10:18, Nazir Bilal Yavuz wrote:
> >>>> FWIW, Bilal and I had, IIRC, explicitly not enabled on windows CI because it
> >>>> made the build process even slower. But perhaps we should re-measure the
> >>>> difference and re-consider?
> >>>>
> >>>> Greetings,
> >>>>
> >>>> Andres Freund
> >>> As long as you use Windows locale names once this patch is in place.
> >>> Posix locale names will still incur the performance hit until the next
> >>> gettext release. Once using the next gettext release there will not be a
> >>> performance penalty for using an invalid locale on Windows.
> >>
> >> What I was referring to was that *building* with NLS support is slower than
> >> building without, which is the reason why CI currently isn't testing NLS in
> >> the "Windows - Server 2022, MinGW64 - Meson" task. Even with ccache, the CI
> >> builds with mingw are pretty darn slow, adding the overhead of creating a good
> >> number of additional files is (or was, haven't retested this recently) making
> >> it even slower.
> >
> > I tested this and the timings (minute:seconds) of running tests:
> >
> > MinGW + NLS [1]: 16:01
> > MinGW + Patch + NLS [2]: 13:57
> >
> > I ran the CI again to make sure and the speed up was similar.
>
> Andres was asking about the build time with and without NLS.

You are right. I run CI again and compared build times now:

MinGW - NLS [1]: 05:59
MinGW + NLS [2]: 05:55
MinGW + Patch - NLS [3]: 06:42
MinGW + Patch + NLS [4]: 06:04

> I have been testing this a bit.  Locally, using MinGW, I was not able to
> detect any significant difference.

I did not detect any on the CI MinGW task either.

> On CI runs, the numbers were to
> erratic to get any consistent sense.

Yes, I realized that. If you have not cleared the task cache, clearing
it helps to some degree.

[1] https://cirrus-ci.com/task/6337023629328384
[2] https://cirrus-ci.com/task/4929648745775104
[3] https://cirrus-ci.com/task/5570413708705792
[4] https://cirrus-ci.com/task/5409537722679296

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

On 10.12.25 01:45, Bryan Green wrote:
> The attached patch takes a pragmatic approach: for gettext 0.20.1+, we
> avoid triggering the bug by using Windows locale format instead of
> calling IsoLocaleName(). This works because gettext 0.20.1+ internally
> converts the Windows format back to POSIX for catalog lookups, whereas
> 0.19.8 and earlier need POSIX format directly.

A few comments on the patch itself:

1) The description from the commit message, 'gettext 0.20.1+ expects 
Windows locale format ("English_United States") not POSIX format 
("en_US")', or similar, should be added as a comment to the code.

2) Is the cutoff actually 0.20.1 or should it be 0.20?

3) Similarly some comment about why "C" and "POSIX" are handled specially.

4) There is already earlier in pg_perm_setlocale() code that handles 
LC_MESSAGES specially on Windows, and it deals with the (locale == NULL 
|| locale[0] == '\0') case, which is then repeated later on in your 
patch.  I wonder if this could be simplified or combined somehow.

5) With the patch applied, building with a new-enough gettext will leave 
the function IsoLocaleName() unused, which could result in a compiler 
warning.  You could add pg_attribute_unused() to avoid that.

On 12/17/2025 3:54 AM, Peter Eisentraut wrote:
> On 10.12.25 01:45, Bryan Green wrote:
>> The attached patch takes a pragmatic approach: for gettext 0.20.1+, we
>> avoid triggering the bug by using Windows locale format instead of
>> calling IsoLocaleName(). This works because gettext 0.20.1+ internally
>> converts the Windows format back to POSIX for catalog lookups, whereas
>> 0.19.8 and earlier need POSIX format directly.
> 
> A few comments on the patch itself:
> 
> 1) The description from the commit message, 'gettext 0.20.1+ expects
> Windows locale format ("English_United States") not POSIX format
> ("en_US")', or similar, should be added as a comment to the code.
> 
> 2) Is the cutoff actually 0.20.1 or should it be 0.20?
> 
> 3) Similarly some comment about why "C" and "POSIX" are handled specially.
> 
> 4) There is already earlier in pg_perm_setlocale() code that handles
> LC_MESSAGES specially on Windows, and it deals with the (locale == NULL
> || locale[0] == '\0') case, which is then repeated later on in your
> patch.  I wonder if this could be simplified or combined somehow.
> 
> 5) With the patch applied, building with a new-enough gettext will leave
> the function IsoLocaleName() unused, which could result in a compiler
> warning.  You could add pg_attribute_unused() to avoid that.
> 

Peter,

I agree with the above changes and have implemented them, including the
correction to the cutoff version.  But, before sharing the patch with
those changes I think we should discuss 1) should we short-circuit
C/POSIX and not ever call gettext in that case, 2) should we try to
convert "ISO" to Windows legacy format.

Even with the patch, if locale is not in Windows legacy locale format we
will pay the performance penalty.  I think we should short-circuit the
C/POSIX variants and just return the message untranslated.  This is the
intent of setting LC_MESSAGES to one of those values.  There is no
point, that I am aware of, in calling gettext in this case.

As for handling the "ISO" locale format if a Windows system has it set--
we could write something to convert from "ISO" to the Windows legacy
format. At first glance it is not obvious to me how to determine the
codepage-- unless we just use the default for the locale setting.
Otherwise, we seem to be at a position to require changes by the user. I
am considering submitting a patch to have the Windows enumeration code
in gettext handle "ISO" as well.

But, there seems to be no timeline for when any of these fixes will be
released for gettext upstream.  I have also found another patch that
needs to happen on gettext and will get that submitted shortly.  A more
radical choice would be to just have our own gettext with these patches
applied until upstream is ready.



BACKGROUND:

We call gettext() with three locale format types:

1) "C"/"POSIX" variants
2) ISO locales (language-region format like "en-US")
3) Windows legacy locales (like "English_United States.1252")

Current Problems
Problem 1: Version incompatibility

Versions before 0.20: Handled ISO locales correctly on Windows
Versions 0.20-0.26: Broke ISO locale handling with flawed enumeration logic

Problem 2: Performance degradation (versions 0.20+)

gettext enumerates all Windows locales on every call to find a match
Enumeration misses are not cached, forcing re-enumeration
Since we convert legacy locales to ISO format, we always miss.
Result: Severe performance penalty (180s → 32s for 1M exceptions when fixed)

A patch fixing the cache issue has been accepted into gettext's
development branch, but no release timeline exists.

Problem 3: C/POSIX locale handling

Windows doesn't recognize "C" or "POSIX" as valid locales
gettext correctly returns untranslated messages for these locales
However, it still performs the full locale enumeration before doing so
This adds unnecessary overhead to every call with C/POSIX locales

I'm preparing a patch for upstream regarding this issue.  Although, I
won't be surprised if they say "why are you setting it to those values
on Windows...those don't make any sense on Windows".

For gettext versions 0.20 and later, we must:

1) Stop converting Windows legacy locales to ISO format
2) Short-circuit gettext calls when locale is C/POSIX variant
3) Document that Windows users must use legacy locale format

Trade-offs

Versions before 0.20: Work correctly but can't upgrade to escape CVEs if
needed.
Versions 0.20+: Require workarounds above to maintain acceptable
performance.



-- 
Bryan Green
EDB: https://www.enterprisedb.com

On 08.01.26 15:57, Bryan Green wrote:
> I agree with the above changes and have implemented them, including the
> correction to the cutoff version.  But, before sharing the patch with
> those changes I think we should discuss 1) should we short-circuit
> C/POSIX and not ever call gettext in that case,

You had written that you had submitted a patch to gettext to handle that 
there.  Has that gotten anywhere?

> 2) should we try to
> convert "ISO" to Windows legacy format.

I don't know.  We can just tell users to set their locale in the right 
format.

On 1/20/2026 2:39 PM, Peter Eisentraut wrote:
> On 08.01.26 15:57, Bryan Green wrote:
>> I agree with the above changes and have implemented them, including the
>> correction to the cutoff version.  But, before sharing the patch with
>> those changes I think we should discuss 1) should we short-circuit
>> C/POSIX and not ever call gettext in that case,
> 
> You had written that you had submitted a patch to gettext to handle that
> there.  Has that gotten anywhere?
> 
>> 2) should we try to
>> convert "ISO" to Windows legacy format.
> 
> I don't know.  We can just tell users to set their locale in the right
> format.
Peter,
I have attached the patch with the changes you suggested/requested.  The
patch was added to gnulib in December.  The latest release of gnu
gettext (1.0) does include the patch.  Yes, they jumped from 0.26 to 1.0.

-- 
Bryan Green
EDB: https://www.enterprisedb.com

On 04.02.26 16:08, Bryan Green wrote:
> On 1/20/2026 2:39 PM, Peter Eisentraut wrote:
>> On 08.01.26 15:57, Bryan Green wrote:
>>> I agree with the above changes and have implemented them, including the
>>> correction to the cutoff version.  But, before sharing the patch with
>>> those changes I think we should discuss 1) should we short-circuit
>>> C/POSIX and not ever call gettext in that case,
>>
>> You had written that you had submitted a patch to gettext to handle that
>> there.  Has that gotten anywhere?
>>
>>> 2) should we try to
>>> convert "ISO" to Windows legacy format.
>>
>> I don't know.  We can just tell users to set their locale in the right
>> format.
> Peter,
> I have attached the patch with the changes you suggested/requested.  The
> patch was added to gnulib in December.  The latest release of gnu
> gettext (1.0) does include the patch.  Yes, they jumped from 0.26 to 1.0.

The newly released gettext 1.0 is now available in MSYS2, so I tested 
this again.  The new gettext indeed makes a significant performance 
improvement compared to my test results with earlier gettext versions 
(about 10x faster).

This is all without any PostgreSQL patch.

But when I apply your patch, it actually makes things worse (by about 
25%).  This is incomprehensible to me, but it's very reproducible.  I'm 
not sure how to proceed now.

On 2/16/2026 9:34 AM, Peter Eisentraut wrote:
> On 04.02.26 16:08, Bryan Green wrote:
>> On 1/20/2026 2:39 PM, Peter Eisentraut wrote:
>>> On 08.01.26 15:57, Bryan Green wrote:
>>>> I agree with the above changes and have implemented them, including the
>>>> correction to the cutoff version.  But, before sharing the patch with
>>>> those changes I think we should discuss 1) should we short-circuit
>>>> C/POSIX and not ever call gettext in that case,
>>>
>>> You had written that you had submitted a patch to gettext to handle that
>>> there.  Has that gotten anywhere?
>>>
>>>> 2) should we try to
>>>> convert "ISO" to Windows legacy format.
>>>
>>> I don't know.  We can just tell users to set their locale in the right
>>> format.
>> Peter,
>> I have attached the patch with the changes you suggested/requested.  The
>> patch was added to gnulib in December.  The latest release of gnu
>> gettext (1.0) does include the patch.  Yes, they jumped from 0.26 to 1.0.
> 
> The newly released gettext 1.0 is now available in MSYS2, so I tested
> this again.  The new gettext indeed makes a significant performance
> improvement compared to my test results with earlier gettext versions
> (about 10x faster).
> 
> This is all without any PostgreSQL patch.
> 
> But when I apply your patch, it actually makes things worse (by about
> 25%).  This is incomprehensible to me, but it's very reproducible.  I'm
> not sure how to proceed now.
> 

Peter,

TLDR; We don't need the patch for gettext < 20 and >= 1.0.  The negative
caching provides a fast failure path.  The more correct windows path
adds some cost.  I think both are faster than the current extreme slowness.

Note: I would like to point out that we currently do not respect the
language preferences on Windows because we set LC_MESSAGES.  In gettext
the MUI flag is disabled by default.  If we wanted to respect the
language preference of the user we would have to enable the flag and not
set LC_MESSAGES.  For the future...



I dug into the gettext 1.0 source code to understand why the patch
actually makes things worse, and I think I can explain what's happening.

It comes down to two different code paths in gnulib's
localename-unsafe.c for resolving locale names on native Windows, and
they have very different performance characteristics even with the cache
fix in place.

Without the patch, PostgreSQL sets LC_MESSAGES to a POSIX name like
"en_US" via IsoLocaleName(). When gettext calls
setlocale(LC_MESSAGES, NULL), it gets back "en_US" and passes it to
get_lcid(). The enum_locales_fn callback only compares against
Windows-format names (built via GetLocaleInfo(LOCALE_SENGLANGUAGE) +
"_" + GetLocaleInfo(LOCALE_SENGCOUNTRY)), so "en_US" never matches.
The lookup fails, returns LCID 0, and with the new LRU cache in gettext
1.0 that failure is cached on the first call. Subsequent calls hit the
cache immediately. The function then falls through to
gl_locale_name_environ(), which just reads the LC_MESSAGES environment
variable and returns the "en_US" string directly. That becomes the
single_locale used to find the .mo file.

With the patch, PostgreSQL passes "English_United States.1252" through.
Now get_lcid() succeeds -- it enumerates all ~259 system locales,
finds a match, and returns a valid LCID (also cached after first call).
But then it has to call gl_locale_name_from_win32_LCID() →
gl_locale_name_from_win32_LANGID() to convert back to POSIX format.
That function calls GetACP() and getenv("GETTEXT_MUI") on every
invocation -- neither is cached -- plus does a large nested switch on
the LANGID. The end result is the same "en_US" string, but obtained
through a more expensive path.

Both paths produce the exact same single_locale value and the same
.mo file lookup. When there's no .mo file, _nl_load_domain() caches
the failure (decided = 1, data = NULL), so subsequent calls don't
touch the filesystem. The only difference is the per-call overhead in
guess_category_value() → gl_locale_name_posix(), which runs on
every gettext() call. The POSIX path resolves cheaply through a
cached get_lcid() failure plus a simple getenv(), while the Windows
path goes through a cached get_lcid() success followed by uncached
GetACP(), getenv("GETTEXT_MUI"), and the LANGID switch table.

That explains the ~25% regression: the "correct" Windows locale path is
paradoxically slower than the "failing" POSIX path, because the failure
is cheap and cached while the success triggers additional uncached work
on every call.

With gettext 1.0's cache fix, I don't think we need the patch at all.
The POSIX names produce the right .mo file paths and perform better.
The original caching bug that motivated the patch -- where failed
lookups caused repeated EnumSystemLocales() calls -- is fixed in
gettext 1.0's LRU cache in get_lcid().


-- 
Bryan Green
EDB: https://www.enterprisedb.com