bugGNU gettext - Bugs: bug #66491, Fallback languages aren't used on...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #66491: Fallback languages aren't used on Windows

Submitter:  Pierre Ossman <cendossm>
Submitted:  Thu 28 Nov 2024 10:28:50 AM UTC
   
 
Category:  End-user / runtime Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Confirmed Privacy:  Public
Assigned to:  haible Open/Closed:  Open

Discussion

Jump to the original submission

Mon 08 Dec 2025 03:35:13 PM UTC, comment #11: 

Here is a quick piece of code I threw together if anyone needs an example of the rough sketch-- not tested in detail, so I decided against a patch of langprefs.c and decided to just include the snippet:

static char *
convert_language_list_to_posix (const WCHAR *wchar_list, ULONG num_languages)
{
  const WCHAR *p;
  char *q;
  ULONG i;
  size_t bufsize;

  /* Calculate buffer size needed.  */
  bufsize = 0;
  p = wchar_list;
  for (i = 0; i < num_languages; i++)
    {
      bufsize += wcslen (p) + 1;  /* +1 for colon or null terminator */
      p += wcslen (p) + 1;
    }

  char languages = (char ) malloc (bufsize + num_languages * 10 + 1);
  if (languages == NULL)
    return NULL;

  p = wchar_list;
  q = languages;

  for (i = 0; i < num_languages; i++)
    {
      char *q_start = q;
      char *q_lang_start;

      if (i > 0)
        *q++ = ':';
      q_lang_start = q;

   
      for (; *p != (WCHAR) '\0'; p++)
        {
          if ((unsigned char) *p != *p || *p == ':')
            {
              /* Non-ASCII or colon - skip this language.  */
              q = q_start;
              break;
            }
          *q++ = (unsigned char) *p;
        }

      if (q == q_start)
        {
          /* Skip to next language.  */
          while (*p != (WCHAR) '\0')
            p++;
          p++;
          continue;
        }

      *q = '\0';
      /* Convert BCP 47 (en-US) to POSIX (en_US).  */
      gl_locale_name_canonicalize (q_lang_start);
      q = q_lang_start + strlen (q_lang_start);
      p++;
    }

  *q = '\0';

  if (q > languages)
    return languages;

  free (languages);
  return NULL;
}

/* Get the preferences list through the registry or MUI APIs.  */
static const char *
_nl_language_preferences_win32_mui (HMODULE kernel32)
{
  HKEY hKey;
  DWORD type, dataSize;

  /* Try registry first - contains unfiltered preference list.  */
  if (RegOpenKeyExW (HKEY_CURRENT_USER,
                     L"Control Panel\\International\\User Profile",
                     0, KEY_QUERY_VALUE, &hKey) == ERROR_SUCCESS)
    {
      dataSize = 0;
      if (RegQueryValueExW (hKey, L"Languages", NULL, &type, NULL, &dataSize)
          == ERROR_SUCCESS
          && type == REG_MULTI_SZ && dataSize > 0)
        {
          WCHAR buffer = (WCHAR ) malloc (dataSize);
          if (buffer != NULL)
            {
              if (RegQueryValueExW (hKey, L"Languages", NULL, &type,
                                   (LPBYTE) buffer, &dataSize) == ERROR_SUCCESS)
                {
                  ULONG num_languages = 0;
                  const WCHAR *p = buffer;
                  while (*p != L'\0')
                    {
                      num_languages++;
                      p += wcslen (p) + 1;
                    }

                  char *languages = convert_language_list_to_posix (buffer,
                                                                    num_languages);
                  free (buffer);
                  RegCloseKey (hKey);
                  if (languages != NULL)
                    return languages;
                }
              else
                free (buffer);
            }
        }
      RegCloseKey (hKey);
    }

  /* Fall back to GetUserPreferredUILanguages() (filters by installed packs).  */
  typedef BOOL (WINAPI *GetUserPreferredUILanguages_func) (DWORD, PULONG, PWSTR, PULONG);
  GetUserPreferredUILanguages_func p_GetUserPreferredUILanguages;

  p_GetUserPreferredUILanguages =
    (GetUserPreferredUILanguages_func)
    GetProcAddress (kernel32, "GetUserPreferredUILanguages");

  if (p_GetUserPreferredUILanguages != NULL)
    {
      ULONG num_languages, bufsize = 0;

      if ((p_GetUserPreferredUILanguages (MUI_LANGUAGE_NAME, &num_languages,
                                         NULL, &bufsize)
           || GetLastError () == STATUS_BUFFER_OVERFLOW)
          && bufsize > 0)
        {
          WCHAR buffer = (WCHAR ) malloc (bufsize * sizeof (WCHAR));
          if (buffer != NULL)
            {
              if (p_GetUserPreferredUILanguages (MUI_LANGUAGE_NAME,
                                                &num_languages, buffer, &bufsize))
                {
                  char *languages = convert_language_list_to_posix (buffer,
                                                                    num_languages);
                  free (buffer);
                  if (languages != NULL)
                    return languages;
                }
              else
                free (buffer);
            }
        }
    }

  return NULL;
}

Bryan Green <lispyone>
Mon 08 Dec 2025 10:54:37 AM UTC, comment #10: 


> The REG_MULTI_SZ(HKEY_CURRENT_USER\Control Panel\International\User Profile\Languages) value contains the complete user preference list regardless of installed Windows language packs - exactly what is needed for application localization.


Many thanks for this hint! This indeed seems to be the right way to get at this priority list. It's ugly to access the registry, but it seems to be the best way, in order to match user expectations.

This stackoverflow page https://stackover ... estions/63877075/ describes an alternative: to use an undocumented function from an undocumented DLL. But that is even worse than reading the registry.

I'll implement it in libintl in a couple of weeks.

Bruno Haible <haible>
Group administrator
Sat 06 Dec 2025 07:26:20 PM UTC, comment #9: 

I looked into this briefly and could only get a list of languages by using the registry.  I wrote some test programs to try GetUserPreferredUILanguages() with the MUI_LANGUAGE_NAME flag and it only returned languages that have Windows UI language packs installed. This is probably by design for Windows UI localization, but it breaks application localization.

The API only returns en-US because that's the only language with a Windows UI pack installed.  You don't have sv_SE.mo, but you do have de_DE.mo.  Unfortunately, you don't have the windows UI pack for de_DE. gettext never sees Swedish or German in the list, so it can't fall back to German when Swedish isn't available.

The REG_MULTI_SZ(HKEY_CURRENT_USER\Control Panel\International\User Profile\Languages) value contains the complete user preference list regardless of installed Windows language packs - exactly what is needed for application localization.


A rough sketch of what_nl_language_preferences_win32_mui() should do:
1. Try reading HKCU\Control Panel\International\User Profile\Languages
2. Convert from BCP 47 format (en-US) to POSIX format (en_US)
3. Build colon-separated list
4. Fall back to GetUserPreferredUILanguages() if registry read fails

Bryan Green <lispyone>
Mon 09 Dec 2024 03:05:57 PM UTC, comment #8: 

I'm afraid I'm not seeing any change in behaviour. Nor any new output.

Pierre Ossman <cendossm>
Mon 09 Dec 2024 02:45:39 PM UTC, comment #7: 


> Unfortunately the issue remains. I'm still getting English if the primary language isn't supported.


OK.

Can you make another test, please? With the same gettext-0.23 binaries, but with the environment variable GETTEXT_MUI set to a non-empty value?

In bash syntax:

export GETTEXT_MUI=yes


In cmd.exe syntax:

set GETTEXT_MUI=yes


Bruno Haible <haible>
Group administrator
Mon 09 Dec 2024 02:30:29 PM UTC, comment #6: 

It's a quirk of our build environment that it wants to rebuild it for all architectures.

I was able to work my way around it, though. Unfortunately the issue remains. I'm still getting English if the primary language isn't supported.

It seems to respect $LANGUAGE better, so some part of the mechanism works.

(I do see one bug with $LANGUAGE as well, in that English is ignored in the priority list)

Any debug settings for gettext that might help?

Pierre Ossman <cendossm>
Fri 06 Dec 2024 04:21:15 PM UTC, comment #5: 


> ../../../gettext-tools/gnulib-tests/file-has-acl.c:177:38: error: 'XATTR_NAME_SMACK' undeclared


This error is in Linux specific code. You are welcome to open another ticket for this compilation error (with details about the Linux distribution, installed Linux kernel version, libacl version, libattr version).

But this bug is about native Windows. So you need a native Windows build of gettext 0.23, in order to test.

Bruno Haible <haible>
Group administrator
Fri 06 Dec 2024 03:45:55 PM UTC, comment #4: 

I tried a quick test build here, but unfortunately the libc/kernel requirements seem to have gone up with gettext 0.23 so it fails to build:

../../../gettext-tools/gnulib-tests/file-has-acl.c:177:38: error: 'XATTR_NAME_SMACK' undeclared (first use in this function)
           if (aclinfo_has_xattr (ai, XATTR_NAME_SMACK))
                                      ^

I'll see if I can work around it next week to get a test build going.

Pierre Ossman <cendossm>
Fri 06 Dec 2024 09:23:50 AM UTC, comment #3: 

Could you please try again with GNU gettext 0.23? It has code that makes use of GetUserPreferredUILanguages; this code did not work in earlier releases (cf. bug #65128).

Bruno Haible <haible>
Group administrator
Thu 28 Nov 2024 12:33:19 PM UTC, comment #2: 

1) See attached screenshot

2) We haven't done any research in to Windows APIs. A quick search suggests that GetUserPreferredUILanguages() might be interesting.


Pierre Ossman <cendossm>
Thu 28 Nov 2024 10:35:08 AM UTC, comment #1: 


> Windows has not just a single language, but a prioritised list of  languages.


1) How can the user set/change this list? (Assuming Windows 10.)

2) How can a program retrieve this list? I'm asking because I vaguely recall having searched for an API that returns this list and found none.

Bruno Haible <haible>
Group administrator
Thu 28 Nov 2024 10:28:50 AM UTC, original submission:  

Windows has not just a single language, but a prioritised list of  languages. Much like $LANGUAGE.

Unfortunately, it seems gettext doesn't look at this list. If the configured primary language is not available in the message catalog, then gettext will fall back to the default English.

Tested with gettext 0.22.5.

Pierre Ossman <cendossm>

 

Attached Files

Attached Files
file #56651:  languages.png added by cendossm (181KiB - image/png - Screenshot of Windows 10 language settings)

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by lispyone (Posted a comment)
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by cendossm (Submitted the item)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    History

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2025-12-08 haible StatusNeed Info Confirmed
    2024-12-06 haible StatusNone Need Info
    2024-11-28 haible StatusNeed Info None
    2024-11-28 cendossm Attached File- Added languages.png, #56651
    2024-11-28 haible StatusNone Need Info
        Assigned toNone haible

    Back to the top

    Powered by Savane 3.16-598c.
    Corresponding source code