bugGNU gettext - Bugs: bug #65053, spurious "hexadecimal escape...


You are not allowed to post comments on this tracker with your current authentication level.

bug #65053: spurious "hexadecimal escape sequence out of range" warnings

Submitter:  Vaclav Slavik <vslavik>
Submitted:  Tue 19 Dec 2023 05:16:01 PM UTC
Category:  Programmer tools Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Sun 31 Dec 2023 12:42:20 PM UTC, comment #4: 

> This is why I cannot ignore the problem with FreeBSD and Solaris.

You're obviously the best authority on what your past self meant, but are you sure we're talking about the same change? For reference, this is the commit: https://git.savannah.gnu.org/gitweb/?p=gettext.git;a=commit;h=0c0345632aedfb254b69f72cce268728113edf2e (and subsequent analog for Vala: https://git.savannah.gnu.org/gitweb/?p=gettext.git;a=commit;h=dd9a098416034695ad3aed8c22631b9ffaa82d58).

The commit seems very clear that it really is only about actual overflow. The warning message says that. It uses the same wording as a C compiler for actual overflow. The commit message says so (with some extra explanation). The code in phase7_getc() clearly does that, it compares to 0x100. There's no indication anywhere that it is related to portability issues with Solaris or FreeBSD, and every indication that it is about 8-bit character overflow.

By your explanation, the code should warn about any non-ASCII \x sequences, i.e. >= 0x80 as they too are locale-dependent on these platforms. But it doesn't do that and happily accepts them. That seems inconsistent with the portability check purpose as well.

I expected this to be a simple case of phase7_getc() being ignorant of wide characters and just having to use a value different from 0x100 for the check. I mean, the relevant invocation of the function even says so ("We could worry about the 'L' before wide character constants, but ignoring it has no effect unless one of the keywords is "L".") and it was true that it didn't matter before 0c0345632aedfb254b69f72cce268728113edf2e, but does now.

Also, it warns on character literals which are never going to by passed to a gettext function, and are therefore non-issue for portability (by way of sharing common parsing code).

> Maybe I should change the warning "hexadecimal escape sequence out of range" to "hexadecimal escape in wide-char literal is unsupported; use \u instead of \x if you meant to designate a Unicode character" ?

If you think this check is correct and should stay as-is, then I'd say yes, the message definitely could use clarity, because as it is now, it is very misleading. A bogus warning with an obvious workaround is mere annoyance; one with no obvious path to silence it, which looks like a bug to a user, is much worse.

But doing what you suggest is going to be non-trivial and probably not worth the effort: the parser doesn't differentiate narrow and wide literals; it does correctly warn about narrow-char overflows and arguably that's what it should continue to say there; it also accepts some \x escapes in wide-char literals (although that's arguably OK, it doesn't have to warn about every instance...).

I suppose the minimal clarifying change would be something like "hexadecimal escape sequence out of range; use \u in wide-char literal if you meant to designate a Unicode character"? Though you could rightfully complain that such wording steers towards using wchar_t...

Vaclav Slavik <vslavik>
Wed 20 Dec 2023 06:37:56 PM UTC, comment #3: 

> it emits them even for literals that aren't used in i18n functions

Yes, that's because composition of literals happens before xgettext decides whether to emit the sequence of literals into the resulting .pot file. I acknowledge that it's a bit unfortunate. But it would require a complex logic to make this warning appear only for strings that get extracted.

> emits warning on perfectly legal, standard-comformant C++ code

The problem behind that is that the expectations regarding xgettext are higher than the expecations regarding a C++ compiler. The C++ compiler only needs to put the string into a read-only portion of the resulting binary, and the result in different locales on various platforms is then the user's business, not the compiler writer's business. Whereas for xgettext the expectation is that it transforms the literal into a byte sequence in the .po file, and that this byte sequence is the correct one regardless of the locale at run-time and regardless of the target platform. This is why I cannot ignore the problem with FreeBSD and Solaris.

> there's a workaround ...: use \u instead of \x.

Indeed. And with this one there is no problem, because with \u the programmer denoted the character in a locale and platform independent way.

Maybe I should change the warning "hexadecimal escape sequence out of range" to "hexadecimal escape in wide-char literal is unsupported; use \u instead of \x if you meant to designate a Unicode character" ?

Bruno Haible <haible>
Group administrator
Wed 20 Dec 2023 04:22:06 PM UTC, comment #2: 

Honestly, both my personal choices I would make when writing new UNIX software or GNU's guidelines for their software are a bit off-topic to the issue. And while I appreciate the non-trivialities of working with wchar_t, calling any use of it not portable is a bit of a stretch (case in point, this sample code is perfectly fine and doesn't hit any of the pain points mentioned in gnulib docs).

The important part, IMHO, is that wchar_t is part of ISO C++ as well as modern C standards and so are widechar literals (regardless of one's opinion on their merit or appropriate use). And as it is, xgettext currently

1. emits warning on perfectly legal, standard-comformant C++ code
2. emits untruthful warning, claiming that the value is out of range even when it isn't

Moreover, it emits them even for literals that aren't used in i18n functions, and there's no way to silence the warning, so one has to change even i18n-unrelated code to accomodate xgettext.

For now at least, there's a workaround that I didn't think about myself; another person alerted me to it: use \u instead of \x.

Vaclav Slavik <vslavik>
Wed 20 Dec 2023 06:34:38 AM UTC, comment #1: 

You are aware that programs that make use of L"\x2396" or L'\x02D9' are not portable? Namely to Solaris and FreeBSD.

See https://www.gnu.org/software/gnulib/manual/html_node/The-wchar_005ft-type.html for details.

Bruno Haible <haible>
Group administrator
Tue 19 Dec 2023 05:16:01 PM UTC, original submission:  

GNU gettext 0.22 introduced a new warning in 0c0345632aedfb254b69f72cce268728113edf2e ("xgettext: In language C, report out-of-range hexadecimal escapes.").

Unfortunately, the warning is produced even in situations when the code is correct, because the string or character literal in question is a C++ wchar_t one. (Notice that the warning is produced for any strings, even those that are not wrapped in a gettext function and --extract-all is not used.)

Attached is a minimalistic example:

int main()
   const wchar_t *test1 = L"1192" L"\x2396" L"59";
   if (c == L'\x02D9')  // U+2009  thin space
       return 1;
   if (c == L'\xff')  // this is ok
       return 0;
   return 0;

Corresponding xgettext 0.22.4's output:

$ xgettext -o /dev/null main.cpp
main.cpp:3: warning: hexadecimal escape sequence out of range
main.cpp:4: warning: hexadecimal escape sequence out of range

These are actually two diffferent misparsings, one in a character literal and one in a string, but what they have in common is that the characted is expected to be < 0xFF, i.e. the parser doesn't recognize these string/character literals as wide ones.

Vaclav Slavik <vslavik>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by vslavik (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.


    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-b921.
    Corresponding source code