bugGNU gettext - Bugs: bug #32505, xgettext on C# file crashes when...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #32505: xgettext on C# file crashes when input file contains Unicode literal with surrogates

Submitter:  Petr Kadlec <mormegil>
Submitted:  Wed 16 Feb 2011 04:57:42 PM UTC
   
 
Category:  C# Severity:  3 - Normal
Item Group:  None Status:  Fix Released
Privacy:  Public Assigned to:  ueno
Open/Closed:  Closed

Mon 01 Dec 2014 07:17:51 AM UTC, comment #3: 
Daiki Ueno <ueno>
Group administrator
Fri 28 Nov 2014 08:57:09 AM UTC, comment #2: 

According to ECMA-334 9.4.1, a Unicode code point in the range U+10000 to U+10FFFF is represented using two Unicode surrogate code units.  I couldn't find any example in the specification, but a quick web search showed that the following expressions represent the same string:


string s1 = "\U00020213";    // Codepoint U+20213
string s2 = "\uD840\uDE13";  // Encoded form of U+20213


This indeed works with Mono, but aborts xgettext.  I'm attaching a tentative patch to fix this.


(file #32566)

Daiki Ueno <ueno>
Group administrator
Thu 28 Feb 2013 10:22:14 AM UTC, comment #1: 

#1 and #3 are implemented currently, but the real problem is #2, because conversion to UTF-8 is done before knowing it is translatable.

This problem is the related to the issues I'm facing with unicode support in C/C++. It will need some design changes, maybe something quite simple like storing an "int *buffer" instead of a "char *buffer" and process it in a later stage, but there are too many places to touch and lots of testing to do before posting any useful patch.

Also, this bug report is old, but if you already need a workaround and this file does not contains any translatable string, you can comment out this file from your POTFILES.in or erase it from xgettext arguments.

Miguel Ángel Arruga Vivas <m1gu3l>
Group Member
Wed 16 Feb 2011 04:57:42 PM UTC, original submission:  

Running xgettext on a C# file which contains a string literal containing a Unicode escape for a surrogate character leads to an abnormal termination (abort()).

This is because the character reference is parsed correctly, then passed to string_buffer_append_unicode() in x-csharp.cs, which uses u8_uctomb() to encode the character to UTF-8. However, u8_uctomb() denies to encode surrogate characters (correctly, we might say, they should not be encoded separately, the “real” Unicode character should be encoded, not its UTF-16 encoding) and returns -1, after which string_buffer_append_unicode() decides to abort(), as “The caller should have ensured that uc is not out-of-range.”

Reproducible:
test.cs:

class C
{
    string s = "\xD800\xDC00";
}

Then run:
xgettext --language=C# test.cs

A few thoughts on solving the problem:

  1. When the string containing the surrogate character is in fact a localizable string, and would get to the .po file, we should ensure the first surrogate is correctly followed by the corresponding low surrogate, and then UTF-8 encode the real Unicode character. But this is probably not completely trivial.
  2. When the string is not localizable and we are parsing just to get around in the file, we could ignore the underlying meaning of the surrogate character and just ignore it altogether, or encode it UTF-8-like (i.e. CESU-8). It is not only easier, it is also necessary in some cases: There is nothing wrong on having something like “private const char firstSurrogate = '\xD800';”.
  3. Leaving the final case: What if the string is localizable, but contains invalid UTF-16 (e.g. a bare surrogate)? In that case, xgettext should probably report an error, similar to when it detects invalid byte sequence for the selected encoding (e.g. “Non-ASCII character at…”).


Right now, I would be happy having just #2 above implemented, because that is my problem right now – xgettext crashing on a file having nothing to do with localization.

Petr Kadlec <mormegil>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by haible (Updated the item)
  • -email is unavailable- added by ueno (Updated the item)
  • -email is unavailable- added by m1gu3l (Updated the item)
  • -email is unavailable- added by mormegil (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-04 haible StatusFixed Fix Released
    2016-11-27 haible CategoryNone C#
    2014-12-01 ueno StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2014-11-28 ueno Attached File- Added 0001-csharp-Recognize-Unicode-surrogate-character-pair.patch, #32566
        StatusConfirmed In Progress
        Assigned tom1gu3l ueno
    2013-02-28 m1gu3l StatusNone Confirmed
    2013-02-28 m1gu3l Assigned toNone m1gu3l

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code