bugGNU gettext - Bugs: bug #32505, xgettext on C# file crashes when...

 
 

bug #32505: xgettext on C# file crashes when input file contains Unicode literal with surrogates

Submitted by:  Petr Kadlec <mormegil>
Submitted on:  Wed 16 Feb 2011 04:57:42 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Confirmed
Privacy: PublicAssigned to: Miguel Ángel Arruga Vivas <m1gu3l>
Open/Closed: Open

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 28 Feb 2013 10:22:14 AM UTC, comment #1:

#1 and #3 are implemented currently, but the real problem is #2, because conversion to UTF-8 is done before knowing it is translatable.

This problem is the related to the issues I'm facing with unicode support in C/C++. It will need some design changes, maybe something quite simple like storing an "int buffer" instead of a "char buffer" and process it in a later stage, but there are too many places to touch and lots of testing to do before posting any useful patch.

Also, this bug report is old, but if you already need a workaround and this file does not contains any translatable string, you can comment out this file from your POTFILES.in or erase it from xgettext arguments.

Miguel Ángel Arruga Vivas <m1gu3l>
Project MemberIn charge of this item.
Wed 16 Feb 2011 04:57:42 PM UTC, original submission:

Running xgettext on a C# file which contains a string literal containing a Unicode escape for a surrogate character leads to an abnormal termination (abort()).

This is because the character reference is parsed correctly, then passed to string_buffer_append_unicode() in x-csharp.cs, which uses u8_uctomb() to encode the character to UTF-8. However, u8_uctomb() denies to encode surrogate characters (correctly, we might say, they should not be encoded separately, the “real” Unicode character should be encoded, not its UTF-16 encoding) and returns -1, after which string_buffer_append_unicode() decides to abort(), as “The caller should have ensured that uc is not out-of-range.”

Reproducible:
test.cs:
Then run:
xgettext --language=C# test.cs

A few thoughts on solving the problem:

  1. When the string containing the surrogate character is in fact a localizable string, and would get to the .po file, we should ensure the first surrogate is correctly followed by the corresponding low surrogate, and then UTF-8 encode the real Unicode character. But this is probably not completely trivial.
  2. When the string is not localizable and we are parsing just to get around in the file, we could ignore the underlying meaning of the surrogate character and just ignore it altogether, or encode it UTF-8-like (i.e. CESU-8). It is not only easier, it is also necessary in some cases: There is nothing wrong on having something like “private const char firstSurrogate = '\xD800';”.
  3. Leaving the final case: What if the string is localizable, but contains invalid UTF-16 (e.g. a bare surrogate)? In that case, xgettext should probably report an error, similar to when it detects invalid byte sequence for the selected encoding (e.g. “Non-ASCII character at…”).

Right now, I would be happy having just #2 above implemented, because that is my problem right now – xgettext crashing on a file having nothing to do with localization.

Petr Kadlec <mormegil>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by m1gu3l (Updated the item)
  • -unavailable- added by mormegil (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 28 Feb 2013 10:22:14 AM UTCm1gu3lStatusNone=>Confirmed
    Thu 28 Feb 2013 10:02:27 AM UTCm1gu3lAssigned toNone=>m1gu3l

    Back to the top


    Powered by Savane 3.1-cleanup