bugGNU gettext - Bugs: bug #32505, xgettext on C# file crashes when...

 
 

bug #32505: xgettext on C# file crashes when input file contains Unicode literal with surrogates

Submitted by:  Petr Kadlec <mormegil>
Submitted on:  Wed 16 Feb 2011 04:57:42 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Daiki Ueno <ueno>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 01 Dec 2014 07:17:51 AM UTC, comment #3:

Pushed a simplified patch, using mixed_string_buffer:
http://git.sv.gnu.org/cgit/gettext.git/commit/?id=6aad665c
with a test:
http://git.sv.gnu.org/cgit/gettext.git/commit/?id=0d2b9cd3

Daiki Ueno <ueno>
Project AdministratorIn charge of this item.
Fri 28 Nov 2014 08:57:09 AM UTC, comment #2:

According to ECMA-334 9.4.1, a Unicode code point in the range U+10000 to U+10FFFF is represented using two Unicode surrogate code units. I couldn't find any example in the specification, but a quick web search showed that the following expressions represent the same string:

This indeed works with Mono, but aborts xgettext. I'm attaching a tentative patch to fix this.

(file #32566)

Daiki Ueno <ueno>
Project AdministratorIn charge of this item.
Thu 28 Feb 2013 10:22:14 AM UTC, comment #1:

#1 and #3 are implemented currently, but the real problem is #2, because conversion to UTF-8 is done before knowing it is translatable.

This problem is the related to the issues I'm facing with unicode support in C/C++. It will need some design changes, maybe something quite simple like storing an "int buffer" instead of a "char buffer" and process it in a later stage, but there are too many places to touch and lots of testing to do before posting any useful patch.

Also, this bug report is old, but if you already need a workaround and this file does not contains any translatable string, you can comment out this file from your POTFILES.in or erase it from xgettext arguments.

Miguel Ángel Arruga Vivas <m1gu3l>
Project Member
Wed 16 Feb 2011 04:57:42 PM UTC, original submission:

Running xgettext on a C# file which contains a string literal containing a Unicode escape for a surrogate character leads to an abnormal termination (abort()).

This is because the character reference is parsed correctly, then passed to string_buffer_append_unicode() in x-csharp.cs, which uses u8_uctomb() to encode the character to UTF-8. However, u8_uctomb() denies to encode surrogate characters (correctly, we might say, they should not be encoded separately, the “real” Unicode character should be encoded, not its UTF-16 encoding) and returns -1, after which string_buffer_append_unicode() decides to abort(), as “The caller should have ensured that uc is not out-of-range.”

Reproducible:
test.cs:
Then run:
xgettext --language=C# test.cs

A few thoughts on solving the problem:

  1. When the string containing the surrogate character is in fact a localizable string, and would get to the .po file, we should ensure the first surrogate is correctly followed by the corresponding low surrogate, and then UTF-8 encode the real Unicode character. But this is probably not completely trivial.
  2. When the string is not localizable and we are parsing just to get around in the file, we could ignore the underlying meaning of the surrogate character and just ignore it altogether, or encode it UTF-8-like (i.e. CESU-8). It is not only easier, it is also necessary in some cases: There is nothing wrong on having something like “private const char firstSurrogate = '\xD800';”.
  3. Leaving the final case: What if the string is localizable, but contains invalid UTF-16 (e.g. a bare surrogate)? In that case, xgettext should probably report an error, similar to when it detects invalid byte sequence for the selected encoding (e.g. “Non-ASCII character at…”).

Right now, I would be happy having just #2 above implemented, because that is my problem right now – xgettext crashing on a file having nothing to do with localization.

Petr Kadlec <mormegil>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by ueno (Updated the item)
  • -unavailable- added by m1gu3l (Updated the item)
  • -unavailable- added by mormegil (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 01 Dec 2014 07:17:51 AM UTCuenoStatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed
    Fri 28 Nov 2014 08:57:09 AM UTCuenoAttached File-=>Added 0001-csharp-Recognize-Unicode-surrogate-character-pair.patch, #32566
      StatusConfirmed=>In Progress
      Assigned tom1gu3l=>ueno
    Thu 28 Feb 2013 10:22:14 AM UTCm1gu3lStatusNone=>Confirmed
    Thu 28 Feb 2013 10:02:27 AM UTCm1gu3lAssigned toNone=>m1gu3l

    Back to the top


    Powered by Savane 3.1-cleanup