bugGNU gettext - Bugs: bug #65778, Strings are not considered UTF-8...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #65778: Strings are not considered UTF-8 encoded in Python 3

Submitter:  Serhii Tereshchenko <partizan>
Submitted:  Thu 23 May 2024 08:05:24 AM UTC
   
 
Category:  Python Severity:  3 - Normal
Item Group:  None Status:  Fix Released
Privacy:  Public Assigned to:  haible
Open/Closed:  Closed

Sun 01 Dec 2024 04:37:45 PM UTC, comment #3: 

The fix is contained in gettext 0.23.

Bruno Haible <haible>
Group administrator
Wed 14 Aug 2024 04:20:35 PM UTC, comment #2: 
Bruno Haible <haible>
Group administrator
Thu 23 May 2024 05:54:41 PM UTC, comment #1: 

Your test.py is to be interpreted as UTF-8 source code, according to https://docs.python.org/3.12/reference/lexical_analysis.html#encoding-declarations "If no encoding declaration is found, the default encoding is UTF-8.".

The escape sequence \xA0 is, according to https://docs.python.org/3.12/reference/lexical_analysis.html#escape-sequences "In a bytes literal, ... escapes denote the byte with the given value. In a string literal, these escapes denote a Unicode character with the given value.", meant to denote U+00A0.

You're right, it's a bug in xgettext.

Bruno Haible <haible>
Group administrator
Thu 23 May 2024 08:05:24 AM UTC, original submission:  

Hello.

To reproduce this bug, create a file "test.py" with the following content:


gettext("Hello\xa0World")


Then, run xgettext:


> xgettext test.py
xgettext: String at test.py:1 is not UTF-8 encoded.
          Please specify the source encoding through --from-code.


Adding 'u' prefix to string, fixes this issue. But it should not be the case for Python 3, as it treats string as Unicode by default.

(And adding --from-code does not work, because Python is exception from this option)

There is related old bug in Django, related to templates:

https://code.djangoproject.com/ticket/26093

They're adding this prefix when converting {% trans ... %} tags to a code that gettext can understand. But this is workaround, and not a real solution.

Real solution should be on the gettext side, treating python string as "utf-8" by default (python 2 reached EOL long time ago). Or at least, adding an option for this.

Serhii Tereshchenko <partizan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by partizan (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-12-01 haible StatusFixed Fix Released
    2024-08-14 haible StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2024-05-23 haible StatusNone Confirmed
        Assigned toNone haible

    Back to the top

    Powered by Savane 3.14-9aa3.
    Corresponding source code