bugGNU gettext - Bugs: bug #50357, xgettext doesn't recognise...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #50357: xgettext doesn't recognise encoding aliases Python supports

Submitter:  Geoffrey Sneddon <gsnedders>
Submitted:  Sun 19 Feb 2017 10:11:55 PM UTC
   
 
Category:  Python Severity:  3 - Normal
Item Group:  None Status:  Not a Bug
Privacy:  Public Assigned to:  haible
Open/Closed:  Closed

Jump to the original submission

Mon 20 Feb 2017 10:17:50 AM UTC, comment #6: 


> name matching is standardised in various places; for example in UTR #22

No, name matching is not standardised. Different charset conversion libraries have different ways of doing it. The UTR #22 probably describes the way ICU does it, but ICU is != glibc, != libiconv, != Windows, != Solaris, etc.

Additionally, the problem with this specification is that it's not future-proof: If before, you have interpreted ISO-8859-1-5 as being equivalent to ISO-8859-15, it will break once an encoding ISO-8859-1-5 is really introduced. The text even states that this has already happened (for ISO-IR-9-1 and ISO-IR-9-2).

Bruno Haible <haible>
Group administrator
Mon 20 Feb 2017 10:12:06 AM UTC, comment #5: 


> all Python implementations to my knowledge support the same set of encodings for Python source files

But the "# coding:" marker is interpreted by different programs: not only by Python when it loads or compiles the file, but also by editors (Emacs et al.) when they display a file, or - here - by xgettext when it extracts internationalized strings.

You cannot expect that all these programs support all the encodings listed in that page, from EBCDIC-CP-HE to kz1048.

That's where standards come into play: to state clear expectations about interoperability between different programs. The standards don't know about an alias "utf8". So it's an unreasonable expectation that editors and xgettext support it.

Bruno Haible <haible>
Group administrator
Sun 19 Feb 2017 11:37:12 PM UTC, comment #4: 

I'll also point out that character set name matching is standardised in various places; for example in UTR #22: <http://www.unicode.org/reports/tr22/tr22-7.html#Charset_Alias_Matching>. Per that, "utf8" and "UTF-8" refer to the same encoding.

Geoffrey Sneddon <gsnedders>
Sun 19 Feb 2017 11:33:33 PM UTC, comment #3: 

https://docs.python.org/3/library/codecs.html#standard-encodings documents the set of encodings Python's standard library supports, and all Python implementations to my knowledge support the same set of encodings for Python source files. Notably, that table includes utf8 as an alias.

As such, this isn't about one implementation of Python; this is about matching all.

Geoffrey Sneddon <gsnedders>
Sun 19 Feb 2017 11:31:14 PM UTC, comment #2: 

In other words, just bite the bullet and change your source code to mention "# coding: utf-8". This is a one-time effort, and it's future-proof because the encoding name 'UTF-8' is standardized.

Bruno Haible <haible>
Group administrator
Sun 19 Feb 2017 11:28:19 PM UTC, comment #1: 

Thanks for the report.

> xgettext should support, for Python code, the same set of aliases as Python does.


I disagree. The set of recognised encoding names is defined by standards, in particular by IANA. The standards define one name for this encoding, namely "UTF-8". No alias.

Why it is not a good idea to invent new aliases for new encodings: because then one program after the other gets requests for supporting this aliases. And then the next program, for interoperability reasons, and so on. So the effect of such requests is to create interoperability problems for several years.

Btw, the web page you cited http://www.python.org/peps/pep-0263.html contains 10 mentions of "UTF-8" and 0 mentions of "UTF8".

Finally, your argument "should support ... the same set of aliases as Python does" talks about an implementation of the Python language. But there are several Python implementations (see https://en.wikipedia.org/wiki/List_of_Python_software#Python_implementations ), and I bet they don't support the exact same set of encodings on all platforms. That's why you need standards.

Bruno Haible <haible>
Group administrator
Sun 19 Feb 2017 10:11:55 PM UTC, original submission:  

Given a file starting with `# coding: utf8` xgettext outputs:

xgettext: bar.py:1: Unknown encoding "utf8". Proceeding with ASCII instead.
xgettext: Non-ASCII string at bar.py:3.
          Please specify the source encoding through --from-code or through a comment
          as specified in http://www.python.org/peps/pep-0263.html.
         
xgettext should support, for Python code, the same set of aliases as Python does.

Geoffrey Sneddon <gsnedders>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by gsnedders (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-02-19 haible StatusNone Not a Bug
        Assigned toNone haible
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code