bugGNU gettext - Bugs: bug #50357, xgettext doesn't recognise...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #50357: xgettext doesn't recognise encoding aliases Python supports

Submitted by:  Geoffrey Sneddon <gsnedders>
Submitted on:  Sun 19 Feb 2017 10:11:55 PM UTC  
 
Category: PythonSeverity: 3 - Normal
Item Group: NoneStatus: Not a Bug
Privacy: PublicAssigned to: Bruno Haible <haible>
Open/Closed: Closed

(Jump to the original submission Jump to the original submission)

Mon 20 Feb 2017 10:17:50 AM UTC, comment #6:

> name matching is standardised in various places; for example in UTR #22

No, name matching is not standardised. Different charset conversion libraries have different ways of doing it. The UTR #22 probably describes the way ICU does it, but ICU is != glibc, != libiconv, != Windows, != Solaris, etc.

Additionally, the problem with this specification is that it's not future-proof: If before, you have interpreted ISO-8859-1-5 as being equivalent to ISO-8859-15, it will break once an encoding ISO-8859-1-5 is really introduced. The text even states that this has already happened (for ISO-IR-9-1 and ISO-IR-9-2).

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Mon 20 Feb 2017 10:12:06 AM UTC, comment #5:

> all Python implementations to my knowledge support the same set of encodings for Python source files

But the "# coding:" marker is interpreted by different programs: not only by Python when it loads or compiles the file, but also by editors (Emacs et al.) when they display a file, or - here - by xgettext when it extracts internationalized strings.

You cannot expect that all these programs support all the encodings listed in that page, from EBCDIC-CP-HE to kz1048.

That's where standards come into play: to state clear expectations about interoperability between different programs. The standards don't know about an alias "utf8". So it's an unreasonable expectation that editors and xgettext support it.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun 19 Feb 2017 11:37:12 PM UTC, comment #4:

I'll also point out that character set name matching is standardised in various places; for example in UTR #22: <http://www.unicode.org/reports/tr22/tr22-7.html#Charset_Alias_Matching>. Per that, "utf8" and "UTF-8" refer to the same encoding.

Geoffrey Sneddon <gsnedders>
Sun 19 Feb 2017 11:33:33 PM UTC, comment #3:

https://docs.python.org/3/library/codecs.html#standard-encodings documents the set of encodings Python's standard library supports, and all Python implementations to my knowledge support the same set of encodings for Python source files. Notably, that table includes utf8 as an alias.

As such, this isn't about one implementation of Python; this is about matching all.

Geoffrey Sneddon <gsnedders>
Sun 19 Feb 2017 11:31:14 PM UTC, comment #2:

In other words, just bite the bullet and change your source code to mention "# coding: utf-8". This is a one-time effort, and it's future-proof because the encoding name 'UTF-8' is standardized.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun 19 Feb 2017 11:28:19 PM UTC, comment #1:

Thanks for the report.

> xgettext should support, for Python code, the same set of aliases as Python does.


I disagree. The set of recognised encoding names is defined by standards, in particular by IANA. The standards define one name for this encoding, namely "UTF-8". No alias.

Why it is not a good idea to invent new aliases for new encodings: because then one program after the other gets requests for supporting this aliases. And then the next program, for interoperability reasons, and so on. So the effect of such requests is to create interoperability problems for several years.

Btw, the web page you cited http://www.python.org/peps/pep-0263.html contains 10 mentions of "UTF-8" and 0 mentions of "UTF8".

Finally, your argument "should support ... the same set of aliases as Python does" talks about an implementation of the Python language. But there are several Python implementations (see https://en.wikipedia.org/wiki/List_of_Python_software#Python_implementations ), and I bet they don't support the exact same set of encodings on all platforms. That's why you need standards.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun 19 Feb 2017 10:11:55 PM UTC, original submission:

Given a file starting with `# coding: utf8` xgettext outputs:

xgettext: bar.py:1: Unknown encoding "utf8". Proceeding with ASCII instead.
xgettext: Non-ASCII string at bar.py:3.
Please specify the source encoding through --from-code or through a comment
as specified in http://www.python.org/peps/pep-0263.html.

xgettext should support, for Python code, the same set of aliases as Python does.

Geoffrey Sneddon <gsnedders>

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by haible (Posted a comment)
  • -unavailable- added by gsnedders (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 19 Feb 2017 11:28:19 PM UTChaibleStatusNone=>Not a Bug
      Assigned toNone=>haible
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1