bugGNU gettext - Bugs: bug #58643, ITS - Double escapes when merging...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #58643: ITS - Double escapes when merging translations back to XML

Submitter:  Samy Mahmoudi <samy_mahmoudi>
Submitted:  Mon 22 Jun 2020 05:31:49 PM UTC
   
 
Category:  XML Severity:  3 - Normal
Item Group:  None Status:  Confirmed
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Jump to the original submission

Mon 05 Sep 2022 03:22:12 AM UTC, comment #16: 

comment #11:

> I agree with you that the current text is misleading. Would you like to propose the doc change as a patch (I believe it wouldn't require copyright assignment)?


Hello Daiki,

I have added a third patch for the documentation. Please see comment #15 together with our previous discussions as a reminder.

The three patches should be fine.

Kind regards,
Samy Mahmoudi

Samy Mahmoudi <samy_mahmoudi>
Sat 24 Oct 2020 09:35:49 AM UTC, comment #15: 

I have checked that ‘msgfmt --xml’ also performs character escaping when ‘escape’ is set to ‘yes’.

To reproduce, follow the instructions of Daiki from comment #7, with metainfo.its-escape-yes.diff instead of metainfo.its.diff in step 3, and "Foo &amp; Bar" as "foo &amp; bar" instead of "Foo & Bar" as "foo & bar" in step 7.

Therefore ‘msgfmt --xml’ performs character escaping without regard to the value of the ‘escape’ attribute.

A third patch is attached.

(file #50083)

Samy Mahmoudi <samy_mahmoudi>
Mon 27 Jul 2020 02:01:59 AM UTC, comment #14: 

I also noticed two typos. Please find attached the proposed patches and do not hesitate to give your feedback.

(file #49550, file #49551)

Samy Mahmoudi <samy_mahmoudi>
Sun 26 Jul 2020 03:57:25 PM UTC, comment #13: 


> ‘msgfmt’’s --xml DOES perform character escaping when merging

When 'escape' is set to 'no', this is confirmed by comment #7 from Daiki.

Samy Mahmoudi <samy_mahmoudi>
Sun 26 Jul 2020 03:52:13 PM UTC, comment #12: 

Hi all,

Sorry for the delay in response.

> Would you like to propose the doc change as a patch (I believe it wouldn't require copyright assignment)?

Yes I would!

A patch will be attached soon but I doubt the following assertion (extracted from the current doc) is true: ‘msgfmt’’s --xml option doesn’t perform character escaping, so translated strings can have arbitrary XML constructs, such as elements for markup. Source: https://www.gnu.org/software/gettext/manual/html_node/Preparing-ITS-Rules.html

Because from what I can remember, ‘msgfmt’’s --xml DOES perform character escaping when merging, independently of the value of the 'escape' attribute. Could you double-check this so that I could possibly add the deletion of that sentence to the patch.

Samy Mahmoudi <samy_mahmoudi>
Mon 29 Jun 2020 05:57:29 PM UTC, comment #11: 

I agree with you that the current text is misleading. Would you like to propose the doc change as a patch (I believe it wouldn't require copyright assignment)?

Daiki Ueno <ueno>
Group administrator
Mon 29 Jun 2020 09:07:12 AM UTC, comment #10: 

I had felt my previous proposal could be wrong (hence the 'might'). Now I just found what I was looking for, namely the following lines in gschema.its:

  <!-- Extracted strings are consumed by the library and are never
       merged back; we don't want to escape special characters.  -->
  <gt:escapeRule selector="/schemalist" escape="no"/>

Therefore correcting the documentation needs more than the proposed inversion.

Samy Mahmoudi <samy_mahmoudi>
Sun 28 Jun 2020 06:19:55 PM UTC, comment #9: 

So we might need to invert 'former' and 'latter' (or alternatively 'shouldn't' and 'should') in the first paragraph of section 16.1.6.1.

Samy Mahmoudi <samy_mahmoudi>
Sun 28 Jun 2020 04:16:25 PM UTC, comment #8: 


> I can't reproduce it. The steps I tried are:

Neither can I (following your steps).

> But now the translations are not merged back

This was due to an incomplete understanding of the fuzzy attribute (fuzzy attributes have been added to the entries that correspond to my change of escape=yes into escape=no). After revision, and deletion of such attributes, translations are indeed merged back as expected.

In conclusion, this boils down to escape=yes being inappropriate when merging back to XML, which bring us back to the initial description of this bug: Section 16.1.6.1 of https://www.gnu.org/software/gettext/manual/html_node/Preparing-ITS-Rules.html seems quite misleading to me! Because if "they should in the latter case" ("they" refers to "special characters in the extracted strings"), then one would set escape to yes, wouldn't one?

Samy Mahmoudi <samy_mahmoudi>
Sun 28 Jun 2020 09:12:01 AM UTC, comment #7: 


> But now the translations are not merged back


I can't reproduce it. The steps I tried are:

  1. git clone --depth 1 https://git.savannah.gnu.org/git/gettext.git
  2. cd gettext
  3. patch -p1 < metainfo.its.diff
  4. cd gettext-tools
  5. cp mf.appdata.xml .
  6. GETTEXTDATADIR=. xgettext -o mf.pot mf.appdata.xml
  7. msginit --input mf.pot -l fr --no-translator
  8. vi fr.po # add translation for "Foo & Bar" as "foo & bar"
  9. GETTEXTDATADIR=. msgfmt --xml --template mf.appdata.xml -l fr fr.po -o merged.xml


Now I have:

$ cat merged.xml
<?xml version="1.0" encoding="UTF-8"?>
<component type="desktop">
  <id>org.gnome.Characters.desktop</id>
  <name>GNOME Characters</name>
  <summary>Character map application</summary>
  <licence>CC0</licence>
  <description>
    <p>Foo &amp; Bar</p>
    <p xml:lang="fr">foo &amp; bar</p>
  </description>
  <url type="homepage">https://wiki.gnome.org/Design/Apps/CharacterMap</url>
  <updatecontact>dueno_at_src.gnome.org</updatecontact>
</component>


Daiki Ueno <ueno>
Group administrator
Wed 24 Jun 2020 01:36:42 PM UTC, comment #6: 

Just to make myself clear: are the following workflows reasonable and/or supposed to be achievable in the current gettext version?


Ideal workflow:

• original XML:             <short>&amp;</short>

• generated *.po:           msgid "&"

• *.po after translations:  msgid "&"
                            msgstr "&"

• merged XML:               <short>&amp;</short>
                            <short xml:lang="aa">&amp;</short>
                            <short xml:lang="bb">&amp;</short>
                            <short xml:lang="cc">&amp;</short>


Not-ideal-but-acceptable workflow:

• original XML:             <short>&amp;</short>

• generated *.po:           msgid "&amp;"

• *.po after translations:  msgid "&amp;"
                            msgstr "&amp;"

• merged XML:               <short>&amp;</short>
                            <short xml:lang="aa">&amp;</short>
                            <short xml:lang="bb">&amp;</short>
                            <short xml:lang="cc">&amp;</short>


As for the achievability, I don't get to achieve any of these. Can you?

Samy Mahmoudi <samy_mahmoudi>
Wed 24 Jun 2020 01:11:31 PM UTC, comment #5: 


> I reproduce the issue even with the default AppData XML rules, with the attached files and these steps:

It seems you considered a different scenario because in my original XML files, &amp; are used instead of & as is. I had that idea of replacing occurences of &amp; with & in my original .xml.in documents, which I tried to justify by the fact that my templates do not have to be valid XML documents. But using & as is obviously results in an error message 'xmlParseEntityRef: no name' whether it be at extraction or when merging. Why don't you have such an error in your example? Anyway, this workaround now looks a bad idea to me, because even if gettext/msgfmt's underlying XML parser let me do this, I would still have to reverse the replacements when internationalization is disabled: .xml.in (invalid-XML XML template) ---> *.xml (valid-XML XML file).

> xgettext should map character entities to plain Unicode characters, and leave other entities unchanged.
> msgfmt should not escape the msgstr when storing it in the merged XML file.

Do you mean we would e.g. store a plain & in the merged XML file? Does not this make it invalid?

>> xgettext should map character entities to plain Unicode characters, and leave other entities unchanged.
>
> This might be already covered by `gt:escapeRule`, as the attached patch.

I had escape set to yes. If I set escape to no and update my *.po files, then all the strings containing &amp; in my original XML become strings containing & in msgid lines, and I just need to update the msgstr lines: in a sense, this is how I thought my problem would be solved when filing this bug report.

But now the translations are not merged back, which makes me realize that may not be the way to solve the problem. So the question is now:

When you meet the following conditions,

• escape set to no
• <short>&amp;</short>; in your original XML file
• *.po files updated to msgid "&" and msgstr "&"

what is the intended behavior of a merge: merging several translations containing "&" into "&amp;" (just below the original "&amp;" in the XML), or not merging at all because the msgid "&" does not match the "&amp;" of the original document?

Samy Mahmoudi <samy_mahmoudi>
Wed 24 Jun 2020 07:31:34 AM UTC, comment #4: 


> xgettext should map character entities to plain Unicode characters, and leave other entities unchanged.


This might be already covered by `gt:escapeRule`, as the attached patch.

(file #49365)

Daiki Ueno <ueno>
Group administrator
Wed 24 Jun 2020 07:15:53 AM UTC, comment #3: 

We also need to respect/consider the ITS standard https://www.w3.org/TR/its/ .

Bruno Haible <haible>
Group administrator
Wed 24 Jun 2020 12:37:28 AM UTC, comment #2: 

As PO files are not XML files, maybe we need to distinguish character entities from other entities (defined through <!ENTITY ...>). The latter frequently occur in Docbook XML files can meaningfully occur in msgids and msgstrs.

So, as I see it:

  • xgettext should map character entities to plain Unicode characters, and leave other entities unchanged.
  • msgfmt should not escape the msgstr when storing it in the merged XML file.


Daiki, what do you think? Is this correct?

Bruno Haible <haible>
Group administrator
Wed 24 Jun 2020 12:25:01 AM UTC, comment #1: 

Thanks for the report.

I reproduce the issue even with the default AppData XML rules, with the attached files and these steps:

1.

xgettext -o mf.pot mf.appdata.xml


2. Copy mf.pot to fr.po and fill in fr.po.

3.

msgfmt --xml --template=mf.appdata.xml -l fr fr.po -o merged.xml



(file #49362, file #49363, file #49364)

Bruno Haible <haible>
Group administrator
Mon 22 Jun 2020 05:31:49 PM UTC, original submission:  

Hi all,

In the second use-case of XML internationalization (as described in 16.1.6.1: https://www.gnu.org/software/gettext/manual/html_node/Preparing-ITS-Rules.html), special characters in the extracted strings should be escaped when they are merged back, and they are indeed escaped as intended.

However, when extracting strings from the original XML document, escaped special characters are not unescaped. I doubt that this is the intended behavior.

For example, <short>foo &amp; bar</short> (in the original XML document) with a short translateRule selector (ITS) successively becomes:

msgid "foo &amp; bar" (in gnu.po, before translation)
msgstr ""

msgid "foo &amp; bar"
msgstr "jed &amp; zvd" (in gnu.po, after translation into the GNU language...)

<short>foo &amp; bar</short>
<short xml:lang="gnu">jed &amp;amp; zvd</short> (in the merged XML)

Would xgettext rather unescape special characters when extracting strings from original XML documents ? Or at least optionally for this use-case ?

In addition to avoiding double escapes in the merged XML document, unescaping special characters when extracting strings from original XML documents would also increase readability in the translation process, since msgid "foo &amp; bar" is less readable than msgid "foo & bar".

Best regards,
Samy Mahmoudi

Samy Mahmoudi <samy_mahmoudi>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #49365:  metainfo.its.diff added by ueno (813B - text/x-patch)
file #49362:  mf.appdata.xml added by haible (406B - text/xml)
file #49363:  fr.po added by haible (807B - text/x-gettext-translation)
file #49364:  merged.xml added by haible (563B - text/xml)
file #49348:  compiz.loc added by samy_mahmoudi (179B - application/octet-stream - .its and .loc files)
file #49349:  compiz.its added by samy_mahmoudi (573B - application/octet-stream - .its and .loc files)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ueno (Updated the item)
  • -email is unavailable- added by haible (Updated the item)
  • -email is unavailable- added by samy_mahmoudi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-10-24 samy_mahmoudi Attached File- Added 0003-doc-Correct-a-note-related-to-msgfmt.patch, #50083
    2020-10-24 samy_mahmoudi Attached File- Added metainfo.its-escape-yes.diff, #50081
    2020-07-27 samy_mahmoudi Attached File- Added 0001-doc-Fix-two-typos.patch, #49550
        Attached File- Added 0002-doc-Correct-suggestion-when-merging-back-to-XML.patch, #49551
    2020-06-24 ueno Attached File- Added metainfo.its.diff, #49365
    2020-06-24 haible Attached File- Added mf.appdata.xml, #49362
        Attached File- Added fr.po, #49363
        Attached File- Added merged.xml, #49364
        StatusNone Confirmed
    2020-06-22 samy_mahmoudi Attached File- Added compiz.loc, #49348
        Attached File- Added compiz.its, #49349

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code