GNU gettext - Bugs: bug #58643, ITS - Double escapes when merging...
You are not allowed to post comments on this tracker with your current authentication level.
bug #58643: ITS - Double escapes when merging translations back to XML
Submitter: | Samy Mahmoudi <samy_mahmoudi> | ||
Submitted: | Mon 22 Jun 2020 05:31:49 PM UTC | ||
Category: | XML | Severity: | 3 - Normal |
Item Group: | None | Status: | Confirmed |
Privacy: | Public | Assigned to: | None |
Open/Closed: | Open |
( Jump to the original submission )
Mon 05 Sep 2022 03:22:12 AM UTC, comment #16: |
Samy Mahmoudi <samy_mahmoudi> |
Sat 24 Oct 2020 09:35:49 AM UTC, comment #15: I have checked that ‘msgfmt --xml’ also performs character escaping when ‘escape’ is set to ‘yes’.
To reproduce, follow the instructions of Daiki from comment #7, with metainfo.its-escape-yes.diff instead of metainfo.its.diff in step 3, and "Foo & Bar" as "foo & bar" instead of "Foo & Bar" as "foo & bar" in step 7.
Therefore ‘msgfmt --xml’ performs character escaping without regard to the value of the ‘escape’ attribute.
A third patch is attached.
|
Samy Mahmoudi <samy_mahmoudi> |
Mon 27 Jul 2020 02:01:59 AM UTC, comment #14: I also noticed two typos. Please find attached the proposed patches and do not hesitate to give your feedback. |
Samy Mahmoudi <samy_mahmoudi> |
Sun 26 Jul 2020 03:57:25 PM UTC, comment #13:
When 'escape' is set to 'no', this is confirmed by comment #7 from Daiki. |
Samy Mahmoudi <samy_mahmoudi> |
Sun 26 Jul 2020 03:52:13 PM UTC, comment #12: Hi all,
Sorry for the delay in response.
Yes I would!
A patch will be attached soon but I doubt the following assertion (extracted from the current doc) is true: ‘msgfmt’’s --xml option doesn’t perform character escaping, so translated strings can have arbitrary XML constructs, such as elements for markup. Source: https://www.gnu.org/software/gettext/manual/html_node/Preparing-ITS-Rules.html
Because from what I can remember, ‘msgfmt’’s --xml DOES perform character escaping when merging, independently of the value of the 'escape' attribute. Could you double-check this so that I could possibly add the deletion of that sentence to the patch. |
Samy Mahmoudi <samy_mahmoudi> |
Mon 29 Jun 2020 05:57:29 PM UTC, comment #11: I agree with you that the current text is misleading. Would you like to propose the doc change as a patch (I believe it wouldn't require copyright assignment)? |
Daiki Ueno <ueno>![]() |
Mon 29 Jun 2020 09:07:12 AM UTC, comment #10: I had felt my previous proposal could be wrong (hence the 'might'). Now I just found what I was looking for, namely the following lines in gschema.its:
<!-- Extracted strings are consumed by the library and are never
Therefore correcting the documentation needs more than the proposed inversion. |
Samy Mahmoudi <samy_mahmoudi> |
Sun 28 Jun 2020 06:19:55 PM UTC, comment #9: So we might need to invert 'former' and 'latter' (or alternatively 'shouldn't' and 'should') in the first paragraph of section 16.1.6.1. |
Samy Mahmoudi <samy_mahmoudi> |
Sun 28 Jun 2020 04:16:25 PM UTC, comment #8:
Neither can I (following your steps).
This was due to an incomplete understanding of the fuzzy attribute (fuzzy attributes have been added to the entries that correspond to my change of escape=yes into escape=no). After revision, and deletion of such attributes, translations are indeed merged back as expected.
In conclusion, this boils down to escape=yes being inappropriate when merging back to XML, which bring us back to the initial description of this bug: Section 16.1.6.1 of https://www.gnu.org/software/gettext/manual/html_node/Preparing-ITS-Rules.html seems quite misleading to me! Because if "they should in the latter case" ("they" refers to "special characters in the extracted strings"), then one would set escape to yes, wouldn't one? |
Samy Mahmoudi <samy_mahmoudi> |
Sun 28 Jun 2020 09:12:01 AM UTC, comment #7:
Now I have:
|
Daiki Ueno <ueno>![]() |
Wed 24 Jun 2020 01:36:42 PM UTC, comment #6: Just to make myself clear: are the following workflows reasonable and/or supposed to be achievable in the current gettext version?
Ideal workflow:
• original XML: <short>&</short>
• generated *.po: msgid "&"
• *.po after translations: msgid "&"
• merged XML: <short>&</short>
Not-ideal-but-acceptable workflow:
• original XML: <short>&</short>
• generated *.po: msgid "&"
• *.po after translations: msgid "&"
• merged XML: <short>&</short>
As for the achievability, I don't get to achieve any of these. Can you? |
Samy Mahmoudi <samy_mahmoudi> |
Wed 24 Jun 2020 01:11:31 PM UTC, comment #5:
It seems you considered a different scenario because in my original XML files, & are used instead of & as is. I had that idea of replacing occurences of & with & in my original .xml.in documents, which I tried to justify by the fact that my templates do not have to be valid XML documents. But using & as is obviously results in an error message 'xmlParseEntityRef: no name' whether it be at extraction or when merging. Why don't you have such an error in your example? Anyway, this workaround now looks a bad idea to me, because even if gettext/msgfmt's underlying XML parser let me do this, I would still have to reverse the replacements when internationalization is disabled: .xml.in (invalid-XML XML template) ---> *.xml (valid-XML XML file).
Do you mean we would e.g. store a plain & in the merged XML file? Does not this make it invalid?
I had escape set to yes. If I set escape to no and update my *.po files, then all the strings containing & in my original XML become strings containing & in msgid lines, and I just need to update the msgstr lines: in a sense, this is how I thought my problem would be solved when filing this bug report.
But now the translations are not merged back, which makes me realize that may not be the way to solve the problem. So the question is now:
When you meet the following conditions,
• escape set to no
what is the intended behavior of a merge: merging several translations containing "&" into "&" (just below the original "&" in the XML), or not merging at all because the msgid "&" does not match the "&" of the original document? |
Samy Mahmoudi <samy_mahmoudi> |
Wed 24 Jun 2020 07:31:34 AM UTC, comment #4:
|
Daiki Ueno <ueno>![]() |
Wed 24 Jun 2020 07:15:53 AM UTC, comment #3: We also need to respect/consider the ITS standard https://www.w3.org/TR/its/ . |
Bruno Haible <haible>![]() |
Wed 24 Jun 2020 12:37:28 AM UTC, comment #2: As PO files are not XML files, maybe we need to distinguish character entities from other entities (defined through <!ENTITY ...>). The latter frequently occur in Docbook XML files can meaningfully occur in msgids and msgstrs.
So, as I see it:
Daiki, what do you think? Is this correct? |
Bruno Haible <haible>![]() |
Wed 24 Jun 2020 12:25:01 AM UTC, comment #1: Thanks for the report.
I reproduce the issue even with the default AppData XML rules, with the attached files and these steps:
1.
2. Copy mf.pot to fr.po and fill in fr.po.
3.
|
Bruno Haible <haible>![]() |
Mon 22 Jun 2020 05:31:49 PM UTC, original submission:
Hi all,
In the second use-case of XML internationalization (as described in 16.1.6.1: https://www.gnu.org/software/gettext/manual/html_node/Preparing-ITS-Rules.html), special characters in the extracted strings should be escaped when they are merged back, and they are indeed escaped as intended.
However, when extracting strings from the original XML document, escaped special characters are not unescaped. I doubt that this is the intended behavior.
For example, <short>foo & bar</short> (in the original XML document) with a short translateRule selector (ITS) successively becomes:
msgid "foo & bar" (in gnu.po, before translation)
msgid "foo & bar"
<short>foo & bar</short>
Would xgettext rather unescape special characters when extracting strings from original XML documents ? Or at least optionally for this use-case ?
In addition to avoiding double escapes in the merged XML document, unescaping special characters when extracting strings from original XML documents would also increase readability in the translation process, since msgid "foo & bar" is less readable than msgid "foo & bar".
Best regards,
|
Samy Mahmoudi <samy_mahmoudi> |
Depends on the following items: None found
Items that depend on this one: None found
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Follow 11 latest changes.
Date | Changed by | Updated Field | Previous Value | => | Replaced by |
---|---|---|---|---|---|
2020-10-24 | samy_mahmoudi | Attached File | - | ![]() |
Added 0003-doc-Correct-a-note-related-to-msgfmt.patch, #50083 |
2020-10-24 | samy_mahmoudi | Attached File | - | ![]() |
Added metainfo.its-escape-yes.diff, #50081 |
2020-07-27 | samy_mahmoudi | Attached File | - | ![]() |
Added 0001-doc-Fix-two-typos.patch, #49550 |
Attached File | - | ![]() |
Added 0002-doc-Correct-suggestion-when-merging-back-to-XML.patch, #49551 | ||
2020-06-24 | ueno | Attached File | - | ![]() |
Added metainfo.its.diff, #49365 |
2020-06-24 | haible | Attached File | - | ![]() |
Added mf.appdata.xml, #49362 |
Attached File | - | ![]() |
Added fr.po, #49363 | ||
Attached File | - | ![]() |
Added merged.xml, #49364 | ||
Status | None | ![]() |
Confirmed | ||
2020-06-22 | samy_mahmoudi | Attached File | - | ![]() |
Added compiz.loc, #49348 |
Attached File | - | ![]() |
Added compiz.its, #49349 |
comment #11:
Hello Daiki,
I have added a third patch for the documentation. Please see comment #15 together with our previous discussions as a reminder.
The three patches should be fine.
Kind regards,
Samy Mahmoudi