bugGNU gettext - Bugs: bug #50920, support Javascript template...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #50920: support Javascript template strings in xgettext

Submitter:  Philip Chimento <ptomato>
Submitted:  Tue 02 May 2017 11:14:55 PM UTC
Votes: 405
 
Category:  JavaScript Severity:  4 - Important
Item Group:  None Status:  Fix Released
Privacy:  Public Assigned to:  haible
Open/Closed:  Closed

Jump to the original submission

Tue 19 May 2020 10:53:29 PM UTC, comment #22: 

As the two test cases that you are showing use a combination of template strings (standardized in ECMAscript) and JSX (not standardized in ECMAscript, cf. bug #57927), I would prefer if you could open a new ticket.

Bruno Haible <haible>
Group administrator
Tue 19 May 2020 09:40:05 PM UTC, comment #21: 

The 0.20.2 release fixes the repro in comment #17 but the parsing short-circuit still happens when the template literal lives inside a tag:


const App = () => (
  <div>
    _('one');
    {`${'foo'}`}
    _('two');
  </div>
)


Also fails as an attribute inside a tag:


_('one');
<div className={`${'foo'}`} />
_('two');


Should we reopen this ticket? Or I'd be happy to open a new ticket if you'd prefer.

(And thank you for your time to get this working with newer JavaScript!)

whiteinge
Tue 14 Apr 2020 12:25:03 PM UTC, comment #20: 

The fix is contained in gettext 0.20.2.

Bruno Haible <haible>
Group administrator
Wed 25 Sep 2019 12:41:44 AM UTC, comment #19: 

In fact, the fix of bug #56678 (already in git) fixes this test case.

Bruno Haible <haible>
Group administrator
Wed 25 Sep 2019 12:31:04 AM UTC, comment #18: 

Thanks for the report and test case. I can reproduce the issue.

Bruno Haible <haible>
Group administrator
Tue 24 Sep 2019 12:30:08 AM UTC, comment #17: 

Sorry for all the noise Bruno, here is the minimal reproducer with 0.20.1:

    _('one');
    {`${}`}
    _('two');

So a template string inside any code block, {}, seems to cause xgettext to skip the rest of the file during parsing without warning or error.

Andy Holmes <andyholmes>
Tue 24 Sep 2019 12:00:00 AM UTC, comment #16: 

Sorry, my previous remark was a bit hasty. There is definitely something that's breaking xgettext, but its clear to me it's not all template literals.

I have a rather large project and I'm in the process of tracking down the offending code; will report back when I do.

Andy Holmes <andyholmes>
Mon 23 Sep 2019 06:58:37 PM UTC, comment #15: 

Hi, sorry to report bad news, but since at least 0.20.1 parsing template literals is entirely broken for me in GJS/JavaScript.

Previously (0.19.?) template literals with unbalanced slashes or oddly nested quotes could cause problems, but now almost any template literal will choke up xgettext. :(

Andy Holmes <andyholmes>
Thu 09 May 2019 07:28:46 PM UTC, comment #14: 

The fix is contained in the gettext-0.20 release.

Bruno Haible <haible>
Group administrator
Mon 08 Apr 2019 12:29:02 AM UTC, comment #13: 

Marco,

Thanks for your patches. I did not use them because they don't provide the result that I'd like to have:

1) For
var s7 = _(tag`A template literal with a tag`);
it extracts an msgid "A template literal with a tag". I prefer not to extract that, because we don't know anything about what the tag function does.

2) For
var s10 = `abc${foo({},_('should be extracted'))}xyz`;
it does not extract an msgid "should be extracted". But the expression between ${ and the matching(!) closing brace is a normal expression; _() calls in it should be handled.

Bruno Haible <haible>
Group administrator
Mon 08 Apr 2019 12:11:53 AM UTC, comment #12: 

Proper parsing of the template literals is implemented through
https://git.savannah.gnu.org/gitweb/?p=gettext.git;a=commitdiff;h=db8e4b76206ec179112b8171b92270e025c62a20

Note: This is just the parsing. If someone makes use of tagged template literals for internationalization, by defining a tag function that
1. converts the given string arguments to a single string with placeholders,
2. looks up the translation of that string from a message catalog,
3. inserts the provided values in place of the placeholders,
such as in
   tag `OK to overwrite file ${source} with ${target}?`
-> "OK to overwrite file {0} with {1}?"
or "OK to overwrite file %s with %s?"
this will require extra support in xgettext. xgettext does not "see" the definition of the tag function, nor can it execute JavaScript.

Bruno Haible <haible>
Group administrator
Sun 07 Apr 2019 10:09:24 PM UTC, comment #11: 


> I think it is not straightforward to determine the end of template strings, because substitutions may be nested


The grammar in http://www.ecma-international.org/ecma-262/6.0/#sec-template-literal is pretty straightforward. Yes, it looks like template strings can be nested, just like expressions can be nested by using parentheses.

> Maybe, it could be based on the backtick support in the shell-script scanner, which takes into account of nesting.


No, the backtick support in the shell-script scanner has additional complexities because a backslash inside `...` in shell scripts is different from a backslash outside `...`. Here, with the JavaScript template strings, it's not that hairy.

Bruno Haible <haible>
Group administrator
Thu 25 Oct 2018 05:03:03 AM UTC, comment #10: 

Thank you for the patches, Marco.  The general approach (discarding template strings with any substitutions) sounds reasonable.

However, I think it is not straightforward to determine the end of template strings, because substitutions may be nested; for example, this is a valid template string:

`a${`b${`c`+d}`}e`


Maybe, it could be based on the backtick support in the shell-script scanner, which takes into account of nesting.

Daiki Ueno <ueno>
Group administrator
Fri 07 Sep 2018 02:21:49 AM UTC, comment #9: 

While I agree that supporting template strings with placeholder is wrong, I think is actually safe to support template strings when they only contain normal text.

In any case, these strings should be properly parsed, and we should not ignore them in case they contain a mix of apices.

For example a code like this:

# xgettext --language=javascript -o - --extract-all /tmp/test.js
let str = `that's a valid string. ` + _('This too');


A part of warn, it generates something like:


#: /tmp/test.js:1
msgid "s a valid string. ` + _("
msgstr ""

#: /tmp/test.js:1
msgid ");"
msgstr ""



Attached here, two patches that fix this.
The 1st one, just add support for the backtick (`) strings, while the second one, discards them when they use javascript placeholders (so basically if there's a "${" inside; and that's enough to respect the JS syntax).

In case --extract-all is passed we warn about this misusage, however I'd love to do this in all the cases a template string is passed to a gettext flag (i.e., only if we do _(`${foo}`), and not if we've just a print(`${foo}`) call).

I've tried to do that, but unfortunately there are some xgettext limitations at core level that make this quite annoying to do only changing the javascript parser.

Ideally, I'd like to have a function that is called once the messageid is extracted, just before the format parsing, and that can be used to verify the content of the string, to check if this is legal or not.
I've tried to get the same with a literalstring_parser, but that's not doable because it doesn't handle nullified msgid's, and that's a too late stage.

Anyway, the two commits here should fix the main issue, in the mean time we don't decide to have something with a nicer feedback.

(file #44949, file #44950)

Marco Trevisan <trevinho>
Wed 25 Jul 2018 03:22:39 AM UTC, comment #8: 

Likewise.

David Mosberger-Tang <dmosberger>
Wed 18 Jul 2018 12:34:56 PM UTC, comment #7: 

Same issue here and it's a real PITA.

Bruno Desthuilliers <brunoblookup>
Wed 04 Oct 2017 03:05:10 PM UTC, comment #6: 

Hello,

We get the same issue, it is quite painful if we want to use recent version of JavaScript.

How can we help to resolve this issue?

Regards,
Nicolas

Nicolas Terray <nterray>
Fri 12 May 2017 12:09:25 PM UTC, comment #5: 

This is a little bit more serious than simply not extracting such strings - it sometimes breaks parsing of the JS file even if a template string is not used in connection with gettext. As an example, feeding xgettext this file:


window.location.href = `/${path}`;
_('hello');


will not extract "hello" (a normal string) or any subsequent _() calls simply because a template string with / in it happened before:


$ xgettext -o- x.js
x.js:3: warning: RegExp literal terminated too early
$


Vaclav Slavik <vslavik>
Wed 03 May 2017 08:38:23 PM UTC, comment #4: 

Ah, I see now, thanks for correcting me.

You mean to support the so-called "template strings" / "template literals":
http://stackoverflow.com/questions/27678052/what-is-the-usage-of-the-backtick-symbol-in-javascript
http://exploringjs.com/es6/ch_template-literals.html
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals
https://docs.microsoft.com/en-us/scripting/javascript/advanced/template-strings-javascript

Makes a lot of sense - with the limitation that the expressions inside braces should be limited to simple variable references.

Bruno Haible <haible>
Group administrator
Wed 03 May 2017 07:09:44 PM UTC, comment #3: 

(To be clear, the difference is the backticks)

Philip Chimento <ptomato>
Wed 03 May 2017 07:08:57 PM UTC, comment #2: 

You might be misunderstanding my motivation for filing this bug report. Believe me I know the problems of format strings getting mangled by translators no matter the programming language, so that's not what the bug report was about

However, Javascript's interpolated strings are not supported in xgettext at all, even if you use only simple variable names or no interpolation at all:

echo '_(`hello`)' | xgettext - --language=javascript -o -

I think xgettext should recognize _(`hello`) as a string marked for translation, even if it is bad practice to use complicated interpolation expressions in it.

Philip Chimento <ptomato>
Wed 03 May 2017 03:15:21 PM UTC, comment #1: 

I will close this as "Not a bug" and merely document that only simple variable names should be used in braces.

Reasons:
1) Translators are (in general) not programmers. You alienate them if you present them programming language expressions; even if all they have to do is to copy or preserve sich expressions.
2) A code refactoring (e.g. rename method 'toUpperCase()' to 'toUpper()') would change the string in the POT file, thus cause work to the translators.

Bruno Haible <haible>
Group administrator
Tue 02 May 2017 11:14:55 PM UTC, original submission:  

In the latest Javascript specification, the language has gained interpolated strings:

    `${greeting}, ${planet.toUpperCase()}!`

They are delineated with backticks, and behave much like interpolated strings in other various languages.

Many widely used Javascript implementations either already support these strings, or are commonly used with preprocessors that perform the required code transformation for implementations without the feature.

It would be nice if xgettext understood these strings.

Philip Chimento <ptomato>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by whiteinge (Posted a comment)
  • -email is unavailable- added by andyholmes (Posted a comment)
  • -email is unavailable- added by ueno (Posted a comment)
  • -email is unavailable- added by michaelgruber
  • -email is unavailable- added by trevinho (Updated the item)
  • -email is unavailable- added by piotrdrag
  • -email is unavailable- added by dmosberger (Posted a comment)
  • -email is unavailable- added by dmosberger (Voted in favor of this item)
  • -email is unavailable- added by brunoblookup (Posted a comment)
  • -email is unavailable- added by zumoshi (Voted in favor of this item)
  • -email is unavailable- added by vslavik (Voted in favor of this item)
  • -email is unavailable- added by erickwilder (Voted in favor of this item)
  • -email is unavailable- added by nterray (Voted in favor of this item)
  • -email is unavailable- added by nterray (Posted a comment)
  • -email is unavailable- added by weekens (Voted in favor of this item)
  • -email is unavailable- added by haible (Updated the item)
  • -email is unavailable- added by ptomato (Submitted the item)
  •  

    There are 405 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 25 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-04 haible StatusFixed Fix Released
    2019-09-25 haible StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2019-09-25 haible StatusFixed Confirmed
        Open/ClosedClosed Open
    2019-04-08 haible StatusNone Fixed
        Assigned toNone haible
        Open/ClosedOpen Closed
    2018-09-13 michaelgruber Carbon-Copy- Added michaelgruber
    2018-09-07 trevinho Attached File- Added 0001-javascript-add-support-for-ES6-template-strings.patch, #44949
        Attached File- Added 0002-javascript-do-not-translate-template-strings-wit-pla.patch, #44950
    2018-09-05 piotrdrag Carbon-Copy- Added piotrdrag
    2018-07-25 dmosberger Carbon-Copy- Added dmosberger
    2018-07-05 zumoshi Carbon-Copy- Added zumoshi
    2018-01-31 vslavik Carbon-Copy- Added vslavik
    2018-01-16 erickwilder Carbon-Copy- Added erickwilder
    2018-01-05 haible Severity3 - Normal 4 - Important
    2017-10-04 nterray Carbon-Copy- Added nterray
    2017-10-03 weekens Carbon-Copy- Added weekens
    2017-05-03 haible Severity1 - Wish 3 - Normal
        StatusNot a Bug None
        Assigned tohaible None
        SummaryJavascript interpolated strings in xgettext support Javascript template strings in xgettext
    2017-05-03 haible StatusNone Not a Bug
        Assigned toNone haible

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code