bugGNU gettext - Bugs: bug #50920, support Javascript template...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #50920: support Javascript template strings in xgettext

Submitted by:  Philip Chimento <ptomato>
Submitted on:  Tue 02 May 2017 11:14:55 PM UTC  
Votes:  405  
 
Category:  JavaScript Severity:  4 - Important
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  Bruno Haible <haible>
Open/Closed:  Closed

( Jump to the original submission)

Thu 09 May 2019 07:28:46 PM UTC, comment #14:

The fix is contained in the gettext-0.20 release.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Mon 08 Apr 2019 12:29:02 AM UTC, comment #13:

Marco,

Thanks for your patches. I did not use them because they don't provide the result that I'd like to have:

1) For
var s7 = _(tag`A template literal with a tag`);
it extracts an msgid "A template literal with a tag". I prefer not to extract that, because we don't know anything about what the tag function does.

2) For
var s10 = `abc${foo({},_('should be extracted'))}xyz`;
it does not extract an msgid "should be extracted". But the expression between ${ and the matching(!) closing brace is a normal expression; _() calls in it should be handled.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Mon 08 Apr 2019 12:11:53 AM UTC, comment #12:

Proper parsing of the template literals is implemented through
https://git.savannah.gnu.org/gitweb/?p=gettext.git;a=commitdiff;h=db8e4b76206ec179112b8171b92270e025c62a20

Note: This is just the parsing. If someone makes use of tagged template literals for internationalization, by defining a tag function that
1. converts the given string arguments to a single string with placeholders,
2. looks up the translation of that string from a message catalog,
3. inserts the provided values in place of the placeholders,
such as in
tag `OK to overwrite file ${source} with ${target}?`
-> "OK to overwrite file {0} with {1}?"
or "OK to overwrite file %s with %s?"
this will require extra support in xgettext. xgettext does not "see" the definition of the tag function, nor can it execute JavaScript.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun 07 Apr 2019 10:09:24 PM UTC, comment #11:

> I think it is not straightforward to determine the end of template strings, because substitutions may be nested


The grammar in http://www.ecma-international.org/ecma-262/6.0/#sec-template-literal is pretty straightforward. Yes, it looks like template strings can be nested, just like expressions can be nested by using parentheses.

> Maybe, it could be based on the backtick support in the shell-script scanner, which takes into account of nesting.


No, the backtick support in the shell-script scanner has additional complexities because a backslash inside `...` in shell scripts is different from a backslash outside `...`. Here, with the JavaScript template strings, it's not that hairy.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Thu 25 Oct 2018 05:03:03 AM UTC, comment #10:

Thank you for the patches, Marco. The general approach (discarding template strings with any substitutions) sounds reasonable.

However, I think it is not straightforward to determine the end of template strings, because substitutions may be nested; for example, this is a valid template string:

Maybe, it could be based on the backtick support in the shell-script scanner, which takes into account of nesting.

Daiki Ueno <ueno>
Project Administrator
Fri 07 Sep 2018 02:21:49 AM UTC, comment #9:

While I agree that supporting template strings with placeholder is wrong, I think is actually safe to support template strings when they only contain normal text.

In any case, these strings should be properly parsed, and we should not ignore them in case they contain a mix of apices.

For example a code like this:

A part of warn, it generates something like:

Attached here, two patches that fix this.
The 1st one, just add support for the backtick (`) strings, while the second one, discards them when they use javascript placeholders (so basically if there's a "${" inside; and that's enough to respect the JS syntax).

In case --extract-all is passed we warn about this misusage, however I'd love to do this in all the cases a template string is passed to a gettext flag (i.e., only if we do _(`${foo}`), and not if we've just a print(`${foo}`) call).

I've tried to do that, but unfortunately there are some xgettext limitations at core level that make this quite annoying to do only changing the javascript parser.

Ideally, I'd like to have a function that is called once the messageid is extracted, just before the format parsing, and that can be used to verify the content of the string, to check if this is legal or not.
I've tried to get the same with a literalstring_parser, but that's not doable because it doesn't handle nullified msgid's, and that's a too late stage.

Anyway, the two commits here should fix the main issue, in the mean time we don't decide to have something with a nicer feedback.

(file #44949, file #44950)

Marco Trevisan <trevinho>
Wed 25 Jul 2018 03:22:39 AM UTC, comment #8:

Likewise.

David Mosberger-Tang <dmosberger>
Wed 18 Jul 2018 12:34:56 PM UTC, comment #7:

Same issue here and it's a real PITA.

Bruno Desthuilliers <brunoblookup>
Wed 04 Oct 2017 03:05:10 PM UTC, comment #6:

Hello,

We get the same issue, it is quite painful if we want to use recent version of JavaScript.

How can we help to resolve this issue?

Regards,
Nicolas

Nicolas Terray <nterray>
Fri 12 May 2017 12:09:25 PM UTC, comment #5:

This is a little bit more serious than simply not extracting such strings - it sometimes breaks parsing of the JS file even if a template string is not used in connection with gettext. As an example, feeding xgettext this file:

will not extract "hello" (a normal string) or any subsequent _() calls simply because a template string with / in it happened before:

Vaclav Slavik <vslavik>
Wed 03 May 2017 08:38:23 PM UTC, comment #4:

Ah, I see now, thanks for correcting me.

You mean to support the so-called "template strings" / "template literals":
http://stackoverflow.com/questions/27678052/what-is-the-usage-of-the-backtick-symbol-in-javascript
http://exploringjs.com/es6/ch_template-literals.html
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals
https://docs.microsoft.com/en-us/scripting/javascript/advanced/template-strings-javascript

Makes a lot of sense - with the limitation that the expressions inside braces should be limited to simple variable references.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Wed 03 May 2017 07:09:44 PM UTC, comment #3:

(To be clear, the difference is the backticks)

Philip Chimento <ptomato>
Wed 03 May 2017 07:08:57 PM UTC, comment #2:

You might be misunderstanding my motivation for filing this bug report. Believe me I know the problems of format strings getting mangled by translators no matter the programming language, so that's not what the bug report was about

However, Javascript's interpolated strings are not supported in xgettext at all, even if you use only simple variable names or no interpolation at all:

echo '_(`hello`)' | xgettext - --language=javascript -o -

I think xgettext should recognize _(`hello`) as a string marked for translation, even if it is bad practice to use complicated interpolation expressions in it.

Philip Chimento <ptomato>
Wed 03 May 2017 03:15:21 PM UTC, comment #1:

I will close this as "Not a bug" and merely document that only simple variable names should be used in braces.

Reasons:
1) Translators are (in general) not programmers. You alienate them if you present them programming language expressions; even if all they have to do is to copy or preserve sich expressions.
2) A code refactoring (e.g. rename method 'toUpperCase()' to 'toUpper()') would change the string in the POT file, thus cause work to the translators.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Tue 02 May 2017 11:14:55 PM UTC, original submission:

In the latest Javascript specification, the language has gained interpolated strings:

`${greeting}, ${planet.toUpperCase()}!`

They are delineated with backticks, and behave much like interpolated strings in other various languages.

Many widely used Javascript implementations either already support these strings, or are commonly used with preprocessors that perform the required code transformation for implementations without the feature.

It would be nice if xgettext understood these strings.

Philip Chimento <ptomato>

 

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ueno (Posted a comment)
  • -email is unavailable- added by michaelgruber
  • -email is unavailable- added by trevinho (Updated the item)
  • -email is unavailable- added by piotrdrag
  • -email is unavailable- added by dmosberger (Posted a comment)
  • -email is unavailable- added by dmosberger (Voted in favor of this item)
  • -email is unavailable- added by brunoblookup (Posted a comment)
  • -email is unavailable- added by zumoshi (Voted in favor of this item)
  • -email is unavailable- added by vslavik (Voted in favor of this item)
  • -email is unavailable- added by erickwilder (Voted in favor of this item)
  • -email is unavailable- added by nterray (Voted in favor of this item)
  • -email is unavailable- added by nterray (Posted a comment)
  • -email is unavailable- added by weekens (Voted in favor of this item)
  • -email is unavailable- added by haible (Updated the item)
  • -email is unavailable- added by ptomato (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 405 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 21 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-04-08 haible StatusNone => Fixed
        Assigned toNone => haible
        Open/ClosedOpen => Closed
    2018-09-13 michaelgruber Carbon-Copy- => Added michaelgruber
    2018-09-07 trevinho Attached File- => Added 0001-javascript-add-support-for-ES6-template-strings.patch, #44949
        Attached File- => Added 0002-javascript-do-not-translate-template-strings-wit-pla.patch, #44950
    2018-09-05 piotrdrag Carbon-Copy- => Added piotrdrag
    2018-07-25 dmosberger Carbon-Copy- => Added dmosberger
    2018-07-05 zumoshi Carbon-Copy- => Added zumoshi
    2018-01-31 vslavik Carbon-Copy- => Added vslavik
    2018-01-16 erickwilder Carbon-Copy- => Added erickwilder
    2018-01-05 haible Severity3 - Normal => 4 - Important
    2017-10-04 nterray Carbon-Copy- => Added nterray
    2017-10-03 weekens Carbon-Copy- => Added weekens
    2017-05-03 haible Severity1 - Wish => 3 - Normal
        StatusNot a Bug => None
        Assigned tohaible => None
        SummaryJavascript interpolated strings in xgettext => support Javascript template strings in xgettext
    2017-05-03 haible StatusNone => Not a Bug
        Assigned toNone => haible
    2017-05-03 haible Severity3 - Normal => 1 - Wish

    Back to the top


    Powered by Savane 3.4