bugGNU roff - Bugs: bug #59814, improve internationalization at...

 
 

bug #59814: improve internationalization at startup

Submitter:  G. Branden Robinson <gbranden>
Submitted:  Mon 04 Jan 2021 07:08:57 AM UTC
   
 
Category:  General Severity:  1 - Wish
Item Group:  Feature change Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.23.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 05 Feb 2023 08:42:22 AM UTC, comment #17: 

I wrote this, so dropping patch annotation.

And recategorizing; this touched multiple parts of the tree.

 G. Branden Robinson <gbranden>
Group administrator
Sat 10 Jul 2021 03:14:57 PM UTC, comment #16: 

Consequent to review, this feature got tweaked.


commit 5b5307bf34277279c292599770c7241ebc6e1af6
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jul 9 04:29:01 2021 +1000

    [docs]: Update to reflect reversion of 12434c13.

    * ChangeLog: Remove no longer applicable chunks of 2021-04-15 entry
      regarding Savannah #59814.

    * NEWS: Revise item regarding presumed input language.

    * doc/groff.texi (Manipulating Hyphenation): As previous item, and
      remove LC_ALL and LANG entries from environment variable index.

    * man/groff_diff.7.man (Language/Extended requests) <hpf>: As NEWS item.
    * man/groff_tmac.5.man (Localization packages): As NEWS item.  Also
      undocument unusability of localization files with compatibility mode;
      the resolution of Savannah #60874 invalidated this caveat.

commit 7eb25c45b5ec67f1037abcc670793b734584987c
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jul 9 03:51:38 2021 +1000

    tmac/troffrc: Load English localization file.

    * tmac/troffrc: Load the English localization file `en.tmac` by default,
      instead of dealing with hyphenation language and patterns directly in
      this file.  (This also replaces a localization-determination mechanism
      that was never part of a groff release.)

commit 3e9250fea4ab3e698cf7c2f903eaa4ed0fabf5c0
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jul 9 03:40:04 2021 +1000

    [tests]: Test localization with `-m` option.

    * src/roff/groff/tests/initialization_is_quiet.sh:
    * src/roff/groff/tests/localization_works.sh: Rewrite test to use
      explicit macro file loading localization mechanism.

commit 5f8bfdf6ded6570989466ce20fd605c36c27dd3c
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jul 9 03:27:37 2021 +1000

    Revert "tmac/troffrc: Derive groff locale from system."

    This reverts commit 12434c13bec939a68e543e6aae758bc7e92c3fb0.

    See:
      https://lists.gnu.org/archive/html/groff/2021-07/msg00013.html
    and follow-ups.

    Thanks to Ingo Schwarze, John Ankarström, Dave Kemper, and James K.
    Lowden for the discussion.

    WARNING: This commit fails a localization test.


 G. Branden Robinson <gbranden>
Group administrator
Thu 15 Apr 2021 10:42:25 AM UTC, comment #15: 


commit 11195e2e4a23bbd8f461892a655a0fa7ab754985 (HEAD -> master, origin/master, origin/HEAD)
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Thu Apr 15 19:46:36 2021 +1000

    {ChangeLog, NEWS}: Report localization change.

    Fixes <https://savannah.gnu.org/bugs/?59814>.

commit a22e2a44333e1456700d531a67b378c721b961c8
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Tue Apr 13 18:26:27 2021 +1000

    [tmac]: Adjust editor file encoding hints.

    * tmac/cs.tmac: Tell Vim the file is "iso-8859-2" (ISO Latin-2).
    * tmac/en.tmac: Stop telling Emacs the file is "latin-1"; it's ASCII.
    * tmac/fr.tmac: Tell Vim the file is "iso-8859-15" (ISO Latin-9).

    Vim users may need to use ":e ++enc=iso-8859-2", for instance, to see
    correct glyphs.

commit b3ed81fe4b8446604e081d47a67125d6abf07379
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Tue Apr 13 18:25:55 2021 +1000

    tmac/LOCALIZATION: Rewrite.

commit 2a047db626b1cf683d440fefe048341a1fef62e5
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Tue Apr 13 15:35:51 2021 +1000

    Test localization refactor.

    * src/roff/groff/tests/localization_works.sh: Add 10 test cases.
    * src/roff/groff/groff.am (groff_TESTS): Run test.

commit 8cbe35b5b6562eb332ee025a9a96ff78fb30f266
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jan 15 03:38:11 2021 +1100

    [docs]: Update hyphenation and localization stuff.

    * doc/groff.texi (Manipulating Hyphenation):
    * man/groff.7.man (Hyphenation):
    * man/groff_diff.7.man (Implementation differences):
      - Refer to "U.S. English" hyphenation patterns as simply "English";
        they will be mostly correct for Commonwealth English as well, and no
        alternative English hyphenation patterns for other territories are
        available.

    * doc/groff.texi (Manipulating Hyphenation):
    * man/groff_diff.7.man (New requests):
      - Note that default hyphenation mode depends on the language used on
        the system.
      - Add concept index entry for localization.
      - Add file index entries for the locale macro files (cs.tmac, etc.).
      - Add environment variable index entries for LANG and LC_ALL.
      - Describe how groff's idea of the locale is determined.
      - Update to reflect rename of English hyphenation patterns and
        .hla identifier from "us" to "en".

commit a60784b82a5c53caff5443fc036b8d13f4084a32
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jan 15 03:27:15 2021 +1100

    tmac/en.tmac: Set hyphenation mode to 4.

commit 12434c13bec939a68e543e6aae758bc7e92c3fb0
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:53:05 2021 +1100

    tmac/troffrc: Derive groff locale from system.

    Determine the groff locale (default input language) using the system
    locale.  Use the environment if possible.  Try LC_ALL first, then LANG.
    "C" means English (en).  Otherwise, only the first two characters of the
    locale name are used.

    Unrecognized locales (those without a supporting xx.tmac file) are
    ignored, and groff falls back to English.

    Those who want groff's default locale to differ from LC_ALL/LANG should
    edit this troffrc to source the appropriate groff locale macro file
    (cs.tmac, de.tmac, den.tmac, fr.tmac, ja.tmac, sv.tmac, zh.tmac).

commit a86d9251ed05cec18f6279a9e613449ae7aa7315
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:44:57 2021 +1100

    tmac/en.tmac: Add English localization file.

commit 1d5a04b6ec08ecfe9e772026a8ce2742e4b0f29f
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:10:20 2021 +1100

    [troffrc]: Change hyphenation language us to en.

commit 4c5a4b92bee4f704175423eca13d3dbfceab90a4
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 16:59:03 2021 +1100

    [tmac]: Rename *.us to *.en.

    The first-order determinant of hyphenation points is language, not
    territory.  Use ISO 639 2-letter language codes for hyphenation and
    exception patterns instead of ISO 3166 2-letter territory codes.


 G. Branden Robinson <gbranden>
Group administrator
Fri 09 Apr 2021 04:09:54 AM UTC, comment #14: 

Bug #59973 is now resolved so I'm returning to this.

 G. Branden Robinson <gbranden>
Group administrator
Thu 11 Feb 2021 05:59:39 AM UTC, comment #13: 

Looks good to me, aside from one grammar quibble in the docs.

The attached file contains the changes implementing the .*soquiet requests (bug #59973) as well as the changes for this bug, although the latter changes use the bare .mso (with a comment that the commit could wait for .msoquiet's availability).  I presume this will all get sorted before the actual commits (especially as comment #7 shows the current bug's changes broken into multiple logical pieces).

The grammar quibble, in text that appears in doc/groff.texi and man/groff_diff.7.man, is subtle and actually easiest to see if the sentence is laid out with programming-style indentation:

If
  the environment variables are
    not set,
    set to "C", or
    a localization file for the locale does not exist,
the English localization file is used.

The "A, B, or C" clause, being a single series, must all grammatically fit into the same part of the sentence--that is, all be on the same nesting level.  But here, A and B grammatically follow the second nesting level and C the first.  I'd probably rework it as

If
  the environment variables are
    not set or
    set to "C",
  or a localization file for the locale does not exist,
the English localization file is used.

 Dave <barx>
Group Member
Wed 10 Feb 2021 06:44:46 PM UTC, comment #12: 


comment #11:

> Sorry for my lack of clarity: it was the implementation I was looking for, and failed to find.  The most recent attachment is that of comment #3, whereas comment #7 describes changes beyond this.
>
> The idea and design as described sound solid.


Ah, right.  Thanks, Dave.  New attachment uploaded.  This is just a "git diff origin/master" against my branch.

(file #50850)

 G. Branden Robinson <gbranden>
Group administrator
Wed 10 Feb 2021 03:58:43 PM UTC, comment #11: 

Sorry for my lack of clarity: it was the implementation I was looking for, and failed to find.  The most recent attachment is that of comment #3, whereas comment #7 describes changes beyond this.

The idea and design as described sound solid.

 Dave <barx>
Group Member
Wed 10 Feb 2021 07:50:52 AM UTC, comment #10: 


comment #9:

> comment #7:
> > Feedback still desired.
>
> On what, specifically?  The commits listed in this comment do not seem to have been applied, and this bug report's latest patch is the one attached to comment #3.


On whether people think:

(1) It's a good idea;
(2) It's a sound design;
(3) It's implemented reasonably.

G. Branden Robinson <gbranden>
Group administrator
Fri 05 Feb 2021 02:28:51 AM UTC, comment #9: 

comment #7:

> Feedback still desired.


On what, specifically?  The commits listed in this comment do not seem to have been applied, and this bug report's latest patch is the one attached to comment #3.

 Dave <barx>
Group Member
Sat 30 Jan 2021 06:43:18 AM UTC, comment #8: 

See bug #59973 for .msoquiet request.

 G. Branden Robinson <gbranden>
Group administrator
Sat 30 Jan 2021 05:44:29 AM UTC, comment #7: 

I think this is as done as I can get it without .soquiet and .msoquiet.

Feedback still desired.



commit 0dd4a43feff2cf8160020e19947c56f3f68498e8 (HEAD -> language-en)
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jan 15 03:38:11 2021 +1100

    [docs]: Update hyphenation and localization stuff.

    * doc/groff.texi (Manipulating Hyphenation):
    * man/groff.7.man (Hyphenation):
    * man/groff_diff.7.man (Implementation differences):
      - Refer to "U.S. English" hyphenation patterns as simply "English";
        they will be mostly correct for Commonwealth English as well, and no
        alternative English hyphenation patterns for other territories are
        available.

    * doc/groff.texi (Manipulating Hyphenation):
    * man/groff_diff.7.man (New requests):
      - Note that default hyphenation mode depends on the language used on
        the system.
      - Add concept index entry for localization.
      - Add file index entries for the locale macro files (cs.tmac, etc.).
      - Add environment variable index entries for LANG and LC_ALL.
      - Describe how groff's idea of the locale is determined.
      - Update to reflect rename of English hyphenation patterns and
        .hla identifier from "us" to "en".

commit 46ba7b2140eec076daee73f8daef20f16e9c2536
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jan 15 03:27:15 2021 +1100

    tmac/en.tmac: Set hyphenation mode to 4.

commit 252f7f8a0aa1f2435b0bc88371faf17f913be533
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:53:05 2021 +1100

    tmac/troffrc: Derive groff locale from system.

    Determine the groff locale (default input language) using the system
    locale.  Use the environment if possible.  Try LC_ALL first, then LANG.
    "C" means English (en).  Otherwise, only the first two characters of the
    locale name are used.

    Unrecognized locales (those without a supporting xx.tmac file) are
    ignored, and groff falls back to English.

    Those who want groff's default locale to differ from LC_ALL/LANG should
    edit this troffrc to source the appropriate groff locale macro file
    (cs.tmac, de.tmac, den.tmac, fr.tmac, ja.tmac, sv.tmac, zh.tmac).

commit 0d4bb5ac2344e09e43e24ac9aed9afac8c5df539
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:44:57 2021 +1100

    tmac/en.tmac: Add English localization file.

commit e148fb564d70214e6e5ab9f9efca0d85c161b55e
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:10:20 2021 +1100

    [troffrc]: Change hyphenation language us to en.

    The first-order determinant of hyphenation points is language, not
    territory.  Use ISO 639 2-letter language codes for hyphenation and
    exception patterns instead of ISO 3166 2-letter territory codes.

commit 81c38680f2bcb0debbfe6a4a135c5e1fb8b1d39f
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 16:59:03 2021 +1100

    tmac: Rename *.us to *.en.

    The first-order determinant of hyphenation points is language, not
    territory.  Use ISO 639 2-letter language codes for hyphenation and
    exception patterns instead of ISO 3166 2-letter territory codes.


 G. Branden Robinson <gbranden>
Group administrator
Sun 17 Jan 2021 06:45:07 AM UTC, comment #6: 

[myself in comment #4 comment #4:]

> This, I'm dubious about.  It would make it impossible to switch languages mid-document and get appropriate hyphenation for the other language(s).


My objection was overstated--I hadn't looked at the patch yet (shame on me).  It safely aliased the internal implementation of the request, which is otherwise irretrievably discarded on redefinition.

However, having to do a dance where you call the "I really mean it" aliased version of the request is also pretty squicky.

comment #5:

> A fair point.  This code was really just a stopgap anyway until the root of bug #57556 (reading the relevant lefthyphenmin and righthyphenmin from the pattern files) can be addressed.


Yes.  I've updated #57556 with some concrete info about what exactly needs to be supported.

 G. Branden Robinson <gbranden>
Group administrator
Fri 15 Jan 2021 04:15:27 AM UTC, comment #5: 

comment #4:

> This, I'm dubious about.  It would make it impossible to switch languages
> mid-document and get appropriate hyphenation for the other language(s).


A fair point.  This code was really just a stopgap anyway until the root of bug #57556 (reading the relevant lefthyphenmin and righthyphenmin from the pattern files) can be addressed.

 Dave <barx>
Group Member
Fri 15 Jan 2021 03:24:54 AM UTC, comment #4: 


comment #2:

> As suggested in bug #53413 comment #13, the new tmac/en.tmac could also call ".hy 4", as the English patterns require.


I've done this.

> tmac/en.tmac could go one step further and incorporate the patch attached to bug #57556.  (The patch as submitted adds the code to tmac/troffrc, but that's no longer the right place under the current bug's new framework.)  The code in this patch sets an initial .hy value and then redefines the .hy request to prevent the user from later calling it with a value incompatible with the English patterns.


This, I'm dubious about.  It would make it impossible to switch languages mid-document and get appropriate hyphenation for the other language(s).

 G. Branden Robinson <gbranden>
Group administrator
Fri 15 Jan 2021 03:21:06 AM UTC, comment #3: 

Updated version.


commit db993c17e02b3924eca88d112c3f0342f1ce7a5d (HEAD -> language-en)
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jan 15 03:38:11 2021 +1100

    [docs]: Update hyphenation and localization stuff.

commit ea58825c604ab677c55fab4476bc6f0b8abfc777
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jan 15 03:27:15 2021 +1100

    tmac/en.tmac: Set hyphenation mode to 4.

commit a704c76b9b909b832f37a47933dae664c5f2e76a
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:53:05 2021 +1100

    tmac/troffrc: Derive groff locale from system.

    Determine the groff locale (default input language) using the system
    locale.  Use the environment if possible.  Try LC_ALL first, then LANG.
    "C" means English (en).  Otherwise, only the first two characters of the
    locale name are used.

    Unrecognized locales (those without a supporting xx.tmac file) are
    ignored, and groff falls back to English.

    Those who want groff's default locale to differ from LC_ALL/LANG should
    edit this troffrc to source the appropriate groff locale macro file
    (cs.tmac, de.tmac, den.tmac, fr.tmac, ja.tmac, sv.tmac, zh.tmac).

commit 55af800b24976861e76cd1f7bee640d50f790ebd
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:44:57 2021 +1100

    tmac/en.tmac: Add English localization file.

commit ad9f00c55d122a8fe44c576798f1aa0abb17e545
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:10:20 2021 +1100

    [troffrc]: Change hyphenation language us to en.

    The first-order determinant of hyphenation points is language, not
    territory.  Use ISO 639 2-letter language codes for hyphenation and
    exception patterns instead of ISO 3166 2-letter territory codes.

commit 7e4a5de02eebc36fb1b590b9d24aaf5d5d8eecaf
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 16:59:03 2021 +1100

    tmac: Rename *.us to *.en.

    The first-order determinant of hyphenation points is language, not
    territory.  Use ISO 639 2-letter language codes for hyphenation and
    exception patterns instead of ISO 3166 2-letter territory codes.


(file #50722)

 G. Branden Robinson <gbranden>
Group administrator
Thu 14 Jan 2021 10:37:27 AM UTC, comment #2: 

As suggested in bug #53413 comment #13, the new tmac/en.tmac could also call ".hy 4", as the English patterns require.

tmac/en.tmac could go one step further and incorporate the patch attached to bug #57556.  (The patch as submitted adds the code to tmac/troffrc, but that's no longer the right place under the current bug's new framework.)  The code in this patch sets an initial .hy value and then redefines the .hy request to prevent the user from later calling it with a value incompatible with the English patterns.

 Dave <barx>
Group Member
Wed 13 Jan 2021 08:47:24 AM UTC, comment #1: 

[said mailing-list thread: http://lists.gnu.org/archive/html/groff/2021-01/msg00007.html]

 Dave <barx>
Group Member
Mon 04 Jan 2021 07:08:57 AM UTC, original submission:  

Savannah #53413 reminded me that I had been wanting to do this.

I'll start a topic on the mailing list to solicit feedback.


commit 4eb1e209c8ebac3c1c62b7a51ce031ed7a6baffb (HEAD -> language-en)
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:53:05 2021 +1100

    tmac/troffrc: Derive groff locale from system.

    Determine the groff locale (default input language) using the system
    locale.  Determine the groff locale from the environment if possible.
    Try LC_ALL first, then LANG.  "C" means English (en).

    Unrecognized locales are ignored, and groff falls back to English.

    Those who want groff's default locale to differ from LC_ALL/LANG should
    edit this troffrc to source the appropriate groff locale macro file
    (cs.tmac, de.tmac, den.tmac, fr.tmac, ja.tmac, sv.tmac, zh.tmac).

    Also wrap a long line.

commit 306296344b2910f20579baa6db6eef1a8a4a3e0f
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:44:57 2021 +1100

    tmac/en.tmac: Add English localization file.

commit e24e939d3518b069b034d9fcd1c25ea97ddc771f
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 17:10:20 2021 +1100

    [troffrc]: Change hyphenation language us to en.

    The first-order determinant of hyphenation points is language, not
    territory.  Use ISO 639 2-letter language codes for hyphenation and
    exception patterns instead of ISO 3166 2-letter territory codes.

commit fde7035d02490c1c055c01718a6645f8b225ed5e
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 4 16:59:03 2021 +1100

    tmac: Rename *.us to *.en.

    The first-order determinant of hyphenation points is language, not
    territory.  Use ISO 639 2-letter language codes for hyphenation and
    exception patterns instead of ISO 3166 2-letter territory codes.


 G. Branden Robinson <gbranden>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50850:  groff-i18n-2021-02-10.diff added by gbranden (18KiB - text/x-patch)
file #50722:  groff-i18n-2.diff added by gbranden (13KiB - text/x-patch)
file #50640:  groff-i18n.diff added by gbranden (6KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by gbranden (Submitted the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-02-05 gbranden CategoryCore General
        Summary[PATCH] improve internationalization at startup improve internationalization at startup
    2021-04-15 gbranden StatusIn Progress Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.23.0
    2021-04-09 gbranden StatusNeed Info In Progress
    2021-02-10 gbranden Attached File- Added groff-i18n-2021-02-10.diff, #50850
    2021-01-15 gbranden Attached File- Added groff-i18n-2.diff, #50722
    2021-01-04 gbranden Attached File- Added groff-i18n.diff, #50640

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-f13d.
    Corresponding source code