bugGNU roff - Bugs: bug #57594, sync hyphenation pattern files...

 
 

bug #57594: sync hyphenation pattern files with TeX hyph-utf8 project and iconv them

Submitter:  Dave <barx>
Submitted:  Tue 14 Jan 2020 01:40:53 AM UTC
   
 
Category:  Macro - others/general Severity:  4 - Important
Item Group:  Feature change Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 30 Jul 2021 06:57:31 AM UTC, comment #14: 

comment #12:

> I suppose comment #4 needs to become part of the larger UTF-8 effort,


Said larger effort is probably under bug #40720's umbrella, though its Summary and description are (intentionally) fairly vague.

Dave <barx>
Group Member
Wed 28 Jul 2021 05:53:35 AM UTC, comment #13: 

Really changing Severity and Status now.  Sometimes Savannah throws away these things.  :-/

G. Branden Robinson <gbranden>
Group administrator
Wed 28 Jul 2021 05:52:39 AM UTC, comment #12: 

comment #11:

> comment #9:
> > Yes.  If the hyphenation pattern files contain non-ASCII
> > characters, the localization file loads latin{1,2,9}.tmac as
> > appropriate before invoking `hpf` to load the pattern file.
>
> Great!
>
> > This mechanism is a bit of a dead end because the TeX hyph-utf8
> > project has migrated to UTF-8, and decoding of multibyte
> > character sequences is far beyond the .trin request's capabilities.
>
> Right, but I asked this in the context of my proposal in comment #7; that is, that in the groff tree we could iconv each UTF-8 TeX file to an appropriate groff-digestible format before git committing it, at least until the more ambitious solution outlined in comment #4 comes to pass.  This would not be the prettiest solution, but it would be trivial to script.


Yes.  This seems like a tractable thing to do for groff 1.23.0.  Un-postponing and goosing the severity.

I suppose comment #4 needs to become part of the larger UTF-8 effort, however long that takes.

G. Branden Robinson <gbranden>
Group administrator
Wed 28 Jul 2021 05:25:01 AM UTC, comment #11: 

comment #9:

> Yes.  If the hyphenation pattern files contain non-ASCII
> characters, the localization file loads latin{1,2,9}.tmac as
> appropriate before invoking `hpf` to load the pattern file.


Great!

> This mechanism is a bit of a dead end because the TeX hyph-utf8
> project has migrated to UTF-8, and decoding of multibyte
> character sequences is far beyond the .trin request's capabilities.


Right, but I asked this in the context of my proposal in comment #7; that is, that in the groff tree we could iconv each UTF-8 TeX file to an appropriate groff-digestible format before git committing it, at least until the more ambitious solution outlined in comment #4 comes to pass.  This would not be the prettiest solution, but it would be trivial to script.

Dave <barx>
Group Member
Wed 21 Jul 2021 03:00:34 AM UTC, comment #10: 

Hello again, Dave!

Sorry I missed this comment.  I dimly recall meaning to reply to it.

comment #6:

> Commit b2284ab0 (referenced in comment #4) removed the groff-specific "Local Variables" section from the end of tmac/hyphen.en (or, rather, it made tmac/hyphen.en exactly match its tex-hyphen counterpart (hyph-utf8/tex/generic/hyph-utf8/patterns/tex/hyph-en-us.tex), which lacks this footer).  Is this intentional?


Yes--when I committed it I had hopes that we'd never have to edit the file.

G. Branden Robinson <gbranden>
Group administrator
Wed 21 Jul 2021 02:57:47 AM UTC, comment #9: 

Hi, Dave!

comment #8:

> comment #7:
> > Does groff read pattern files in Latin-1 encoding?
>
> Answering my own question: it must read not only Latin-1, but other members of the ISO 8859 family as well, since tmac/hyphen.den and tmac/hyphen.det are already in Latin-1 encoding, tmac/hyphen.cs is in Latin-2, and tmac/hyphen.fr is in Latin-9.


Yes.  If the hyphenation pattern files contain non-ASCII characters, the localization file loads latin{1,2,9}.tmac as appropriate before invoking `hpf` to load the pattern file.

These latin*.tmac files are each a long set of .trin requests to map the printing characters from the upper half of ISO 8859 to the correct groff special character escape sequences.

In principle, possibly the localization files should clear these translations after reading the pattern files.

In practice, it doesn't seem to matter too much, since it seems likely that users of the relevant locale will either already be using the corresponding input character set (so the mappings won't be wrong for input characters in the document), or they'll be using UTF-8, and will have run their input through preconv(1) already which will bust it down to ASCII and special character escapes.

If I had to solve the "in principle" problem, I'd change the character-set tmac files to move the existing .trin requests into a macro named something like "load-charset", add another macro "unload-charset" that undoes the .trin requests, and have the localization files call these macros in appropriate places.

This mechanism is a bit of a dead end because the TeX hyph-utf8 project has migrated to UTF-8, and decoding of multibyte character sequences is far beyond the .trin request's capabilities.

Regards,
Branden

G. Branden Robinson <gbranden>
Group administrator
Wed 21 Jul 2021 01:33:53 AM UTC, comment #8: 

comment #7:

> Does groff read pattern files in Latin-1 encoding?


Answering my own question: it must read not only Latin-1, but other members of the ISO 8859 family as well, since tmac/hyphen.den and tmac/hyphen.det are already in Latin-1 encoding, tmac/hyphen.cs is in Latin-2, and tmac/hyphen.fr is in Latin-9.

Dave <barx>
Group Member
Sat 10 Jul 2021 08:19:32 PM UTC, comment #7: 

comment #4:

> To do more is going to require adding support for UTF-8 input
> in the hyphenation pattern files, as that's what the TeX
> hyph-utf8 project uses for Czech, French, German, and Swedish.


Does groff read pattern files in Latin-1 encoding?  It seems like it ought to, since it reads its input stream as Latin-1.

This wouldn't help for the Czech and French pattern files, which use characters that have no Latin-1 representation.  But the Swedish (hyph-utf8/tex/generic/hyph-utf8/patterns/tex/hyph-sv.tex) and German (hyph-utf8/tex/generic/hyph-utf8/patterns/tex/hyph-de-1996.tex) files can be converted losslessly to Latin-1.  (Lossless in terms of data content, anyway.  The German file uses directional quotation marks in its comments, but these can become ASCII straight quotes with no functional consequences.)

Obviously, this doesn't solve the general problem, but the general solution may be some time away, while running the transliterable files through iconv before commiting them to the groff tree is a quick and easy way to make those files usable.

Dave <barx>
Group Member
Sat 10 Jul 2021 06:32:14 PM UTC, comment #6: 

Commit b2284ab0 (referenced in comment #4) removed the groff-specific "Local Variables" section from the end of tmac/hyphen.en (or, rather, it made tmac/hyphen.en exactly match its tex-hyphen counterpart (hyph-utf8/tex/generic/hyph-utf8/patterns/tex/hyph-en-us.tex), which lacks this footer).  Is this intentional?

Dave <barx>
Group Member
Sat 10 Jul 2021 03:12:43 PM UTC, comment #5: 


comment #3:

> We should also write script(s) to verify the groff-cleanliness of the hyphenation files.  This can easily take the form of one or more regression tests like the 48 others we have.  The most obvious thing I can think of to do is to have one test file for each supported groff locale which then calls .hla (or uses my in-development internationalization patch, bug #59814).  If my understanding is correct, an otherwise empty groff document will suffice to provoke diagnostics from .hla if the hyphenation or exception files contain unsupported constructs.  See src/roff/troff/env.cpp:hyphen_trie::hpf_getc and src/roff/troff/env.cpp:hyphen_trie::read_patterns_file.


The foregoing is done.


commit 3e9250fea4ab3e698cf7c2f903eaa4ed0fabf5c0
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Fri Jul 9 03:40:04 2021 +1000

    [tests]: Test localization with `-m` option.

    * src/roff/groff/tests/initialization_is_quiet.sh:
    * src/roff/groff/tests/localization_works.sh: Rewrite test to use
      explicit macro file loading localization mechanism.


I haven't updated FOR-RELEASE yet because we're blocked on UTF-8 input handling as noted in my previous comment, and the process is not streamlined enough to become robotic.  I had to manually do some tomfoolery to deal with our English hyphenation exceptions file (on the bright side, it's much shorter now).

G. Branden Robinson <gbranden>
Group administrator
Sat 10 Jul 2021 03:08:17 PM UTC, comment #4: 

I've carried this as far as I can for the moment.

To do more is going to require adding support for UTF-8 input in the hyphenation pattern files, as that's what the TeX hyph-utf8 project uses for Czech, French, German, and Swedish.

We got lucky with English and Italian, which use ASCII.

On the bright side, finding or writing a UTF-8 input parser we can use for reading hyphenation pattern files would, unless we're very unlucky, go a long way toward equipping us to handle UTF-8 generally.  If we stick the routines in libgroff, several pieces of the GNU roff system can use them.  The big lift for troff itself is going to be moving the stuff in src/roff/troff/input.h to Unicode-safe code points, something in the Private Use Area, I reckon.

Dropping the severity because we've done what we can for now, I think.


commit b2284ab01d2d87507f3bcbd7de2a081efb6528a6
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sun Jul 11 00:50:27 2021 +1000

    Update English hyphenation patterns.

    * NEWS: Add item.
    * tmac/hyphen.en: Update file using `hyph-en-us.tex` patterns file from
      the TeX hyph-utf8 project.
    * tmac/hyphenex.en: Remove explicit hyphenations for words that no
      longer require them when using the new patterns.  Add one item scraped
      from an erratum comment in hyphen.en ("dem-o-crat").

    The new patterns likely _will_ change the automatic hyphenation break
    points of your English documents.  Here is a sample of affected words
    found within groff's own documentary corpus.

    OLD                     NEW
    ===                     ===
    ar‐range‐ment           arrange‐ment
    col‐umns                columns
    con‐struc‐ted           con‐structed
    cus‐tom‐ized            cus‐tomized
    def‐i‐ni‐tions          de‐f‐i‐n‐i‐tions
    der‐i‐va‐tions          de‐riva‐tions
    hy‐phen‐a‐tion          hy‐phen‐ation
    ma‐te‐rial              ma‐te‐r‐ial
    Mi‐cro‐soft             Mi‐crosoft
    pipe‐lines              pipelines
    post‐pro‐ces‐sors       post‐proces‐sors
    pro‐cessed              processed
    pro‐cesses              processes
    spa‐ces                 spaces
    Wer‐ner                 Werner

    Partially addresses <https://savannah.gnu.org/bugs/?57594>.


G. Branden Robinson <gbranden>
Group administrator
Thu 14 Jan 2021 02:53:00 PM UTC, comment #3: 

If this should go into the release checklist then we should add it there, in FOR-RELEASE.

We should also write script(s) to verify the groff-cleanliness of the hyphenation files.  This can easily take the form of one or more regression tests like the 48 others we have.  The most obvious thing I can think of to do is to have one test file for each supported groff locale which then calls .hla (or uses my in-development internationalization patch, bug #59814).  If my understanding is correct, an otherwise empty groff document will suffice to provoke diagnostics from .hla if the hyphenation or exception files contain unsupported constructs.  See src/roff/troff/env.cpp:hyphen_trie::hpf_getc and src/roff/troff/env.cpp:hyphen_trie::read_patterns_file.

Once both of these are done, this ticket can be resolved.

In the meantime, it's a blocker.

G. Branden Robinson <gbranden>
Group administrator
Mon 12 Oct 2020 10:30:46 AM UTC, comment #2: 

With 'massaging' I mean to remove all the TeX stuff that can't be digested by groff.

Werner LEMBERG <wl>
Group administrator
Mon 12 Oct 2020 09:55:41 AM UTC, comment #1: 

original submission:

> If groff has some sort of "release checklist,"


It does now!  Thanks, commit 0907598d.

Since I don't know what Werner means by "the necessary massaging for groff," I don't think I'm qualified to write this item of the list, however.

Dave <barx>
Group Member
Tue 14 Jan 2020 01:40:53 AM UTC, original submission:  

Groff's hyphenation files are copied from their TeX counterparts.  About a year ago, Werner Lemberg wrote on the groff email list (http://lists.gnu.org/archive/html/groff/2018-11/msg00071.html):

    "Please update all pattern files from the data in the `tex-hyphen' repository, for example,

    https://github.com/hyphenation/tex-hyphen/blob/master/hyph-utf8/tex/generic/hyph-utf8/patterns/tex/hyph-sv.tex

    (and applying the necessary massaging for groff)."

There appears to be no systematic way to refresh these groff files when updates are made on the TeX side.  If groff has some sort of "release checklist," perhaps making appropriate updates to these pattern files could be added to that.

(These files would have to be updated anyway as part of the fix for bug #57556, but I am filing it as a separate bug because fixing 57556 will be significantly more work and thus may get deferred, whereas this is probably straightforward.  But also, 57556 is a one-time fix, whereas this bug includes an ongoing synchronization aspect, so it's worth tracking separately.)

Dave <barx>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by wl (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-08-19 barx Dependencies- bugs #57556 is dependent
    2022-01-11 gbranden Item GroupNone Feature change
    2021-07-28 gbranden Severity3 - Normal 4 - Important
        StatusPostponed None
    2021-07-28 gbranden Summarysync hyphenation pattern files with TeX versions sync hyphenation pattern files with TeX hyph-utf8 project and iconv them
    2021-07-10 gbranden Severity5 - Blocker 3 - Normal
        StatusNone Postponed
        Summarysync groff hyphenation-pattern files with upstream TeX versions sync hyphenation pattern files with TeX versions
    2021-01-14 gbranden Severity3 - Normal 5 - Blocker

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code