bugGNU troff - Bugs: bug #57556, Enforce left and right minimums as...

 
 

bug #57556: Enforce left and right minimums as required by hyphenation patterns

Submitted by:  Dave <barx>
Submitted on:  Wed 08 Jan 2020 08:13:48 AM UTC  
 
Category:  Core Severity:  3 - Normal
Item Group:  New feature Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Mon 18 Jan 2021 05:45:28 AM UTC, comment #7: 

comment #6:

> Design question: since a minimum hyphenation mode can be
> computed based upon this characteristic of the pattern
> files, should reading such a pattern file effectively set .hy?


Yes, if you want to fix this bug as outlined in the last paragraph of comment #0. :-)

What alternative design do you have in mind?  troff could make the left and right minimum values available to the user (in registers, for instance), and then require the user to compute a .hy value based on this data (not a straightforward matter before bug #55070 is addressed), but I see little value to that approach unless someone feels it's important to continue to allow users to use .hy values that are incompatible with their language's hyphenation data.

Or am I misunderstanding the question?

Dave <barx>
Project Member
Sun 17 Jan 2021 06:42:31 AM UTC, comment #6: 

comment #1:

> See
>
> https://github.com/hyphenation/tex-hyphen/tree/master/hyph-utf8/tex/generic/hyph-utf8/patterns/tex
>
> for patterns with YAML metadata containing hyphenmin values.


Below I'm inlining what those look like.  Integrating YAML parsing into the troff executable seems like overkill.  We just need a little state machine to read and match the relevant chunk ("hyphenmins") of the comment header and complain if one isn't found.

Design question: since a minimum hyphenation mode can be computed based upon this characteristic of the pattern files, should reading such a pattern file effectively set .hy?

I'm thinking "probably not" and that such a thing can be rolled into bug #55070, if that ever happens, but it's worth considering.

% title: Hyphenation patterns for American English
% copyright: Copyright (C) 1990, 2004, 2005 Gerard D.C. Kuiken
% notice: This file is part of the hyph-utf8 package.
%     See http://www.hyphenation.org/tex for more information.
% language:
%     name: English, American spelling
%     tag: en-us
% version: 2005-05-30
% authors:
%   -
%     name: Gerard D.C. Kuiken
% licence:
%     text: >
%         Copying and distribution of this file, with or without modification,
%         are permitted in any medium without royalty provided the copyright
%         notice and this notice are preserved.
% hyphenmins:
%     typesetting:
%         left: 2
%         right: 3
% changes:
%     March 1, 1990 Initial release
%     May 30, 2005 Added copyright notice, no patterns change.
% texlive:
%     encoding: ascii
%     babelname: usenglishmax
%     legacy_patterns: ushyphmax.tex
%     message: Hyphenation patterns for American English
%     package: english
% known_bugs:
%     de-mo-c-rat: 'instead of dem-o-crat (see GitHub issue #15)'
% ==========================================
%
% ushyphmax.tex -- patterns for more hyphenation pattern memory (12000+).
% Also known as ushyphen.max.
%
% Needs extended pattern memory.
% Hyphenation trie becomes 7283 with 377 ops.
%
% These patterns are based on the Hyphenation Exception Log
% published in TUGboat, Volume 10 (1989), No. 3, pp. 337-341,
% and a large number of incorrectly hyphenated words not yet published.
% If added to Liang's before the closing bracket } of \patterns,
% the patterns run errorfree as far as known at this moment.
%
% These patterns find all admissible hyphens of the words in
% the Exception Log.  ushyph2.tex is a smaller set.
%
% Please send bugs or suggestions to tex-live (at) tug.org.
%
% 2005-05-30 (karl): in the past, ushyphmax.tex was a file containing
% only the additional patterns, without the \patterns command, etc.
% This turned out not to be very useful, since in practice the TeX
% distributions need one self-contained file for a language.  Therefore,
% ushyphmax.tex now contains both the additional patterns from
% Dr. Kuiken, and the original patterns and hyphenations from Knuth's
% hyphen.tex.
%
% The Plain TeX hyphenation tables.
\patterns{ % just type <return> if you're not using INITEX
.ach4
[...]

G. Branden Robinson <gbranden>
Project Administrator
Thu 14 Jan 2021 10:44:37 AM UTC, comment #5: 

comment #3:

> Of particular note is the observation that groff has several language-specific setup files but lacks one for English


This disparity is addressed as part of the proposed change in bug #59814.  Under the new 59814 framework, the code in the patch attached to comment #3 would more properly belong in the new tmac/en.tmac.

Dave <barx>
Project Member
Thu 05 Nov 2020 02:20:39 AM UTC, comment #4: 

An additional observation about the posted patch: it has no effect when groff is run in compatibility mode.  I don't know why--I never use compatibility mode in real life, so I'm hardly an expert in debugging problems with it--but this isn't necessarily a showstopper, since the compatibility-mode behavior is the same with or without the patch.  So it's helping some users and not regressing the rest.

Dave <barx>
Project Member
Mon 02 Nov 2020 07:56:31 AM UTC, comment #3: 

Subsequent comments in that bug are also relevant to this one (and probably really belong here).  Of particular note is the observation that groff has several language-specific setup files but lacks one for English, but that tmac/troffrc seems to fulfill this role: it unconditionally sets the hyphenation language to "us" and loads hyphenation patterns and exceptions files hyphen.us and hyphenex.us.  Thus tmac/troffrc seems like a reasonable place to enforce the .hy restrictions the English patterns require.  But I confess I don't know how non-English groffs are set up (Do they skip troffrc altogether?  Does the installation process modify this file to load different hyphenation patterns?), so this is conjecture.

In case it's correct, however, I submit the attached patch as a workaround when using the English patterns until there's a proper fix for this bug.  Some notes on it:

  • As far as I can tell, the roff language lacks bitwise operators, making the logic ugly.
  • This macro emits a warning when it changes the value that the user requests.  I'm not sure it should, but it's easy to remove.
  • The warning may not follow the style of groff's current warning system.
  • I retained one .tm I used for debugging, hidden behind an ".if 0".  Committer's choice whether to include this in the commit (if the patch is accepted at all).

This patch could reasonably be deferred until after the 1.23 release, since it involves (minor) code changes that will have little chance for testing.  But it does address a legitimate bug currently existing in groff, that some .hy values cause invalid hyphenations in English.

The patch may not be considered complete anyway, as it does nothing with the documentation.  The .hy section of doc/groff.texi tries to be language-agnostic--it covers hyphenation parameters of various languages, including English, without assuming a particular language--so it's not clear this is the place to introduce English-specific changes to the .hy defaults.  On the other hand, it's not clear what place is.  The language-specific setup files for some other languages change the initial value of .hy (though do not restrict what the user subsequently does with it) without seeming to explicitly document this fact, so doing the same for English might be OK.

(file #50188)

Dave <barx>
Project Member
Mon 10 Feb 2020 07:06:44 AM UTC, comment #2: 
Dave <barx>
Project Member
Wed 08 Jan 2020 08:32:18 AM UTC, comment #1: 

See

https://github.com/hyphenation/tex-hyphen/tree/master/hyph-utf8/tex/generic/hyph-utf8/patterns/tex

for patterns with YAML metadata containing hyphenmin values.

Werner LEMBERG <wl>
Project Administrator
Wed 08 Jan 2020 08:13:48 AM UTC, original submission:  

For most supported languages, groff's hyphenation data assumes a minimum number of letters that must appear at the beginning or end of a word before or after a hyphenation point.  The hyphenation algorithm attempting to leave fewer letters than this minimum will result in erroneous hyphenations.  (The groff manual's .hy section contains an example of an erroneous division of the word "splitting" resulting from a .hy value invalid for the English patterns.)

These required minimums vary depending on the language.  Currently, groff has a default .hy value that is used regardless of language, and which is invalid for two of its supported languages (French and English).  By default, groff should set these minimums based on the selected language.

The hyphenation data historically did not include these minimum values -- even though they are essential for using the patterns correctly.  Werner Lemberg points out (http://lists.gnu.org/archive/html/groff/2018-03/msg00002.html) that since the patterns originate in TeX and not groff, that's the proper place for this metadata to be introduced.  Later in the thread he adds, "The TeX hyphenation project is going to [explicitly provide lefthyphenmin and righthyphenmin values] by introducing YAML headers to all pattern files."  (A tex-hyphen thread about this can be found at http://tug.org/pipermail/tex-hyphen/2018-March/001650.html.)

In turn, groff should use these newly supplied values to (1) set .hy to a valid default for the chosen language, and (2) reject any .hy value incompatible with the language's hyphenation data.  (The user is free, of course, to request a value of .hy that specifies greater hyphenation restrictions than the minimums the language requires.)

Dave <barx>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50188:  57556.workaround.patch added by barx (1KiB - text/x-diff)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by wl (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-11-05 barx SummaryFeature request: enforce left and right minimums as required by hyphenation patterns => Enforce left and right minimums as required by hyphenation patterns
    2020-11-02 barx Attached File- => Added 57556.workaround.patch, #50188

    Back to the top


    Powered by Savane 3.6