bugGNU roff - Bugs: bug #57556, [troff] enforce left and right...

 
 

bug #57556: [troff] enforce left and right minimums as required by hyphenation patterns

Submitter:  Dave <barx>
Submitted:  Wed 08 Jan 2020 08:13:48 AM UTC
   
 
Category:  Core Severity:  1 - Wish
Item Group:  Feature change Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 23 Mar 2025 01:26:55 AM UTC, comment #13: 


comment #12:

> I did read through #55070 and saw nothing that addressed the idea of separating pattern minimums from user minimums.  By my reading, it concerned only separating the beginning and ending minimums from each other, and from cross-column/page restrictions.  If there's a more comprehensive straw-man proposal, I don't know where it lives.


That turned out to be my problem.  I pushed "looking for it" onto the stack, and couldn't find it.  I thought I had pitched it on the groff@ list recently, but fell into a tar pit with onf, but that seems to have been a discussion of my "next-generation alignment and adjustment" reform instead.  Or maybe there was more than one tarpit.

Befuddled, I neglected to update my comment before posting it.

I'll have to look harder.  Or maybe I'm misremembering and I never actually sketched out a solution.  Despite the reputation for aggressive changes I fear I've acquired, many of my big ideas await feedback for literally years before being implemented or discarded.

G. Branden Robinson <gbranden>
Group administrator
Sun 23 Mar 2025 01:03:19 AM UTC, comment #12: 

comment #11:

> I'll advise anyone just joining us of bug #55070 (already
> mentioned a couple of times), and also my straw-man proposal
> that attacks at least some of the 


Oh, the suspense!

I did read through #55070 and saw nothing that addressed the idea of separating pattern minimums from user minimums.  By my reading, it concerned only separating the beginning and ending minimums from each other, and from cross-column/page restrictions.  If there's a more comprehensive straw-man proposal, I don't know where it lives.

Dave <barx>
Group Member
Sun 23 Mar 2025 12:06:13 AM UTC, comment #11: 

comment #10:

> comment #6:
> > Design question: since a minimum hyphenation mode can be
> > computed based upon this characteristic of the pattern
> > files, should reading such a pattern file effectively set .hy?
>
> I want to revise my comment #7 answer to this by zooming out.


Since that comment is over 3 years old, I'll advise anyone just joining us of bug #55070 (already mentioned a couple of times), and also my straw-man proposal that attacks at least some of the 

>
> From a design standpoint, a weakness of the current .hy mechanism is that it's attempting to control two different things:
> 0 What hyphenation limitations are encoded into the active hyphenation patterns
> 0 What hyphenation restrictions the document author wants
> This tension is especially observable in a multilingual document: the values of both 1 and 2 above might change over the course of the document, and not necessarily in tandem.


Right.

Also, the "suppress hyphenation before a vertical position trap" flag reveals a subdivision of your second category.  It's a restriction the author might want to configure, but separately from others like "don't hyphenate between the 2nd and 3rd letters of a word even the patterns permit it".

G. Branden Robinson <gbranden>
Group administrator
Sat 22 Mar 2025 10:32:27 PM UTC, comment #10: 

comment #6:

> Design question: since a minimum hyphenation mode can be
> computed based upon this characteristic of the pattern
> files, should reading such a pattern file effectively set .hy?


I want to revise my comment #7 answer to this by zooming out.

From a design standpoint, a weakness of the current .hy mechanism is that it's attempting to control two different things:

  1. What hyphenation limitations are encoded into the active hyphenation patterns
  2. What hyphenation restrictions the document author wants

This tension is especially observable in a multilingual document: the values of both 1 and 2 above might change over the course of the document, and not necessarily in tandem.

A ground-up redesign would ideally address this by decoupling this .hy overload.

  1. Loading a new set of hyphenation patterns would automatically and invisibly update the pattern-dependent hyphenation minimums that groff uses.  The document author could discover but not alter these minimums.
  2. A user-level request would let document authors specify their preferred hyphenation restrictions.  Depending on the automatic setting of point 1, groff might actually use more restrictive settings than this.  But hyphenation could never be more lax than what the user specifies.


This would allow item 1 (pattern-based restrictions) to change over the course of a multilingual document, but item 2 (author preference) to remain constant throughout the document without additional author intervention.  It would losslessly retain the user's preference, which would not necessarily be the case if reading a new patterns file altered the user-preference register directly.

So, in comment #7, when I answered yes to your design question, I was thinking within the current hyphenation framework, where .hy is the only mechanism for controlling hyphenation minimums.  But on further reflection, when reading a hyphenation pattern file, groff should automatically enforce that file's requirements independently of the user-settable .hy mechanism.

Dave <barx>
Group Member
Thu 10 Aug 2023 02:05:06 PM UTC, comment #9: 

Unpostponing, since Savannah has grown a bug-dependency feature in the past 2 years, and Dave has already snuck usage of it into this one (a year ago!).

G. Branden Robinson <gbranden>
Group administrator
Wed 30 Jun 2021 05:14:46 PM UTC, comment #8: 

This feature depends on bug #57594: the data needed to implement this will only be available after the hyphenation-pattern files are updated.

Since savannah lacks a Blocked state or other mechanism to show a dependency on another bug, I'm following Branden's lead in bug #59738 and marking this Postponed.

Dave <barx>
Group Member
Mon 18 Jan 2021 05:45:28 AM UTC, comment #7: 

comment #6:

> Design question: since a minimum hyphenation mode can be
> computed based upon this characteristic of the pattern
> files, should reading such a pattern file effectively set .hy?


Yes, if you want to fix this bug as outlined in the last paragraph of comment #0. :-)

What alternative design do you have in mind?  troff could make the left and right minimum values available to the user (in registers, for instance), and then require the user to compute a .hy value based on this data (not a straightforward matter before bug #55070 is addressed), but I see little value to that approach unless someone feels it's important to continue to allow users to use .hy values that are incompatible with their language's hyphenation data.

Or am I misunderstanding the question?

Dave <barx>
Group Member
Sun 17 Jan 2021 06:42:31 AM UTC, comment #6: 

comment #1:

> See
>
> https://github.com/hyphenation/tex-hyphen/tree/master/hyph-utf8/tex/generic/hyph-utf8/patterns/tex
>
> for patterns with YAML metadata containing hyphenmin values.


Below I'm inlining what those look like.  Integrating YAML parsing into the troff executable seems like overkill.  We just need a little state machine to read and match the relevant chunk ("hyphenmins") of the comment header and complain if one isn't found.

Design question: since a minimum hyphenation mode can be computed based upon this characteristic of the pattern files, should reading such a pattern file effectively set .hy?

I'm thinking "probably not" and that such a thing can be rolled into bug #55070, if that ever happens, but it's worth considering.


% title: Hyphenation patterns for American English
% copyright: Copyright (C) 1990, 2004, 2005 Gerard D.C. Kuiken
% notice: This file is part of the hyph-utf8 package.
%     See http://www.hyphenation.org/tex for more information.
% language:
%     name: English, American spelling
%     tag: en-us
% version: 2005-05-30
% authors:
%   -
%     name: Gerard D.C. Kuiken
% licence:
%     text: >
%         Copying and distribution of this file, with or without modification,
%         are permitted in any medium without royalty provided the copyright
%         notice and this notice are preserved.
% hyphenmins:
%     typesetting:
%         left: 2
%         right: 3
% changes:
%     March 1, 1990 Initial release
%     May 30, 2005 Added copyright notice, no patterns change.
% texlive:
%     encoding: ascii
%     babelname: usenglishmax
%     legacy_patterns: ushyphmax.tex
%     message: Hyphenation patterns for American English
%     package: english
% known_bugs:
%     de-mo-c-rat: 'instead of dem-o-crat (see GitHub issue #15)'
% ==========================================
%
% ushyphmax.tex -- patterns for more hyphenation pattern memory (12000+).
% Also known as ushyphen.max.
%
% Needs extended pattern memory.
% Hyphenation trie becomes 7283 with 377 ops.
%
% These patterns are based on the Hyphenation Exception Log
% published in TUGboat, Volume 10 (1989), No. 3, pp. 337-341,
% and a large number of incorrectly hyphenated words not yet published.
% If added to Liang's before the closing bracket } of \patterns,
% the patterns run errorfree as far as known at this moment.
%
% These patterns find all admissible hyphens of the words in
% the Exception Log.  ushyph2.tex is a smaller set.
%
% Please send bugs or suggestions to tex-live (at) tug.org.
%
% 2005-05-30 (karl): in the past, ushyphmax.tex was a file containing
% only the additional patterns, without the \patterns command, etc.
% This turned out not to be very useful, since in practice the TeX
% distributions need one self-contained file for a language.  Therefore,
% ushyphmax.tex now contains both the additional patterns from
% Dr. Kuiken, and the original patterns and hyphenations from Knuth's
% hyphen.tex.
%
% The Plain TeX hyphenation tables.
\patterns{ % just type <return> if you're not using INITEX
.ach4
[...]


G. Branden Robinson <gbranden>
Group administrator
Thu 14 Jan 2021 10:44:37 AM UTC, comment #5: 

comment #3:

> Of particular note is the observation that groff has several language-specific setup files but lacks one for English


This disparity is addressed as part of the proposed change in bug #59814.  Under the new 59814 framework, the code in the patch attached to comment #3 would more properly belong in the new tmac/en.tmac.

Dave <barx>
Group Member
Thu 05 Nov 2020 02:20:39 AM UTC, comment #4: 

An additional observation about the posted patch: it has no effect when groff is run in compatibility mode.  I don't know why--I never use compatibility mode in real life, so I'm hardly an expert in debugging problems with it--but this isn't necessarily a showstopper, since the compatibility-mode behavior is the same with or without the patch.  So it's helping some users and not regressing the rest.

Dave <barx>
Group Member
Mon 02 Nov 2020 07:56:31 AM UTC, comment #3: 

Subsequent comments in that bug are also relevant to this one (and probably really belong here).  Of particular note is the observation that groff has several language-specific setup files but lacks one for English, but that tmac/troffrc seems to fulfill this role: it unconditionally sets the hyphenation language to "us" and loads hyphenation patterns and exceptions files hyphen.us and hyphenex.us.  Thus tmac/troffrc seems like a reasonable place to enforce the .hy restrictions the English patterns require.  But I confess I don't know how non-English groffs are set up (Do they skip troffrc altogether?  Does the installation process modify this file to load different hyphenation patterns?), so this is conjecture.

In case it's correct, however, I submit the attached patch as a workaround when using the English patterns until there's a proper fix for this bug.  Some notes on it:

  • As far as I can tell, the roff language lacks bitwise operators, making the logic ugly.
  • This macro emits a warning when it changes the value that the user requests.  I'm not sure it should, but it's easy to remove.
  • The warning may not follow the style of groff's current warning system.
  • I retained one .tm I used for debugging, hidden behind an ".if 0".  Committer's choice whether to include this in the commit (if the patch is accepted at all).


This patch could reasonably be deferred until after the 1.23 release, since it involves (minor) code changes that will have little chance for testing.  But it does address a legitimate bug currently existing in groff, that some .hy values cause invalid hyphenations in English.

The patch may not be considered complete anyway, as it does nothing with the documentation.  The .hy section of doc/groff.texi tries to be language-agnostic--it covers hyphenation parameters of various languages, including English, without assuming a particular language--so it's not clear this is the place to introduce English-specific changes to the .hy defaults.  On the other hand, it's not clear what place is.  The language-specific setup files for some other languages change the initial value of .hy (though do not restrict what the user subsequently does with it) without seeming to explicitly document this fact, so doing the same for English might be OK.

(file #50188)

Dave <barx>
Group Member
Mon 10 Feb 2020 07:06:44 AM UTC, comment #2: 
Dave <barx>
Group Member
Wed 08 Jan 2020 08:32:18 AM UTC, comment #1: 

See

https://github.com/hyphenation/tex-hyphen/tree/master/hyph-utf8/tex/generic/hyph-utf8/patterns/tex

for patterns with YAML metadata containing hyphenmin values.

Werner LEMBERG <wl>
Group administrator
Wed 08 Jan 2020 08:13:48 AM UTC, original submission:  

For most supported languages, groff's hyphenation data assumes a minimum number of letters that must appear at the beginning or end of a word before or after a hyphenation point.  The hyphenation algorithm attempting to leave fewer letters than this minimum will result in erroneous hyphenations.  (The groff manual's .hy section contains an example of an erroneous division of the word "splitting" resulting from a .hy value invalid for the English patterns.)

These required minimums vary depending on the language.  Currently, groff has a default .hy value that is used regardless of language, and which is invalid for two of its supported languages (French and English).  By default, groff should set these minimums based on the selected language.

The hyphenation data historically did not include these minimum values -- even though they are essential for using the patterns correctly.  Werner Lemberg points out (http://lists.gnu.org/archive/html/groff/2018-03/msg00002.html) that since the patterns originate in TeX and not groff, that's the proper place for this metadata to be introduced.  Later in the thread he adds, "The TeX hyphenation project is going to [explicitly provide lefthyphenmin and righthyphenmin values] by introducing YAML headers to all pattern files."  (A tex-hyphen thread about this can be found at http://tug.org/pipermail/tex-hyphen/2018-March/001650.html.)

In turn, groff should use these newly supplied values to (1) set .hy to a valid default for the chosen language, and (2) reject any .hy value incompatible with the language's hyphenation data.  (The user is free, of course, to request a value of .hy that specifies greater hyphenation restrictions than the minimums the language requires.)

Dave <barx>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50188:  57556.workaround.patch added by barx (1KiB - text/x-diff)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by wl (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2025-03-23 barx Severity3 - Normal 1 - Wish
    2025-03-23 gbranden SummaryEnforce left and right minimums as required by hyphenation patterns [troff] enforce left and right minimums as required by hyphenation patterns
    2023-08-10 gbranden StatusPostponed None
    2022-08-19 barx Dependencies- Depends on bugs #57594
    2021-06-30 barx StatusNone Postponed
    2020-11-05 barx SummaryFeature request: enforce left and right minimums as required by hyphenation patterns Enforce left and right minimums as required by hyphenation patterns
    2020-11-02 barx Attached File- Added 57556.workaround.patch, #50188

    Back to the top

    Powered by Savane 3.15.
    Corresponding source code