bugGNU roff - Bugs: bug #61919, are the new English hyphenation...

 
 

bug #61919: are the new English hyphenation patterns really an improvement?

Submitter:  Bjarni Ingi Gislason <bjarniig>
Submitted:  Tue 25 Jan 2022 10:06:31 PM UTC
   
 
Category:  Core Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  Need Info
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 26 Jan 2022 07:38:22 PM UTC, comment #6: 

Maybe as part of the RC2 announcement we can solicit the opinions of the user base on this point.  That might be good marketing (people love to share opinions), help get the RC into peoples' hands for actual evaluation, which RC1 seems not to have managed to do, and also paves the way for reverting them for the final release if that's the way the wind of opinion blows.

An interestingly ironic possible future would be for groff to get an influx of TeX refugees who can't stand the K-K patterns, and defect to us because we keep the old pure Knuthian flame burning.

G. Branden Robinson <gbranden>
Group administrator
Wed 26 Jan 2022 07:28:43 PM UTC, comment #5: 

Oh, absolutely, the only two reasonable options open to groff are keeping the Knuth+Kuiken patterns, or reverting to Knuth only.  If it seemed like I was suggesting some other course, like tweaking the patterns to improve results, I apologize for giving the wrong impression.  That phantom email I swear I will finish one day would have made this clearer, I think.

But yes, fully agreed that the content of the pattern sets is outside groff's purview and probably also its sphere of expertise.

Dave <barx>
Group Member
Wed 26 Jan 2022 05:52:25 PM UTC, comment #4: 

Hi, Dave!

comment #3:

> comment #2:
> > Not only is the subject of this ticket subsumed by existing
> > bug #57594,
>
> Hmm, the last sentence of the original submission is subsumed by #57594, but it's not clear the rest is.  Though maybe we have different perspectives on what constitutes "the rest":
>
> > but the exact issue, down to the very word, that Bjarni
> > raises was addressed in a commit message and ChangeLog entry.
>
> The issue was certainly mentioned in the commit message.  But that the original Knuth hyphenation patterns gave results for "material" (and other words) that agreed with the dictionary, and that the current Knuth+Kuiken ones violate the typical dictionary hyphenation for "material" (and other words, some far more egregiously), remains unaddressed.
>
> This is a much larger issue than the single word Bjarni highlighted, but that word is symptomatic of a legitimate problem that warrants some discussion amongst a larger audience.  (Yes, yes, I need to revisit and finish that email draft.)
>
> But just as a sanity check: do you see any of the divisions under the NEW column as improvements over their OLD brethren?


Mostly not.  I agreed with you on this point and nearly said so when resolving the ticket.

I had/have two biases operating:

  • a desire to see bugs against the project closed
  • a desire to rule out issues as "not our problem"


The second is perhaps more defensible in this case, as it's questionable whether we have the human power to maintain groff as-is, let alone take on an independent multilingual hyphenation data base/algorithm project.

So I do have an inclination to say, "if the Knuth+Kuiken hyphenations are bad, take it up with the TeX hyph-utf8 project".  I really don't want that mess on our plate.

But alternative resolutions may exist; we could go back to the old patterns, for instance.

G. Branden Robinson <gbranden>
Group administrator
Wed 26 Jan 2022 05:40:39 PM UTC, comment #3: 

comment #2:

> Not only is the subject of this ticket subsumed by existing
> bug #57594,


Hmm, the last sentence of the original submission is subsumed by #57594, but it's not clear the rest is.  Though maybe we have different perspectives on what constitutes "the rest":

> but the exact issue, down to the very word, that Bjarni
> raises was addressed in a commit message and ChangeLog entry.


The issue was certainly mentioned in the commit message.  But that the original Knuth hyphenation patterns gave results for "material" (and other words) that agreed with the dictionary, and that the current Knuth+Kuiken ones violate the typical dictionary hyphenation for "material" (and other words, some far more egregiously), remains unaddressed.

This is a much larger issue than the single word Bjarni highlighted, but that word is symptomatic of a legitimate problem that warrants some discussion amongst a larger audience.  (Yes, yes, I need to revisit and finish that email draft.)

But just as a sanity check: do you see any of the divisions under the NEW column as improvements over their OLD brethren?

Dave <barx>
Group Member
Wed 26 Jan 2022 07:53:33 AM UTC, comment #2: 

Thanks, Dave.  I thought this stuff sounded familiar.

Not only is the subject of this ticket subsumed by existing bug #57594, but the exact issue, down to the very word, that Bjarni raises was addressed in a commit message and ChangeLog entry.


commit b2284ab01d2d87507f3bcbd7de2a081efb6528a6
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sun Jul 11 00:50:27 2021 +1000

    Update English hyphenation patterns.

    * NEWS: Add item.
    * tmac/hyphen.en: Update file using `hyph-en-us.tex` patterns file from
      the TeX hyph-utf8 project.
    * tmac/hyphenex.en: Remove explicit hyphenations for words that no
      longer require them when using the new patterns.  Add one item scraped
      from an erratum comment in hyphen.en ("dem-o-crat").

    The new patterns likely _will_ change the automatic hyphenation break
    points of your English documents.  Here is a sample of affected words
    found within groff's own documentary corpus.

    OLD                     NEW
    ===                     ===
    ar‐range‐ment           arrange‐ment
    col‐umns                columns
    con‐struc‐ted           con‐structed
    cus‐tom‐ized            cus‐tomized
    def‐i‐ni‐tions          de‐f‐i‐n‐i‐tions
    der‐i‐va‐tions          de‐riva‐tions
    hy‐phen‐a‐tion          hy‐phen‐ation
    ma‐te‐rial              ma‐te‐r‐ial
    Mi‐cro‐soft             Mi‐crosoft
    pipe‐lines              pipelines
    post‐pro‐ces‐sors       post‐proces‐sors
    pro‐cessed              processed
    pro‐cesses              processes
    spa‐ces                 spaces
    Wer‐ner                 Werner

    Partially addresses <https://savannah.gnu.org/bugs/?57594>.


G. Branden Robinson <gbranden>
Group administrator
Wed 26 Jan 2022 06:28:46 AM UTC, comment #1: 

original submission:

> Wrong(?) hyphenation (word division) of "material"


This is one of a number of hyphenations changed by commit b2284ab0.  That commit from last summer added a new set of patterns (the Kuiken patterns) to the English hyphenation patterns file; the commit log shows 15 words whose hyphenation points were altered--all, IMHO, for the worse.

I started writing a script at one point to go through an English word list and emit all the words whose hyphenation points were changed by this commit, intending to post the results to the email list for a discussion about whether this change was a net improvement to automatic hyphenation.  That project has been pushed aside many times.

Preliminary results are inconclusive; a lot of words do fare better, but seemingly at least as many were made worse.  While it violates the dictionary, "material" isn't actually terrible if it's divided as mater-ial; the one-syllable "drives" is pretty awful with its new break point of dri-ves.

> TeX dictates minimum of three letters at the end.


groff also requires this for the English patterns, though does not enforce it; see bug #57556.

> Hyphenation patterns need to be updated with each release;
> put that in the file "FOR-RELEASE".


The opening salvo of bug #57594 also mentions this.

Dave <barx>
Group Member
Tue 25 Jan 2022 10:06:31 PM UTC, original submission:  

Subject: Wrong(?) hyphenation (word division) of "material" (for
"FOR-RELEASE"))

  Input:

.ll 1n
material

  Command: test-nroff -men <input>

  Result:

ma-
te-
r-
ial


  Dictionary shows: ma-te-ri-al

  TeX dictates minimum of three letters at the end.

  Hyphenation patterns need to be updated with each release;
 put that in the file "FOR-RELEASE".

Bjarni Ingi Gislason <bjarniig>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by bjarniig (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-01-26 gbranden StatusDuplicate Need Info
        Open/ClosedClosed Open
        Summarywrong(?) hyphenation (word division) of &quot;material&quot; (for &quot;FOR-RELEASE&quot;)) are the new English hyphenation patterns really an improvement?
    2022-01-26 gbranden CategoryGeneral Core
        Item GroupDocumentation Incorrect behaviour
        StatusNone Duplicate
        Open/ClosedOpen Closed
        SummaryWrong(?) hyphenation (word division) of &quot;material&quot; (for &quot;FOR-RELEASE&quot;)) wrong(?) hyphenation (word division) of "material" (for "FOR-RELEASE"))

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code