bugGNU roff - Bugs: bug #53413, add English hyphenation patterns...

 
 

bug #53413: add English hyphenation patterns to enable use of ".hy 48"

Submitter:  Bjarni Ingi Gislason <bjarniig>
Submitted:  Thu 22 Mar 2018 12:48:08 AM UTC
   
 
Category:  Macro - others/general Severity:  3 - Normal
Item Group:  Feature change Status:  Rejected
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 30 Mar 2023 05:01:02 PM UTC, comment #17: 

Dropping patch annotation from rejected ticket.

G. Branden Robinson <gbranden>
Group administrator
Tue 19 Jan 2021 05:03:35 AM UTC, comment #16: 

Hi Dave!

comment #15:

> Is there anything further to be done for this bug?  There seems to be no support for incorporating the patch in the initial submission, and all the ancillary issues brought up have been spun off into their own reports.


Right.  I don't think there is.  Resolving.

I'm persuaded by Werner's position.  Next-generation hyphenation control should absorb all the within-word positional parameterization of what is now the hyphenation mode into separate requests that are coupled to hyphenation pattern files; if someone tries to request hyphenation at positions that are incompatible with the pattern files for the current language, the request should be rejected with a diagnostic.

I interpret Bjarni's project in this ticket as being motivated by the observation that the software gave him a knob to twist, so he twisted it.  This is a reasonable desire; software should not expose nonsensical configuration options.

Unfortunately due to technical debt, that is what we are doing today.

So until we get that next-generation hyphenation control, our answer to users wanting to use .hy 48 with English must be "don't do that".  (Or we can point them at a mountain named "write your own English hyphenation patterns".)

This is already documented in Git HEAD, and perhaps with .hy 4 in en.tmac being loaded by default for English-localized users--_if_ I get that branch merged in time for 1.23.0--people will be less tempted to fiddle with it.

> comment #13:
> > I reckon I would add '.hy 4' to my proposed new en.tmac file in #59814.
>
> That seems best given the design proposed in that bug.


Yup, and this is on the branch now as seen in the ticket.

> > I think we should have English localization per se done in a separate
> > file.  Its concerns are distinguishable from those in the rest of troffrc.
>
> Eh, well, they are as of when the 59814 patch is applied.  Before that, one of troffrc's concerns was loading English hyphenation patterns.


Yes.  I use parallel brains to contemplate and discuss currently-released-groff (1.22.4), Git HEAD, and my branch-in-development.   It would be helpful to others if I mentioned which brain I was thinking with in a given discussion.  :D

G. Branden Robinson <gbranden>
Group administrator
Thu 14 Jan 2021 10:04:09 AM UTC, comment #15: 

Is there anything further to be done for this bug?  There seems to be no support for incorporating the patch in the initial submission, and all the ancillary issues brought up have been spun off into their own reports.



comment #13:

> I reckon I would add '.hy 4' to my proposed new en.tmac file in #59814.


That seems best given the design proposed in that bug.

> I think we should have English localization per se done in a separate
> file.  Its concerns are distinguishable from those in the rest of troffrc.


Eh, well, they are as of when the 59814 patch is applied.  Before that, one of troffrc's concerns was loading English hyphenation patterns.

Dave <barx>
Group Member
Mon 04 Jan 2021 07:23:13 AM UTC, comment #14: 

Sigh. I forgot to preview.  The new bug is bug #59814.

G. Branden Robinson <gbranden>
Group administrator
Mon 04 Jan 2021 07:22:24 AM UTC, comment #13: 


comment #12:

> comment #11:
> > Probably, or at least en_US.tmac to keep the Commonwealthers from screaming too loudly.
>
> Groff already makes no distinction between US and UK English in its hyphenation handling, so whatever screaming they want to do, it's over something more fundamental than the name of the .tmac file.


That's true.  But it annoyed me so I created #59814.

> > The setup for English is handled by troffrc and hyphen{,ex}.us.
> >
> > Unless I'm misunderstanding you.
>
> I don't know, because I'm not sure I'm following you.
>
> hyphen{,ex}.us, despite living in tmac/, aren't tmac files, so the default .hy can't be set there.


That's true.  I reckon I would add '.hy 4' to my proposed new en.tmac file in #59814.
 

> troffrc looks like it unconditionally sets the language to "us" and loads those two hyphenation files.  So this might be a good place to go ahead and set .hy as well, rather than creating a new .tmac file.


On the contrary, I think we should have English localization per se done in a separate file.  Its concerns are distinguishable from those in the rest of troffrc.

>  But I don't know how non-English groffs are set up (Do they skip troffrc altogether?  Does the installation process modify this file to load different hyphenation patterns?), so I'm probably not the best person to say.


Non-English groff is...not set up by default.  You have to hack your troffrc or load one of the localizing tmac files yourself in a macro package or document.

G. Branden Robinson <gbranden>
Group administrator
Fri 07 Aug 2020 11:32:55 PM UTC, comment #12: 

comment #11:

> Probably, or at least en_US.tmac to keep the Commonwealthers from screaming too loudly.


Groff already makes no distinction between US and UK English in its hyphenation handling, so whatever screaming they want to do, it's over something more fundamental than the name of the .tmac file. 

> The setup for English is handled by troffrc and hyphen{,ex}.us.
>
> Unless I'm misunderstanding you.


I don't know, because I'm not sure I'm following you.

hyphen{,ex}.us, despite living in tmac/, aren't tmac files, so the default .hy can't be set there.

troffrc looks like it unconditionally sets the language to "us" and loads those two hyphenation files.  So this might be a good place to go ahead and set .hy as well, rather than creating a new .tmac file.  But I don't know how non-English groffs are set up (Do they skip troffrc altogether?  Does the installation process modify this file to load different hyphenation patterns?), so I'm probably not the best person to say.

Dave <barx>
Group Member
Thu 06 Aug 2020 03:15:51 PM UTC, comment #11: 

comment #10:

> Curiously, there is no such setup file for English, even though the groff default (.hy 1) is invalid for the English hyphenation patterns.  Until a fix for bug 57556 offers a more robust solution to this, perhaps there should be a tmac/en.tmac that is called by default in English environments?


Probably, or at least en_US.tmac to keep the Commonwealthers from screaming too loudly.

The setup for English is handled by troffrc and hyphen{,ex}.us.

Unless I'm misunderstanding you.


G. Branden Robinson <gbranden>
Group administrator
Sun 12 Jul 2020 11:42:23 PM UTC, comment #10: 

comment #9:

> My suggestion is to make groff look into the meta information
> of the pattern themselves while they are loaded to set those
> parameters.  The `.hy' command could then become a user-only
> request restricted by the meta information – for example,
> setting hy=48 would have no effect because the patterns don't
> allow it.


I've opened bug #57556 with this suggestion.

> Right now, these parameters are hardcoded using `.hy', this
> is, they are part of groff's language setup files.


I take it you're referring here to the tmac/{cs|de|fr|ja|sv|zh}.tmac files?


$ shopt -s extglob
$ grep '\. *hy' tmac/@(cs|de|fr|ja|sv|zh).tmac
tmac/cs.tmac:.hy 1
tmac/de.tmac:.hy 1
tmac/fr.tmac:.hy 4
tmac/sv.tmac:.hy 32
$


Curiously, there is no such setup file for English, even though the groff default (.hy 1) is invalid for the English hyphenation patterns.  Until a fix for bug 57556 offers a more robust solution to this, perhaps there should be a tmac/en.tmac that is called by default in English environments?

Dave <barx>
Group Member
Thu 17 May 2018 04:58:50 AM UTC, comment #9: 

If the user asks to hyphenate after the first letter, groff should indeed refuse in case current language's hyphenation parameters don't allow it.

Right now, these parameters are hardcoded using `.hy', this is, they are part of groff's language setup files.  My suggestion is to make groff look into the meta information of the pattern themselves while they are loaded to set those parameters.  The `.hy' command could then become a user-only request restricted by the meta information – for example, setting hy=48 would have no effect because the patterns don't allow it.

Werner LEMBERG <wl>
Group administrator
Wed 16 May 2018 09:57:54 PM UTC, comment #8: 


> It's outside of groff's scope to decide that since it is
> meta information bound to a given language.


Well, yes and no.  Looking at the bigger picture:

In English, a single letter is not a valid hyphenation breakpoint, even if it is a syllable breakpoint.  This is a longstanding typographic convention and a fairly ironclad rule in modern publishing.

However, what if the user asks the typesetter to override this rule?  Should the typesetter acquiesce, because the user is ultimately in charge, and it's not software's job to protect users from themselves?  This appears to be Bjarni's position.

Or should the typesetting software refuse, because knowing typesetting best practices is its job, despite what the user asks for?  This seems to be Werner's position.

Still not taking sides, just trying to clarify what the sides are.

Dave <barx>
Group Member
Tue 15 May 2018 05:43:14 AM UTC, comment #7: 

I can only repeat that there is no `failure' in hyphenation patterns.  It's outside of groff's scope to decide that since it is meta information bound to a given language.

What's missing is a mechanism for groff to get the correct minmax hyphenation values.  Fortunately, there are efforts to improve that: Look at `yaml-headers' branch of the central repository of TeX hyphenation patterns

  https://github.com/hyphenation/tex-hyphen/tree/yaml-headers

which adds the necessary information to all available hyphenation patterns!  AFAIK, this will eventually be merged into `master' (as soon as the maintainer has time to do that).

I suggest to add code to groff to make use of this information, for example, by replacing the current `hyphen.xxx' files with the corresponding files from the `tex-hyphen' repository, then parsing the YAML headers for the necessary information.

Werner LEMBERG <wl>
Group administrator
Mon 14 May 2018 10:38:05 PM UTC, comment #6: 


> And `generally' using `.hy 48' is a bad idea. Values of
> `.hy' are always bound to the currently selected hyphenation
> patterns.


Yes, but the entire reason .hy 48 produces erroneous output is that the hyphenation patterns were not crafted with this setting in mind.  This strikes me as a failure of the hyphenation patterns, not of the user's .hy selection.

The proposed patch (haphazardly) addresses that, but for English only.  Werner's point, in comment #1, is that it's imprudent to fix a problem for one language that ships with groff while leaving the same problem in the others.

I suppose it's a philosophical question whether it's better to partially fix a shortcoming but create an inconsistency between languages (what this patch does), or maintain consistency and address the shortcoming in the documentation (the status quo).  Werner also declined to make this call in comment #1.  And I'm not a groff developer, so it's not really my place to have an opinion, either.  But I hope this clarifies the issue.

Dave <barx>
Group Member
Tue 24 Apr 2018 01:37:07 AM UTC, comment #5: 

  The number 8 is high enough to prohibit wrong hyphenations caused by
odd numbers less than 9.

  It is more important to avoid wrong hyphenations than finding
patterns for rare ones.

  And the last used number should be an even number to eliminate wrong
patterns found earlier.

  [If 9 is used, maybe '0' could be used as a counterbalance?]

  I am not saying that hy=48 should be default.  Users may (are free
to, this is "free" software) choose for themselves any defined value
(irrespective of the used main-hyphenation file).

  I use hy=48 with the additional files for my own writing; for reading
man pages, no hyphenations (waste of resources).
I have not noticed any irregularities.

  The additional files just lessen the chance of seeing wrong
hyphenations, as long as the algorithm does not take care of the issue.

N.B.
  The files need a prolog to explain it, like

% These patterns avoid false hyphenations for US English with the
% current (April 2018) algorithm in groff and values of ".hy" that allow
% splitting of one character.

Bjarni Ingi Gislason <bjarniig>
Mon 23 Apr 2018 12:02:09 PM UTC, comment #4: 

Bjarni,

Can you address Werner's last comment, please?  This bug is stuck in limbo.

G. Branden Robinson <gbranden>
Group administrator
Thu 22 Mar 2018 11:08:04 AM UTC, comment #3: 

German patterns use numbers up to value 8 – this means that such a `safety addition' as you suggest isn't possible, because values can't be larger than 9.

> "*roff" has the 'hy=1' which TeX does not(?).


It's not clear what you want to say.  If you mean that TeX doesn't support \{left,right}hyphenmin=1, this is not correct.  You can set those two parameters to any value.  However, if you select a too low value, you also get invalid hyphenation points.

And `generally' using `.hy 48' is a bad idea.  Values of `.hy' are always bound to the currently selected hyphenation patterns.

Werner LEMBERG <wl>
Group administrator
Thu 22 Mar 2018 10:50:25 AM UTC, comment #2: 

  The number 4 is not high enough to forbid the hyphenation as there are
patterns of type "5x$" and "^x5", so it must be changed to 6.

  The fundamental flaw of the used algorithms is to compare patterns to
the beginning and end of words without the anchor point '.'.

  "*roff" has the 'hy=1' which TeX does not(?).

  The new additional pattern files are simply a choice for those who
generally want to use hy=48 (current development state of groff) or
hy=1 (current stable version) with the current implementation of the
hyphenation algorithm.

Bjarni Ingi Gislason <bjarniig>
Thu 22 Mar 2018 05:18:31 AM UTC, comment #1: 

Thanks.  However, I really wonder whether we should add this to groff.

Compare this to a red traffic light in the US.  You tell all passengers that it is forbidden to cross the street in that case.  However, at the same time you provide an armor to them as a protection so that they can still cross the street anytime without being hurt.  As soon as they visit a different country like Germany they think the armor still works but no, this is no longer the case...

This doesn't make sense to me.

I'm leaving a decision to other people.

Werner LEMBERG <wl>
Group administrator
Thu 22 Mar 2018 12:48:08 AM UTC, original submission:  

From c49578e37518fea91aaed737f4cfa50405b4b8a4 Mon Sep 17 00:00:00 2001
From: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
Date: Thu, 22 Mar 2018 00:29:08 +0000
Subject: [PATCH] Add hyphenation patterns to use with US and hy=48

  To inhibit the splitting of one character when hy=48 (formerly hy=1)
is used, provide extra files "hyphen-first.us" and "hyphen-last.us",
that prohibit this.

Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
---
 tmac/hyphen-first.us | 29 +++++++++++++++++++++++++++++
 tmac/hyphen-last.us  | 29 +++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
 create mode 100644 tmac/hyphen-first.us
 create mode 100644 tmac/hyphen-last.us

diff --git a/tmac/hyphen-first.us b/tmac/hyphen-first.us
new file mode 100644
index 00000000..e5ca0b5b
--- /dev/null
+++ b/tmac/hyphen-first.us
@@ -0,0 +1,29 @@
+% a-fraid a-fresh a-shore a-side
+.a4
+.b4
+.c4
+.d4
+% e-mit
+.e4
+.f4
+.g4
+.h4
+.i4
+.j4
+.k4
+.l4
+.m4
+.n4
+% o-mit o-me-ga
+.o4
+.p4
+.q4
+.r4
+.s4
+.t4
+.u4
+.v4
+.w4
+.x4
+.y4
+.z4
diff --git a/tmac/hyphen-last.us b/tmac/hyphen-last.us
new file mode 100644
index 00000000..076bce9a
--- /dev/null
+++ b/tmac/hyphen-last.us
@@ -0,0 +1,29 @@
+% me-di-a cri-te-ri-a
+4a.
+4b.
+4c.
+4d.
+4e.
+4f.
+4g.
+4h.
+4i.
+4j.
+4k.
+4l.
+4m.
+4n.
+% dem-o vid-e-o
+4o.
+4p.
+4q.
+4r.
+4s.
+4t.
+4u.
+4v.
+4w.
+4x.
+% cloud-y dream-y drift-y meat-y
+4y.
+4z.
--
2.16.2


Bjarni Ingi Gislason <bjarniig>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by wl (Posted a comment)
  • -email is unavailable- added by bjarniig (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-03-30 gbranden Summary[PATCH] Add hyphenation patterns to use with US and hy=48 add English hyphenation patterns to enable use of ".hy 48"
    2021-01-19 gbranden StatusNeed Info Rejected
        Assigned toNone gbranden
        Open/ClosedOpen Closed
    2018-04-23 gbranden StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code