bugGNU troff - Bugs: bug #53413, [PATCH] Add hyphenation patterns...

 
 

bug #53413: [PATCH] Add hyphenation patterns to use with US and hy=48

Submitted by:  Bjarni Ingi Gislason <bjarniig>
Submitted on:  Thu 22 Mar 2018 12:48:08 AM UTC  
 
Category:  Macro - others Severity:  3 - Normal
Item Group:  New feature Status:  Need Info
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Fri 07 Aug 2020 11:32:55 PM UTC, comment #12: 

comment #11:

> Probably, or at least en_US.tmac to keep the Commonwealthers from screaming too loudly.


Groff already makes no distinction between US and UK English in its hyphenation handling, so whatever screaming they want to do, it's over something more fundamental than the name of the .tmac file. 

> The setup for English is handled by troffrc and hyphen{,ex}.us.
>
> Unless I'm misunderstanding you.


I don't know, because I'm not sure I'm following you.

hyphen{,ex}.us, despite living in tmac/, aren't tmac files, so the default .hy can't be set there.

troffrc looks like it unconditionally sets the language to "us" and loads those two hyphenation files.  So this might be a good place to go ahead and set .hy as well, rather than creating a new .tmac file.  But I don't know how non-English groffs are set up (Do they skip troffrc altogether?  Does the installation process modify this file to load different hyphenation patterns?), so I'm probably not the best person to say.

Dave <barx>
Project Member
Thu 06 Aug 2020 03:15:51 PM UTC, comment #11: 

comment #10:

> Curiously, there is no such setup file for English, even though the groff default (.hy 1) is invalid for the English hyphenation patterns.  Until a fix for bug 57556 offers a more robust solution to this, perhaps there should be a tmac/en.tmac that is called by default in English environments?


Probably, or at least en_US.tmac to keep the Commonwealthers from screaming too loudly.

The setup for English is handled by troffrc and hyphen{,ex}.us.

Unless I'm misunderstanding you.

G. Branden Robinson <gbranden>
Project Administrator
Sun 12 Jul 2020 11:42:23 PM UTC, comment #10: 

comment #9:

> My suggestion is to make groff look into the meta information
> of the pattern themselves while they are loaded to set those
> parameters.  The `.hy' command could then become a user-only
> request restricted by the meta information – for example,
> setting hy=48 would have no effect because the patterns don't
> allow it.


I've opened bug #57556 with this suggestion.

> Right now, these parameters are hardcoded using `.hy', this
> is, they are part of groff's language setup files.


I take it you're referring here to the tmac/{cs|de|fr|ja|sv|zh}.tmac files?

$ shopt -s extglob
$ grep '\. *hy' tmac/@(cs|de|fr|ja|sv|zh).tmac
tmac/cs.tmac:.hy 1
tmac/de.tmac:.hy 1
tmac/fr.tmac:.hy 4
tmac/sv.tmac:.hy 32
$

Curiously, there is no such setup file for English, even though the groff default (.hy 1) is invalid for the English hyphenation patterns.  Until a fix for bug 57556 offers a more robust solution to this, perhaps there should be a tmac/en.tmac that is called by default in English environments?

Dave <barx>
Project Member
Thu 17 May 2018 04:58:50 AM UTC, comment #9: 

If the user asks to hyphenate after the first letter, groff should indeed refuse in case current language's hyphenation parameters don't allow it.

Right now, these parameters are hardcoded using `.hy', this is, they are part of groff's language setup files.  My suggestion is to make groff look into the meta information of the pattern themselves while they are loaded to set those parameters.  The `.hy' command could then become a user-only request restricted by the meta information – for example, setting hy=48 would have no effect because the patterns don't allow it.

Werner LEMBERG <wl>
Project Administrator
Wed 16 May 2018 09:57:54 PM UTC, comment #8: 

> It's outside of groff's scope to decide that since it is
> meta information bound to a given language.


Well, yes and no.  Looking at the bigger picture:

In English, a single letter is not a valid hyphenation breakpoint, even if it is a syllable breakpoint.  This is a longstanding typographic convention and a fairly ironclad rule in modern publishing.

However, what if the user asks the typesetter to override this rule?  Should the typesetter acquiesce, because the user is ultimately in charge, and it's not software's job to protect users from themselves?  This appears to be Bjarni's position.

Or should the typesetting software refuse, because knowing typesetting best practices is its job, despite what the user asks for?  This seems to be Werner's position.

Still not taking sides, just trying to clarify what the sides are.

Dave <barx>
Project Member
Tue 15 May 2018 05:43:14 AM UTC, comment #7: 

I can only repeat that there is no `failure' in hyphenation patterns.  It's outside of groff's scope to decide that since it is meta information bound to a given language.

What's missing is a mechanism for groff to get the correct minmax hyphenation values.  Fortunately, there are efforts to improve that: Look at `yaml-headers' branch of the central repository of TeX hyphenation patterns

  https://github.com/hyphenation/tex-hyphen/tree/yaml-headers

which adds the necessary information to all available hyphenation patterns!  AFAIK, this will eventually be merged into `master' (as soon as the maintainer has time to do that).

I suggest to add code to groff to make use of this information, for example, by replacing the current `hyphen.xxx' files with the corresponding files from the `tex-hyphen' repository, then parsing the YAML headers for the necessary information.

Werner LEMBERG <wl>
Project Administrator
Mon 14 May 2018 10:38:05 PM UTC, comment #6: 

> And `generally' using `.hy 48' is a bad idea. Values of
> `.hy' are always bound to the currently selected hyphenation
> patterns.


Yes, but the entire reason .hy 48 produces erroneous output is that the hyphenation patterns were not crafted with this setting in mind.  This strikes me as a failure of the hyphenation patterns, not of the user's .hy selection.

The proposed patch (haphazardly) addresses that, but for English only.  Werner's point, in comment #1, is that it's imprudent to fix a problem for one language that ships with groff while leaving the same problem in the others.

I suppose it's a philosophical question whether it's better to partially fix a shortcoming but create an inconsistency between languages (what this patch does), or maintain consistency and address the shortcoming in the documentation (the status quo).  Werner also declined to make this call in comment #1.  And I'm not a groff developer, so it's not really my place to have an opinion, either.  But I hope this clarifies the issue.

Dave <barx>
Project Member
Tue 24 Apr 2018 01:37:07 AM UTC, comment #5: 

  The number 8 is high enough to prohibit wrong hyphenations caused by
odd numbers less than 9.

  It is more important to avoid wrong hyphenations than finding
patterns for rare ones.

  And the last used number should be an even number to eliminate wrong
patterns found earlier.

  [If 9 is used, maybe '0' could be used as a counterbalance?]

  I am not saying that hy=48 should be default.  Users may (are free
to, this is "free" software) choose for themselves any defined value
(irrespective of the used main-hyphenation file).

  I use hy=48 with the additional files for my own writing; for reading
man pages, no hyphenations (waste of resources).
I have not noticed any irregularities.

  The additional files just lessen the chance of seeing wrong
hyphenations, as long as the algorithm does not take care of the issue.

N.B.
  The files need a prolog to explain it, like

% These patterns avoid false hyphenations for US English with the
% current (April 2018) algorithm in groff and values of ".hy" that allow
% splitting of one character.

Bjarni Ingi Gislason <bjarniig>
Mon 23 Apr 2018 12:02:09 PM UTC, comment #4: 

Bjarni,

Can you address Werner's last comment, please?  This bug is stuck in limbo.

G. Branden Robinson <gbranden>
Project Administrator
Thu 22 Mar 2018 11:08:04 AM UTC, comment #3: 

German patterns use numbers up to value 8 – this means that such a `safety addition' as you suggest isn't possible, because values can't be larger than 9.

> "*roff" has the 'hy=1' which TeX does not(?).


It's not clear what you want to say.  If you mean that TeX doesn't support \{left,right}hyphenmin=1, this is not correct.  You can set those two parameters to any value.  However, if you select a too low value, you also get invalid hyphenation points.

And `generally' using `.hy 48' is a bad idea.  Values of `.hy' are always bound to the currently selected hyphenation patterns.

Werner LEMBERG <wl>
Project Administrator
Thu 22 Mar 2018 10:50:25 AM UTC, comment #2: 

  The number 4 is not high enough to forbid the hyphenation as there are
patterns of type "5x$" and "^x5", so it must be changed to 6.

  The fundamental flaw of the used algorithms is to compare patterns to
the beginning and end of words without the anchor point '.'.

  "*roff" has the 'hy=1' which TeX does not(?).

  The new additional pattern files are simply a choice for those who
generally want to use hy=48 (current development state of groff) or
hy=1 (current stable version) with the current implementation of the
hyphenation algorithm.

Bjarni Ingi Gislason <bjarniig>
Thu 22 Mar 2018 05:18:31 AM UTC, comment #1: 

Thanks.  However, I really wonder whether we should add this to groff.

Compare this to a red traffic light in the US.  You tell all passengers that it is forbidden to cross the street in that case.  However, at the same time you provide an armor to them as a protection so that they can still cross the street anytime without being hurt.  As soon as they visit a different country like Germany they think the armor still works but no, this is no longer the case...

This doesn't make sense to me.

I'm leaving a decision to other people.

Werner LEMBERG <wl>
Project Administrator
Thu 22 Mar 2018 12:48:08 AM UTC, original submission:  

From c49578e37518fea91aaed737f4cfa50405b4b8a4 Mon Sep 17 00:00:00 2001
From: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
Date: Thu, 22 Mar 2018 00:29:08 +0000
Subject: [PATCH] Add hyphenation patterns to use with US and hy=48

  To inhibit the splitting of one character when hy=48 (formerly hy=1)
is used, provide extra files "hyphen-first.us" and "hyphen-last.us",
that prohibit this.

Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
---
 tmac/hyphen-first.us | 29 +++++++++++++++++++++++++++++
 tmac/hyphen-last.us  | 29 +++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
 create mode 100644 tmac/hyphen-first.us
 create mode 100644 tmac/hyphen-last.us

diff --git a/tmac/hyphen-first.us b/tmac/hyphen-first.us
new file mode 100644
index 00000000..e5ca0b5b
--- /dev/null
+++ b/tmac/hyphen-first.us
@@ -0,0 +1,29 @@
+% a-fraid a-fresh a-shore a-side
+.a4
+.b4
+.c4
+.d4
+% e-mit
+.e4
+.f4
+.g4
+.h4
+.i4
+.j4
+.k4
+.l4
+.m4
+.n4
+% o-mit o-me-ga
+.o4
+.p4
+.q4
+.r4
+.s4
+.t4
+.u4
+.v4
+.w4
+.x4
+.y4
+.z4
diff --git a/tmac/hyphen-last.us b/tmac/hyphen-last.us
new file mode 100644
index 00000000..076bce9a
--- /dev/null
+++ b/tmac/hyphen-last.us
@@ -0,0 +1,29 @@
+% me-di-a cri-te-ri-a
+4a.
+4b.
+4c.
+4d.
+4e.
+4f.
+4g.
+4h.
+4i.
+4j.
+4k.
+4l.
+4m.
+4n.
+% dem-o vid-e-o
+4o.
+4p.
+4q.
+4r.
+4s.
+4t.
+4u.
+4v.
+4w.
+4x.
+% cloud-y dream-y drift-y meat-y
+4y.
+4z.
--
2.16.2

Bjarni Ingi Gislason <bjarniig>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by wl (Posted a comment)
  • -email is unavailable- added by bjarniig (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-04-23 gbranden StatusNone => Need Info

    Back to the top


    Powered by Savane 3.5