bugGNU roff - Bugs: bug #42870, [troff] revise hyphenation code...

 
 

bug #42870: [troff] revise hyphenation code management

Submitter:  None
Submitted:  Tue 29 Jul 2014 09:28:30 AM UTC
   
 
Category:  Core Severity:  1 - Wish
Item Group:  Feature change Status:  None
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 09 Aug 2024 06:17:06 PM UTC, comment #21: 

So, as I read this, the upshot is that .hw has always (for some value of "always") accepted special character escapes.  When using them seemed to have no effect (as in comment #16, clarified in comment #17), it's because the appropriate .hcodes had not been established.

This matches results from groff 1.22.4.

$ file hw_test
hw_test: troff or preprocessor input, ISO-8859 text
$ iconv -f iso-8859-1 hw_test
.hcode á á
.hw h\['a]-worth
.ll 5n
Jack H\['a]worth is an agent of destruction.
$ groff --version | head -1
GNU groff version 1.22.4
$ nroff -Wbreak hw_test | cat -s
Jack
Há-
worth
is an
agent
of
de-
struc-
tion.

In fact, unlike .hcode, whose input-character restrictions have long been documented, nothing in .hw's documentation has ever mention such a restriction.  The .hw claim seems to originate in Werner's email cited in comment #0.

Dave <barx>
Group Member
Fri 09 Aug 2024 05:38:38 AM UTC, comment #20: 

This works with simple and composite Unicode-style special character escape sequences too.


$ cat EXPERIMENTS/hw-with-special-character.groff
.\" groff -K utf8
.na
.ll 5n
.hw h\['a]-worth
Jack H\['a]worth is an agent of destruction.
Jack H\[a aa]worth is an agent of destruction.
Jack H\[u00E1]worth is an agent of destruction.
Jack H\[u0061_0301]worth is an agent of destruction.
Jack Háworth is an agent of destruction.
.pl \n[nl]u
$ ./build/test-groff -T utf8 -Wbreak -K utf8 EXPERIMENTS/hw-with-special-character.groff
Jack
Há‐
worth
is an
agent
of
de‐
struc‐
tion.
Jack
Há‐
worth
is an
agent
of
de‐
struc‐
tion.
Jack
Há‐
worth
is an
agent
of
de‐
struc‐
tion.
Jack
Há‐
worth
is an
agent
of
de‐
struc‐
tion.
Jack
Há‐
worth
is an
agent
of
de‐
struc‐
tion.


G. Branden Robinson <gbranden>
Group administrator
Fri 09 Aug 2024 05:33:53 AM UTC, comment #19: 

Evidence of the input from comment #16 working:


$ ./build/test-groff -T utf8 -Wbreak -K utf8 EXPERIMENTS/hw-with-special-character.groff
Jack
Háworth
is an
agent
of
de‐
struc‐
tion.
Jack
Háworth
is an
agent
of
de‐
struc‐
tion.
Jack
Háworth
is an
agent
of
de‐
struc‐
tion.


G. Branden Robinson <gbranden>
Group administrator
Fri 09 Aug 2024 05:32:05 AM UTC, comment #18: 


0629380a9d464c74d6dd205384bc255e1015c2f4 is the first hyphen commit
commit 0629380a9d464c74d6dd205384bc255e1015c2f4
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sat Aug 3 18:36:42 2024 -0500

    tmac/*: Refactor hyphenation code declarations.

    Move `hcode` invocations from language-specific macro files (those that
    define a "groff locale", such as "english" or "german") to the macro
    files that support 8-bit character encodings, because in the formatter,
    hyphenation codes are global, not environment-specific like other
    hyphenation data.  (That design decision is worth reconsidering in the
    future, but feels less urgent than adding UTF-8 input support to the
    formatter, which may obviate some encoding-related issues.)

    * tmac/LOCALIZATION: Relieve groff locale contributor of hyphenation
      code definition duty.
    * tmac/de.tmac: Move Latin-1 hyphenation code definitions from here...
    * tmac/latin1.tmac: ...to here.  Reorganize and cover _every_ letter
      glyph in ISO 8859-1, including Icelandic letters that German
      hyphenation patterns don't require.
    * tmac/ru.tmac: Move KOI8-R hyphenation code definitions from here...
    * tmac/koi8-r.tmac: ...to here.
    * tmac/latin2.tmac: Define Latin-2 hyphenation codes.
    * tmac/latin5.tmac: Define Latin-5 hyphenation codes.  Include the
      notorious special case for the Turkish capital dotted I, and annotate
      it for migration in the event we ever get a "tr.tmac" file, because
      this is a language-specific case mapping, not one strictly implied by
      the character encoding.
    * tmac/latin9.tmac: Define Latin-9 (ISO 8859-15) hyphenation codes.

    Thanks to Werner Lemberg for the discussion in <https://lists.gnu.org/
    archive/html/groff/2024-08/msg00001.html>.

 ChangeLog         | 35 ++++++++++++++++++++
 tmac/LOCALIZATION |  8 ++---
 tmac/de.tmac      | 18 -----------
 tmac/koi8-r.tmac  | 42 +++++++++++++++++++++++-
 tmac/latin1.tmac  | 67 +++++++++++++++++++++++++++++++++++++++
 tmac/latin2.tmac  | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 tmac/latin5.tmac  | 75 ++++++++++++++++++++++++++++++++++++++++++-
 tmac/latin9.tmac  | 77 +++++++++++++++++++++++++++++++++++++++++++-
 tmac/ru.tmac      | 36 +--------------------
 9 files changed, 391 insertions(+), 62 deletions(-)


G. Branden Robinson <gbranden>
Group administrator
Fri 09 Aug 2024 05:23:22 AM UTC, comment #17: 

Whoops.  I can't read my own output.

It was true.

It just recently became false, in Git HEAD only.

I'll bisect why.

G. Branden Robinson <gbranden>
Group administrator
Fri 09 Aug 2024 05:22:25 AM UTC, comment #16: 

original submission:

> `.hw' [is] limited to raw 8bit characters.


This is not true, and seems not to have been true as far back as groff 1.22.3.


$ cat EXPERIMENTS/hw-with-special-character.groff
.\" groff -K utf8
.na
.ll 5n
.hw h\['a]-worth
Jack H\['a]worth is an agent of destruction.
Jack H\[a aa]worth is an agent of destruction.
Jack Háworth is an agent of destruction.
.pl \n[nl]u
$ ~/groff-1.22.3/bin/groff -T utf8 -Wbreak -K utf8 EXPERIMENTS/hw-with-special-character.groff
Jack
Háworth
is an
agent
of
de‐
struc‐
tion.
Jack
Háworth
is an
agent
of
de‐
struc‐
tion.
Jack
Háworth
is an
agent
of
de‐
struc‐
tion.


I get the same results with groff 1.22.4, 1.23.0, and Git HEAD.

G. Branden Robinson <gbranden>
Group administrator
Wed 07 Aug 2024 03:17:44 PM UTC, comment #15: 

This turns out to require rearchitecting the `symbol` class and things that use it.  See bug #66051.  It's hard.

G. Branden Robinson <gbranden>
Group administrator
Thu 01 Aug 2024 08:20:13 PM UTC, comment #14: 

Retitling didn't take.  Trying again.

G. Branden Robinson <gbranden>
Group administrator
Thu 01 Aug 2024 08:17:10 PM UTC, comment #13: 

Since this ticket has prompted a rethink of hyphenation code management, I'm retitling it with a broader scope.

Adding dependencies.

G. Branden Robinson <gbranden>
Group administrator
Mon 29 Jul 2024 06:30:30 PM UTC, comment #12: 

comment #8:

> I cannot reproduce the complaint about `hw` not working with 8-bit characters.


OK, but that's not the complaint.  The complaint is that it requires 8-bit characters when specifying something in the Latin-1 Supplement block, rather than accepting the equivalent escape.

For .hcode, this is demonstrable with the test case I posted in bug #66040.

$ file resume
resume: troff or preprocessor input, ISO-8859 text
$ iconv -f iso-8859-1 resume
.ll 1n
r\['e]sum\['e]
.hcode \['e] \['e]
r\['e]sum\['e]
.hcode \['e] é
r\['e]sum\['e]
$ groff -a -ww -Wbreak resume
<beginning of page>
r<'e>sum<'e>
r<'e>sum<'e>
r<'e><hy>
sum<'e>

Specifically, the complaint is that the second .hcode in the test file has an effect while the first one--nominally equivalent--does not.

Dave <barx>
Group Member
Mon 29 Jul 2024 05:42:33 PM UTC, comment #11: 

I think this report was just mistaken in the first place.

Very few changes were made in "src/roff/troff" in 2014, and those that I see have no bearing on hyphenation as far as I can tell.

Here's a counterexample for the `hcode` complaint.


$ cat EXPERIMENTS/weird-kindergarten.groff
.if r HCODE .hcode @ a
.ll 62n
\c
.nr oldline \n[nl]
.while (\n[oldline] = \n[nl]) kinderg@rten
.pl \n[nl]u
$ ~/groff-HEAD/bin/groff -T utf8 EXPERIMENTS/weird-kindergarten.groff
kinderg@rten     kinderg@rten     kinderg@rten    kinderg@rten
kinderg@rten
$ ~/groff-HEAD/bin/groff -r HCODE=1 -T utf8 EXPERIMENTS/weird-kindergarten.groff
kinderg@rten  kinderg@rten  kinderg@rten  kinderg@rten kinder‐
g@rten
$ ~/groff-stable/bin/groff -T utf8 EXPERIMENTS/weird-kindergarten.groff
kinderg@rten     kinderg@rten     kinderg@rten    kinderg@rten
kinderg@rten
$ ~/groff-stable/bin/groff -r HCODE=1 -T utf8 EXPERIMENTS/weird-kindergarten.groff
kinderg@rten  kinderg@rten  kinderg@rten  kinderg@rten kinder‐
g@rten
$ /usr/bin/groff -T utf8 EXPERIMENTS/weird-kindergarten.groff
kinderg@rten     kinderg@rten     kinderg@rten    kinderg@rten
kinderg@rten
$ /usr/bin/groff -r HCODE=1 -T utf8 EXPERIMENTS/weird-kindergarten.groff
kinderg@rten  kinderg@rten  kinderg@rten  kinderg@rten kinder‐
g@rten
$ ~/groff-1.22.3/bin/groff -T utf8 EXPERIMENTS/weird-kindergarten.groff
kinderg@rten     kinderg@rten     kinderg@rten    kinderg@rten
kinderg@rten
$ ~/groff-1.22.3/bin/groff -r HCODE=1 -T utf8 EXPERIMENTS/weird-kindergarten.groff
kinderg@rten  kinderg@rten  kinderg@rten  kinderg@rten kinder‐
g@rten


Resolving as invalid.

G. Branden Robinson <gbranden>
Group administrator
Mon 29 Jul 2024 05:32:41 PM UTC, comment #10: 

The example in comment #8 works for me with groff 1.22.3 as well.

That was released in November 2014, a mere four months after the mailing list thread linked in comment #1.

It's possible Werner was mistaken, and he misremembered a feature as missing when it was present.

Or he swiftly fixed it for the 1.22.3 release.

G. Branden Robinson <gbranden>
Group administrator
Mon 29 Jul 2024 05:02:37 PM UTC, comment #9: 

The example in comment #8 works for me with groff 1.22.4 and groff 1.23.0.

Possibly Werner fixed this a long time ago?

G. Branden Robinson <gbranden>
Group administrator
Mon 29 Jul 2024 05:01:07 PM UTC, comment #8: 

I cannot reproduce the complaint about `hw` not working with 8-bit characters.


$ cat EXPERIMENTS/escape-kindergarten.groff
.if r HW .hw kin-der-g\[a ad]r-ten
.ll 62n
\c
.nr oldline \n[nl]
.while (\n[oldline] = \n[nl]) kinderg\[a ad]rten
.pl \n[nl]u
$ ./build/test-groff -T utf8 EXPERIMENTS/escape-kindergarten.groff
kindergärten     kindergärten     kindergärten    kindergärten
kindergärten
$ ./build/test-groff -r HW=1 -T utf8 EXPERIMENTS/escape-kindergarten.groff
kindergärten kindergärten kindergärten kindergärten kindergär‐
ten


Looping in original reporter.


G. Branden Robinson <gbranden>
Group administrator
Mon 29 Jul 2024 04:57:13 PM UTC, comment #7: 

In case anyone's curious, here's a demonstration of "raw 8-bit characters" working with `hw`.  I know that's not what's complained about but since I'm digging into this, I thought I would share it.


$ cat EXPERIMENTS/kindergarten-utf8.groff
.if r HW .hw kin-der-gär-ten
.ll 62n
\c
.nr oldline \n[nl]
.while (\n[oldline] = \n[nl]) kindergärten
.pl \n[nl]u
$ iconv -f utf-8 -t iso-8859-1 EXPERIMENTS/kindergarten-utf8.groff >| EXPERIMENTS/kindergarten.groff
$ ./build/test-groff -T utf8 EXPERIMENTS/kindergarten.groff
kindergärten     kindergärten     kindergärten    kindergärten
kindergärten
$ ./build/test-groff -r HW=1 -T utf8 EXPERIMENTS/kindergarten.groff
kindergärten kindergärten kindergärten kindergärten kindergär‐
ten


G. Branden Robinson <gbranden>
Group administrator
Mon 29 Jul 2024 04:16:50 PM UTC, comment #6: 

(at least for `hcode`)

G. Branden Robinson <gbranden>
Group administrator
Mon 29 Jul 2024 04:15:42 PM UTC, comment #5: 


original submission:

> `.hcode' and `.hw' are limited to raw 8bit characters.


This doesn't seem to be the case?


c65ea0c8fb tmac/ps.tmac (Werner LEMBERG      2003-12-22 10:49:55 +0000  68) .fchar \[S ,] \o'S\[ac]'
c65ea0c8fb tmac/ps.tmac (Werner LEMBERG      2003-12-22 10:49:55 +0000  69) .hcode \[S ,]s
c65ea0c8fb tmac/ps.tmac (Werner LEMBERG      2003-12-22 10:49:55 +0000  70) .fchar \[s ,] \o's\[ac]'
c65ea0c8fb tmac/ps.tmac (Werner LEMBERG      2003-12-22 10:49:55 +0000  71) .hcode \[s ,]s


G. Branden Robinson <gbranden>
Group administrator
Mon 02 Nov 2020 09:08:41 AM UTC, comment #4: 

Comment #2 and comment #3, on a related but distinct issue from this bug report's topic, have now spun off into bug #59397.

Dave <barx>
Group Member
Wed 20 Mar 2019 04:28:46 PM UTC, comment #3: 

To address the point raised in comment #2, init_charset_table() in src/roff/troff/input.cpp appears to be what defines the default hcode values, in particular the lines:

  for (int i = 0; i < 256; i++) {
...
    if (csalpha(i))
      charset_table[i]->set_hyphenation_code(cmlower(i));
  }


So the csalpha() call must be returning false for any characters that are ISO-8859-1 (a.k.a. Latin-1) alphabetic characters but outside the ASCII range.

Indeed, a peek into cset_init::cset_init() in src/libs/libgroff/cset.cpp supports this:

  for (int i = 0; i <= UCHAR_MAX; i++) {
    csalpha.v[i] = ISASCII(i) && isalpha(i);
...
  }

The isalpha() call is part of the C standard library's <ctype.h>.  Its return value depends on the current locale.  In groff, which lives in the ISO-8859-1 locale, it's undesirable for this function's behavior to change based on the user's environment; it's for this reason, I presume, that the additional test ISASCII() is imposed, to force non-ASCII characters to return 0 regardless of what isalpha() returns.  And in the ASCII range, isalpha() should function the same no matter the current locale.

But a more robust solution may be to call <ctype.h>'s isalpha_l() instead, so that the ISO-8859-1 locale can be enforced.  By doing this and removing the ISASCII() test (from the csalpha.v[i] line and all the following lines setting other attributes), the character attributes set in cset_init::cset_init() would be accurate for all ISO-8859-1 characters, not just ASCII ones.

This could have implications beyond the hcode values, of course, and I confess I'm not familiar enough with groff's internals to determine what they might be.

Dave <barx>
Group Member
Sun 13 Dec 2015 09:47:28 PM UTC, comment #2: 

Having groff's defaults in alignment would ameliorate this bug.  That is, because latin-1 is the default input encoding, by default groff should assign hyphenation codes to all alphabetic characters in the latin-1 character set, rather than to only ASCII alphabetic characters.

Anonymous
Tue 29 Jul 2014 11:11:18 AM UTC, comment #1: 
Ralph Corderoy <ralph>
Tue 29 Jul 2014 09:28:30 AM UTC, original submission:  

`.hcode' and `.hw' are limited to raw 8bit characters.  Instead, they should accept any characters entities like `\[u00AD]'.  Similarly, `.hpfcode' should be modified to allow entities, too. [1]

With the current implementation it may be required to use latin1 or other non-7-bit coded input files.  utf-8 does not work directly for these requests.  This may limit the portabilty.  Extending these requests to accept codings like \[u00AD] would allow to use standard ASCII input files for all purposes.

[1] See email from Werner LEMBERG to -email is unavailable- on 2014-07-29

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by ralph (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 20 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-08-08 gbranden DependenciesRemoved dependency from bugs #65099 -
    2024-08-01 gbranden Dependencies- Depends on bugs #66054
    2024-08-01 gbranden Dependencies- bugs #65099 is dependent
    2024-08-01 gbranden Summary[troff] `.hcode' and `.hw' are limited to ordinary characters [troff] revise hyphenation code management
    2024-08-01 gbranden DependenciesRemoved dependency to bugs #66040 -
    2024-08-01 gbranden Dependencies- Depends on bugs #59397
    2024-08-01 gbranden Dependencies- Depends on bugs #66051
        Dependencies- Depends on bugs #66040
    2024-07-29 barx StatusInvalid None
        Open/ClosedClosed Open
    2024-07-29 gbranden StatusUnreproducible Invalid
        Assigned tocarstenkunze gbranden
        Open/ClosedOpen Closed
        Summary`.hcode' and `.hw' are limited to raw 8bit characters but should accept any characters entities. [troff] `.hcode' and `.hw' are limited to ordinary characters
    2024-07-29 gbranden StatusNeed Info Unreproducible
        Assigned toNone carstenkunze
    2024-07-29 gbranden StatusNone Need Info
    2020-11-02 barx CategoryNone Core
        Severity3 - Normal 1 - Wish
        Item GroupNone Feature change

    Back to the top

    Powered by Savane 3.15-e6e5.
    Corresponding source code