bugGNU roff - Bugs: bug #59397, [troff] want English hyphenation...

 
 

bug #59397: [troff] want English hyphenation codes for Latin-1 Supplement characters

Submitter:  Dave <barx>
Submitted:  Mon 02 Nov 2020 08:59:58 AM UTC
   
 
Category:  Core Severity:  1 - Wish
Item Group:  Feature change Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.24.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 19 Aug 2024 05:14:37 PM UTC, comment #13: 

Comment #10 quoted the bug-fixing commit log but didn't reveal the commit's identity: it's commit 0629380a9d464c74d6dd205384bc255e1015c2f4.

Dave <barx>
Group Member
Fri 09 Aug 2024 07:38:31 AM UTC, comment #12: 

comment #11:

> One step remains for this:


Handled.

Commit 56e793e73.

Dave <barx>
Group Member
Wed 07 Aug 2024 01:41:02 PM UTC, comment #11: 
Dave <barx>
Group Member
Sun 04 Aug 2024 09:16:40 PM UTC, comment #10: 

The original summary of this bug was:

"Assign default .hcode values to alphabetic characters in groff's default character set"

This is done.


2024-08-03  G. Branden Robinson <g.branden.robinson@gmail.com>

        tmac/*: Refactor hyphenation code declarations.

        Move `hcode` invocations from language-specific macro files
        {those that define a "groff locale", such as "english" or
        "german"} to the macro files that support 8-bit character
        encodings, because in the formatter, hyphenation codes are
        global, not environment-specific like other hyphenation data.
        {That design decision is worth reconsidering in the future,
        but feels less urgent than adding UTF-8 input support to the
        formatter, which may obviate some encoding-related issues.}

        * tmac/LOCALIZATION: Relieve groff locale contributor of
        hyphenation code definition duty.
        * tmac/de.tmac: Move Latin-1 hyphenation code definitions from
        here...
        * tmac/latin1.tmac: ...to here.  Reorganize and cover _every_
        letter glyph in ISO 8859-1, including Icelandic letters that
        German hyphenation patterns don't require.
        * tmac/ru.tmac: Move KOI8-R hyphenation code definitions from
        here...
        * tmac/koi8-r.tmac: ...to here.
        * tmac/latin2.tmac: Define Latin-2 hyphenation codes.
        * tmac/latin5.tmac: Define Latin-5 hyphenation codes.  Include
        the notorious special case for the Turkish capital dotted I, and
        annotate it for migration in the event we ever get a "tr.tmac"
        file, because this is a language-specific case mapping, not one
        strictly implied by the character encoding.
        * tmac/latin9.tmac: Define Latin-9 (ISO 8859-15) hyphenation
        codes.

        Fixes <https://savannah.gnu.org/bugs/?59397>.  Thanks to Werner
        Lemberg for the discussion in <https://lists.gnu.org/archive/
        html/groff/2024-08/msg00001.html>.


The stuff Dave raised regarding hyphenation codes for special character escape sequences is within the scope of bug #66051.

G. Branden Robinson <gbranden>
Group administrator
Thu 01 Aug 2024 09:50:49 PM UTC, comment #9: 

Either place is probably better than my suggested modification in comment #0 of defining them in the formatter, which doesn't take into account different .hcode requirements for different languages.

Dave <barx>
Group Member
Thu 01 Aug 2024 08:08:02 PM UTC, comment #8: 

I raised this on the list.  Werner Lemberg liked my idea of moving `hcode` invocations into the encoding macro files instead of the localization ones.

https://lists.gnu.org/archive/html/groff/2024-08/msg00001.html

G. Branden Robinson <gbranden>
Group administrator
Thu 01 Aug 2024 12:16:31 AM UTC, comment #7: 

comment #5:

> groff, it's just going to make its best guess,


Sloppy wording.  The humans setting up the hyphenation patterns and exceptions files are making their best guesses--or predictions, more accurately.  Groff itself merely follows an algorithm that uses that data, no guesswork involved.

Dave <barx>
Group Member
Thu 01 Aug 2024 12:15:26 AM UTC, comment #6: 

Retitling.

The following specimen works the same on groff 1.22.3, 1.22.4, 1.23.0, and Git HEAD.


$ cat EXPERIMENTS/hw-with-special-characters.groff
.ll 1n
r\['e]sum\['e]
.hcode \['e]e
r\['e]sum\['e]
.pl \n[nl]u
$ ~/groff-1.22.3/bin/groff -ww -W break -T utf8 EXPERIMENTS/hw-with-special-characters.groff
résumé
ré‐
sumé


This is good fodder for a future regression test (in order words, it works now, but could conceivably break in future refactorings).  So I'll probably add a test based the foregoing.

My plan for groff 1.24 had been to not load "latin1.tmac" by default anymore, to get groff users shifted to ISO 646/"ASCII" by default so that we can compatibly change the input character encoding to UTF-8 for groff 1.25.  I'm not sure how that plan interacts with the above; maybe it doesn't.

G. Branden Robinson <gbranden>
Group administrator
Thu 01 Aug 2024 12:04:16 AM UTC, comment #5: 

comment #3:

> I think it might be an open question as to whether letters
> from outside the basic Latin alphabet should necessarily be
> hyphenated like their basic Latin "base characters".


There's a little fuzz in any automated hyphenation system.  When encountering the string "project", groff can't know whether it's the noun, which is broken "proj-ect", or the verb, which would be "pro-ject".  An LLM could probably figure it out, but short of integrating one of those into groff, it's just going to make its best guess, and rely on the user to override it if it's wrong.

On the other hand, when a diacritic changes the syllabication, such as "expose" vs "exposé", it will pretty much (I hedge, but can't think of any exceptions) always do so by adding a syllable, and thus a potential break point.  The patterns, presumably, are set up for the unaccented form, meaning groff will never use the additional break point offered by the accented form.  But that's fine: it's better to not break a word in an acceptable spot than to break one in an unacceptable spot.

And anyway, those are the rarer cases.  More commonly, the break points won't change, such as whether "coöperate", "doppelgänger", or "débâcle" are written with or without the diacritics.

But in order for any of this to work, the adorned letters need hyphenation codes, which they don't have by default, hence this ticket.

Dave <barx>
Group Member
Mon 09 Nov 2020 04:39:15 AM UTC, comment #4: 

comment #2:

> So the example should be
>


> .hcode \['e] \['e]


This is not valid syntax:


$ echo ".hcode \['e] \['e]" | groff -Wall
troff: backtrace: file '<standard input>':1
troff: <standard input>:1: error: hyphenation code must be ordinary character


as documented in the info manual: "A hyphenation code must be a single input character (not a special character)."  Bug #42870 (referenced in comment #0) seeks to remedy this limitation.

Putting a literal Latin-1 é in place of the second \['e] does work, however.

Correcting the syntax is the easy part, though; I don't yet have good answers to the conceptual questions Branden raises in comment #3.

Dave <barx>
Group Member
Sat 07 Nov 2020 06:44:21 AM UTC, comment #3: 

I still have a lot to learn about hyphenation, so feel free to point me to the manuals on this, but I think Bjarni may be mistaken.

The hyphenation patterns we use for English (tmac/hyphen.us, which should probably be hyphen.en), since language, not territory, is the first-order determinant of hyphenation break points) don't have any non-basic-Latin letters in them.

Therefore, for the Latin-1 Supplement letters (or any others) to be treated like their base characters from basic Latin, their hcodes will need to be set as Dave advised.

I think if we do as Bjarni advises, there would be no effect.

So we probably want something like:

.hcode e ['e] e ['E]

...right?

Come to think of it I don't think the documentation is crystal clear here on the ordering.  I think the odd-parity arguments are the codes-that-already-exist when the elements of an argument pair are not identical (establishing a new hyphenation code).

I'll have to dig into the source, or be clued in by the lurking Werner, to figure this out, because I've already processed this part of the manual, and if I can't answer this I don't think it's documented.

Finally I think it might be an open question as to whether letters from outside the basic Latin alphabet should necessarily be hyphenated like their basic Latin "base characters".  And there remains the problem of what to do when, say, the ae or oe ligatures, or the Icelandic eth and thorn letters show up.  Let alone stuff from outside Latin-1 or Latin-9.

It is words, not letters, that get hyphenated, and that process occurs within the context of a language, and...well, you should be able to temporarily change languages. You should be able to do it with a correctly-sequenced set of .hla and .hpf requests.  But you'd probably want to macrofy that.  Exceptions wouldn't get lost if the manual is correct when it says "[h]yphenation exceptions specified with the 'hw' request are associated with the current hyphenation language".

G. Branden Robinson <gbranden>
Group administrator
Sat 07 Nov 2020 12:49:17 AM UTC, comment #2: 


  As shown in the documentation,
(for example "info groff |& less", search for "\.hcode")
each lower case 8-bit character is its own hyphenation code.

  So the example should be


.hcode \['e] \['e]


Bjarni Ingi Gislason <bjarniig>
Thu 05 Nov 2020 05:03:16 PM UTC, comment #1: 

A test case to illustrate the problem:


$ cat test.59397
.hy 4
.ll 1u
resume
r\['e]sum\['e]
$ nroff -Wall test.59397 | uniq
re-
sume
résumé


(The output is the same whether the last line of the input file is expressed as above (using groff escapes for the accented characters), or with Latin-1 characters as "résumé".)

The user can correct the hyphenation of the second word by adding a .hcode request to the top of the file:


.hcode \['e] e


But groff natively accepts Latin-1 as input, so it ought to correctly hyphenate by default words that use any letters from this character set.

Dave <barx>
Group Member
Mon 02 Nov 2020 08:59:58 AM UTC, original submission:  

This is copied wholesale from two comments in bug #42870, both of which I think I wrote, that are only tangentially related to the topic of that bug.  This is really a separate issue deserving of its own report.

The problem


Groff's default input character set, Latin-1, does not align with its default hyphenation codes, which are assigned only to ASCII alphabetic characters.  By default groff should assign hyphenation codes to all alphabetic characters in the Latin-1 character set, to reflect the default input character set.

Analysis


init_charset_table() in src/roff/troff/input.cpp appears to be what defines the default hcode values, in particular the lines:


  for (int i = 0; i < 256; i++) {
...
    if (csalpha(i))
      charset_table[i]->set_hyphenation_code(cmlower(i));
  }


So the csalpha() call must be returning false for any characters that are ISO-8859-1 (a.k.a. Latin-1) alphabetic characters but outside the ASCII range.

Indeed, a peek into cset_init::cset_init() in src/libs/libgroff/cset.cpp supports this:


  for (int i = 0; i <= UCHAR_MAX; i++) {
    csalpha.v[i] = ISASCII(i) && isalpha(i);
...
  }


The isalpha() call is part of the C standard library's <ctype.h>.  Its return value depends on the current locale.  In groff, which operates in the ISO-8859-1 locale, it's undesirable for this function's behavior to change based on the user's environment; it's for this reason, I presume, that the additional test ISASCII() is imposed, to force non-ASCII characters to return 0 regardless of what isalpha() returns.  And in the ASCII range, isalpha() should function the same no matter the current locale.

But a more robust solution may be to call <ctype.h>'s isalpha_l() instead, so that the ISO-8859-1 locale can be enforced.  By doing this and removing the ISASCII() test (from the csalpha.v[i] line and all the following lines setting other attributes), the character attributes set in cset_init::cset_init() would be accurate for all ISO-8859-1 characters, not just ASCII ones.

This could have implications beyond the hcode values, of course, and I confess I'm not familiar enough with groff's internals to determine what they might be.

Dave <barx>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by bjarniig (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-08-04 gbranden StatusNone Fixed
        Assigned toNone gbranden
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.24.0
    2024-08-01 gbranden Dependencies- bugs #42870 is dependent
    2024-08-01 barx StatusNeed Info None
        SummaryAssign default .hcode values to alphabetic characters in groff's default character set [troff] want English hyphenation codes for Latin-1 Supplement characters
    2024-08-01 barx StatusNone Need Info
        Summary[troff] want English hyphenation codes for Latin-1 Supplement characters Assign default .hcode values to alphabetic characters in groff's default character set
    2024-08-01 gbranden StatusNeed Info None
        SummaryAssign default .hcode values to alphabetic characters in groff's default character set [troff] want English hyphenation codes for Latin-1 Supplement characters
    2020-11-28 gbranden StatusNone Need Info

    Back to the top

    Powered by Savane 3.15-e6e5.
    Corresponding source code