bugGNU roff - Bugs: bug #59397, Assign default .hcode values to...

 
 

bug #59397: Assign default .hcode values to alphabetic characters in groff's default character set

Submitter:  Dave <barx>
Submitted:  Mon 02 Nov 2020 08:59:58 AM UTC
   
 
Category:  Core Severity:  1 - Wish
Item Group:  Feature change Status:  Need Info
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 09 Nov 2020 04:39:15 AM UTC, comment #4: 

comment #2:

> So the example should be
>


> .hcode \['e] \['e]


This is not valid syntax:


$ echo ".hcode \['e] \['e]" | groff -Wall
troff: backtrace: file '<standard input>':1
troff: <standard input>:1: error: hyphenation code must be ordinary character


as documented in the info manual: "A hyphenation code must be a single input character (not a special character)."  Bug #42870 (referenced in comment #0) seeks to remedy this limitation.

Putting a literal Latin-1 é in place of the second \['e] does work, however.

Correcting the syntax is the easy part, though; I don't yet have good answers to the conceptual questions Branden raises in comment #3.

Dave <barx>
Group Member
Sat 07 Nov 2020 06:44:21 AM UTC, comment #3: 

I still have a lot to learn about hyphenation, so feel free to point me to the manuals on this, but I think Bjarni may be mistaken.

The hyphenation patterns we use for English (tmac/hyphen.us, which should probably be hyphen.en), since language, not territory, is the first-order determinant of hyphenation break points) don't have any non-basic-Latin letters in them.

Therefore, for the Latin-1 Supplement letters (or any others) to be treated like their base characters from basic Latin, their hcodes will need to be set as Dave advised.

I think if we do as Bjarni advises, there would be no effect.

So we probably want something like:

.hcode e ['e] e ['E]

...right?

Come to think of it I don't think the documentation is crystal clear here on the ordering.  I think the odd-parity arguments are the codes-that-already-exist when the elements of an argument pair are not identical (establishing a new hyphenation code).

I'll have to dig into the source, or be clued in by the lurking Werner, to figure this out, because I've already processed this part of the manual, and if I can't answer this I don't think it's documented.

Finally I think it might be an open question as to whether letters from outside the basic Latin alphabet should necessarily be hyphenated like their basic Latin "base characters".  And there remains the problem of what to do when, say, the ae or oe ligatures, or the Icelandic eth and thorn letters show up.  Let alone stuff from outside Latin-1 or Latin-9.

It is words, not letters, that get hyphenated, and that process occurs within the context of a language, and...well, you should be able to temporarily change languages. You should be able to do it with a correctly-sequenced set of .hla and .hpf requests.  But you'd probably want to macrofy that.  Exceptions wouldn't get lost if the manual is correct when it says "[h]yphenation exceptions specified with the 'hw' request are associated with the current hyphenation language".

G. Branden Robinson <gbranden>
Group administrator
Sat 07 Nov 2020 12:49:17 AM UTC, comment #2: 


  As shown in the documentation,
(for example "info groff |& less", search for "\.hcode")
each lower case 8-bit character is its own hyphenation code.

  So the example should be


.hcode \['e] \['e]


Bjarni Ingi Gislason <bjarniig>
Thu 05 Nov 2020 05:03:16 PM UTC, comment #1: 

A test case to illustrate the problem:


$ cat test.59397
.hy 4
.ll 1u
resume
r\['e]sum\['e]
$ nroff -Wall test.59397 | uniq
re-
sume
résumé


(The output is the same whether the last line of the input file is expressed as above (using groff escapes for the accented characters), or with Latin-1 characters as "résumé".)

The user can correct the hyphenation of the second word by adding a .hcode request to the top of the file:


.hcode \['e] e


But groff natively accepts Latin-1 as input, so it ought to correctly hyphenate by default words that use any letters from this character set.

Dave <barx>
Group Member
Mon 02 Nov 2020 08:59:58 AM UTC, original submission:  

This is copied wholesale from two comments in bug #42870, both of which I think I wrote, that are only tangentially related to the topic of that bug.  This is really a separate issue deserving of its own report.

The problem


Groff's default input character set, Latin-1, does not align with its default hyphenation codes, which are assigned only to ASCII alphabetic characters.  By default groff should assign hyphenation codes to all alphabetic characters in the Latin-1 character set, to reflect the default input character set.

Analysis


init_charset_table() in src/roff/troff/input.cpp appears to be what defines the default hcode values, in particular the lines:


  for (int i = 0; i < 256; i++) {
...
    if (csalpha(i))
      charset_table[i]->set_hyphenation_code(cmlower(i));
  }


So the csalpha() call must be returning false for any characters that are ISO-8859-1 (a.k.a. Latin-1) alphabetic characters but outside the ASCII range.

Indeed, a peek into cset_init::cset_init() in src/libs/libgroff/cset.cpp supports this:


  for (int i = 0; i <= UCHAR_MAX; i++) {
    csalpha.v[i] = ISASCII(i) && isalpha(i);
...
  }


The isalpha() call is part of the C standard library's <ctype.h>.  Its return value depends on the current locale.  In groff, which operates in the ISO-8859-1 locale, it's undesirable for this function's behavior to change based on the user's environment; it's for this reason, I presume, that the additional test ISASCII() is imposed, to force non-ASCII characters to return 0 regardless of what isalpha() returns.  And in the ASCII range, isalpha() should function the same no matter the current locale.

But a more robust solution may be to call <ctype.h>'s isalpha_l() instead, so that the ISO-8859-1 locale can be enforced.  By doing this and removing the ISASCII() test (from the csalpha.v[i] line and all the following lines setting other attributes), the character attributes set in cset_init::cset_init() would be accurate for all ISO-8859-1 characters, not just ASCII ones.

This could have implications beyond the hcode values, of course, and I confess I'm not familiar enough with groff's internals to determine what they might be.

Dave <barx>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by bjarniig (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-11-28 gbranden StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code