bugGNU roff - Bugs: bug #42870, `.hcode' and `.hw' are limited to...

 
 

bug #42870: `.hcode' and `.hw' are limited to raw 8bit characters but should accept any characters entities.

Submitter:  None
Submitted:  Tue 29 Jul 2014 09:28:30 AM UTC
   
 
Category:  Core Severity:  1 - Wish
Item Group:  Feature change Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 02 Nov 2020 09:08:41 AM UTC, comment #4: 

Comment #2 and comment #3, on a related but distinct issue from this bug report's topic, have now spun off into bug #59397.

Dave <barx>
Group Member
Wed 20 Mar 2019 04:28:46 PM UTC, comment #3: 

To address the point raised in comment #2, init_charset_table() in src/roff/troff/input.cpp appears to be what defines the default hcode values, in particular the lines:

  for (int i = 0; i < 256; i++) {
...
    if (csalpha(i))
      charset_table[i]->set_hyphenation_code(cmlower(i));
  }


So the csalpha() call must be returning false for any characters that are ISO-8859-1 (a.k.a. Latin-1) alphabetic characters but outside the ASCII range.

Indeed, a peek into cset_init::cset_init() in src/libs/libgroff/cset.cpp supports this:

  for (int i = 0; i <= UCHAR_MAX; i++) {
    csalpha.v[i] = ISASCII(i) && isalpha(i);
...
  }

The isalpha() call is part of the C standard library's <ctype.h>.  Its return value depends on the current locale.  In groff, which lives in the ISO-8859-1 locale, it's undesirable for this function's behavior to change based on the user's environment; it's for this reason, I presume, that the additional test ISASCII() is imposed, to force non-ASCII characters to return 0 regardless of what isalpha() returns.  And in the ASCII range, isalpha() should function the same no matter the current locale.

But a more robust solution may be to call <ctype.h>'s isalpha_l() instead, so that the ISO-8859-1 locale can be enforced.  By doing this and removing the ISASCII() test (from the csalpha.v[i] line and all the following lines setting other attributes), the character attributes set in cset_init::cset_init() would be accurate for all ISO-8859-1 characters, not just ASCII ones.

This could have implications beyond the hcode values, of course, and I confess I'm not familiar enough with groff's internals to determine what they might be.

Dave <barx>
Group Member
Sun 13 Dec 2015 09:47:28 PM UTC, comment #2: 

Having groff's defaults in alignment would ameliorate this bug.  That is, because latin-1 is the default input encoding, by default groff should assign hyphenation codes to all alphabetic characters in the latin-1 character set, rather than to only ASCII alphabetic characters.

Anonymous
Tue 29 Jul 2014 11:11:18 AM UTC, comment #1: 
Ralph Corderoy <ralph>
Tue 29 Jul 2014 09:28:30 AM UTC, original submission:  

`.hcode' and `.hw' are limited to raw 8bit characters.  Instead, they should accept any characters entities like `\[u00AD]'.  Similarly, `.hpfcode' should be modified to allow entities, too. [1]

With the current implementation it may be required to use latin1 or other non-7-bit coded input files.  utf-8 does not work directly for these requests.  This may limit the portabilty.  Extending these requests to accept codings like \[u00AD] would allow to use standard ASCII input files for all purposes.

[1] See email from Werner LEMBERG to -email is unavailable- on 2014-07-29

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by ralph (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-11-02 barx CategoryNone Core
        Severity3 - Normal 1 - Wish
        Item GroupNone Feature change

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code