bugGNU roff - Bugs: bug #66051, [troff] permit special characters...

 
 

bug #66051: [troff] permit special characters to have bespoke hyphenation codes

Submitter:  G. Branden Robinson <gbranden>
Submitted:  Wed 31 Jul 2024 09:08:38 PM UTC
   
 
Category:  Core Severity:  1 - Wish
Item Group:  Feature change Status:  Postponed
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 07 Aug 2024 03:13:20 PM UTC, comment #7: 

Rut-roh, Raggy.

So, yeah, let's put in the missing bits.


diff --git a/src/roff/troff/input.cpp b/src/roff/troff/input.cpp
index 44e0981f6..6d9ee85c6 100644
--- a/src/roff/troff/input.cpp
+++ b/src/roff/troff/input.cpp
@@ -7275,12 +7275,15 @@ static void set_character_flags()
   skip_line();
 }

+// We use this to mint hyphenation codes for special characters.
+static int hyphenation_code_counter = 256;
+
 static void set_hyphenation_codes()
 {
   tok.skip();
   if (tok.is_newline() || tok.is_eof()) {
-    warning(WARN_MISSING, "hyphenation code configuration request"
-           " expects arguments");
+    warning(WARN_MISSING, "hyphenation code assignment request expects"
+           " arguments");
     skip_line();
     return;
   }
@@ -7309,24 +7312,38 @@ static void set_hyphenation_codes()
       error("cannot use the hyphenation code of a numeral");
       break;
     }
-    unsigned char new_code = 0; // TODO: int
+    int new_code = 0;
     charinfo *cisrc = tok.get_char();
-    if (csrc != 0)
-      new_code = csrc;
-    else {
+    if (cisrc != 0 /* nullptr */)
+      // Common case: assign destination character the hyphenation code
+      // of another character that already has one.
+      new_code = cisrc->get_hyphenation_code();
+    if (0 == csrc) {
       if (0 /* nullptr */ == cisrc) {
        error("expected ordinary or special character, got %1",
              tok.description());
        break;
       }
        break;
       }
       // source character is special
-      if (0 == cisrc->get_hyphenation_code()) {
-       error("second member of hyphenation code pair must be an"
-             " ordinary character, or a special character already"
-             " assigned a hyphenation code");
-       break;
+      if (strcmp(cidst->nm.contents(), cisrc->nm.contents()) == 0) {
+       debug("GBR: mint new");
+       if (hyphenation_code_counter == INT_MAX) {
+         error("supply of hyphenation codes exhausted");
+         break;
+       }
+       new_code = hyphenation_code_counter++;
+       debug("GBR: new code is %1", int(new_code));
       }
-      new_code = cisrc->get_hyphenation_code();
+      else {
+       debug("GBR: copy code");
+       new_code = cisrc->get_hyphenation_code();
+      }
+    }
+    else {
+      // If assigning a code to itself, use its character encoding value
+      // to mint a new code.
+      if (csrc == cdst)
+       new_code = tok.ch();
     }
     cidst->set_hyphenation_code(new_code);
     if (cidst->get_translation()


No problem.  Compiles fine.

Let's test it.


$ nl EXPERIMENTS/phcode-works.groff
     1  .phcode a
     2  .phcode A
     3  .phcode $
     4  .tm invoking .hcode \['a] $
     5  .hcode \['a] $
     6  .phcode \['a]
     7  .tm invoking .hcode \['a] \['a]
     8  .hcode \['a] \['a]
     9  .phcode \['a]
    10  .tm invoking .hcode \['a] \['A]
    11  .hcode \['a] \['A]
    12  .phcode \['a]
    13  .phcode \[vS]
$ ./build/test-groff EXPERIMENTS/phcode-works.groff
a       97
A       97
$       0
invoking .hcode \['a] $
\['a]   0
invoking .hcode \['a] \['a]
troff:EXPERIMENTS/phcode-works.groff:8: debug: GBR: mint new
troff:EXPERIMENTS/phcode-works.groff:8: debug: GBR: new code is 256
\['a]   0
invoking .hcode \['a] \['A]
troff:EXPERIMENTS/phcode-works.groff:11: debug: GBR: copy code
\['a]   225
\[vS]   0


"new code is 256"--great!

Next line...


\['a]   0


Wha..?  Well, yeah.  Because the hyphenation code is a `char`, so when we assigned an `int` to it, the upper bits got masked off, and this isn't warned about because pfffffffft, we're C/C++ programmers--we live close to the metal where no one ever worries about narrowing type conversions.  😐

All right, fine, a hyphenation code is just a numeric type.  Let's widen it.


diff --git a/src/roff/troff/charinfo.h b/src/roff/troff/charinfo.h
index 02be5aa95..6b05c1dc2 100644
--- a/src/roff/troff/charinfo.h
+++ b/src/roff/troff/charinfo.h
@@ -76,7 +76,7 @@ public:
   int prohibit_break_before();
   int prohibit_break_after();
   int inter_char_space();
-  unsigned char get_hyphenation_code();
+  int get_hyphenation_code();
   unsigned char get_ascii_code();
   unsigned char get_asciify_code();
   int get_unicode_code();
@@ -212,7 +212,7 @@ inline charinfo *charinfo::get_translation(int transparent_throughput)
          : translation);
 }

-inline unsigned char charinfo::get_hyphenation_code()
+inline int charinfo::get_hyphenation_code()
 {
   return hyphenation_code;
 }
diff --git a/src/roff/troff/env.cpp b/src/roff/troff/env.cpp
index ca69e38fa..8d80ad2ba 100644
--- a/src/roff/troff/env.cpp
+++ b/src/roff/troff/env.cpp
@@ -3634,7 +3634,7 @@ static void add_hyphenation_exceptions()
     skip_line();
     return;
   }
-  char buf[WORD_MAX + 1];
+  int buf[WORD_MAX + 1];
   unsigned char pos[WORD_MAX + 2];
   for (;;) {
     tok.skip();
@@ -3655,7 +3655,7 @@ static void add_hyphenation_exceptions()
          pos[npos++] = i;
       }
       else {
-       unsigned char c = ci->get_hyphenation_code();
+       int c = ci->get_hyphenation_code();
        if (c == 0)
          break;
        buf[i++] = c;
@@ -3664,9 +3664,9 @@ static void add_hyphenation_exceptions()
     if (i > 0) {
       pos[npos] = 0;
       buf[i] = 0;
-      unsigned char *tem = new unsigned char[npos + 1];
+      int *tem = new int[npos + 1];
       memcpy(tem, pos, npos + 1);
-      tem = (unsigned char *)current_language->exceptions.lookup(symbol(buf),
+      tem = (int *)current_language->exceptions.lookup(symbol(buf),
                                                                 tem);
       if (tem)
        delete[] tem;


Easy-peasy.  Now...


$ make -C build troff
make: Entering directory '/home/branden/src/GIT/groff/build'
  CXX      src/roff/troff/env.o
../src/roff/troff/env.cpp: In function ‘void add_hyphenation_exceptions()’:
../src/roff/troff/env.cpp:3669:66: error: no matching function for call to ‘symbol::symbol(int [257])’
 3669 |       tem = (int *)current_language->exceptions.lookup(symbol(buf),
      |                                                                  ^
In file included from ../src/include/color.h:22,
                 from ../src/roff/troff/troff.h:32,
                 from ../src/roff/troff/env.cpp:19:
../src/include/symbol.h:46:8: note: candidate: ‘symbol::symbol()’
   46 | inline symbol::symbol() : s(0)
      |        ^~~~~~
../src/include/symbol.h:46:8: note:   candidate expects 0 arguments, 1 provided
../src/include/symbol.h:32:3: note: candidate: ‘symbol::symbol(const char*, int)’
   32 |   symbol(const char *p, int how = 0);
      |   ^~~~~~
../src/include/symbol.h:32:22: note:   no known conversion for argument 1 from ‘int [257]’ to ‘const char*’
   32 |   symbol(const char *p, int how = 0);
      |          ~~~~~~~~~~~~^
../src/include/symbol.h:24:7: note: candidate: ‘constexpr symbol::symbol(const symbol&)’
   24 | class symbol {
      |       ^~~~~~
../src/include/symbol.h:24:7: note:   no known conversion for argument 1 from ‘int [257]’ to ‘const symbol&’
../src/include/symbol.h:24:7: note: candidate: ‘constexpr symbol::symbol(symbol&&)’
../src/include/symbol.h:24:7: note:   no known conversion for argument 1 from ‘int [257]’ to ‘symbol&&’
make: *** [Makefile:8689: src/roff/troff/env.o] Error 1
make: Leaving directory '/home/branden/src/GIT/groff/build'


All right, it's our old friend the `symbol` class, which is used all over the place.

Suddenly following this thread around the corner reveals the hulking beast known as "change groff's internal character representation to a wide type", which is also a prerequisite for internal Unicode support.

So, uh, this one's screeching to a halt for 1.24.

I'll land what I can (without changing any classes), declare defeat, and move on to the next formatter change.

G. Branden Robinson <gbranden>
Group administrator
Wed 31 Jul 2024 11:13:15 PM UTC, comment #6: 

comment #4:

> Whereas this one will make possible:
>
> .hcode \[ss] \[ss]
>
> The fix I have in mind for bug #42780 does not.


s/bug #42780/bug #42870/

If ultimately both are to be fixed, I suppose it's academic how the individual tickets get scoped.  But I don't read anything in the #42870 description that would rule out the .hcode line above.  (Werner even contemplates entities of the form \[u00AD], which would let .hcode process text that preconv has already gotten its mitts on--one element of the real-world problem in the email thread that spawned the ticket.)

Dave <barx>
Group Member
Wed 31 Jul 2024 10:44:28 PM UTC, comment #5: 

Maybe bug #42870's future is to become a bug that depends on more recent ones.  And in that sense perhaps this one really is "more modest".

G. Branden Robinson <gbranden>
Group administrator
Wed 31 Jul 2024 10:43:38 PM UTC, comment #4: 

comment #2:

> It makes total sense and also seems a duplicate of bug #42870.  I guess I'm not seeing what's "more modest" about that one.


The fix I have in mind for bug #42870 will address (I have it pending, with passing regression test) this issue:


input=".
.ec @
.ll 1n
r@['e]sum@['e]
.hcode @['e] e
r@['e]sum@['e]
.hcode @['E] @['e]
R@['E]SUM@['E]
.pl @n[nl]u
."

output=$(echo "$input" | "$groff" -a -ww -Wbreak)
echo "$output"

# Expected output:
#
# <beginning of page>
# r<'e>sum<'e>
# r<'e><hy>
# sum<'e>
# R<'E><hy>
# SUM<'E>


Whereas this one will make possible:


.hcode \[ss] \[ss]


The fix I have in mind for bug #42780 does not.

> If anything, the way it's worded, this one seems more modest, as it's asking for support of only special characters, and only for .hcode, whereas #42870 (per its original submission wording) asks for support of "any characters entities," for .hw and .hcode.


I may still be having trouble understanding completely what that one's asking for, and/or whether making the change here will cause the desired `hw` functionality to just "shake out" or not.

I'll need another test for that.

G. Branden Robinson <gbranden>
Group administrator
Wed 31 Jul 2024 10:42:57 PM UTC, comment #3: 

comment #2:

> #42870 (per its original submission wording)


...wording that, by the way, comes originally from Werner: http://lists.gnu.org/r/groff/2014-07/msg00127.html

Dave <barx>
Group Member
Wed 31 Jul 2024 10:38:21 PM UTC, comment #2: 

It makes total sense and also seems a duplicate of bug #42870.  I guess I'm not seeing what's "more modest" about that one.

If anything, the way it's worded, this one seems more modest, as it's asking for support of only special characters, and only for .hcode, whereas #42870 (per its original submission wording) asks for support of "any characters entities," for .hw and .hcode.

Dave <barx>
Group Member
Wed 31 Jul 2024 09:11:07 PM UTC, comment #1: 

Added Dave to CC list; Dave, please comment if I'm making sense, and especially if I'm not.

G. Branden Robinson <gbranden>
Group administrator
Wed 31 Jul 2024 09:08:38 PM UTC, original submission:  

This idea is a descendant of bug #42870, which asked for something a little more modest.

The concept is this.

If we can do this...


.hcode ß ß


...why can't we do this?


.hcode \[ss] \[ss]


This has long produced a diagnostic.


$ printf '.hcode \\[ss] \\[ss]\n' | ~/groff-stable/bin/groff
troff:<standard input>:1: error: hyphenation code must be ordinary character


I suggested an answer in bug #66040, comment 9.

> Because the formatter doesn't know what [hyphenation code] value to give [the special character].  Under the hood, [a hyphenation code] is just a character code--in other words, on an ISO 8859 system, the hyphenation codes for 'a' through 'z' are 97 through 122--but our documentation stands on its head to avoid saying that.  The trouble is that there is a potentially larger space of sui generis special characters, by which I mean ones that don't belong to an equivalence class of a Basic Latin letter.  [... The] German Eszett [for example] is not.  If we had an Icelandic locale, thorn and eth would similarly have to have hyphenation codes above 127 decimal.
>
> The real fun comes when you add letters from multiple ISO 8859 character sets.  Before long you're going to have collisions.
>
> So it's good that our documentation does the headstand.  We should not disclose what the hyphenation code values are, we need only to ensure that they sort into the correct equivalence classes, so that they then interoperate as desired with the hyphenation patterns.
>
> When we get support for UTF-8-encoded hyphenation pattern files, things will become straightforward again.
>
> In the meantime, what I think I will do is use a `static int` to mint a sequence number (starting at 256) for hyphenation codes any time a special character needs one sui generis.

G. Branden Robinson <gbranden>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by gbranden (discussant of Savannah #42870 and #66040)
  • -email is unavailable- added by gbranden (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-08-07 gbranden StatusNone Postponed
    2024-08-01 gbranden Dependencies- bugs #42870 is dependent
    2024-07-31 gbranden Carbon-Copy- Added barx

    Back to the top

    Powered by Savane 3.14-3b9d.
    Corresponding source code