bugGNU roff - Bugs: bug #55154, .tr has undocumented and...

 
 

bug #55154: .tr has undocumented and inconsistent space-character restrictions

Submitter:  Dave <barx>
Submitted:  Mon 03 Dec 2018 12:18:36 PM UTC
   
 
Category:  Core Severity:  2 - Minor
Item Group:  Incorrect behaviour Status:  Need Info
Privacy:  Public Assigned to:  barx
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 18 Jul 2024 04:54:13 AM UTC, comment #16: 

But zooming out:

Comment #13 was my attempt to test my hypothesis of "why .char doesn't work for this" as presented in comment #11.

This is all tangential to the purpose of this ticket, which covers the .tr request and various types of spaces.  Everything through comment #10 touches on this.

Comment #11 onward is where I veered off course, covering the parallel situation with .char.  This might be more relevant to bug #65829 (which didn't exist till comment #12).

Does this clarify the status for you?

Dave <barx>
Group Member
Wed 17 Jul 2024 10:57:20 PM UTC, comment #15: 

comment #14:

> Try this patch:


Just tried to apply it, but it appears to have already been committed as commit 84bf4520b.

Dave <barx>
Group Member
Thu 11 Jul 2024 05:34:33 PM UTC, comment #14: 

Try this patch:


diff --git a/src/roff/troff/node.cpp b/src/roff/troff/node.cpp
index b00f8377e..2c6cbbec7 100644
--- a/src/roff/troff/node.cpp
+++ b/src/roff/troff/node.cpp
@@ -2183,6 +2183,8 @@ void glyph_node::ascii_print(ascii_output_file *ascii)
     ascii->outs(ci->nm.contents());
 }

+// XXX: This and `composite_node::dump_node()` are identical.  C++
+// presumably has several different solutions for this.  Pick one.
 void glyph_node::dump_node()
 {
   unsigned char c = ci->get_ascii_code();
@@ -4279,6 +4281,7 @@ public:
   bool is_tag();
   void vertical_extent(vunits *, vunits *);
   vunits vertical_width();
+  void dump_node();
 };

 composite_node::composite_node(node *p, charinfo *c, tfont *t, statem *s,
@@ -4918,6 +4921,26 @@ void composite_node::tprint(troff_output_file *out)
     out->right(track_kern);
 }

+// XXX: This and `glyph_node::dump_node()` are identical.  C++
+// presumably has several different solutions for this.  Pick one.
+void composite_node::dump_node()
+{
+  unsigned char c = ci->get_ascii_code();
+  fprintf(stderr, "{type: %s, character: ", type());
+  if (c)
+    fprintf(stderr, "\"%c\"", c);
+  else
+    fprintf(stderr, "\"\\%s\"", ci->nm.contents());
+  fputs(", ", stderr);
+  if (push_state)
+    fprintf(stderr, "push_state, ");
+  if (state)
+    state->display_state();
+  fprintf(stderr, "diversion level: %d", div_nest_level);
+  fputs("}", stderr);
+  fflush(stderr);
+}
+
 static node *make_composite_node(charinfo *s, environment *env)
 {
   int fontno = env_definite_font(env);


Here's your exhibit.


$ ./build/test-groff -z EXPERIMENTS/55154.tr
{type: line_start_node, diversion level: 0},
{type: glyph_node, character: "a", diversion level: 0},
{type: composite_node, character: "b", diversion level: 0},
{type: glyph_node, character: "c", diversion level: 0},
{type: word_space_node, diversion level: 0},
{type: glyph_node, character: "c", diversion level: 0},
{type: composite_node, character: "b", diversion level: 0},
{type: glyph_node, character: "a", diversion level: 0}

{type: line_start_node, diversion level: 0},
{type: glyph_node, character: "a", diversion level: 0},
{type: unbreakable_space_node, diversion level: 0},
{type: glyph_node, character: "c", diversion level: 0},
{type: word_space_node, diversion level: 0},
{type: glyph_node, character: "c", diversion level: 0},
{type: unbreakable_space_node, diversion level: 0},
{type: glyph_node, character: "a", diversion level: 0}


Assigning back to you because I'm not clear where we are on this, or whether the foregoing illuminates anything.

G. Branden Robinson <gbranden>
Group administrator
Tue 11 Jun 2024 06:59:56 PM UTC, comment #13: 

comment #11:

> I presume this is due to this explanation in the Texinfo manual:
>
>  -- Request: .char c ['"'][contents]
>      Every time C is to be output, CONTENTS is processed in a
> temporary environment and the result encapsulated in a node.
>
> The temporary environment being unaware of the rest of the line,
> it can only turn \~ into a node that is the width of an ordinary
> unbreakable space.


The below (requiring a recent groff build, as it uses .pline) is consistent with this hypothesis (though not proof of it, as the precise content of the two composite_node nodes is unknown).

$ cat 55154.tr
.char b \~
abc cba
.pline
.brp
.tm
a\~c c\~a
.pline
.brp
$ groff 55154.tr > /dev/null
{type: line_start_node, diversion level: 0},
{type: glyph_node, character: "a", diversion level: 0},
{type: composite_node, diversion level: 0},
{type: glyph_node, character: "c", diversion level: 0},
{type: word_space_node, diversion level: 0},
{type: glyph_node, character: "c", diversion level: 0},
{type: composite_node, diversion level: 0},
{type: glyph_node, character: "a", diversion level: 0}

{type: line_start_node, diversion level: 0},
{type: glyph_node, character: "a", diversion level: 0},
{type: unbreakable_space_node, diversion level: 0},
{type: glyph_node, character: "c", diversion level: 0},
{type: word_space_node, diversion level: 0},
{type: glyph_node, character: "c", diversion level: 0},
{type: unbreakable_space_node, diversion level: 0},
{type: glyph_node, character: "a", diversion level: 0}


Dave <barx>
Group Member
Mon 03 Jun 2024 05:11:12 AM UTC, comment #12: 

comment #11:

> While everything here appears to be working as designed, I'm
> tempted to open a new bug report anyway,


Succumbed to temptation: bug #65829

Dave <barx>
Group Member
Fri 17 May 2024 05:08:33 AM UTC, comment #11: 

comment #10:

> Bizarrely, while it accepts the second translation, it doesn't
> actually honor it.


It gets worse: even .char fails at this.

$ cat char-test
.char b \~
abc cba\p
$ nroff char-test | cat -s
a c                                                           c a

$

I presume this is due to this explanation in the Texinfo manual:

 -- Request: .char c ['"'][contents]
     Every time C is to be output, CONTENTS is processed in a temporary environment and the result encapsulated in a node.

The temporary environment being unaware of the rest of the line, it can only turn \~ into a node that is the width of an ordinary unbreakable space.

This is frustrating, because it means there is no way within groff to work around bug #62300 (fixed in 1.23.0) for UTF-8 documents that need to work under earlier preconvs.  Another tool has to preprocess the file before preconv gets to it.  While everything here appears to be working as designed, I'm tempted to open a new bug report anyway, because the design thwarts such a seemingly straightforward way to handle pre-1.23 preconv output for the input character U+00A0.

Hopefully I'm wrong about something above, and someone wiser will set me straight.

Dave <barx>
Group Member
Wed 15 May 2024 08:27:45 PM UTC, comment #10: 

original submission:

> .tr a
> .tr b\~
> .tr c\
> .tr d\|
> .tr e\^
> .tr f\0
>
> This attempts to translate six alphabetic characters to six
> different types of space characters.  What it does instead is
> accept the first two translations and reject the last four:


Bizarrely, while it accepts the second translation, it doesn't actually honor it.

$ cat tr-test
.tr b\~
abc cba\p
$ nroff tr-test | cat -s
a c                                                           c a

$

I'm not sure what to think of this.  On the one hand, comment #2 argues this shouldn't work; even the texinfo manual says a .tr target should be a glyph, which \~ isn't.  On the other hand, given that fact, the translation failing outright (as the subsequent lines of the original submission do) would make sense, whereas silently converting a stretchable space to an unstretchable one surely does not.

The deprecation proposed in bug #64337 would make this moot.

Dave <barx>
Group Member
Thu 28 Dec 2023 12:56:45 AM UTC, comment #9: 

comment #7:

> > I don't think \| and \^ are too much of a challenge here.
>
> I recall you've spoken about the font file's ability to alter
> their sizes being a feature you've never seen used in practice,
> and maybe floated the idea of deprecating this as well, but if
> there's a ticket for that proposal, I couldn't find it.


I found the discussion, buried in bug #58930 (in a paragraph beginning "The two exceptions").  You did say you'd never seen this feature used, but came short of calling for its deprecation.  So my half-memory was half correct.

Dave <barx>
Group Member
Sat 24 Jun 2023 03:35:26 PM UTC, comment #8: 

Everything else I wanted to say (here and in bug #64337) I ended up talking myself out of.  All I have left are two minor comments on the documentation update.

comment #6:

> > This feature replaces the odd-parity tr mapping trick


"replaces" implies that the odd-parity trick has gone away.  Maybe "augments" or "parallels"?

> > Translation of a character to \~ is unnecessary.


This seems unnecessarily opinionated for technical documentation.  Arguably all translations are unnecessary.  But they're a useful tool, allowing you to (1) simplify or make more legible your input, or (2) treat certain characters conditionally (e.g. based on the output device).  I'm not sure it's fair to characterize mapping to \~ as any more or less necessary than mapping to any other character.

The goal presumably being to steer users away from a mapping that may be deprecated, saying so directly probably serves the reader better, e.g.: "Translation of a character to \~ may be deprecated in a future release."

Dave <barx>
Group Member
Thu 22 Jun 2023 09:52:37 PM UTC, comment #7: 

comment #6:

> Possibly, we should try deprecating both of these uses of `tr`,
> so that we can say simply that `tr` remaps characters (ordinary
> or special).  Full stop.


Now that this proposal has its own ticket (bug #64337), I'll respond there to the parts of this comment specifically related to that.

> I don't think \| and \^ are too much of a challenge here.
> They're still not characters, and we could introduce new
> directives to the font description file syntax to replace them.


I recall you've spoken about the font file's ability to alter their sizes being a feature you've never seen used in practice, and maybe floated the idea of deprecating this as well, but if there's a ticket for that proposal, I couldn't find it.

Dave <barx>
Group Member
Thu 22 Jun 2023 01:40:17 PM UTC, comment #6: 

I may be groping toward an apology (in the formal, rhetorical, not conversational, sense) in the course of recent plus pending changes to groff_diff(7).

I had forgotten, and thus was semi-surprised, that ".tr a\ " (note trailing space) was not accepted.

But that surprise was only semi- because as James Clark pointed out, ".tr a", more often seen as ".tr ~" is a truly bizarre feature.

I suspect this feature crept in very early, before the "\ " escape sequence, which gets you the same result without having to bother with a translation.

Clark stayed precisely within this non-orthogonality when supporting ".tr a\~".

Possibly, we should try deprecating both of these uses of `tr`, so that we can say simply that `tr` remaps characters (ordinary or special).  Full stop.

I don't think \| and \^ are too much of a challenge here.  They're still not characters, and we could introduce new directives to the font description file syntax to replace them.


hair-space-width 10
thin-space-width 20


I observe that the "\| and \^ in the font description file" trick is not documented or implied in CSTR #54, though it does appear in CSTR #97.  Perhaps by 1992, Kernighan began to think better of it.

Anyway, here's what is in my working copy.

> In GNU troff, the tr request can map characters to the unbreakable
> space escape sequence \~ as a special case (tr normally operates
> only on characters).  This feature replaces the odd-parity tr
> mapping trick used in AT&T troff documents, where a character, often
> ~, was "sacrificed" by mapping it to "nothing", drafting it into use
> as an unadjustable, unbreakable space.  (This feature was gratuitous
> even in early AT&T troff, which supported the \space escape sequence
> by 1976.)  Often, it makes more sense to use GNU troff's \~ escape
> sequence instead, which has been adopted by every other active troff
> implementation except that of Illumos, as well as by the non-troff
> mandoc.  Translation of a character to \~ is unnecessary.


Thoughts?

G. Branden Robinson <gbranden>
Group administrator
Wed 24 May 2023 01:05:52 PM UTC, comment #5: 

A subset of escape sequences interpolates special characters.

\` \' \- \_ \(xx \[xxx]

No escape sequences interpolate ordinary characters.

(I'm ignoring \*(xx and \*[xxx], which interpolate strings potentially containing characters.)

G. Branden Robinson <gbranden>
Group administrator
Wed 24 May 2023 10:56:16 AM UTC, comment #4: 

I also wonder whether this statement in groff.texi should be expanded: "The following characters can't be translated: space (with one exception, see below), backspace, newline, leader (and '\a'), tab (and '\t')."

It is not just ordinary space characters that cannot be translated: it is all spaces (such as \0, \|, \^):

$ echo '.tr \|a' | groff
troff:<standard input>:1: error: expected ordinary or special character, got a horizontal motion

One can argue that the current statement is sufficient:

  1. Comment #2 points out that these other "spaces" are instead actually horizontal motions.
  2. The docs term these "escapes" rather than "characters," and .tr is documented to work only on characters.


But there are potential rebuttals to both of these:

  1. \| and \^ fall into a gray area, as they can be defined as characters in a font file.
  2. The above-quoted list of exceptions already includes two escapes.


I have no firm conclusion yet.

Dave <barx>
Group Member
Thu 07 Jul 2022 09:56:29 AM UTC, comment #3: 

Yes, that makes sense; the documentation changes you speak of (commit bf6ff8f2) refute my "illogically inconsistent" claim in the initial report.

Dave <barx>
Group Member
Wed 06 Jul 2022 06:27:55 AM UTC, comment #2: 

Hi Dave,

With my recent terminological revisions and clearer distinction between "spaces" and (horizontal) "motions", this puzzling behavior becomes more understandable, I think.

Character translation targets have to be input sequences that are constitutive of text.

That includes spaces, which are discardable and may be breakable or not, but not motions, which are never either.

If we pretend that a font's word space is one en wide, and its numerals 1 em wide, then the following inputs are equivalent.


.tr c\ \" translate to escaped space
.tr d\|
.tr e\^
.tr f\0

.tr c\h'1n'
.tr d\h'1m/6u'
.tr e\h'1m/12u'
.tr f\h'1m'


No one attempts to translate characters to horizontal or vertical motions, or crazier things like device control escape sequences.  Thus the acceptance and rejection pattern you see.

Do you buy this?  If so, I can update the document to make the corresponding clarfication.

G. Branden Robinson <gbranden>
Group administrator
Mon 03 Dec 2018 12:53:07 PM UTC, comment #1: 

It turns out this bug tracker has stripped the trailing spaces from my sample code, so pasting it from the bug description won't work.  I've put a correct copy in an attachment.

(file #45569)

Dave <barx>
Group Member
Mon 03 Dec 2018 12:18:36 PM UTC, original submission:  

This bug exists in groff 1.22.2 and in a groff built from the latest source in git.

Consider this groff input.  (The first and third lines end in a space.)

.tr a
.tr b\~
.tr c\
.tr d\|
.tr e\^
.tr f\0

This attempts to translate six alphabetic characters to six different types of space characters.  What it does instead is accept the first two translations and reject the last four:

test:3: normal or special character expected (got `\ ')
test:4: normal or special character expected (got a horizontal space)
test:5: normal or special character expected (got a horizontal space)
test:6: normal or special character expected (got a horizontal space)

The documentation says that the space character cannot be translated, but puts no restrictions on any other character being translated to any type of space character.

And this is not just undocumented but illogically inconsistent: why should a printable character be translatable to one type of space and not another?

Even if groff has worked this way since day one, fixing it won't break backwards compatibility, since such translations were previously disallowed.

Dave <barx>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45569:  test added by barx (45B - application/octet-stream - input file demonstrating the bug)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-07-11 gbranden Assigned toNone barx
    2022-07-06 gbranden StatusNone Need Info
    2021-08-25 barx Summary.tr has undocumented and inconsistennt space-character restrictions .tr has undocumented and inconsistent space-character restrictions
    2021-05-14 barx CategoryNone Core
        Severity3 - Normal 2 - Minor
        Item GroupNone Incorrect behaviour
    2018-12-03 barx Attached File- Added test, #45569

    Back to the top

    Powered by Savane 3.13-b921.
    Corresponding source code