bugGNU roff - Bugs: bug #66481, [troff] `\w|x|` no longer works on...

 
 

bug #66481: [troff] `\w|x|` no longer works on the bleeding edge of Git

Submitter:  Paul Eggert <eggert>
Submitted:  Sun 24 Nov 2024 02:08:29 AM UTC
   
 
Category:  Core Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.24.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 28 Jan 2025 08:38:20 AM UTC, comment #11: 


commit 39c1176bfa9ba84d519b015b1453f316e013d1de
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jan 27 18:52:39 2025 -0600

    [troff]: Fix Savannah #66686 (`\w|foo|` blues).

    * src/roff/troff/input.cpp (is_char_usable_as_delimiter): Restore `|`
      character as an invalid delimiter when not in compatibility mode.
      This would regress the fix for Savannah #66481, but...

      (do_overstrike, do_bracket, do_name_test, do_zero_width_output)
      (read_size, do_register, do_width, do_device_extension)
      (read_drawing_command): Throw warning in "delimiter" category and
      explain ambiguity of delimiter instead of emitting error and refusing
      further interpretation of the escape sequence being parsed.  Leave
      behind "#if"ed code for restoration of former stricter behavior in a
      future groff release (which would fix Savannah #66009).

      (is_conditional_expression_true): Stop special-casing an exception for
      `|` that permitted it to be used as a formatted output comparison
      operator.  Savannah #66481 complained only about groff's rejection of
      `|` to delimit the argument to the `\w` (width measurement) escape
      sequence, not in general, and was seen in some man pages.  The usage
      Paul Eggert reported remains accepted, albeit warned about, per
      `do_width()` above.

    * src/roff/groff/tests/check-delimiter-validity.sh: Update test
      expectations.  We now expect `|` to be invalid once again to delimit a
      line-drawing escape sequence.

    Fixes <https://savannah.gnu.org/bugs/?66686>.  Thanks to Dave Kemper for
    the report.  Savannah #66526 is implicated.

    The uses of `\w|whatever|` cited by Eggert appear to be a chunk of
    boilerplate passed around by some man page authors for GNU man pages.
    Eggert's "tz" package doesn't itself use `|` as a delimiter, but I did
    verify their use in gawk, grep, and rcs.

    gawk:
    .if !\n(.g \{\
    .       if !\w|\*(lq| \{\
    .               ds lq ``
    .               if \w'\(lq' .ds lq "\(lq
    .       \}
    .       if !\w|\*(rq| \{\
    .               ds rq ''
    .               if \w'\(rq' .ds rq "\(rq
    .       \}
    .\}

    grep:
    .if !\w|\*(lq| \{\
    .\" groff an-old.tmac does not seem to be in use, so define lq and rq.
    .       ie \n(.g \{\
    .               ds lq \(lq\"
    .               ds rq \(rq\"
    .       \}
    .       el \{\
    .               ds lq ``
    .               ds rq ''
    .       \}
    .\}

    rcs:
    .if !\n(.g \{\
    .       if !\w|\*(lq| \{\
    .               ds lq ``
    .               if \w'\(lq' .ds lq "\(lq
    .       \}
    .       if !\w|\*(rq| \{\
    .               ds rq ''
    .               if \w'\(rq' .ds rq "\(rq
    .       \}
    .\}

    One observes that only grep(1) doesn't feature-gate its use of `\w|foo|`
    behind a test of the `.g` register (groff-compatible formatter) for
    falsity.  It is therefore the only one of these three that would have
    provoked a warning.  (What the foregoing do is reimplement the `lq` and
    `rq` quotation strings, an extension from 4BSD (1980)--my guess is to
    compensate for System V-based troffs missing them).

    The rcs man pages otherwise use `\w` pretty extensively (which can be
    its own portability worry), but in every other case with the unambiguous
    `'` delimiter.

commit 1eed3cceaaa38e095e2b265cda846842853525e4
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Tue Jan 28 00:57:03 2025 -0600

    [doc,man]: Report new delimiter warning scenario.


G. Branden Robinson <gbranden>
Group administrator
Sat 30 Nov 2024 09:07:49 AM UTC, comment #10: 


commit d052cb31d9982ef2ad1d776d828bd4370ce5e43e
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Nov 25 17:13:56 2024 -0600

    [troff]: Fix Savannah #66481 and unfix #66099.

    * src/roff/troff/input.cpp (is_char_usable_as_delimiter): Revert fix for
      bug #66009.  Unfortunately, `|` is in use in the wild as a delimiter,
      for instance in man pages for GNU awk, GNU grep, and GNU rcs.  Weaning
      people off of it (because it is a valid character in a numeric
      expression, and GNU troff has never accepted most other such
      characters as delimeters,{*} whereas AT&T troff accepted them all)
      looks to be a multi-stage, multi-year process.

    Fixes <https://savannah.gnu.org/bugs/?66481>.  Thanks to Paul Eggert for
    the report.

    {*} For distorted values of "most"--both GNU and AT&T troffs accept any
        basic Latin letter ([A-Za-z]) as a delimiter, a collection of 52
        exceptions that quantitatively swallows the rule.  Pragmatically,
        few *roff document authors past or present seem to have been
        adventurous enough to exercise this freedom.


G. Branden Robinson <gbranden>
Group administrator
Mon 25 Nov 2024 10:26:19 PM UTC, comment #9: 

Hi Paul,

At 2024-11-25T15:09:02-0500, Paul Eggert wrote:

> Follow-up Comment #8, bug #66481 (group groff):
> > +  case '|':
> > +    error("support for '|' as an argument delimiter is deprecated and"
> > +         " will be withdrawn in a future release");
>
> A quick survey of what's installed on my desktop suggests that this
> will cause diagnostics to be issued for the man pages for gawk, grep,
> and rcs.


I'm glad to hear it's not worse!  This bug report has caused me some
disquiet.

> The uses I see are at the top level, so how about if groff issues the
> warning only when \w is nested inside some other quoted construct? I'd
> rather not have to go through those long-working man pages to change
> '|' to some other character. And there should be no problem with \w|X|
> when it's not nested.


As I understand the code, this may be possible--there is a member
function `input_stack::get_level()` that I think would facilitate this.

However, having interpolation-depth-dependent grammar seems almost as
bad to me as having ambiguous grammar.

Here's the solution I'm considering now, having slept on the problem:

1.  Go back to accepting `|` for groff 1.24, without diagnostics.
2.  We're planning on adding a `-wstyle` option to GNU troff for
    groff 1.25 (bug #62776).  This can become one.  That way people
    can run groff 1.25 with that option over corpora of man pages and
    see where this problem shows up.  I (and fingers crossed, others)
    start submitting patches to affected man pages.
3.  Stick the above-quoted deprecation message in for groff 1.26.
4.  Withdraw support for `|` as a delimiter in groff 1.27 (but see
    next paragraph).

Relatedly but distinguishably, it looks to me like I can make GNU
troff more AT&T-compatible in compatibility mode (the `-C` option)
by skipping this entire `switch` statement when that mode is enabled.
I'm inclined to make that change adjacently to this one (item 1 above).

While I'm keen to reform *roff grammar in ways that sand down the warts
and sharp edges, I also want GNU troff to render well documents
prepared with AT&T troff in mind, as far as practicable.  (I believe I
am following in James Clark's tradition by not, in general, aiming for
bug-compatibility or identical indulgence of undefined behavior.)

> PS. I see that some traditional troff macro libraries in Solaris 10
> /usr/lib/tmac use control-G instead of ', under the theory that user
> strings never contain control-G.


Yes.  It was a strategem that was (a) unergonomic, (b) esoteric, and (c)
inadequate to the purpose.  If a finite-state automaton could simulate a
pushdown automaton, Chomsky would have told us so.

> I'd hate to have to do that sort of thing.


Fear not--I don't wish to impose that sort of yuckiness on anyone.

Regards,
Branden

G. Branden Robinson <gbranden>
Group administrator
Mon 25 Nov 2024 08:09:00 PM UTC, comment #8: 


> +  case '|':
> +    error("support for '|' as an argument delimiter is deprecated and"
> +         " will be withdrawn in a future release");


A quick survey of what's installed on my desktop suggests that this will cause diagnostics to be issued for the man pages for gawk, grep, and rcs.

The uses I see are at the top level, so how about if groff issues the warning only when \w is nested inside some other quoted construct? I'd rather not have to go through those long-working man pages to change '|' to some other character. And there should be no problem with \w|X| when it's not nested.

PS. I see that some traditional troff macro libraries in Solaris 10 /usr/lib/tmac use control-G instead of ', under the theory that user strings never contain control-G. I'd hate to have to do that sort of thing.

Paul Eggert <eggert>
Mon 25 Nov 2024 05:35:00 AM UTC, comment #7: 

comment #6:

> In "fallbacks.tmac" we have the following.
>


> .  fchar \[u2012] \^\v'-.3m'\l'\w"\0"u'\v'+.3m'\^\" figure dash


Bug #63354 proposes to overhaul this definition.

(which I note for completeness, not because it affects your point about nesting escapes)

Dave <barx>
Group Member
Mon 25 Nov 2024 04:35:28 AM UTC, comment #6: 

My concern is a real-world one, and groff itself trips over it.

Exhibit:

In "fallbacks.tmac" we have the following.


.  fchar \[u2012] \^\v'-.3m'\l'\w"\0"u'\v'+.3m'\^\" figure dash


That's a "general argument" (`\w`) inside a (partially) "numeric argument" (`\l`).

Are people willing to live with this proposed patch?


diff --git a/src/roff/troff/input.cpp b/src/roff/troff/input.cpp
index 61029bab6..36eecdc3f 100644
--- a/src/roff/troff/input.cpp
+++ b/src/roff/troff/input.cpp
@@ -2609,8 +2609,11 @@ static bool is_char_usable_as_delimiter(int c)
   case '(':
   case ')':
   case '.':
-  case '|':
     return false;
+  case '|':
+    error("support for '|' as an argument delimiter is deprecated and"
+         " will be withdrawn in a future release");
+    // fall through
   default:
     return true;
   }


G. Branden Robinson <gbranden>
Group administrator
Mon 25 Nov 2024 04:22:35 AM UTC, comment #5: 

Ugh.  Because escape sequences can be nested, to fix this I have to track more state in the formatter: whether the current input level is "general" or "numeric".

It would be nice if we could wean people off of using "|" as a delimiter just as we did with:


  case '0':
  case '1':
  case '2':
  case '3':
  case '4':
  case '5':
  case '6':
  case '7':
  case '8':
  case '9':
  case '+':
  case '-':
  case '/':
  case '*':
  case '%':
  case '<':
  case '>':
  case '=':
  case '&':
  case ':':
  case '(':
  case ')':
  case '.':


G. Branden Robinson <gbranden>
Group administrator
Mon 25 Nov 2024 12:28:25 AM UTC, comment #4: 

comment #3:

> $ printf '\\l95n\\&*9\n' | ~/groff-1.22.3/bin/nroff | cat -s
> <standard input>:1: cannot use character `9' as a starting delimiter
> 5n*9


>
> If I had to wager, I'd bet that the foregoing input has been
> rejected by GNU troff since day one.


For over 19 years, at least.

$ nroff --version
GNU nroff (groff) version 1.19.2
$ printf '\\l95n\\&*9\n' | nroff | cat -s
<standard input>:1: cannot use character `9' as a starting delimiter
5n*9


Dave <barx>
Group Member
Mon 25 Nov 2024 12:10:02 AM UTC, comment #3: 

I can fix this but GNU troff is still going to be stricter than AT&T troff, albeit not with respect to the `\w` escape sequence, which accepts what we might call a "general" argument rather than a numeric expression argument.

Consider use of the `\l` escape sequence, which accepts a numeric expression argument, and optionally, a character to draw the line with.  If said character would be valid in a numeric expression, it must be preceded by a dummy character escape sequence. However, AT&T troff still accepts (apparently) any delimiter that would be valid for more general input.


$ printf '\\l"5n\\&*"\n' | DWBHOME=~/dwb ~/dwb/bin/nroff | cat -s
*****

$ printf '\\l|5n\\&*|\n' | DWBHOME=~/dwb ~/dwb/bin/nroff | cat -s
*****

$ printf '\\l95n\\&*9\n' | DWBHOME=~/dwb ~/dwb/bin/nroff | cat -s
*****

$ printf '\\l95n\\&*9\n' | nroff | cat -s
troff:<standard input>:1: error: character '9' is not allowed as a delimiter
5n*9


Moreover, in anticipation of gripes from certain quarters, I observe that this behavior is not an instance of me being a jerkfaced fascist with respect to input cleanliness.  Jackboot prints have long gouged the earth of our input parser's hiking trail.


$ printf '\\l95n\\&*9\n' | ~/groff-1.22.3/bin/nroff | cat -s
<standard input>:1: cannot use character `9' as a starting delimiter
5n*9


If I had to wager, I'd bet that the foregoing input has been rejected by GNU troff since day one.  If this sort of tomfoolery was once "commonly used" as Paul said of `\w|`, 35 years of groff influence have, I suspect, boiled it away.

`printf '\\l95n\\&*9\n'` strikes me as too perverse a use case to support.  Even if people lean on me to support it, I feel confident that I'd stuff it behind AT&T compatibility mode.  There are few, if any, reasons to use exotic delimiters in GNU troff.[1]

So I will do as Paul suggests, but I will not go further at this time, and maybe not ever.


5.6.5 Delimiters
----------------

[snip]

   Delimiter syntax is complex and flexible primarily for historical
reasons; the foregoing restrictions need be kept in mind mainly when
using 'groff' in AT&T compatibility mode.  GNU 'troff' keeps track of
the nesting depth of escape sequence interpolations, so the only
characters you need to avoid using as delimiters are those that appear
in the arguments you input, not any that result from interpolation.
Typically, ''' works fine.  *Note Implementation Differences::.


(The foregoing language also appears in section "Delimiters" of groff(7).)

[1] And if you want exotic, do what GNU tbl does.  Select a bespoke special character to serve as your delimiter.  (You don't even have to define this special character with the `char` request--since it is never used to format anything, but only to delimit input, it never provokes a diagnostic.  Neat trick!)

https://git.savannah.gnu.org/cgit/groff.git/tree/src/preproc/tbl/table.cpp?h=1.23.0#n29

G. Branden Robinson <gbranden>
Group administrator
Sun 24 Nov 2024 11:05:30 PM UTC, comment #2: 

...and in DWB nroff as well (from which Heirloom apparently descends, but which has a less buggy mm package [not relevant here]).


$ printf '\\w|x|\n' | DWBHOME=~/dwb ~/dwb/bin/nroff | cat -s
24


Thanks for catching this.  I'll put it on the 1.24.0 release goal list so we don't release with this regression.

G. Branden Robinson <gbranden>
Group administrator
Sun 24 Nov 2024 03:50:45 AM UTC, comment #1: 

original submission:

> outputs "24" in traditional (Solaris 10) nroff,


and Heirloom nroff.

Dave <barx>
Group Member
Sun 24 Nov 2024 02:08:29 AM UTC, original submission:  

The following shell command:

printf '\\w|x|\n' | nroff | uniq

outputs "24" in traditional (Solaris 10) nroff, as well as in groff 1.23.0. However, bleeding-edge groff outputs the following diagnostic:

troff:<standard input>:1: error: character '|' is not allowed as a delimiter

and then outputs "x|" to standard output. This behavior is incorrect, as many longstanding troff documents use "\w|X|" when X might contain an apostrophe.

This bug was apparently introduced by the bug #66009 fix, which overenthusiastically banned '|' as a delimiter even in non-numeric contexts, where '|' works just fine and is commonly used.

I found out about this groff bug when Bjarni Ingi Gislason reported that the man page for GNU grep stopped working <https://bugs.gnu.org/74502>.

Paul Eggert <eggert>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by eggert (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-11-30 gbranden StatusIn Progress Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.24.0
    2024-11-25 gbranden StatusConfirmed In Progress
    2024-11-24 gbranden Summary\w|x| no longer works in bleeding-edge groff [troff] `\w|x|` no longer works on the bleeding edge of Git
    2024-11-24 gbranden Dependencies- bugs #65099 is dependent
    2024-11-24 gbranden StatusNone Confirmed
        Assigned toNone gbranden
    2024-11-24 barx CategoryGeneral Core

    Back to the top

    Powered by Savane 3.14-9aa3.
    Corresponding source code