bugGNU roff - Bugs: bug #60571, Footnote markers defeat...

 
 

bug #60571: Footnote markers defeat end-of-sentence recognition

Submitter:  G. Branden Robinson <gbranden>
Submitted:  Mon 10 May 2021 02:55:16 PM UTC
   
 
Category:  Core Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 12 Feb 2023 04:28:34 PM UTC, comment #11: 

Now that I am being forced to document our two dummy characters \& and \( formally, it appears to me that almost no one seems to use \).  Within the groff tree only mdoc uses it consistently, and it seems to do so mostly, perhaps exclusively, as a superseding form of \&.  (I haven't checked every case, and should.)

I therefore wonder if any serious disruption would be caused by ladling my proposed semantics for \] onto \), preserving a precious slot in the escape sequence name space while disrupting (I conjecture) few to no groff users.

G. Branden Robinson <gbranden>
Group administrator
Tue 11 Oct 2022 11:41:53 PM UTC, comment #10: 

comment #3:

> Offhand, < seems the best candidate for a double-agent escape
> character / begin delimiter, since it has an obvious companion
> for an end delimiter


Bug #60060 points out that neatroff uses \< and \> escapes for a different purpose.

Dave <barx>
Group Member
Thu 03 Feb 2022 06:22:10 PM UTC, comment #9: 

comment #8:

> You can use '\' in an identifier name, but you have to escape
> it, turn the escape mechanism off, or change the escape character.


It occurred to me later that that might be the case, and I sat down to devise a better test case, but found this new comment that beat me to the punch.

It would've helped to read the error message more carefully: it didn't say, as I thought, "error: '\' is not allowed," but "error: '\ ' is not allowed," an important distinction.

Dave <barx>
Group Member
Thu 03 Feb 2022 07:27:33 AM UTC, comment #8: 

comment #7:

> My fleeting concern with that was potential parsing ambiguity if a long register name ends in a backslash: are there cases where the \] signifying the end of the register name in the \n[] escape are confusable with the proposed \] escape?
>
> To see if this was an actual concern, I first looked in the Identifiers node of the Texinfo document.  It claims that of printable characters, only those considered whitespace are ineligible in identifiers.  So the backslash should be legal.
>
> But it isn't:


> $ groff <<< '.nr a\ 1'
> troff:<standard input>:1: error: '\ ' is not allowed in a name
> troff:<standard input>:1: warning: numeric expression expected (got '\ ')


>
> In conclusion, I'm good with \] as an escape, but need to open a new documentation bug.


You can use '\' in an identifier name, but you have to escape it, turn the escape mechanism off, or change the escape character.


$ cat EXPERIMENTS/backslash-in-identifier.groff
.nr a\\b 1
\n[a\\b]
.ec @
.nr c\d 2
@n[c\d]
.eo
.nr e\f 3
.ec \" no way to interpolate that without turning escapes back on
\n[e\\f]
$ ./build/test-groff -Tutf8 | cat -s
1 2 3


That isn't to say I think it's a wise practice, or should be recommended.

The way I think about escaping in *roff is that the escape character, if defined, is lexically supreme--it is always interpreted as starting an escape sequence, unless it can't start one.  And the only way it can't (again, if defined), is if it is already immediately preceded by...the escape character.

Basically, once the escape character (if defined) is seen, the lexer/parser goes into "eat one character" mode.  That character determines the escape function to be performed (interpolate a register, start a comment, etc.), and it is only that context in which hitting another escape character is not special.

So far this rule has not led me astray when interpreting *roff input.  (Admittedly, I've cheated and looked at the parser's implementation in src/roff/troff/input.cpp many times, but I could not claim to have memorized it!)

G. Branden Robinson <gbranden>
Group administrator
Wed 02 Feb 2022 08:34:50 PM UTC, comment #7: 

My fleeting concern with that was potential parsing ambiguity if a long register name ends in a backslash: are there cases where the \] signifying the end of the register name in the \n[] escape are confusable with the proposed \] escape?

To see if this was an actual concern, I first looked in the Identifiers node of the Texinfo document.  It claims that of printable characters, only those considered whitespace are ineligible in identifiers.  So the backslash should be legal.

But it isn't:

$ groff <<< '.nr a\ 1'
troff:<standard input>:1: error: '\ ' is not allowed in a name
troff:<standard input>:1: warning: numeric expression expected (got '\ ')


In conclusion, I'm good with \] as an escape, but need to open a new documentation bug.

Dave <barx>
Group Member
Wed 02 Feb 2022 04:08:23 PM UTC, comment #6: 

I propose `\]` for this purpose.  Mnemonic: it resembles the closing bracket of a common footnote or bibliographic citation marker, and those are the motivating use cases for it.

If you place a `\]` anywhere between a candidate end-of-sentence character and the next potential sentence boundary (a newline or second consecutive input space), that boundary token is forcibly recognized as the end of a sentence.

Interaction with \& and \) is to be determined.  Possibly, \& causes no interference, and \) and \] can override each other ad libitum.  (Imagine macros that keep tacking stuff on to the end of a string or input text; the document author may not have any idea what's going on, so we don't necessarily want to warn.)

Obviously this will need a unit test that exercises some tricky cases as well as straightforward ones.

G. Branden Robinson <gbranden>
Group administrator
Sat 15 May 2021 03:41:37 AM UTC, comment #5: 

comment #4:

> Yeah, I just don't know where the forum for modern *roff
> standardization is.  Is it groff at gnu?


http://github.com/n-t-roff/heirloom-doctools does direct users there for discussion, but I'm not sure how much developer representation there is on the list.  I'm also not aware of any better forum, though.

> within a project, like groff, I think it behooves us to have
> consistent semantics for them.


Absolutely.  But when letters are running short, and none of the handful remaining seems to fit the new use you have in mind, you might have to be less stringent than you'd like.

Dave <barx>
Group Member
Sat 15 May 2021 12:39:17 AM UTC, comment #4: 

comment #3:

> comment #2:
> > The following remain.
>
> And of those. Heirloom uses  
>
> ; @ j J P T U


Checking the doc.ps from their git repository, I see they've claimed

I W

as well.  :-/

This is particularly perverse is their \I is our \A--they wanted \A for URI "a"nchors, apparently, which \W connects to.

> > I think before long we want to propose to the *roff community
> > the reservation of one of the above characters for an extended
> > escape syntax of much the same form as \D and \X.
>
> That makes a great deal of sense to me.


Yeah, I just don't know where the forum for modern *roff standardization is.  Is it groff at gnu?
 

> Offhand, < seems the best candidate for a double-agent escape character / begin delimiter, since it has an obvious companion for an end delimiter (at the risk of maybe making roff code look a little HTMLy).


Agreed.

> Given the paucity of available letters, we may be in the Post-Mnemonic Era for assigning new roff escapes.  Traditional Unix commands with a multitude of single-letter command-line switches have felt this crunch as well.


Yes; I recall Ingo Schwarze seeking to impose a semantic discipline over the approximately 52-character name space of Unix option letters. (I remember how "-?" was going to save the world as a universal way to get a usage message...except, whoops, the Unix shell had already seized "?" for its own purposes, and any novice who needed help would simply stare with incomprehension if they were told they needed to quote the "?".)

I think that's hopeless, although he feels a greater urgency than I do because he also has antipathy for GNU-style long options--but within a project, like groff, I think it behooves us to have consistent semantics for them.  I have not evaluated how well we adhere to that, but in my many passes over our man page corpus I can't remember any clashes leaping out at me.

G. Branden Robinson <gbranden>
Group administrator
Tue 11 May 2021 10:17:39 PM UTC, comment #3: 

comment #2:

> The following remain.


And of those. Heirloom uses  

; @ j J P T U

> I think before long we want to propose to the *roff community
> the reservation of one of the above characters for an extended
> escape syntax of much the same form as \D and \X.


That makes a great deal of sense to me.

Offhand, < seems the best candidate for a double-agent escape character / begin delimiter, since it has an obvious companion for an end delimiter (at the risk of maybe making roff code look a little HTMLy).

> none of the symbols in the above list have any obvious mnemonic
> value to me for the purpose the Heirloom bug submitter and I want.


Given the paucity of available letters, we may be in the Post-Mnemonic Era for assigning new roff escapes.  Traditional Unix commands with a multitude of single-letter command-line switches have felt this crunch as well.

Dave <barx>
Group Member
Mon 10 May 2021 09:55:32 PM UTC, comment #2: 

Hi, Dave!

comment #1:

> In the Heirloom troff bug tracker, a contributor suggested creating a mechanism to explicitly indicate end of sentence that would augment the existing heuristics.
>
> From http://github.com/n-t-roff/heirloom-doctools/issues/83:
>
> "I would like... a way to actively identify the end of a sentence, in addition to the passive mechanism we have now. ... It isn't really practical to identify every character that could be a transparent character--superscripts, for example, could be nearly anything [0-9, A-Z, a-z], and they might come from the ASCII set (reduced in size and shifted up) or be dedicated glyphs.... Then there are the various quotation marks, not to mention hot links (and, of course, macros that process note numbers)."


Indeed, it appears that the original Heirloom report happened across precisely the same problem I did.

> It's not clear to me whether Heirloom troff is actively maintained anymore, but if anyone is nominally in charge of its future direction, it would be good to coordinate with them to come up with a portable system.


It has gotten awfully quiet over there.  Things aren't dead but they are pretty sleepy.  On the other hand 2021 is already much more active than 2020 was (with only 2 commits in 13 months).

The idea of Yet Another New Escape to force end-of-sentence recognition did occur to me, but it makes me a little anxious because the number of free keycaps in the US-ASCII repertoire to which we an assign an escape function after the escape character is rapidly approaching zero.  Here, let me count 'em...

The following remain.


G i I j J K P T U W y
1 2 3 4 5 6 7 8 9
+ ; < > = @ ]


Hmm, I guess that's not as bad as I thought.  Still, I think before long we want to propose to the *roff community the reservation of one of the above characters for an extended escape syntax of much the same form as \D and \X.  A delimiter follows, enclosing some arbitrary-length sequence that is decoded by a dedicated function.

The two candidates that leap to mind are "G", for "GNU" or "groff" if we have to go our own way on this, and "@" if we don't.

In any case, none of the symbols in the above list have any obvious mnemonic value to me for the purpose the Heirloom bug submitter and I want. ("\y" for "yes, damn it, this is the end of a sentence" did occur to me, but alas that is a highly flexible sentiment.)  \; would seem to be an anti-choice; I think it would imply "_not_ the end of a sentence" more strongly than the actual escape for doing so (\&) does.  \., \?, and \! are all taken.

Suggestions?

G. Branden Robinson <gbranden>
Group administrator
Mon 10 May 2021 07:39:55 PM UTC, comment #1: 

In the Heirloom troff bug tracker, a contributor suggested creating a mechanism to explicitly indicate end of sentence that would augment the existing heuristics.

From http://github.com/n-t-roff/heirloom-doctools/issues/83:

"I would like... a way to actively identify the end of a sentence, in addition to the passive mechanism we have now. ... It isn't really practical to identify every character that could be a transparent character--superscripts, for example, could be nearly anything [0-9, A-Z, a-z], and they might come from the ASCII set (reduced in size and shifted up) or be dedicated glyphs.... Then there are the various quotation marks, not to mention hot links (and, of course, macros that process note numbers)."

It's not clear to me whether Heirloom troff is actively maintained anymore, but if anyone is nominally in charge of its future direction, it would be good to coordinate with them to come up with a portable system.

Dave <barx>
Group Member
Mon 10 May 2021 02:55:16 PM UTC, original submission:  

I see the following in ms.ms.


The ms macros are the oldest surviving package for roff systems.[1] While the  man  package
...
    Longer  documents  include a table of contents, which you can invoke by placing the .TC
    macro at the end of your document.[2] Printing the table of contents at the end is nec‐
...
On  occasion,  you  may want to keep multiple lines of text together on the same page.  For
example, ms enables you to prevent a page break between two paragraphs, or between a  para‐
graph and an adjacent list, table, or other item it discusses.[3] Material thus grouped  is


I am ending my input lines after the footnote marker, but by having characters that don't have the .cflags option set after my end-of-sentence punctuation, detection of the end of the sentence is defeated.

The macro package user shouldn't have to care about this, but in any case there doesn't seem to be a mechanism to work around it.  There is no escape sequence that enables us to bracket a sequence of input characters as transparent to end-of-sentence detection.

Manipulating the .cflags of numerals inside the * string definition is not syntactically possible.  (How would I do this with .ds?)  And we don't want the macro package set the .cflags for the numerals in a more general case because that would just be wrong.  Consider this sentence.


I'm spitballing here; in other words, it's just my .02
bucks.


Making the numerals transparent would create a false positive end-of-sentence detection after "my .".

I'm thinking we need yet another read-only register exposing internal troff state, that being the end of sentence detection flag.  We could call it .eos maybe.  It could be saved before the footnote marker is written, then restored afterwards.  But that would still mean turning ms's * from a string into a macro.  Hmm.

Maybe someone else has a better idea.

G. Branden Robinson <gbranden>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by gbranden (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-10-08 barx DependenciesRemoved dependency from bugs #59738 -
    2022-08-17 barx Dependencies- bugs #59738 is dependent

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code