bugGNU roff - Bugs: bug #65108, [troff] support construction of...

 
 

bug #65108: [troff] support construction of general file name request arguments

Submitter:  G. Branden Robinson <gbranden>
Submitted:  Tue 02 Jan 2024 10:59:42 PM UTC
   
 
Category:  Core Severity:  1 - Wish
Item Group:  Feature change Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 25 Jul 2024 10:53:38 PM UTC, comment #9: 


comment #5:

> So I'm not sure whether or not you advocate retaining soelim's current escape mechanism.  Maybe you're not yet either.


It's a question that tempts me to dither.
 

> My suggestion in comment #2 that support for soelim's escapes might need to be dropped was based my concern that one syntax requiring backslashes for spaces and one not would result in ambiguities in representing edge-case filenames, such as ones containing a backslash followed by a space.  But I think you've eliminated this possibility by not allowing a bare backslash to represent itself, requiring it be doubled if the filename contains a backslash.  (What miscreant named this file anyway?)


Right.

> But if soelim is changed to no longer recognize "\ ", then rule 5a is unnecessary and even a little counterintuitive.


Agreed.

G. Branden Robinson <gbranden>
Group administrator
Thu 25 Jul 2024 10:50:55 PM UTC, comment #8: 

comment #7:

> One additional comment on the proposal:
>
> comment #3:
> > Only codes in the range 00-1F and 80-FF are accepted in
> > [`\[u00XX]`] syntax; those in the range 20-7F are ignored with a
> > diagnostic advising the user to deobfuscate their inputs.
>
> I realize there's no good reason for a user to type "\[u0045]" instead of "E"


There may in fact be one.  It could be a means of obtaining an ordinary character (or the handful of special characters in Unicode Basic Latin) when said characters in their conventional forms are at that time subject to `tr` translation.

I don't know if this is feasible, as I still haven't mastered the character-to-glyph resolution process.  It's one of the more complex aspects of the formatter.
 

> ... but at the same time there seems no reason for groff to object to it.  It's ugly but not ambiguous or any harder to parse than the accepted ranges; if anything, a diagnostic seems to complicate the code, which could otherwise handle every \[u00XX] the same way.
>
> Even if you're wedded to the diagnostic, I'd say at least process the character.  Ignoring it seems needlessly punitive.


I have a very tall prescription pad.  But I'll hold my fire for now.  :)

> (Taking ticket out of "Need Info" assuming comment #5 addressed your questions; let me know if I've overlooked anything.)


That's fine.

I think this is just waiting on me now to start implementing and decide:

A.  what to do about `\ ` in GNU soelim and troff.
B.  whether to accept `\[u0021]` (or `\[u0020`?) through `\[u007E]` (or `\[u007F]`?)
C.  if the answer to "B" is "yes", whether to warn about them

G. Branden Robinson <gbranden>
Group administrator
Sat 20 Jul 2024 08:20:06 PM UTC, comment #7: 

One additional comment on the proposal:

comment #3:

> Only codes in the range 00-1F and 80-FF are accepted in
> [`\[u00XX]`] syntax; those in the range 20-7F are ignored with a
> diagnostic advising the user to deobfuscate their inputs.


I realize there's no good reason for a user to type "\[u0045]" instead of "E"... but at the same time there seems no reason for groff to object to it.  It's ugly but not ambiguous or any harder to parse than the accepted ranges; if anything, a diagnostic seems to complicate the code, which could otherwise handle every \[u00XX] the same way.

Even if you're wedded to the diagnostic, I'd say at least process the character.  Ignoring it seems needlessly punitive.

(Taking ticket out of "Need Info" assuming comment #5 addressed your questions; let me know if I've overlooked anything.)

Dave <barx>
Group Member
Sat 20 Jul 2024 07:50:09 PM UTC, comment #6: 

comment #1:

> It would seem that AT&T troff users (and groff users to date)
> have been pretty conservative about the file names they pass to
> these requests.


In my experience, users who work a lot at the command line (which I bet covers almost all *roff users) tend to avoid filename characters that require escaping.  Users who interact with files exclusively through GUIs have no incentive to avoid certain characters in filenames; the only time they'll ever type a file's name is upon creating it, and even then no extra thought or effort is required to use characters that don't play well with shells.  So the above observation doesn't surprise me much.

Even I would name an mp3 of Tom Jones' most famous song its_not_unusual.mp3, and missing apostrophes drive me up the wall.

Dave <barx>
Group Member
Sat 20 Jul 2024 06:48:09 PM UTC, comment #5: 

Your plan looks solid!

I do have one question about two lines at opposite ends of comment #3 that seem to be in opposition.

> let's rough out a syntax that would work both for existing uses
> of `so` as soelim(1) understands it and for formatter syntax,


and

> Since backslash-space is apparently a GNU extension in the
> first place, we might consider dropping it.


So I'm not sure whether or not you advocate retaining soelim's current escape mechanism.  Maybe you're not yet either.

My suggestion in comment #2 that support for soelim's escapes might need to be dropped was based my concern that one syntax requiring backslashes for spaces and one not would result in ambiguities in representing edge-case filenames, such as ones containing a backslash followed by a space.  But I think you've eliminated this possibility by not allowing a bare backslash to represent itself, requiring it be doubled if the filename contains a backslash.  (What miscreant named this file anyway?)

But if soelim is changed to no longer recognize "\ ", then rule 5a is unnecessary and even a little counterintuitive.

Dave <barx>
Group Member
Thu 18 Jul 2024 10:01:48 PM UTC, comment #4: 

I forgot case #1 for Solaris 10 troff soelim.


printf '.so foo bar file.troff\n' | soelim
foo: No such file or directory
.so foo
bar file.troff


So just no space-in-file-name support of any kind.

Also, I cheated here with an example that I don't plan to work:


.so foo\u[0020]bar\u[0020]file.troff


Because of rule 5d:

> Only codes in the range 00-1F and 80-FF are accepted in this syntax; those in the range 20-7F are ignored with a diagnostic advising the user to deobfuscate their inputs.


...but it should get the idea across.

G. Branden Robinson <gbranden>
Group administrator
Thu 18 Jul 2024 09:54:05 PM UTC, comment #3: 

Well, let's rough out a syntax that would work both for existing uses of `so` as soelim(1) understands it and for formatter syntax, which interprets the `so` under slightly different rules (since it brings to bear the full power of the troff lexical analyzer).

1.  An argument of type `file` (as described in groff(7)) to a request consumes the rest of the rest of the line.
2.  Unescaped spaces can therefore populate the argument.
3.  A leading double quote is recognized and removed; a file name can thus start with spaces.
4.  Any other/remaining double quotes are not treated specially.
5.  Only the following escape sequences are recognized.

5a. `\ ` (backslash-space) represents a space.  It is not necessary in troff, but is recognized to avoid disrupting existing soelim(1) usage.
5b. `\"` ends the file name argument and starts a comment.
5c. `\\` represents a (single) literal backslash.  It is handled however the system's standard C library wants to handle it.
5d. `\[u00XX]` where each X is an uppercase hexadecimal digit encodes a character.  Only codes in the range 00-1F and 80-FF are accepted in this syntax; those in the range 20-7F are ignored with a diagnostic advising the user to deobfuscate their inputs.

How are these handled today?

Specimen:


$ cat EXPERIMENTS/extending-so-syntax.troff
.so foo bar file.troff
.so foo\ bar\ file.troff
.so "foo bar file.troff
.so foo.troff\" comment
.so foo\u[0020]bar\u[0020]file.troff


groff _soelim_:


$ soelim EXPERIMENTS/extending-so-syntax.troff
.lf 1 ./EXPERIMENTS/extending-so-syntax.troff
soelim:./EXPERIMENTS/extending-so-syntax.troff:1: error: can't open 'foo': No such file or directory
.so foo bar file.troff
soelim:./EXPERIMENTS/extending-so-syntax.troff:2: error: can't open 'foo bar file.troff': No such file or directory
.so foo\ bar\ file.troff
soelim:./EXPERIMENTS/extending-so-syntax.troff:3: error: can't open '"foo': No such file or directory
.so "foo bar file.troff
.so foo.troff\" comment
.so foo\u[0020]bar\u[0020]file.troff


DWB 3.3 soelim:

...never mind, DWB 3.3 troff has no soelim.  Wow!  Learned something new today.

Heirloom Doctools soelim:


$ ./bin/soelim ./extending-so-syntax.troff
foo: No such file or directory
.so foo
bar file.troff
foo\: No such file or directory
.so foo\
bar\ file.troff
"foo: No such file or directory
.so "foo
bar file.troff
foo.troff\": No such file or directory
.so foo.troff\"
comment
foo\u[0020]bar\u[0020]file.troff: No such file or directory
.so foo\u[0020]bar\u[0020]file.troff


Uh, that's a little hard to interpret.


$ printf '.so foo bar file.troff\n' | ./bin/soelim
foo: No such file or directory
.so foo
bar file.troff


Interesting that it transforms the input in this way, by adding a newline where it decided to stop lexing the file name.  I'm tempted to call that a bug.


0000000   .   s   o       f   o   o  \n   b   a   r       f   i   l   e
0000020   .   t   r   o   f   f  \n
0000026


The other cases:


$ printf '.so foo\\ bar\\ file.troff\n' | ./bin/soelim
foo\: No such file or directory
.so foo\
bar\ file.troff

$ printf '.so "foo bar file.troff\n' | ./bin/soelim
"foo: No such file or directory
.so "foo
bar file.troff

$ printf '.so "foo.troff\\"comment\n' | ./bin/soelim
"foo.troff\"comment: No such file or directory
.so "foo.troff\"comment

$ printf '.so foo\u[0020]bar\u[0020]file.troff\n' | ./bin/soelim
printf '.so foo\\u[0020]bar\\u[0020]file.troff\n' | ./bin/soelim
foo\u[0020]bar\u[0020]file.troff: No such file or directory
.so foo\u[0020]bar\u[0020]file.troff


There seem to be no further surprises here.

Unix V7 did not have soelim, either.

Let me check Solaris 10.


$ printf '.so foo\\ bar\\ file.troff\n' | soelim
foo\: No such file or directory
.so foo\
bar\ file.troff

$ printf '.so "foo bar file.troff\n' |soelim
"foo: No such file or directory
.so "foo
bar file.troff

$ printf '.so "foo.troff\\"comment\n' |soelim
"foo.troff\"comment: No such file or directory
.so "foo.troff\"comment

$ printf '.so foo\u[0020]bar\u[0020]file.troff\n' |soelim
foo\u[0020]bar\u[0020]file.troff: No such file or directory
.so foo\u[0020]bar\u[0020]file.troff


These look identical to Heirloom to me.  I guess we know now where Heirloom got its inspiration, and perhaps even code, for soelim from.

Since backslash-space is apparently a GNU extension in the first place, we might consider dropping it.  It wasn't portable, and even the rest of the groff ecosystem struggled to handle files with spaces in their names.

I further venture that this exact same syntax could be applied to the `sy`/`pso` problem in bug #62787 and to user-constructed diagnostic messages in bug #64071.

I highly value the prospect of having a parallel syntax for these 3 issues if we can get it.

For soelim(1) itself I would further add that this program will continue to recognize only backslash as an escape character, but GNU troff will recognize the configured escape character.

Thoughts?

G. Branden Robinson <gbranden>
Group administrator
Thu 18 Jul 2024 05:32:55 AM UTC, comment #2: 

comment #1:

> Decide what to do about spaces and tabs.  One can argue about tabs
> being reasonable in file names, but spaces are a fact of life.
>
> As it happens, all requests that take file arguments
> always do so as the their last argument, thus could reuse
> optional-strippable-leading-quote syntax.  This would be friendly
> to users, I think.  No new rule to remember for this argument type.


Friendly to some users, but less friendly to those who use soelim's syntax for handling spaces and backslashes in filenames.

I argued in bug #59442 that groff and soelim shouldn't use competing filename syntaxes, as this would not be user-friendly.  I still think that's true, but the right solution might be to make soelim also use the syntax you've outlined here, breaking back compatibility in exchange for a more scalable solution.

Dave <barx>
Group Member
Tue 09 Jul 2024 12:46:45 PM UTC, comment #1: 

This would affect the `\O` escape sequence and the `cf`, `fp`, `hpf`, `hpfa`, `mso`, `msoquiet`, `nx`, `open`, `opena`,  `psbb`, `so`, `soquiet`, and `trf` requests.

Right now at least some of these use the internal function `get_long_name()`, which is inappropriately loose--it's made for reading groff identifiers, which can contain slashes.

A new, more specialized function for reading file name arguments should:

1.  Reject leading slashes, either when not in unsafe mode, or altogether; and

2.  Reject an argument with any slashes in it in the case of `fp` at least, to avoid having to draw mysterious inferences as in the fix for bug #64577.

Decide what to do about spaces and tabs.  One can argue about tabs being reasonable in file names, but spaces are a fact of life.

As it happens, all requests that take file arguments always do so as the their last argument, thus could reuse optional-strippable-leading-quote syntax.  This would be friendly to users, I think.  No new rule to remember for this argument type.

We might have to punt on \O5, and have "pre-html.cpp" reject an input document that has spaces or tabs in its name.  I know I don't want to add much machinery just to support generalized file arguments in this one documentedly internal-use-only escape sequence.

It would seem that AT&T troff users (and groff users to date) have been pretty conservative about the file names they pass to these requests.

G. Branden Robinson <gbranden>
Group administrator
Tue 02 Jan 2024 10:59:42 PM UTC, original submission:  

See the related bug #64071, which aims to do something similar for `sy` and `pso` requests.


Quoting bug #59442, where I got carried away and brainstormed this.

*

[Something] I want to do is specialize the formatter's logic when
handling file name arguments given to requests.

Presently, GNU troff calls the same internal function to gather an
argument that is a file name as it does to gather a roff identifier.

Maybe that made sense in 1990, but it doesn't today.  File names can
contain spaces and non-ASCII characters (in whatever encoding the file
system happens to support).

Since these arguments used mainly as-is, handed off to standard C
library functions like `fopen()`, I don't anticipate many problems here
(O Fortuna, seize my hostage).  The only exception to that I can think
of off the top of my head is the value of the `.F` register, which
interpolates a file name.  We will need some way to represent this such
things as output.  At first blush, it seems to me that we can
interpolate spaces as-is (if you want the argument quoted, do that
yourself in context), and any unprintable non-Basic Latin bytes in
groff's \[u00xx] notation.

I say "\[u00xx]" instead of "\[uXXXX]" because we have no way of knowing
what the file system's character encoding is.  Might be ISO 8859-1,
UTF-8, UTF-16BE/LE, or something else entirely.

What would be affected by this:

Requests:

cf
fp (when invoked with a 3rd argument)
hpf
hpfa
lf (when invoked with a 2nd argument)
mso
msoquiet
nx (when invoked with a 2nd argument)
open
opena
psbb
so
soquiet
trf

Escape sequences:

\O5 (but since this is mainly used internally to manage temporary files
    by grohtml, maybe lazily postponing this in hope that my Grand Plan
    to revise grohtml to no longer use a dedicated preprocessor is a
    better idea)


Registers:

.F

[snip]

And probably step 1 would be a simple refactor to introduce file name
argument-gathering and -interpolating functions which initially behave
no differently than the status [quo], but simply wrap existing logic for
identifier gathering and whatever one-off thing the `.F` interpolator
does.

*

G. Branden Robinson <gbranden>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by gbranden (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-07-26 gbranden Dependencies- bugs #66027 is dependent
    2024-07-20 barx StatusNeed Info None
        Assigned tobarx None
    2024-07-18 gbranden StatusNone Need Info
        Assigned toNone barx
    2024-01-02 gbranden Dependencies- bugs #65099 is dependent

    Back to the top

    Powered by Savane 3.13-3e34.
    Corresponding source code