comment #17:

> Which emails notifications I receive seems to be pretty erratic.

Savannah seems to be under a lot of churn lately--which is overall a good thing, as useful features are being added.  But yeah, I did notice http://savannah.nongnu.org/support/?111148 yesterday myself.

comment #8:

> A.  what to do about `\ ` in GNU soelim and troff.

The soelim part is now covered by bug #66027.  (This bug was modified to depend on 66027 the same day comment #8 and comment #9 were posted, but if there was an email notification of this, bug-groff did not archive it.)

comment #13:

> comment #12:
> > I feel like we're saying the same thing, or compatible things.
>
> Quite possibly.
>
> > A file named "résumé1.ms" might be stored on the file system
> > using either character encoding,
>
> ...or, as my example attempted to illustrate, two files might be stored, each using a different encoding.

Yes, a better point than I initially gave it credit for.  So, ideally, we want GNU troff requests to be able to refer unambiguously to either one.

> Similar to the contents of a file, a filename is just a string of bytes.  What characters those bytes mean is defined by the encoding.

This, I'll quibble with.  An encoding is simply a map between integers and abstract characters.  Nowadays, in the post-ISO 8859 watershed when encoding designers got more woke to the difficulties of large character sets and clashing cultural interpretations of certain symbols, these abstract characters tend to have names.  In the innocent days of USAS X.34-1968, one simply printed a chart with numbered boxes and unnamed glyphs, implying that a rendering device should "make the characters look just like that!"

Importantly, what distinguishes ISO 10646 from Unicode is that the former is only a character encoding standard--the aforementioned mapping--whereas Unicode is a character set standard, the normative responsibilities of which have cast a surprisingly large penumbra regarded from the perspective of more innocent 7- and 8-bit character days.
 

> A file can contain metadata to indicate its encoding; if not, there's often enough context for tools like preconv (or even the system's "file" command) to correctly guess it.

Right.  But a file name can't; not on POSIX systems.  There's no "resource fork" to indicate this.  The file system may impose an encoding (maybe), but as far as I know there's no portable way to query such information.
 

> The settings of one's terminal and LC_CTYPE environment variable affect how the string of bytes in a filename is interpreted.

Not always.  And there's the rub.

fopen(3):
       #include <stdio.h>

       FILE *fopen(const char *pathname, const char *mode);

$ sed -n '/^static void do_open/,/^}/p' src/roff/troff/input.cpp
static void do_open(bool append)
{
  symbol stream = get_name(true /* required */);
  if (!stream.is_null()) {
    symbol filename = get_long_name(true /* required */);
    if (!filename.is_null()) {
      errno = 0;
      FILE *fp = fopen(filename.contents(), append ? "a" : "w");
      if (0 /* nullptr */ == fp) {
        error("cannot open file '%1' for %2: %3",
              filename.contents(),
              append ? "appending" : "writing",
              strerror(errno));
        fp = (FILE *)stream_dictionary.remove(stream);
      }
      else
        fp = (FILE *)stream_dictionary.lookup(stream, fp);
      if (fp)
        fclose(fp);
    }
  }
  skip_line();
}

> There may not be enough context to guess.  There's no metadata (that I'm aware of, though I'd be happy to be wrong) to make the name's encoding definitive.

Precisely.

The way we're getting at file names is a C string with no implied encoding.

They're just bytes.  And GNU troff requests are not expressive enough, at present, to supply fopen() with a sequence of "just bytes".  Mostly, that's a good thing, because it keeps the formatter's own language more sane.  But we're limited to printable ASCII characters (with fuzz around the edges, like space 0x20 and delete 0x7F).  Tabs are right out.  Backslashes...should work?  Theoretically?  If doubled?  Do we need to double them again for C's sake, given that it's an escape character there too?  CSTR #54 offers no specification in this area.

We need an escape hatch, as Kernighan famously noted when critiquing Pascal's lack of them in CSTR #100.

That escape hatch is what I mean to provide, by repurposing GNU troff's Unicode special character escape sequence syntax.  That choice I knew would pinch a little when I made it, because it's not actually representing special characters here...or even, in this application, Unicode, due to the range limitation--and that pinch is something I'm feeling now while trying to reach a meeting of the minds with Deri over what we mean we type these things in non-formatting contents.
 

> > That's why I want to be able to support:
> >

> > $ grep -F .so résumé.ms
> > .so r\[u00E9]sum\[u00E9]1.ms
> > .so r\[u00E9]sum\[u00E9]2.ms
> > .so r\[u00E9]sum\[u00E9]3.ms

>
> Agreed, but I think it's ambiguous which of the two files I created in comment #11 a construction like this refers to.

My answer is straightforward.  I mean to apply a transformation to `filename.contents()` in the `do_open()` function above (actually via a helper function, because I'll need it for bug #64071 too) such that sequences matching `\[u0000]..\[u00FF]` map to C language octal escapes in the range \000 to \377.  That transformed string is what I would hand to fopen().

Some complications arise:

• \000 itself won't work as "desired".  But this is not a practical problem, as 50+ years of Unix and C have led no one to expect that they can infix nulls in any file name anywhere.

• The matter of other C0 controls (so, \001 to \037) is a vexing one.  I would strongly prefer to stay out of the morass altogether.  To see what I mean, and if you have an hour or so to spare, peruse Austin Group ticket 251.  This issue has received deep attention from experts.

Consequently my plan right now is to reject `\[u0000]` through `\[u001F]`, inclusive--meaning throw an error diagnostic and abort the request.

> They both, from some viewpoint, have the base filename "résumé".

That viewpoint is not the one taken by fopen(), which sees only a sequence of 8-bit bytes, to which it ascribes no particular meaning.  From that stance, the Latin-1 vs. UTF-8 encodings of "résumé" plainly differ.

> They can both coexist on the same file system, even in the same directory.

Yes!  And that's why it's good that fopen() can tell them apart, and so can we, if we will meet it on its own terms!

comment #11:

> original submission:
> > we have no way of knowing what the file system's character encoding is.
> > Might be ISO 8859-1, UTF-8, UTF-16BE/LE, or something else entirely.
>
> I'm not sure now if that's a meaningful question.  The file system seems to just store a string of bytes as the file name, and leave it up to the shell how to interpret that.

> $ mkdir foo
> $ cd foo
> $ echo résumé | iconv -tutf8 | xargs touch
> $ echo résumé | iconv -tlatin1 | xargs touch
> $ echo * | od -c
> 0000000   r 303 251   s   u   m 303 251       r 351   s   u   m 351  \n
> 0000020

> Then a UTF-8 shell produces:

> $ ls
>  résumé  'r'$'\351''sum'$'\351'

> and a Latin-1 shell produces:

> $ ls
> résumé  résumé

> That is, both filenames are valid (but different) strings of Latin-1 characters.  In UTF-8, one of them is a string of valid characters, and one has two invalid bytes in it.

It's also valid Latin-2, Latin-5, Latin-9, and KOI8-R, to name four other encodings supported by groff.
 

> This is an ext4 file system, but I would imagine any other Unix-based one would have to work the same in order to interact with shells consistently.

I feel like we're saying the same thing, or compatible things.

A file named "résumé1.ms" might be stored on the file system using either character encoding, or, on a Widows system, using UTF-16LE.  A groff user with a document that wants to `so` that file name:

$ grep -F .so résumé.ms
.so résumé1.ms
.so résumé2.ms
.so résumé3.ms

...is going to need either an encoding match between résumé.ms's contents and their file system, or some sophistication about character encodings.

That's why I want to be able to support:

$ grep -F .so résumé.ms
.so r\[u00E9]sum\[u00E9]1.ms
.so r\[u00E9]sum\[u00E9]2.ms
.so r\[u00E9]sum\[u00E9]3.ms

That way a person doesn't have to preconv their document.

Or did preconv their document and this is what the program left them with because that tool has no sense of context regarding requests that take file name arguments: `so`, `soquiet`, `mso`, `msoquiet`, `open`, `opena`, `psbb`, `cf`, `fp`, `hpf`, `hpfa`, `nx`, or `trf`.

I feel like we might be talking past each other...?

comment #8:

> comment #7:
> > One additional comment on the proposal:
> >
> > comment #3:
> > > Only codes in the range 00-1F and 80-FF are accepted in
> > > [`\[u00XX]`] syntax; those in the range 20-7F are ignored with a
> > > diagnostic advising the user to deobfuscate their inputs.
> >
> > I realize there's no good reason for a user to type "\[u0045]" instead of "E"
>
> There may in fact be one.  It could be a means of obtaining an ordinary character (or the handful of special characters in Unicode Basic Latin) when said characters in their conventional forms are at that time subject to `tr` translation.

This was a bogus digression.  `tr` affects only characters that are sent to the output for transformation to glyphs, and only at the time that this happens.

$ cat EXPERIMENTS/tr-works-only-on-output.roff
.nf
.tr ab
.ds a aunt
\*a
.tr aa
\*a
.pl \n(nlu
$ nroff EXPERIMENTS/tr-works-only-on-output.roff
bunt
aunt

So, disregard.

comment #7:

> One additional comment on the proposal:
>
> comment #3:
> > Only codes in the range 00-1F and 80-FF are accepted in
> > [`\[u00XX]`] syntax; those in the range 20-7F are ignored with a
> > diagnostic advising the user to deobfuscate their inputs.
>
> I realize there's no good reason for a user to type "\[u0045]" instead of "E"

There may in fact be one.  It could be a means of obtaining an ordinary character (or the handful of special characters in Unicode Basic Latin) when said characters in their conventional forms are at that time subject to `tr` translation.

I don't know if this is feasible, as I still haven't mastered the character-to-glyph resolution process.  It's one of the more complex aspects of the formatter.
 

> ... but at the same time there seems no reason for groff to object to it.  It's ugly but not ambiguous or any harder to parse than the accepted ranges; if anything, a diagnostic seems to complicate the code, which could otherwise handle every \[u00XX] the same way.
>
> Even if you're wedded to the diagnostic, I'd say at least process the character.  Ignoring it seems needlessly punitive.

I have a very tall prescription pad.  But I'll hold my fire for now.  :)

> (Taking ticket out of "Need Info" assuming comment #5 addressed your questions; let me know if I've overlooked anything.)

That's fine.

I think this is just waiting on me now to start implementing and decide:

A.  what to do about `\ ` in GNU soelim and troff.
B.  whether to accept `\[u0021]` (or `\[u0020`?) through `\[u007E]` (or `\[u007F]`?)
C.  if the answer to "B" is "yes", whether to warn about them

comment #1:

> It would seem that AT&T troff users (and groff users to date)
> have been pretty conservative about the file names they pass to
> these requests.

In my experience, users who work a lot at the command line (which I bet covers almost all *roff users) tend to avoid filename characters that require escaping.  Users who interact with files exclusively through GUIs have no incentive to avoid certain characters in filenames; the only time they'll ever type a file's name is upon creating it, and even then no extra thought or effort is required to use characters that don't play well with shells.  So the above observation doesn't surprise me much.

Even I would name an mp3 of Tom Jones' most famous song its_not_unusual.mp3, and missing apostrophes drive me up the wall.

I forgot case #1 for Solaris 10 troff soelim.

printf '.so foo bar file.troff\n' | soelim
foo: No such file or directory
.so foo
bar file.troff

So just no space-in-file-name support of any kind.

Also, I cheated here with an example that I don't plan to work:

.so foo\u[0020]bar\u[0020]file.troff

Because of rule 5d:

> Only codes in the range 00-1F and 80-FF are accepted in this syntax; those in the range 20-7F are ignored with a diagnostic advising the user to deobfuscate their inputs.

...but it should get the idea across.

comment #1:

> Decide what to do about spaces and tabs.  One can argue about tabs
> being reasonable in file names, but spaces are a fact of life.
>
> As it happens, all requests that take file arguments
> always do so as the their last argument, thus could reuse
> optional-strippable-leading-quote syntax.  This would be friendly
> to users, I think.  No new rule to remember for this argument type.

Friendly to some users, but less friendly to those who use soelim's syntax for handling spaces and backslashes in filenames.

I argued in bug #59442 that groff and soelim shouldn't use competing filename syntaxes, as this would not be user-friendly.  I still think that's true, but the right solution might be to make soelim also use the syntax you've outlined here, breaking back compatibility in exchange for a more scalable solution.

See the related bug #64071, which aims to do something similar for `sy` and `pso` requests.


Quoting bug #59442, where I got carried away and brainstormed this.

*

[Something] I want to do is specialize the formatter's logic when
handling file name arguments given to requests.

Presently, GNU troff calls the same internal function to gather an
argument that is a file name as it does to gather a roff identifier.

Maybe that made sense in 1990, but it doesn't today.  File names can
contain spaces and non-ASCII characters (in whatever encoding the file
system happens to support).

Since these arguments used mainly as-is, handed off to standard C
library functions like `fopen()`, I don't anticipate many problems here
(O Fortuna, seize my hostage).  The only exception to that I can think
of off the top of my head is the value of the `.F` register, which
interpolates a file name.  We will need some way to represent this such
things as output.  At first blush, it seems to me that we can
interpolate spaces as-is (if you want the argument quoted, do that
yourself in context), and any unprintable non-Basic Latin bytes in
groff's \[u00xx] notation.

I say "\[u00xx]" instead of "\[uXXXX]" because we have no way of knowing
what the file system's character encoding is.  Might be ISO 8859-1,
UTF-8, UTF-16BE/LE, or something else entirely.

What would be affected by this:

Requests:

cf
fp (when invoked with a 3rd argument)
hpf
hpfa
lf (when invoked with a 2nd argument)
mso
msoquiet
nx (when invoked with a 2nd argument)
open
opena
psbb
so
soquiet
trf

Escape sequences:

\O5 (but since this is mainly used internally to manage temporary files
    by grohtml, maybe lazily postponing this in hope that my Grand Plan
    to revise grohtml to no longer use a dedicated preprocessor is a
    better idea)


Registers:

.F

[snip]

And probably step 1 would be a simple refactor to introduce file name
argument-gathering and -interpolating functions which initially behave
no differently than the status [quo], but simply wrap existing logic for
identifier gathering and whatever one-off thing the `.F` interpolator
does.

*