commit 635cd005577c8aa76fe6acf86e9d3af578a471d4
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Thu Nov 14 10:30:17 2024 -0600

    Fix Savannah #63074.

    * NEWS: Explain changes to `\X` and `.device` argument processing.

    Fixes <https://savannah.gnu.org/bugs/?63074>.  Thanks to Nikita
    Ivanov for the report.

Hi Deri,

At 2024-11-12T13:47:33-0500, Deri James wrote:

> Follow-up Comment #40, bug #63074 (group groff):
>
> comment #38:
> > One could envision three levels of support for encoding arbitrary
> > characters.
> >
> > 1.  By Unicode code point.  Reusing groff's own syntax for Unicode special
> > character escape sequences was irresistibly tempting, so that's what I
> > implemented.  We have that in Git HEAD.
> > 2.  By (simple) groff special character escape sequence, like \['o'] (in
> > "Cicerón").  We have that in Git HEAD too.
> > 3.  By composite special character escape sequence, like "\[o aa]", which we
> > might also use to write "Cicerón"--"Cicer\[o aa]n".  We don't have that.  It
> > proved to be difficult.  (The formatter warns if it encounters this syntax
> > where it can't handle it.)
>
> If you implement (3) you realise that searching a document for
> "Cicerón" (which was formed using \[o aa]) may not be found.

I think that's not correct, except in `output`/`\!` arguments, where you
get exactly the "grout" you ask for.

Here's the commented out test, preceded by the corresponding input:

input='.
.ds h Caf\[e aa] Hyphen-Minus and \[rs]\[u2010]
\X"ps:exec 5:\\X     [/Dest /pdf:bm1 /Title (\*[h]) /Level 1 /OUT pdfmark"
\!x X ps:exec 6:\!     [/Dest /pdf:bm1 /Title (\*[h]) /Level 1 /OUT pdfmark
.device ps:exec 7:device [/Dest /pdf:bm1 /Title (\*[h]) /Level 1 /OUT pdfmark
.output x X ps:exec 8:output [/Dest /pdf:bm1 /Title (\*[h]) /Level 1 /OUT pdfmark
.'

#echo "checking practical bookmarking with device request" >&2
#printf "%s\n" "$output" \
#  | grep -Fqx 'x X ps:exec 7:device [/Dest /pdf:bm1 /Title (Caf\[u00E9] Hyphen-Minus and \\[u2010]) /Level 1 /OUT pdfmark' \
#  || wail

As you can see, I expect `\[e aa]` to be transformed to \[u00E9].  But
this doesn't work presently, because the `device` request reads its
argument in copy mode, unlike the `\X` escape sequence.

The goal is to have `\[u00E9]` show up in the output no matter how it
was spelled in the input: `\['e]`, `\[e aa]`, `\[e ']`, `\[u0065_0301]`,
or indeed `\[u00E9]`.

That should present no complications for searching.

(We may want to someday migrate to a different policy for Unicode
decomposition--I leave that problem for when it ripens.)

> So I prefer not allowing composites, unless you have a zinger argument
> for them.

My zinger is simply that no user should have to remember this absurdly
esoteric detail.  Right now, they get a warning if they bump into it.

> Yes, both of these work perfectly:-
>
> printf "Caf\\['e]\n.br\n.output x X ps:exec [/Dest /pdf:bm1 /Title (Eat at
> Joe's Caf\\['e].) /Level 1 /OUT pdfmark\n" | test-groff -T pdf | okular -
>
> printf "Caf\\['e]\n.br\n.device ps:exec [/Dest /pdf:bm1 /Title (Eat at Joe's
> Caf\\['e].) /Level 1 /OUT pdfmark\n" | test-groff -T pdf | okular -

That was one of my goals!

I'm pleased that something worked out after all this struggle.  :-O

Best,
Branden

Further development filed as bug #66432.

Another small step.

commit 2548c4659cf7ceaa43bde7d9b5857a3e1412f553
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sat Sep 21 20:32:04 2024 -0500

    [troff]: Interpret spec'l chars in `device` reqs.

    * src/roff/troff/input.cpp (device_request): Add extensive new logic to
      subvert copy mode so as to interpret special character escape
      sequences.  We don't yet support _composite_ special character escape
      sequences.

    * src/roff/groff/tests/device-control-special-character-handling.sh:
      Update and expand test cases.  Comment out one new case that doesn't
      yet work.

    This change is in further service of the grueling march toward
    resolution of <https://savannah.gnu.org/bugs/?63074>.

    What's it called when you put _yourself_ on a death march?

Deri persuaded me not to go one of the directions I headed.

Which is fine, because that puts me back on a path I wanted to be on anyway: copy mode (or an ersatz simulacrum thereof) for `\X`.

commit a1ced0efe59519afa0aa00e049b7ef6f3627260f
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sat Sep 7 20:01:10 2024 -0500

    Revert "[troff]: \X maps some basic Latin chars to specs."

    This reverts commit dcae60b0fb1ad3fa3314fdfdbecb973961a40410.

    Reverting the change better simulates copy mode, which is eventually
    where we want to take `\X`'s interpretation of its delimited argument.
    There's simply no convenient infrastructure for that in "input.cpp" yet.

    Output drivers that need to distinguish characters in device extension
    commands destined for "document text" from those intended for
    "application text" (or some other external purpose, like constructing a
    shell command for execution) have to use out-of-band means to achieve
    this distinction.  Documentation of recommended practices for achieving
    this are being hashed out and will soon, it is hoped, be documented.

    Fixes <https://savannah.gnu.org/bugs/?66165>.  Thanks to Deri James for
    the report.

This more or less does it for `\X` escape sequences, without (as far as I can tell [and we do have 200+ automated tests]) regressing anything.

But there's much more ground to cover.  See my discussion with Deri in bug #64484.

commit c8332c5c1acdd16072672f2fa401cd77805e7264
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sun Aug 25 18:56:54 2024 -0500

    [troff]: Begin fixing Savannah #63074.

    Support construction of arbitrary byte sequences in device control
    commands.

    * src/roff/troff/input.cpp (encode_special_character_for_device_output):
      Enhance.  When constructing the content of a device control escape
      sequence (and, in the future, that of a `device` request), try harder
      to convert special characters into something meaningful.  Is a special
      character identifier looks like something other than an attempt at a
      Unicode special character escape sequence already, try to convert it
      into one.  Otherwise, write any valid Unicode special character
      identifier (in groff notation: `\[u123ABC]`) to the macro being
      assembled (and thence the `special` node, and ultimately the `x X`
      command this node type produces).
    * src/roff/groff/tests/device-control-special-character-handling.sh:
      Update test expectations.  "\[u1F6C3]" is now correctly passed
      through, and "\[`a]" correctly coverted to "\[u00E0]".  Shorten test
      cases a little.

And of course NOW I see a typo.  Sigh.

My new idea is to use the same machinery we develop for bug #65108.

This could apply not only to `\X` (or `device`, whichever is easier to do first), but to `output`, `\!`, and `\?`.

Self-corrections...

> And since I changed `device` to stop readings its argument (there's really only one, scanned until the end of the line, line `tm`), here is the "NEWS" entry.

That was a botch.  It should say:

And since I changed `device` to stop reading its argument (there's really only one, scanned until the end of the line, like `tm`) in copy mode, here is the "NEWS" entry.

Also, there is goofery in the synopsis of `device` and `\X` in our Texinfo manual (we need just @var{contents} and no ellipses), and I already have a TODO screenshot on my tablet to remind me to fix that.

comment #25:

> I hear your expression of urgency but I don't think "stringhex" is good long-term solution to what ails us.  You are correct in comment #22 that I did not correctly apprehend at first what it was for.  I thought you developed it because we had no way to reliably transmit arbitrary byte sequences to device control commands.  But we did, sort of--it just needed to be made consistent and reliable.  That it wasn't is what my test case attempts to illustrate and what the fix to bug #64484 attempts to prove.

Yes, I want .device to pass everything exactly "as-is" just as it does now, all escapes left untouched. I don't understand your opposition to stringhex. If I remember correctly you didn't like "shouty" capitals in .SH/Sh and people generally agreed, so you added .stringup/down to enforce the case change - fair enough. Similarly people on the list have asked for pdf support for non-latin languages, and to support all the features currently supported for latin languages I need stringhex or a suitable alternative. So far none of your suggestions will solve the problem, which means that perhaps I have not explained it well enough. Let's look at a real example in the file mom-pdf.mom in the mom/examples directory, it contains:-

.HEADING 2 NAMED internal "Creating internal links"
....
.PP
Furthermore, \*[cod]NAMED\|<id>\*[codx] stores the text of the
heading for use later on when linking to it (see
.PDF_LINK internal SUFFIX ). +
If headings are being numbered, the heading number is prepended.

Which produces what is shown in the attached png file. However, if this was written by a greek person (or google translate!) it might look more like:-

.HEADING 2 NAMED εσωτερικό "Δημιουργία εσωτερικών συνδέσεων"
....
.PP
Επιπλέον, \*[cod]NAMED\|<id>\*[codx] αποθηκεύει το κείμενο του επικεφαλίδα
για χρήση αργότερα κατά τη σύνδεση με αυτήν (βλ
.PDF_LINK εσωτερικό SUFFIX ). +
Εάν οι επικεφαλίδες αριθμούνται, προηγείται ο αριθμός της επικεφαλίδας.

In either case the string at the end of the HEADING line has to be available (using the .PDF_LINK key) so that the "+" symbol can be replaced by the given text. This means that whatever is given as the NAMED <id> has to be used as part of the array which holds the heading strings and for text converted to \[uXXXX] form this would not be a legal identifier name, hence the need to munge it into something acceptable which can be reconstituted.

Your example using \A'' is not helpful since it would reject any non-latin language being used as a NAMED <id>.

> No, I accept your premise that the main driver behind "stringhex" was this:
>
> > The problem lies in the original pdfmark API, if you look at the pdfmark.pdf you will see that in the sections describing .pdfhref M and .pdfhref L which both refer to a "dest-name" and "descriptive text", it says that if a dest-name is not given the first word in the description is used as the dest-name.
>
> I appreciate your explanation.  If the problem was with the pdfmark API, then let's fix the pdfmark API.
>
> In particular, this:
>
> > if a dest-name is not given the first word in the description is used as the dest-name
>
> ...strikes me a short-sighted, especially without any validation going on.  A textual description of a hyperlink/bookmark might contain all sorts of crazy stuff.  (Like Cyrillic or CJK characters or, worse, motion or type-size or font-selection escape sequences.)

The pdfmark API was around well before I wrote gropdf, so I just had to support it.

What about embedded calls to macros (i.e. \*[macro arg]) I deal with that as well. Here's an example from mom-pdf.mom again:-

.AUTHOR "Deri James" "\*[UP .5p]and" "Peter Schaffter"

Mom's AUTHOR macro predates pdf integration so it made sense to just add:-

.pdfinfo /Author \\$*

(or similar) to the macro. Which produces in the grout file:-

x X ps:exec [/Author (Deri James, \v'-.5p'and, Peter Schaffter,) /DOCINFO pdfmark

(using the pdf.tmac on my branch) and if you look at the finished pdf you will see a pristine author attribution in the document properties.

>Assuming that it was going to be a well-behaved sequence of ASCII bytes or even that one could "sanitize" or "cln" one's way through was a hopeless notion.  That won't be practical until we have a string iterator and more conditional expressions that enable the user of an iterator to identify the type of each item in an iterated string/macro/diversion.  But if I understand you correctly, we don't need that fancy new stuff to solve the present problem, with stringhex or without.

No such assumption is made, I expect the data to be dirty, I have removed an*cln, asciify and never used sanitize. Stringhex is to make the NAMED <id> acceptable to be used as a string name. I need none of your "fancy new stuff".

> It would probably benefit me to look up Peter's documentation on mom's "HEADING" macro.  It is a bit baffling to me that one has to repeat arguments like this:
>

> .HEADING 1 NAMED Гуляйпольщина "Гуляйпольщина"
> ...
> .PDF_LINK Гуляйпольщина PREFIX ( SUFFIX ) "see: +"

This is a bad example. Perhaps the real life examples above will help.

> > Where the "+" is replaced by the contents of the string register pdf:look(Гуляйпольщина), which would actually be a string of \[uXXXX] nodes, so would generate an error. This is what stringhex is for, to hide the contents so that groff does not see it as a sequence of nodes. The ideal solution would be to allow string registers to have an attribute (say "glass") which signals that groff should never try to interpret its contents, i.e. operate as if the escape mechanism was turned off just for the contents of that register, and have a way of turning that attribute on/off or an escape which sets the attribute for the enclosed string.
>
> Right now I don't understand why we would need to elaborate a fairly fundamental *roff language data type (the string) with a "glass" attribute when, if you have a list indexed by a number or a valid identifier, you can simply define a string using a list item's index as a prefix.

A numeric id is no good since you don't know what refno has been assigned to each destination target, unless you count the HEADINGs and PDF_TARGETs from the top of the document and insert that number in a PDF_LINK. And, as I hoped you have grasped by now, it would be a bit rich to insist users must use english for their named destinations, simply to ensure a valid identifier.

> .nr refno 1
> .de DEFREF
> .  nr refno +1
> .  ds ref*id!\n[refno]!tag \\$1
> .  ds ref*id!\n[refno]!author \\$2
> .  ds ref*id!\n[refno]!desc \\$3
> .  ds ref*id!\n[refno]!year \\$4
> ..
> .DEFREF story "Dupr\[e aa]" "Best \%Story\%Book Ever" 1989

>
> That's a simplified example of how macro packages have been implementing arrays of data structures for decades, complete with idioms for "*" and "!", which are not imposed by the language in any way.  Maybe I'm missing something.

Yes. The fact that users may wish to use their own language for target destinations.

> As it happens, this bug is probably fixed, too--I simply need to come up with a convincing acceptance criterion for it.  A bit tough without adding a feature to an existing output driver.  I trust it's obvious that, with appropriate escaping, one can transmit "\000\001..\377" or "\x00..\xff" or "\[u0000]..\[u00FF]".

Transmit! Where!! Oh, you mean this bug, about the pdf outline not capturing cyrillic text, which was fixed in my branch 6 months ago, what feature and output driver are you talking about?

> I will try to make some time to reply to comment #22 more thoughtfully soon.  Leaving in "Need Info" status and assigned to myself for that reason.

comment #24:

> Bug #64484 is marked as fixed.

Right, but I believe there was a relationship nevertheless.

> I already have a reliable way to pass byte sequences in device control commands, .stringhex.

Okay.  But it didn't do anything about this failing test case (which admittedly didn't exist until I started to research this issue).

https://git.savannah.gnu.org/cgit/groff.git/diff/src/roff/groff/tests/device-control-special-character-handling.sh?id=974c063f0a9e1ef6c0d2cac4755a3b9d6e925b0d

Of which the salient part is the actual test input:

input='.nf
\X#bogus1: esc \%man-beast\[u1F63C]\\[u1F00] -\[aq]\[dq]\[ga]\[ha]\[rs]\[ti]#
.device bogus1: req \%man-beast\[u1F63C]\\[u1F00] -\[aq]\[dq]\[ga]\[ha]\[rs]\[ti]
.ec @
@X#bogus2: esc @%man-beast@[u1F63C]@@[u1F00] -@[aq]@[dq]@[ga]@[ha]@[rs]@[ti]#
.device bogus2: req @%man-beast@[u1F63C]@@[u1F00] -@[aq]@[dq]@[ga]@[ha]@[rs]@[ti]'

...which looks pretty noisy but tests several things.

1.  Use of \X escape sequences versus `device` requests.
2.  Use of \% escape sequences in device control commands (do they get removed?).
3.  Use of ordinary hyphens in device control commands (do they get converted to some crazy Unicode thing?).
4.  Use of special character escape sequences to represent ASCII characters in device control commands and which should therefore be passed through as ASCII.
5.  Robustness in the face of a changed roff escape character.  This did not work prior to the bug #64484 fix.

> This bug was previously named "warning messages when using special characters in TITLE or AUTHOR" and the attached cyrillic.pdf shows both the pdf title and author shown with cyrillics and no warnings. So I would say this one is dependent on bug #65098, i.e. merge the rest of my branch.

I hear your expression of urgency but I don't think "stringhex" is good long-term solution to what ails us.  You are correct in comment #22 that I did not correctly apprehend at first what it was for.  I thought you developed it because we had no way to reliably transmit arbitrary byte sequences to device control commands.  But we did, sort of--it just needed to be made consistent and reliable.  That it wasn't is what my test case attempts to illustrate and what the fix to bug #64484 attempts to prove.

No, I accept your premise that the main driver behind "stringhex" was this:

> The problem lies in the original pdfmark API, if you look at the pdfmark.pdf you will see that in the sections describing .pdfhref M and .pdfhref L which both refer to a "dest-name" and "descriptive text", it says that if a dest-name is not given the first word in the description is used as the dest-name.

I appreciate your explanation.  If the problem was with the pdfmark API, then let's fix the pdfmark API.

In particular, this:

> if a dest-name is not given the first word in the description is used as the dest-name

...strikes me a short-sighted, especially without any validation going on.  A textual description of a hyperlink/bookmark might contain all sorts of crazy stuff.  (Like Cyrillic or CJK characters or, worse, motion or type-size or font-selection escape sequences.)  Assuming that it was going to be a well-behaved sequence of ASCII bytes or even that one could "sanitize" or "cln" one's way through was a hopeless notion.  That won't be practical until we have a string iterator and more conditional expressions that enable the user of an iterator to identify the type of each item in an iterated string/macro/diversion.  But if I understand you correctly, we don't need that fancy new stuff to solve the present problem, with stringhex or without.

It would probably benefit me to look up Peter's documentation on mom's "HEADING" macro.  It is a bit baffling to me that one has to repeat arguments like this:

.HEADING 1 NAMED Гуляйпольщина "Гуляйпольщина"
...
.PDF_LINK Гуляйпольщина PREFIX ( SUFFIX ) "see: +"

> Where the "+" is replaced by the contents of the string register pdf:look(Гуляйпольщина), which would actually be a string of \[uXXXX] nodes, so would generate an error. This is what stringhex is for, to hide the contents so that groff does not see it as a sequence of nodes. The ideal solution would be to allow string registers to have an attribute (say "glass") which signals that groff should never try to interpret its contents, i.e. operate as if the escape mechanism was turned off just for the contents of that register, and have a way of turning that attribute on/off or an escape which sets the attribute for the enclosed string.

Right now I don't understand why we would need to elaborate a fairly fundamental *roff language data type (the string) with a "glass" attribute when, if you have a list indexed by a number or a valid identifier, you can simply define a string using a list item's index as a prefix.

.nr refno 1
.de DEFREF
.  nr refno +1
.  ds ref*id!\n[refno]!tag \\$1
.  ds ref*id!\n[refno]!author \\$2
.  ds ref*id!\n[refno]!desc \\$3
.  ds ref*id!\n[refno]!year \\$4
..
.DEFREF story "Dupr\[e aa]" "Best \%Story\%Book Ever" 1989

That's a simplified example of how macro packages have been implementing arrays of data structures for decades, complete with idioms for "*" and "!", which are not imposed by the language in any way.  Maybe I'm missing something.

As it happens, this bug is probably fixed, too--I simply need to come up with a convincing acceptance criterion for it.  A bit tough without adding a feature to an existing output driver.  I trust it's obvious that, with appropriate escaping, one can transmit "\000\001..\377" or "\x00..\xff" or "\[u0000]..\[u00FF]".

I will try to make some time to reply to comment #22 more thoughtfully soon.  Leaving in "Need Info" status and assigned to myself for that reason.

Adding dependency on bug #64484, because to achieve the aim of this ticket, a reliable means of transmitting encoded byte sequences to the output device is necessary.  Not all devices need to use the same convention, though that would be nice.

I still owe Deri a reply to comment #22.

comment #20:

> I'll bolster the unit tests I have pending to make sure this works okay for all output devices.

Bolstered and confirmed.

Setting ticket to "Need Info" status to solicit feedback from Deri in particular, and anyone else who wants to comment.

Tricky.  By the time `encode_char` sees the input it has already been tokenized.  This means that what is of interest for my approach is a "token" node, and that in turn means that input that looks like

\X'\[u007E]'

has already effectively been rewritten as if it were

\X'\[ti]'

...so that sucks.  (AIUI, "uniglyph.cpp" does this remapping.)

More thinking required.  A different escape notation might be required after all, but I'm not resigned to that possibility YET.

The new gropdf handles this by not asciifying device parameters any more.

comment #8:

> Nikita,
>
> If you've done any switching between groff 1.22.4 and a recent git build (specifically, any since commit 557bc055 six months ago), the newer code disables these warnings by default, so perhaps this accounts for the difference you were seeing?

I was using the most recent groff build in both of those cases, so I don't think that it explains why the patch stopped working.

comment #12:

> comment #11:
> > The messages which started this bug: "special characters are not defined", have very little to do with the message you recently suppressed.
>
> I followed most of the rest of this message but I didn't see where anyone suppressed anything!  Can you clarify?

I read Deri's statement as a response to my comment #8, wherein I suggested that whether Nikita was seeing the warning was in response to the setting of the recently introduced GROFF_ENABLE_TRANSPARENCY_WARNINGS environment variable.  But of course Deri is correct: the warning message originally reported is not one of the ones under the control of that variable, so my comment was a red herring.

The messages which started this bug: "special characters are not defined", have very little to do with the message you recently suppressed. This message is because groff starts up with the TR font, which has no Cyrillic glyphs, and if the macro package includes a .if 'text'text' statement before the font is switched to U-TR, and compares the output of the two text portions, if either "text" includes glyphs not present in the font, you will receive this warning.

You can prove this with:-

[derij@pip bug-63074]$ printf ".if \'\\[u041D]\'\\[u041D]\' .nop" | groff -Tpdf -z
troff: <standard input>:1: warning: can't find special character 'u041D'

But if you add -f U-T to the groff command there is no error.

[derij@pip bug-63074]$ printf ".if \'\\[u041D]\'\\[u041D]\' .nop" | groff -Tpdf -z -f U-T

The mom macro set has an .if statement in the .TITLE macro which is called before the .FAMILY takes effect.

Now to deal with why Cyrillic glyphs do not appear in the bookmark panel, but do appear in the text of the document. The text is using the embedded fonts which contain the Cyrillic glyphs mapped to appropriate code points. The bookmark panel is using whatever system font you have configured for window text. The system font will have Cyrillic glyphs but they will be using UTF code points, not the 8-bit codes available to a type 1 postscript font.

I suspect, although I have not attempted, that if you set your system to legacy Russian rather than the UTF variant it would be possible to get cyrillics into the bookmark panel.

The pdf standard allows two encodings for strings in pdfs. We are using PDFDocEncoding which is a superset of ISO Latin 1 and does not include cyrillics. The alternative is UTF-16 (UTF-8 is not supported), the string must start with a BOM character, and this would allow any UTF glyph to appear in bookmarks. The reason I used the 8-bit encoding is because the groff .asciify command converts the \[UXXXX] back to ascii for me and as a bonus dropped all other escapes from the string which could not be represented as ascii. So a string such as "\fB\s'+2p'foo\s'-2p'\fP" would be converted to "foo". The only niggle was the warning message (now suppressed) each time it dropped a node such as "\fB".

If I dropped the .asciify from pdf.tmac it would mean all the \[uXXXX] strings would reach the post processor gropdf, which could then assemble a UTF-16 string from the hex numbers. As a proof of concept I made some changes to pdf.tmac and gropdf and pdfmom -k -f U-T mom-ru.mom produced the attached pdf. Still a fair bit to do, the biggest job is to sanitise the string to remove unwanted escapes, convert any glyph producing escapes such as \C and \N back to a UTF-16 character, and convert basic latin characters to UTF-16. I suspect a deep dive into the asciify routine in groff will be helpful.


(file #53761)

Thanks for re-opening this, Dave.

Peter rightly observed that this is not a problem with mom(7).

It is an absent feature in the formatter itself.

I am going to quote in slightly edited form my earlier research to the groff list on this issue.

https://lists.gnu.org/archive/html/groff/2022-09/msg00077.html

...this is our old friend "can't output node in transparent throughput".

...I recently disabled these diagnostics by default in groff Git.

Try regenerating the document with

  GROFF_ENABLE_TRANSPARENCY_WARNINGS=1

(actually, you can set the variable to anything) in your environment.

The problem, I think, is that PDF bookmark generation, like the `pdfinfo`
macro defined in _pdf.tmac_ to include document author and title
metadata, and maybe some other advanced features, relies upon use of
device control escape sequences.  That means '\X' stuff.

In device-independent output ("grout", as I term it), these become "x X"
commands, and the arguments to the escape sequence are, you'd think,
passed through as-is.

The trouble comes with the assumption people make about what "as-is"
means.

The problem is this: what if we want to represent a non-ASCII character
in the device control escape sequence?

groff's device-independent output is, up to a point, strictly ISO Basic
Latin, a property we inherited from AT&T troff.

Except in device control escape sequences.

We have the same problem with the requests that write to the standard
error stream, like `tm`.  I'm not sure that problem is worth solving;
groff's own diagnostic messages are not i18n-ed.  Even if it is worth
solving, teaching device control commands how to interpret more kinds of
"node" seems like a higher priority.

We don't have any infrastructure for handling any character encoding but
the default for input.  That's ISO Latin-1 for most platforms, but IBM
code page 1047 for OS/390 Unix (I think--no one who runs groff on such a
machine has ever spoken with me of their experiences).  And in practice
GNU troff doesn't, as far as I can tell, ever write anything but the 94
graphical code points in ASCII, spaces, and newlines to its output.

I imagine a lot of people's first instinct to fix this is to say, "just
give groff full Unicode support and enable input and output of UTF-8"!

That's a huge ask--bug #40720.

A shorter pole might be to establish a protocol for communication of
Unicode code points within device control commands.  Portability isn't
much of an issue here: as far as I know there has been no effort to
achieve interoperation of device control escape sequences among troffs.

That convention even _could_ be UTF-8, but my initial instinct is _not_
to go that way.  I like the 7-bit cleanliness of GNU troff output, and
when I've mused about solving The Big Unicode Problem I have given
strong consideration to preserving it, or enabling tricked-out UTF-8
"grout" only via an option for the kids who really like to watch their
chrome rims spin.  I realize that Heirloom and neatroff can both boast
of this, but how many people _really_ look at device-independent troff
output?  A few curious people, and the poor saps who are stuck
developing and debugging the implementations, like me.  For the latter
community, a modest and well-behaved format saves a lot of time.

Concretely, when I run the following command:

  GROFF_ENABLE_TRANSPARENCY_WARNINGS=1 ./test-groff -Z -mom -Tpdf -pet \
  -Kutf8 ../contrib/mom/examples/mon_premier_doc.mom

I get the following diagnostics familiar to all who have build groff
1.22.4 from source.

troff:../contrib/mom/examples/mon_premier_doc.mom:28: error: can't
transparently output node at top level

(The foregoing is document metadata going wrong, tripping over the "é"
in "Cicéron".)

troff:../contrib/mom/examples/mon_premier_doc.mom:13: error: can't translate
character code 233 to special character ''e' in transparent throughput
troff:../contrib/mom/examples/mon_premier_doc.mom:30: error: can't translate
character code 233 to special character ''e' in transparent throughput
troff:../contrib/mom/examples/mon_premier_doc.mom:108: error: can't translate
character code 233 to special character ''e' in transparent throughput
troff:../contrib/mom/examples/mon_premier_doc.mom:136: error: can't translate
character code 232 to special character '`e' in transparent throughput

(These are section headings of the document being made into PDF bookmarks.
The headings that happen to be in plain basic Latin have no such trouble.)

More tellingly, if I page the foregoing output with "less -R", I see
non-ASCII code points screaming out their rage in reverse video.

x X ps:exec [/Author (Cic<E9>ron,) /DOCINFO pdfmark

x X ps:exec [/Dest /pdf:bm4 /Title (1. Les diff<E9>rentes versions) /Level 2
/OUT pdfmark

x X ps:exec [/Dest /evolution /Title (2. Les <E9>volutions du Lorem) /Level 2
/OUT pdfmark

x X ps:exec [/Dest /pdf:bm8 /Title (Table des mati<E8>res) /Level 1 /OUT pdfmark

It therefore appears to me that the pdfmark extension to PostScript, or
PostScript itself, happily processes Latin-1...but that means that it
accept _only_ Latin-1, which forecloses the use of Cyrillic code points.

I'm a little concerned that we're blindly _feeding_ the device control
commands characters with the eighth bit set.  It's obviously a useful
expedient for documents like mon_premier_doc.mom.  I am curious to know
why instead of getting no text for headings and titles in the Cyrillic
PDF outline, you didn't get horrendous mojibake garbage--but plainly
Latin-1 garbage at that.

Anyway, some type of mode switching or alternative notation within the
PostScript command stream is required for us to be able to encode
Cyrillic code points.

And once we've figured out what that is, maybe we can teach GNU troff
something about it.  The answer might be to do just whatever works for
PostScript and PDF, since I assume this problem has been solved already,
but it also might mean having our own escaping protocol, which the
output drivers then interpret.

I know of three places it would make sense to support the output of UTF-8, and until I encounter a problem I see no reason not to employ the same solution for all three.

1.  We need to be able to put multibyte UTF-8 sequences into PDFs.  That means encoding them in grout as "x X" device commands.  That in turn means being able to encode them in `\X` escape sequences and `.device` requests.  `\Y` and `.devicem` may have to wait for the resolution of bug #40720, since they will require groff to be able to store UTF-8 code points internally.  Or maybe not, since we already have \[uXXXX].

2. The `tm`, `tm1`, `tmc`, `ab`, and `rd` requests all write to standard error.

3. The `cf`, `lf`, `nx`, `open`, `opena`, `psbb`, and `trf` requests all expect to be able to express (to standard error, in the case of `lf`) or, importantly, open files by name from the filesystem.  Right now groff doesn't have a story for being able to open UTF-8-encoded file names that use continuation bytes.

I can think of two approaches to take.

A. Re-use Unicode named glyph notation \[uXXXX] in these contexts.  The advantages are that we don't need to track any kind of shift state while processing them (see below), the notation will be familiar to experienced groff users, is already explained in groff_char(7) and our Texinfo manual, and its purpose is deducible by novices who haven't read the documentation.

B. We could employ a couple of C0 control characters that groff doesn't already use internally, like STX and ETX (Control+B and Control+C) to shift in and out of a "verbatim" mode where any bytes encountered between the shift characters are given as-is to the next layer of the interface.  (So, they'd appear in grout [case 1] or would end up in arguments to C library calls: fprintf(stderr, ...) [case 2, and `lf` in case 3], and fopen() or similar [the rest of case 3].

(A) has disadvantages.  One is that it's kind of an abuse of the special character/named glyph notation; the whole point of these is that they don't become formatted glyphs.  They are merely a way to encode integers.  Another problem is that there's no obvious way to adapt this to any encoding but UTF-8.  A weisenheimer can say that someone can always re-encode the output if they need to, but that remedy is not available for the file-opening case.  I take a dark view of advising groff users to write an open()-intercepting wrapper in C to be used with LD_PRELOAD.

(B) is more flexible--more accommodating of other character encodings--and seems more like the old-school Unix way to handle the problem, especially back in the days of sundry character encodings warring for supremacy, but, at least as I've sketched it, has the problem that you can't encode the ETX (Control+C) character in the "verbatim region".  Eventually, that limitation will bite someone.

I'm leaning toward (A) because the importance of all encodings other than UTF-8 is dwindling.  But there is plenty of time to wrangle over this and for brilliant new ideas to be pitched, since I see no prospect of this work being undertaken for the groff 1.23 release.

comment #6:

> The problem is with groff itself, not mom... The same issue came
> up just a few days ago and the groff developers are aware of it.

If this is the "PDF outline not capturing Cyrillic text" thread beginning at http://lists.gnu.org/r/groff/2022-09/msg00074.html, there appears to be no existing Savannah bug for this issue, so I'm changing the scope of this one to indicate that this is a core groff issue, not a -mom one.

I've tested my patch again and now it doesn't work to some reason (the warnings are still printed). I don't understand the internals of MOM well enough to figure out what's wrong, so I think it would be better if you come up with your solution.

I get the same behavior with my fonts as you get with yours: the text is rendered correctly but some warnings are still printed (which seems to happen because of the PDF outline, indeed). So I believe that using different fonts doesn't affect anything. If you still want to test with the TM font, it's Microsoft Times font family, installed with your script and obtained from here: http://corefonts.sourceforge.net/. I guess it's considered a proprietary font, so I'm not sure if I'm allowed to share the font files directly here.

comment #2:

> It's not entirely clear what you mean by "characters that are not defined in default fonts."

Oh, I mistyped there. I meant that I get warnings when using Cyrillic characters (that are not defined in default fonts).

I've attached a sample file. Note that the TM font is an installed font that contains Cyrillic symbols.

(file #53732)

This is Peter's dominion; adding him to cc.