Lacking feedback, I reckon I'll move ahead with mostly groff 1.23.0 (and 1.22.4, and 1.22.3) -compatible behavior.

comment #30:

> At 2024-11-16T12:22:25-0500, Deri James wrote:
> > Follow-up Comment #29, bug #64484 (group groff):
> >
> > Branden leans towards rejecting things even though the users intention
> > is clear. For groff there are no spaces, only horizontal movements,
>
> I reject this claim as false.
>
> It is rebuttable in at least 3 respects:

Hi Branden,

It is a bit odd that you rebut my claim "For groff there are no spaces, only horizontal movements" by showing that the statement is in fact correct!

> 1.  the formatter's input language;
> 2.  the formatter's internal representation form ("nodes"); and
> 3.  the formatter's output page description language.
>

[...]

>
> > so should the bookmark "Chapter 1" be rejected because it does contain
> > a horizontal movement!
>
> I'd warn about it, at least.

So, to be clear, if a user entered:-

.pdfbookmark 1 "Chapter 1"

You would warn about the space being a horizontal movement. What about "Lake Attica’s Shores", would that receive two warnings?
 

> > Elsewhere, in terminal output, space and \0 both result in a one
> > character cell space.
>
> As you know well, there are output devices that are not terminals.

A user may expect input destined for a bookmark to be processed similarly to how groff outputs to a terminal, i.e. \0 is delivered as a space.

> > I am sure the user would expect the same when outputting text to a
> > bookmark.
>
> Users expect their desires to be perfectly enacted no matter what the
> quality of the input they give the system.  There are limits to how well
> we can accommodate such expectations, particularly when two different
> users have differerent, but defensible, ones from the same inputs.

I am afraid I can't imagine another expectation other than \0  will result in a space in the text, it is unlikely the expectation would be that it is simply ignored.

[...]

> > Using HEAD & .output
> >
> > x X ps:exec [/Dest /pdf:bm1 /Title (Ph: 01632\0\&444666) /Level 1 /OUT
> > pdfmark
> >
> > The only one which "works" is the final one, i.e. produces what the
> > author would reasonably expect - a space within the phone number - for
> > both the text and the bookmark.
>
> It looks to me like you've added a feature (I assume this is gropdf) to
> interpret a bespoke selection of *roff input escape sequences within
> certain device extension commands tagged "ps:exec".

You have been aware of this for a long time, since, when we discussed this, well before you embarked on your "death march" (as you call it), and I told you that I wanted you to pass through text "as is" (if valid), your advice was that I should not need to do that and you planned some looping construct. Which never appeared.
 

> Where is this behavior documented?  Should *roff users in general expect
> output drivers to implement *roff language interpreters, in whole or
> part?

Previous pdf.tmac used .asciify so this prevented unicode bookmarks, it is documented that these are now accepted, which entails parsing strings passed to gropdf.

> > Worryingly .device in HEAD manages to convert \0 to just "0"
> > which does seem wrong.
>
> Yes, that does seem odd and is worth investigating.  Probably the dummy
> character should be silently discarded too.  And thus likely some other
> things, like `\)`.  Argh.  This is the "switch out of copy mode back
> into interpretation mode, or some kind of 'mode 3'" death march you
> counseled me not to waste my time with, characterizing it as a
> self-imposed goal that would only delay the release.  I believe the
> implication was that no one else would care about it.
>
> Well, on the bright(?) side, the release might be delayed for a while
> anyway, given that I have some documentation to write and, in your
> assessment, a low order of intellect with which to compose it.
>
> https://lists.gnu.org/archive/html/groff/2024-11/msg00131.html

I don't think I have ever cast aspersions about your ability to document groff, so I have no doubt you would make a great job replacing pdfmark.ms, combining stuff from gropdf.1 and comments in pdf.tmac. In fact I know you are so thorough it is likely you are likely to find plenty of bugs around the seldom used edges of pdf.tmac.

The message you reference does not question your intellect. On a number of occasions you have stated you found the coding style of pdfmark.tmac/pdf.tmac opaque and difficult to navigate. I sympathise - it is - it took me awhile to fully understand Keiths code, but I do admire it. Any criticism in that post is not aimed at your intellect, rather it is a disappointment you fired off about 16 commits to pdf.tmac just after I went on sabbatical, so no discussion, to pdf.tmac, a file which you said you had difficulty with, which resulted in bugs being introduced.

The fault is not a lack of intellect, if there is one, it may be in over confidence in changing a file which you claimed to not  fully understand.

> > There have always been differences between \X and .device but after
> > Branden's extensive changes in this area I'm not sure our
> > documentation has caught up, since it appears to now say there is no
> > difference,
>
> Where does it say that?

I read the entry in groff.pdf too fast, missed the fact that later paragraphs only applied to \X.
 

> > commit 2548c4659c appears to be the culprit, also affects -T html.
>
> How does grohtml interpret the following?

grohtml uses use_charnames_in_special so I was trying to give more information to pin down the particular code path causing the issue.
 
[...]

Branden leans towards rejecting things even though the users intention is clear. For groff there are no spaces, only horizontal movements, so should the bookmark "Chapter 1" be rejected because it does contain a horizontal movement! Elsewhere, in terminal output, space and \0 both result in a one character cell space. I am sure the user would expect the same when outputting text to a bookmark.

Let's look at an example:-

.ds dj "Ph: 01632\0\&444666
\*[dj]
\X'ps:exec [/Dest /pdf:bm1 /Title (\*[dj]) /Level 1 /OUT pdfmark'
.\" .device ps:exec [/Dest /pdf:bm1 /Title (\*[dj]) /Level 1 /OUT pdfmark
.\" .output x X ps:exec [/Dest /pdf:bm1 /Title (\*[dj]) /Level 1 /OUT pdfmark

Using HEAD & .device

tPh:
wh2500
t01632
h5000
t444666
wh2500
x X ps:exec [/Dest /pdf:bm1 /Title (Ph: 016320\&444666) /Level 1 /OUT pdfmark

1.23.0 & .device

tPh:
wh2500
t01632
h5000
t444666
wh2500
V12000
H150340
x X ps:exec [/Dest /pdf:bm1 /Title (Ph: 01632\0\&444666) /Level 1 /OUT pdfmark

Using HEAD & \X''

troff:0.trf:3: warning: a horizontal motion is not encodable in device-independent output

tPh:
wh2500
t01632
h5000
t444666
wh2500
x X ps:exec [/Dest /pdf:bm1 /Title (Ph: 01632444666) /Level 1 /OUT pdfmark

Using 1.23.0 & \X''

troff:0.trf:3: warning: a horizontal motion is not encodable in device-independent output

tPh:
wh2500
t01632
h5000
t444666
wh2500
V12000
H150340
x X ps:exec [/Dest /pdf:bm1 /Title (Ph: 01632444666) /Level 1 /OUT pdfmark

Using HEAD & .output

x X ps:exec [/Dest /pdf:bm1 /Title (Ph: 01632\0\&444666) /Level 1 /OUT pdfmark

The only one which "works" is the final one, i.e. produces what the author would reasonably expect - a space within the phone number - for both the text and the bookmark. Worryingly .device in HEAD manages to convert \0 to just "0" which does seem wrong. There have always been differences between \X and .device but after Branden's extensive changes in this area I'm not sure our documentation has caught up, since it appears to now say there is no difference, but the above shows there is.

commit 2548c4659c appears to be the culprit, also affects -T html.


(file #56617)

Bug #63074 covered most of this.

Re-scoping this ticket in light of Deri's comment #26.

We need to decide how to handle non-special character nodes as arguments to `.device` and `\X`, and document this handling sufficiently that the user has a fighting chance to intelligently act on any diagnostics produced.

Hi Deri,

At 2024-09-08T19:22:57-0400, Deri James wrote:

> Follow-up Comment #24, bug #64484 (group groff):
>
> Just a couple of small problems:-
>
> A bookmark like "Minus-hyphens are replaced by \[rs][u2010] in
> bookmark text" will look one way in document text but different in a
> bookmark (same is true using \E).

It sounds like it would be helpful to me to write a unit test
illustrating this very point.  So I'll do that.

> We need a way to differenciate when \E | \(rs are being used to
> prevent me from doing the conversion to a UTF-16 character. You can
> choose.

Yeah, once I see the problem concretely in front of me I'll see what I
can think up.

> The special character \0 is blocked as a horizontal motion, which is
> true as are all space characters in groff, but \~ and "\ " are passed
> as a single space, so should \0 be treated the same, if used in \X.

Maybe "\ " shouldn't be...

Horizontal motions' semantics differ slightly from spaces in the roff
language itself.  Whether they should collapse together in device
extension command arguments...is a good question.  I realize you're
concerned only with `\0` here, but it makes me wonder why we shouldn't
treat it (and "\ ") the same as `\h`.

I don't think we're ever going to purge all cases of needing to rewrite
a string that is used both as document text and metadata.  That is why I
was so gung-ho to implement a `for` request for iterating strings.

For example, if someone does this:

.ds bm this is my foo\h'1i'bar heading
.sp
.ps +2
.pdfbookmark 1 "\*[bm]"
.ft B
\*[bm]
.sp
.ft
.ps

They're just not going to get exact parity between the section heading
appearing in the document and that in the navigation pane.

At some point we have to document the limitations and reduce our
exposure to potential bug report complaining that we aren't DWIM.
roff isn't a DWIM language.

> Also, we need to coordinate when you make .device, .output and \! work
> same way as \X,

Definitely.  However, things are looking to me, right this second, less
dramatic than I feared (planned?).

I still think `output` and `\!` should validate their inputs and not
pass through garbage bytes (this prohibition is already in place [mostly
or totally], and productive of the infamously inscrutable diagnostics),
but I finally perceive a documentable use case for ".output x X" and
"\!x X"; they should lack any of the "help a brother out with his
Unicode" facilities I've been putting into `\X` (and plan for `device`).
`output` and `\!` really should just put ASCII characters into
device-independent with as little transformation as possible.

A concrete, if whimsical, example might be:

\!x X pdf: exec [/Dest /bm1 /Title (30\[en]50 feral hogs) /Level 1 /OUT pdfmark

...where I would expect '\[en]' to show up as-is.  No translation.

It's expert mode for experts.

But maybe a little less demanding of extreme expertise once use cases
are documented.

> since I will need to supply a new gropdf and pdf.tmac at the same
> time. If you can either start a new branch on Savannah, which you can
> cherry pick or rebase from when everything is working, or you can live
> with regressions on master for a month or so.

The first two of those (branch+cherry-pick or rebase) sound okay to me.
While you're hammering on gropdf and pdf.tmac, maybe I can get some of
the other 1.24 release goals sorted out.  For the moment I am concerning
myself with the grout state-flushing problem, which to my relief is
looking like it will require minimal changes outside of the formatter:
one additional request to the definition of the `pdfbackground` macro.
And maybe a cautionary note in the gropdf(1) man page about robotically
going to `\X` for that case; the user will then have to accept
responsibility for flushing the fill color.

I may have to come up with a way to kick the node emitter extra hard,
though, to convince it to write out a 'D F' command when it doesn't
think it applies to anything.  I'll see.

You can watch me thinking out loud at

https://savannah.gnu.org/bugs/?66187

if you like.

Possibly we can get away with, for groff 1.24, only a mechanism for
flushing the fill color.  (I'm not happy with my to-do-for-release list
getting longer while I'm trying to shorten it.)  This stuff is so
esoteric that if anyone we don't already hear from on the groff list
is using it, I'd kind of like to provoke them into complaining to us
so we can work out what the use cases and consequent API should be, so
that we can document and test these things.  What a concept, I know.

Regards,
Branden

Updating summary to reflect new, narrower scope.

Restoring original summary, more or less, and un-postponing.

1.  \X should encode special characters in its argument in `\[uXXXX]` form.  This is done.

2.  The `device` request should do the same.

3.  Neither `\X` or `device` should cause a blanket flush of 5 different things:

   set_font(tf);
   glyph_color(gcol);
   fill_color(fcol);
   flush_tbuf();
   do_motion();

Right now they both do, and this causes documents to regress.  People have undertaken baffling workarounds like:

.      nop \!x X ps:exec [/Dest /\\*[PDFBOOKMARK.NAME] /Title (\\*[pdf:title]) /Level \\n[pdf:bm.lev] /OUT pdfmark

...which really should be expressible as:

.      device ps:exec [/Dest /\\*[PDFBOOKMARK.NAME] /Title (\\*[pdf:title]) /Level \\n[pdf:bm.lev] /OUT pdfmark

(That's from "pdf.tmac"'s `pdfbookmark` macro definition, which is why the backslashes are doubled.)

Extend the `fl` request to take letter arguments encoding which bits of state should marked dirty, or flushed, depending on how you look at it.

f set_font()
m glyph_color()
M fill_color()
t flush_tbuf()
p do_motion()

These aren't the world's best mnemonics.  Lots of collisions here: font/fill/flush, m/M/motion.  But we need something.

Maybe we can read an "identifier" (word) at a time and `strcmp()` it.  That'd be less gross.

Most device extension commands won't need to perform a flush, because they don't dirty any of these states.  sboxes is an exception.  But all it needs to dirty is the fill color (and maybe the glyph color, because of the box edges?)...but not the rest.  `do_motion()` in particular appears to cause grief, and breaks Peter's mom documents, unaligning the shaded boxes with the text that's supposed to be in them.

Possibly once all this is done we won't need the long-undocumented `tag` and `taga` ("tag append") requests anymore.  I get the feeling they were stuck in to avoid the state-flushing problems described above.  One bit I don't understand is what's being "appended": both write out "x X" commands, seemingly right away.  `taga` has very limited use: one invocation in "devtag.tmac", which calls it from one macro "DEVTAG-NEXT", which itself has only one caller anywhere, "DEVTAG-COL-NEXT".  me and ms (and only those) call that, but I don't know if columnated HTML output works with those packages in the first place.  Find out.  If it doesn't, nothing is lost if we "break" it.

Bah, nevermind, I got confused, the difference was no difference.

Time for bed, I think!

Hi Deri,

comment #16:

> Hi Branden,
>
> Thanks for this. I appreciate you explaining what you are trying to do. ... I'd love to see the sboxes issue.

Sure--you'll find it below.
 

> Running your for-deri.man example on current git I also get exactly the same results, so I am not sure what effect your changes are making. If you add "-fU-T" to the command you may see proper greek glyphs rather than slanted stuff from the symbol font, it depends, not all URW fonts are the same, some have proper greek glyphs, some don't. We ought to install the SS font for devpdf, part of the deri-gropdf-ng branch.
>
> I have a couple of one liners to illustrate current \X and .device behaviour, ...

> printf  ".ft U-TR\n\X'ps: αβγabc'\nαβγabc"|test-groff -Tpdf -Kutf8 -Z
>
> printf  ".ft U-TR\n.device ps: αβγabc\nαβγabc"|test-groff -Tpdf -Kutf8 -Z

With my pending changes, including bleeding-edge hacked-up working copy stuff, I get a difference.  Which is bad, but it's not where you're worried about it.

$ printf  ".ft U-TR\n\X'ps: αβγabc'\nαβγabc"|./build/test-groff -Tpdf -Kutf8 -Z > deri-64484-escape.grout
## 2024-08-26 21:37:21 bash-5.1 [1] {0} (0) branden@illithid:~/src/GIT/groff !5831$ printf  ".ft U-TR\n.device ps: αβγabc\nαβγabc"|./build/test-groff -Tpdf -Kutf8 -Z > deri-64484-request.grout
## 2024-08-26 21:37:46 bash-5.1 [1] {0} (0) branden@illithid:~/src/GIT/groff !5832$ diff -u deri-64484-*
--- deri-64484-escape.grout     2024-08-26 21:37:21.629078727 -0500
+++ deri-64484-request.grout    2024-08-26 21:37:46.965034227 -0500
@@ -5,15 +5,14 @@
 V12000
 H72000
 x X ps: \[u03B1]\[u03B2]\[u03B3]abc
-V12000
-H72000
-DFd
-wx font 11 S
+x font 11 S
 f11
 s10000
 x Slant 16
-h2500
+V12000
+H72000
 md
+DFd
 C*a
 h6310
 C*b

I screwed up earlier and shared the same patch twice--the rewritten `device_request()` function.

I mean to share in the first verbatim block a different patch that changed way a "special node" works.

commit 90a7146a91391a257a323895a2bb5b75fb561eb4
Author:     G. Branden Robinson <g.branden.robinson@gmail.com>
AuthorDate: Mon Aug 26 13:08:39 2024 -0500
Commit:     G. Branden Robinson <g.branden.robinson@gmail.com>
CommitDate: Mon Aug 26 17:05:20 2024 -0500

    XXX Fix Savannah #63074 (1/x).

diff --git a/src/roff/troff/node.cpp b/src/roff/troff/node.cpp
index 0b3cbaaea..9e6d962c4 100644
--- a/src/roff/troff/node.cpp
+++ b/src/roff/troff/node.cpp
@@ -886,19 +886,23 @@ inline void troff_output_file::put(unsigned int i)
 void troff_output_file::start_special(tfont *tf, color *gcol, color *fcol,
                                      int no_init_string)
 {
+#if 0
   set_font(tf);
   glyph_color(gcol);
   fill_color(fcol);
   flush_tbuf();
   do_motion();
+#endif
   if (!no_init_string)
     put("x X ");
 }

 void troff_output_file::start_special()
 {
+#if 0
   flush_tbuf();
   do_motion();
+#endif
   put("x X ");
 }

So I think I know why I have the diff above.

But I also suspect this is where I'm running into trouble with sboxes.

Here's a diff of "gropdf -d" output (using Savannah Git HEAD, but I haven't made any changes in my working copy) when I feed it each of the foregoing grouts.

--- esc.pdf     2024-08-26 21:43:17.248345518 -0500
+++ req.pdf     2024-08-26 21:43:08.052366869 -0500
@@ -8,7 +8,7 @@
 /Type /Page
 >>
 endobj
-4 0 obj << /Length 1133
+4 0 obj << /Length 1217
 >>
 stream
 q 1 0 0 1 0 0 cm
@@ -18,25 +18,31 @@
 % V12000
 % H72000
 % x X ps: \[u03B1]\[u03B2]\[u03B3]abc
-%% V12000
-% V12000
-% H72000
-% DFd
-0 g
-% wx font 11 S
+%% x font 11 S
+% x font 11 S
 % f11
 q BT
 1 0 0 1 72.000 780.000 Tm
-
 0 Tc
 % s10000
 % x Slant 16
-% h2500
+% V12000
+1 0 0.287 1 72.000 780.000 Tm
+0 Tc
+% H72000
+0.000 0 Td
 % md
-1 0 0.287 1 74.500 780.000 Tm
+1 0 0.287 1 72.000 780.000 Tm
 0 Tc
 0 g
+% DFd
+ET Q
+0 G
+0 g
 % C*a
+q BT
+1 0 0.287 1 72.000 780.000 Tm
+0 Tc
 % Assign: /alpha to 0/97
 /F11 10 Tf
 % h6310
@@ -48,14 +54,14 @@
 % x font 39 U-TR
 % f39
 %! wht0sz=2.500, wt=--
-%! PutLine: XPOS=74.5, CHR=0/97(a)[/alpha], CWID=6.31, HWID=6.31, NOMV=1
-%! PutLine: XPOS=80.81, CHR=0/98(b)[/beta], CWID=5.49, HWID=5.49, NOMV=1
-%! PutLine: XPOS=86.3, CHR=0/103(g)[/gamma], CWID=4.11, HWID=0, NOMV=1
+%! PutLine: XPOS=72, CHR=0/97(a)[/alpha], CWID=6.31, HWID=6.31, NOMV=1
+%! PutLine: XPOS=78.31, CHR=0/98(b)[/beta], CWID=5.49, HWID=5.49, NOMV=1
+%! PutLine: XPOS=83.8, CHR=0/103(g)[/gamma], CWID=4.11, HWID=0, NOMV=1
 0.000 Tw [ (abg) 411.000 ] TJ
 % x Slant 0
 % h4110
 % tabc
-1 0 0 1 90.410 780.000 Tm
+1 0 0 1 87.910 780.000 Tm
 0 Tc
 % Assign: /a to 0/97
 /F39 10 Tf
@@ -63,23 +69,22 @@
 % Assign: /c to 0/99
 % n12000 0
 %! wht0sz=2.500, wt=--
-%! PutLine: XPOS=90.41, CHR=0/97(a)[/a], CWID=4.44, HWID=4.44, NOMV=0
-%! PutLine: XPOS=94.85, CHR=0/98(b)[/b], CWID=5, HWID=5, NOMV=0
-%! PutLine: XPOS=99.85, CHR=0/99(c)[/c], CWID=4.44, HWID=4.44, NOMV=0
+%! PutLine: XPOS=87.91, CHR=0/97(a)[/a], CWID=4.44, HWID=4.44, NOMV=0
+%! PutLine: XPOS=92.35, CHR=0/98(b)[/b], CWID=5, HWID=5, NOMV=0
+%! PutLine: XPOS=97.35, CHR=0/99(c)[/c], CWID=4.44, HWID=4.44, NOMV=0
 0.000 Tw [ (abc)] TJ
 % x trailer
 % V792000
 % x stop
-1 0 0 1 104.290 0.000 Tm
+1 0 0 1 101.790 0.000 Tm
 0 Tc
 ET Q
-0 G
 Q
 endstream
 endobj
-6 0 obj << /CreationDate (D:20240826214317-05'00')
+6 0 obj << /CreationDate (D:20240826214308-05'00')
 /Creator (groff version 1.23.0.1813-52e3b)
-/ModDate (D:20240826214317-05'00')
+/ModDate (D:20240826214308-05'00')
 /Producer (gropdf version 1.23.0.1813-52e3b)
 >>
 endobj
@@ -230,7 +235,7 @@
 /StemV 0
 /Type /FontDescriptor
 >>
-<< /BaseFont /ANFMBC+NimbusRomNo9L-Regu
+<< /BaseFont /LNRAZN+NimbusRomNo9L-Regu
 /Encoding 12 0 R
 /FirstChar 32
 /FontDescriptor 13 0 R
@@ -249,7 +254,7 @@
 /Flags 32
 /FontBBox [0 -281 1053 924 ]
 /FontFile 10 0 R
-/FontName /ANFMBC+NimbusRomNo9L-Regu
+/FontName /LNRAZN+NimbusRomNo9L-Regu
 /ItalicAngle 0
 /StemV 0
 /Type /FontDescriptor
@@ -264,7 +269,7 @@
 /W [1 4 1 ]
 >>
 stream
-^@^@^@^@^@^@^A^@^@^E<D7>^@^A^@^@^F     ^@^A^@^@^@^O^@^A^@^@^@^?^@^A^@^@^F<CC>^@^A^@^@^E^_^@^B^@^@^@^N^@^B^@^@^@^N^A^B^@^@^@^N^B^A^@^@^HY^@^B^@^@^@^N^C^B^@^@^@^N^D^B^@^@^@^N^E^A^@^@*<D7>^@^A^@^@/<93>^@endstream
+^@^@^@^@^@^@^A^@^@^F+^@^A^@^@^F]^@^A^@^@^@^O^@^A^@^@^@^?^@^A^@^@^G ^@^A^@^@^Es^@^B^@^@^@^N^@^B^@^@^@^N^A^B^@^@^@^N^B^A^@^@^H<AD>^@^B^@^@^@^N^C^B^@^@^@^N^D^B^@^@^@^N^E^A^@^@++^@^A^@^@/<E7>^@endstream
 endobj
 trailer
 <<
@@ -272,5 +277,5 @@
 /Size 16
 >>
 startxref
-12179
+12263
 %%EOF

In okular(1), the two PDFs look the same, but only to casual inspection.  I understand little of PDF, but what I see suggests to me that there are differences in glyph placement here, which are a showstopper.

I suspect I will be grubbing around in node construction and node `tprint`ing code of GNU troff to try and figure out where the different motions are coming from.

> I wonder if it would be possible for you to setup a branch on Savannah with your code changes so we can both investigate for any problems.

The only objection I have to this is a permission or procedural one; Savannah Git is not set up to let me force-push branches.  Or at least it wasn't the last time I did any work on a branch.

My working copy churns like hell and I spend considerable time, once I've cracked a problem, rebasing and massaging my hackery into something resembling coherent change sets.

That's also what I would need to do rebase the changes onto the master branch.  We don't want the master branch's history to look like a chaotic wreck, but the product of at least somewhat considered changes, without too many false starts or trips down the rabbit hole.

But I don't have any problem sharing my work as it stands right now.  I'll generate some patches and attach them.

With all of these attached, all tests pass.

However, there is still a problem to be solved and a decision to be made.

1.  The corruption of the page background in msboxes.pdf.  We don't have an automated test to detect this sort of wrongness.  When we've root-caused it, it would be good to write a minimal case to provoke it since this is a somewhat startling breakage.

2.  Producing groff-man-pages.pdf screams to holy hell over all the `\%` and `\:` tokens appearing in device control commands.  This is of course a variant of the problem you wrote `pdfclean` to solve.
Right now, I'm thinking I will just suppress the diagnostic from being thrown for these known harmless cases.  If I get around to writing my long-promised string iterator (not for 1.24), we can reactivate the diagnostics while giving users a tool to sanitize grout-bound strings.

> which is why I get palpitations when you write (after talking about \X) that "I've just about got the `device` request converted over to the same thing"!! I realise you are talking about the "third way" (which could be a very good thing) but it points to bug #63074 which in turn points to bug #65108 where rule 5d restricts unicode to \[u00XX] 00-1f and 80-FF, a minute subset of unicode.

Sorry, I should have addressed this when you first raised it.

I have no intention of imposing such a code point range restriction on device control commands.  In the context of bug #65108 I was thinking of two other loci of external representation: (a) bytes that get stuffed into `fprintf(stderr, "%s", whatever, ...)` messages, and (b) bytes that get stuffed into `fopen(filename, mode)` calls.  Apart from the basic matter of these things taking arguments of type `const char *` (wide characters go home!) I don't want to add support for POSIX locales to GNU troff for these auxiliary aspects of the formatter.  (And besides, your operating environment's POSIX locale's character encoding and that of the file system you're using might not be same, a whole 'nother class of headache.)  So for `tm` and `so` requests and similar, my intention is that you (the generic user) can specify any character encoding you like as long as you express it in GNU troff's terms as single bytes.  If those form UTF-8, or some ISO 2022 monstrosity, that's between you and your OS.

I have 12 commits in my working tree, but 9 of them aren't material to this discussion:

$ git log --oneline origin..HEAD
5d2401c84 (HEAD -> master) XXX [troff]: Fix Savannah #63074 (3/x).
fdea8aff7 [troff]: Fix Savannah #63074 (2/x).
90a7146a9 XXX Fix Savannah #63074 (1/x).
7a504b6e2 XXX honor_vertical_position_traps
577fdf0c8 [troff]: Trivially refactor (bool `get_location`).
60999fcb0 [troff]: Trivially refactor (`do_underline`).
e2faabad6 [troff]: Migrate class backing `.T` register.
c5d22791b [troff]: Trivially refactor (boolify main() vars).
14aa1ac11 [troff]: Make usage message more helpful.
6b1ff2677 [docs]: Tweak "mac" warning category description.
9a50259a7 refer(1): Drop extraneous word.
aff143e1b HACKING: Fix omitted commands in examples.

I'll attach the last 3 (the first 3 in that reverse-chronological list).

Finally, here's a diff of the grout produced for msboxes.pdf by Savannah Git HEAD and my working copy's HEAD.

$ diff -u msboxes-pdf.grout*
--- msboxes-pdf.grout1  2024-08-26 15:25:09.692875149 -0500
+++ msboxes-pdf.grout2  2024-08-26 22:28:26.429669799 -0500
@@ -281,10 +281,9 @@
 n13000 0
 V250792
 H91692
+x X pdf: background fillbox 80692z 48692z 514581z 757308z 1p
 V250792
 H91692
-mr 42405 10794 10794
-x X pdf: background fillbox 80692z 48692z 514581z 757308z 1p
 n13000 0
 V261792
 H56692
@@ -293,13 +292,10 @@
 n13000 0
 V271792
 H91692
-V271792
-H91692
-md
-DFr 65535 64250 61680
 x X pdf: background pagefill
 f20
-DFd
+V271792
+H91692
 Crs
 h3058
 tX
@@ -1453,9 +1449,9 @@
 n13000 0
 V694292
 H135692
+x X pdf: background off
 V694292
 H135692
-x X pdf: background off
 n13000 0
 V711192
 H56692
@@ -1636,10 +1632,9 @@
 n13000 0
 V64492
 H91692
+x X pdf: background fillbox 80692z 48692z 514581z 757308z 1p
 V64492
 H91692
-mr 42405 10794 10794
-x X pdf: background fillbox 80692z 48692z 514581z 757308z 1p
 n13000 0
 V75492
 H56692
@@ -1653,10 +1648,9 @@
 n13000 0
 V72492
 H91692
+x X pdf: background fill 89692z 56692z 505581z 748308z 0
 V72492
 H91692
-md
-x X pdf: background fill 89692z 56692z 505581z 748308z 0
 n13000 0
 V74492
 H56692
@@ -1706,11 +1700,11 @@
 n13000 0
 V89492
 H91692
-f23
+x X pdf: background off
 V89492
 H91692
-x X pdf: background off
 n13000 0
+f23
 V106392
 H91692
 tbegins
@@ -2014,9 +2008,9 @@
 n13000 0
 V218192
 H91692
+x X pdf: background off
 V218192
 H91692
-x X pdf: background off
 n13000 0
 V235092
 H56692
@@ -2272,10 +2266,9 @@
 n13000 0
 V298792
 H91692
+x X pdf: background fillbox 80692z 48692z 514581z 757308z 1p
 V298792
 H91692
-mr 42405 10794 10794
-x X pdf: background fillbox 80692z 48692z 514581z 757308z 1p
 n13000 0
 V309792
 H56692
@@ -2289,10 +2282,9 @@
 n13000 0
 V306792
 H91692
+x X pdf: background fill 89692z 56692z 505581z 748308z 0
 V306792
 H91692
-md
-x X pdf: background fill 89692z 56692z 505581z 748308z 0
 n13000 0
 V308792
 H56692
@@ -2310,11 +2302,11 @@
 n13000 0
 V323792
 H91692
-f23
+x X pdf: background off
 V323792
 H91692
-x X pdf: background off
 n13000 0
+f23
 V340692
 H91692
 ttak
@@ -2377,9 +2369,9 @@
 n13000 0
 V366692
 H91692
+x X pdf: background off
 V366692
 H91692
-x X pdf: background off
 n13000 0
 V383592
 H56692
@@ -2702,21 +2694,17 @@
 n13000 0
 V481092
 H56692
-x font 6 CR
-f6
+x X pdf: background fillbox 45692z 48692z 549581z 757308z 1p
 V481092
 H56692
-mr 42405 10794 10794
-DFr 65535 65535 65535
-x X pdf: background fillbox 45692z 48692z 549581z 757308z 1p
 n13000 0
 V492092
 H56692
 n13000 0
+x font 6 CR
+f6
 V502092
 H56692
-md
-DFd
 t.
 Crs
 h6600
@@ -5915,9 +5903,9 @@
 n13000 0
 V641692
 H56692
+x X pdf: background off
 V641692
 H56692
-x X pdf: background off
 n13000 0
 V776654
 H538581

That 'mr' command configuring one of the colors (cornsilk, maybe) going missing distresses me.  I suspect it's a side effect of my machete-whack change to `troff_output_file::start_special()` in the first diff above.  But that chop solves other problems--it brings us the "miles better" result in comment #15, which is why I'd like to make "specials" ('x X' commands) more orthogonal.  Maybe there is another way to get the "grout" output to recognize the color selection.

Come to think of it, I was startled to see that there's no `defcolor` request anywhere in "msboxes.ms.in" or "sboxes.tmac".  I guess "ps.tmac" takes care of that, but, (after my change) nothing ever "pushes" that color to the grout unless a glyph gets written with it (or it's used as the fill color in a geometric object).  A background shading or box drawn without roff drawing commands, but by gropdf(1), is not visible to the formatter.

This is the sort of thing I'm gesturing at when I say that maybe the `fl` request should be extended to permit us to express the "dirtying" or usage of some property of the troff environment that is otherwise invisible to the formatter since the grout language has no concept of environments.

If something leaps out at you, I'm keen to hear it.

Regards,
Branden

(file #56386, file #56387, file #56388)

Hi Deri,

comment #14:

> I am a bit concerned about this. pdf.tmac contains various .device commands, which, if replaced by \X stop it working properly.

An understandable concern; one of the reasons this feature is taking a while to land is that I am trying to figure out what the true operational semantics of these direct-grout-generating requests and escape sequences are.

Theoretically, there are four different ways to inject stuff into device-independent output.  (And that's leaving aside the `cf` and `trf` requests, so there are six.  At least.)

\X'this is a device control command'
.br
.device this is a device control command
.br
\!x X this is a device control command
.br
.output x X this is a device control command

Are all of these equivalent?  Not quite--there are subtleties involving line breaks, and even deeper ones involving state transitions of drawing parameters.

In my opinion, a device control command per se shouldn't imply a change of any drawing parameters.  If there is a need to record the fact that a device control command "dirtied" the drawing position, font selection, color configuration, and so on, there should be some separate mechanism of telling the formatter that.

This is causing me frustration right now with "sboxes.tmac"; maybe the `fl` request should be given some kind of state-dirtying semantics.  (At present, and historically, it doesn't do that.)  I won't push until I have it sorted out; I'm trying to avoid asking you to change anything in any macro package.

> I have shown previously that gropdf is quite happy to receive groff nodes as 7 bit ascii i.e. the character "â" can be sent to gropdf as \[u00E2] (preconv) or \[^a] (groff special), both 7 bit clean. However, \X blocks both these uses with an error, which is a little confusing to a user because it refers to "special character '\^a'" even if \[u00E2] appears in the input to groff.

I agree that that's confusing.  You might be pleased to know of a change I have pending.

diff --git a/src/roff/troff/input.cpp b/src/roff/troff/input.cpp
index 89f4518c1..d08fe5e4c 100644
--- a/src/roff/troff/input.cpp
+++ b/src/roff/troff/input.cpp
@@ -5829,6 +5829,7 @@ static node *do_device_control() // \X
   return new special_node(mac);
 }

+# if 0
 static void device_request()
 {
   if (!has_arg(true /* peek; we want to read in copy mode */)) {
@@ -5849,15 +5850,49 @@ static void device_request()
   }
   if (curdiv == topdiv && topdiv->before_first_page)
     topdiv->begin_page();
-  // Null characters can correspond to node types like vmotion_node that
-  // are unrepresentable in a device control command, and got scrubbed
-  // by `asciify`.
-  for (; c != '\0' && c != '\n' && c != EOF;
+  for (; c != '\n' && c != EOF;
        c = get_copy(0 /* nullptr */))
-    mac.append(c);
+    encode_character_for_device_output(&mac, c);
   curenv->add_node(new special_node(mac));
   tok.next();
 }
+#endif
+
+static void device_request()
+{
+  if (!has_arg()) {
+    warning(WARN_MISSING, "device control request expects arguments");
+    skip_line();
+    return;
+  }
+  macro mac;
+  while (tok.is_space() || tok.is_tab())
+    tok.next();
+  if ('"' == tok.ch())
+    tok.next();
+  for (;;) {
+    unsigned char c;
+    if (tok.is_newline() || tok.is_eof())
+      break;
+    if (tok.is_space())
+      c = ' ';
+    else if (tok.is_tab())
+      c = '\t';
+    else if (tok.is_leader())
+      c = '\001';
+    else if (tok.is_backspace())
+      c = '\b';
+    else
+      c = tok.ch();
+    //assert(c != 0); // XXX: a node?
+    encode_character_for_device_output(&mac, c);
+    tok.next();
+  }
+  if (curdiv == topdiv && topdiv->before_first_page)
+    topdiv->begin_page();
+  curenv->add_node(new special_node(mac));
+  skip_line();
+}

 static void device_macro_request()
 {

I've enhanced a regression test I added in January that attempts to ensure that processing of various special character escape sequences comes through in device-independent output unmolested.

Here's the test input:

input='.
.nf
\X#bogus1: esc \%to-do\[u1F63C]\\[u1F00] -\[aq]\[dq]\[ga]\[ha]\[rs]\[ti]\[`a]#
.device bogus1: req \%to-do\[u1F63C]\\[u1F00] -\[aq]\[dq]\[ga]\[ha]\[rs]\[ti]\[`a]
.ec @
@X#bogus2: esc @%to-do@[u1F63C]@@[u1F00] -@[aq]@[dq]@[ga]@[ha]@[rs]@[ti]@[`a]##
.device bogus2: req @%to-do@[u1F63C]@@[u1F00] -@[aq]@[dq]@[ga]@[ha]@[rs]@[ti]@[`a]
.'

...and the results.[1]

$ (cd build && ../src/roff/groff/tests/device-control-special-character-handling.sh)
x X bogus1: esc to-do\[u1F00] -'"`^\~
x X bogus1: req @%to-do\[u1F63C]\[u1F00] -\[aq]\[dq]\[ga]\[ha]\[rs]\[ti]\[`a]
x X bogus2: esc to-do\[u1F00] -'"`^\~
x X bogus2: req @%to-do@[u1F63C]@[u1F00] -@[aq]@[dq]@[ga]@[ha]@[rs]@[ti]@[`a]
troff:<standard input>:2: error: special character 'u1F63C' cannot be used within a device control escape sequence
troff:<standard input>:2: error: special character '`a' cannot be used within a device control escape sequence
troff:<standard input>:5: error: special character 'u1F63C' cannot be used within a device control escape sequence
troff:<standard input>:5: error: special character '`a' cannot be used within a device control escape sequence
checking X escape sequence, default escape character
...FAILED
checking X escape sequence, alternate escape character
...FAILED
checking for errors on unsupported special character escapes

That doesn't look good, but when I add some code...

diff --git a/src/roff/troff/input.cpp b/src/roff/troff/input.cpp
index 229e7956e..041a455e7 100644
--- a/src/roff/troff/input.cpp
+++ b/src/roff/troff/input.cpp
@@ -5832,6 +5832,7 @@ static node *do_device_control() // \X
   return new special_node(mac);
 }

+# if 0
 static void device_request()
 {
   if (!has_arg(true /* peek; we want to read in copy mode */)) {
@@ -5852,15 +5853,49 @@ static void device_request()
   }
   if (curdiv == topdiv && topdiv->before_first_page)
     topdiv->begin_page();
-  // Null characters can correspond to node types like vmotion_node that
-  // are unrepresentable in a device control command, and got scrubbed
-  // by `asciify`.
-  for (; c != '\0' && c != '\n' && c != EOF;
+  for (; c != '\n' && c != EOF;
        c = get_copy(0 /* nullptr */))
-    mac.append(c);
+    encode_character_for_device_output(&mac, c);
   curenv->add_node(new special_node(mac));
   tok.next();
 }
+#endif
+
+static void device_request()
+{
+  if (!has_arg()) {
+    warning(WARN_MISSING, "device control request expects arguments");
+    skip_line();
+    return;
+  }
+  macro mac;
+  while (tok.is_space() || tok.is_tab())
+    tok.next();
+  if ('"' == tok.ch())
+    tok.next();
+  for (;;) {
+    unsigned char c;
+    if (tok.is_newline() || tok.is_eof())
+      break;
+    if (tok.is_space())
+      c = ' ';
+    else if (tok.is_tab())
+      c = '\t';
+    else if (tok.is_leader())
+      c = '\001';
+    else if (tok.is_backspace())
+      c = '\b';
+    else
+      c = tok.ch();
+    //assert(c != 0); // XXX: a node?
+    encode_character_for_device_output(&mac, c);
+    tok.next();
+  }
+  if (curdiv == topdiv && topdiv->before_first_page)
+    topdiv->begin_page();
+  curenv->add_node(new special_node(mac));
+  skip_line();
+}

 static void device_macro_request()
 {

...and with which the `BOXSTART` macro is unhappy (the page background turns completely black), I get the following.

$ (cd build && ../src/roff/groff/tests/device-control-special-character-handling.sh)
x X bogus1: esc to-do\[u1F63C]\[u1F00] -'"`^\~\[u00E0]
x X bogus1: req to-do\[u1F63C]\[u1F00] -'"`^\~\[u00E0]
x X bogus2: esc to-do\[u1F63C]\[u1F00] -'"`^\~\[u00E0]
x X bogus2: req to-do\[u1F63C]\[u1F00] -'"`^\~\[u00E0]

checking X escape sequence, default escape character
checking X escape sequence, alternate escape character
checking for errors on unsupported special character escapes

That's just miles better.

> If I understand correctly, your plan is linked to the work you have been doing with filenames (bug #65108, comment 3), which outlines your parsing rules. The restriction in your rule 5d would prevent "strings" of characters in other languages to be used. Many file systems allow utf-8 in filenames:-

> -rw-r--r-- 1 derij derij         0 Aug 26 21:40  αβγ.greek

> Which would fail 5d, and would prevent pdf bookmarks in any language except basic latin or latin-1 supplement, if these rules are extended to .device.
>
> Have I understood this correctly?

I'm not sure.  I don't think so.

Running my working copy with above BOXSTART-breaking patch applied, I can do this.

$ cat ATTIC/for-deri.man
.TH for\-deri 1 2024-08-26 "a demo for Deri"
.SH Name
for\-deri \- a sample command
.SH Description
What program requires documentation?
.SH αβγ.greek
That was some Greek,
and it will end up in a device control command when we format this man
page for PDF.
$ ./build/test-groff -K utf8 -man -T pdf -Z ATTIC/for-deri.man | grep '^x X'
x X ps:exec [/Dest /for\-deri(1) /View [/FitH -26000 u] /DEST pdfmark
x X ps:exec [/Dest /for\-deri(1) /Title (for\-deri(1)) /Level 1 /OUT pdfmark
x X pdf: markrestart
x X ps:exec [/Dest /pdf:bm2 /View [/FitH -57000 u] /DEST pdfmark
x X ps:exec [/Dest /pdf:bm2 /Title (Name) /Level 2 /OUT pdfmark
x X devtag:.NH 1
x X devtag:.eo.h
x X ps:exec [/Dest /pdf:bm3 /View [/FitH -85800 u] /DEST pdfmark
x X ps:exec [/Dest /pdf:bm3 /Title (Description) /Level 2 /OUT pdfmark
x X devtag:.NH 1
x X devtag:.eo.h
x X ps:exec [/Dest /pdf:bm4 /View [/FitH -114600 u] /DEST pdfmark
x X ps:exec [/Dest /pdf:bm4 /Title (\[u03B1]\[u03B2]\[u03B3].greek) /Level 2 /OUT pdfmark
x X devtag:.NH 1
x X devtag:.eo.h
x X pdf: marksuspend

...so the Greek seems to show up fine.

Actually, the "grout" output is unchanged from what's on Savannah's HEAD right now.  So, that extent, my plans are to not break what you're afraid I'm going to break.

I think.

Does this illuminate things?

Despite this ticket's postponed status, it might end up fixed as part of getting bug #63074 over the finish line.  But only as much of it as I need for that purpose.  Time will tell if that's the whole enchilada for this ticket.

Regards,
Branden

[1] In case anyone's curious what older groffs did with that...

$ (cd build && ../src/roff/groff/tests/device-control-special-character-handling.sh)
GNU groff version 1.23.0
x X bogus1: esc to-do\[u1F00] -'"`^\~
x X bogus1: req @%to-do\[u1F63C]\[u1F00] -\[aq]\[dq]\[ga]\[ha]\[rs]\[ti]\[`a]
x X bogus2: esc to-do@[u1F00] -'"`^\~
x X bogus2: req @%to-do@[u1F63C]@[u1F00] -@[aq]@[dq]@[ga]@[ha]@[rs]@[ti]@[`a]

checking X escape sequence, default escape character
...FAILED
checking X escape sequence, alternate escape character
...FAILED
checking for errors on unsupported special character escapes

$ (cd build && ../src/roff/groff/tests/device-control-special-character-handling.sh)
GNU groff version 1.22.4
x X bogus1: esc to-do\[u1F00] -
x X bogus1: req @%to-do\[u1F63C]\[u1F00] -\[aq]\[dq]\[ga]\[ha]\[rs]\[ti]\[`a]
x X bogus2: esc to-do@[u1F00] -
x X bogus2: req @%to-do@[u1F63C]@[u1F00] -@[aq]@[dq]@[ga]@[ha]@[rs]@[ti]@[`a]
troff: <standard input>:3: a special character is invalid within \X
troff: <standard input>:3: a special character is invalid within \X
troff: <standard input>:3: a special character is invalid within \X
troff: <standard input>:3: a special character is invalid within \X
troff: <standard input>:3: a special character is invalid within \X
troff: <standard input>:3: a special character is invalid within \X
troff: <standard input>:3: a special character is invalid within \X
troff: <standard input>:3: a special character is invalid within \X
troff: <standard input>:6: a special character is invalid within \X
troff: <standard input>:6: a special character is invalid within \X
troff: <standard input>:6: a special character is invalid within \X
troff: <standard input>:6: a special character is invalid within \X
troff: <standard input>:6: a special character is invalid within \X
troff: <standard input>:6: a special character is invalid within \X
troff: <standard input>:6: a special character is invalid within \X
troff: <standard input>:6: a special character is invalid within \X
checking X escape sequence, default escape character
...FAILED
checking X escape sequence, alternate escape character
...FAILED
checking for errors on unsupported special character escapes

Waffling on this.  `\X` seems to be mutating in such a way that it has grown, alas, a third mode of argument interpretation, neither "normal" ("interpretation") or "copy".

I've just about got the `device` request converted over to the same thing (for bug #63074) and nearly everything looks very nice.

I blanch at the pedagogical challenge I've created for myself; whatever this new mode is, likely it will need to apply to the `output` request and `\!` escape sequence as well.  All of these for the same reason: there's stuff, foremost "nodes", that have representations inside the formatter but none outside of it.  And the whole point of these escape sequences and requests is to slip the surly bonds of the formatter, seizing the yoke from it and attempting to fly the device-independent page description language oneself.

Little wonder that such attempts usually end with a swift embrace of the ground.

Postponing; I don't have hope of getting this sorted out in time for groff 1.24.

Two differences between the `device` request and `\X` escape sequence concern me and, at the very least, should be documented.

1.  `device` reads its argument in copy mode; `\X` doesn't.
2.  `\X` is accepted, and works, at the beginning of input; `device` does not.

Here are some illustrative cases.

$ printf '\\X@tty: foo\\h#1i#bar@\n' | groff -T ascii -Z
troff:<standard input>:1: error: a horizontal motion is invalid within device control escape sequence
x T ascii
x res 240 24 40
x init
p1
V40
H0
x font 1 R
f1
s10
V40
H0
md
DFd
x X tty: foobar
n40 0
x trailer
V2640
x stop
$ printf '.device tty: foo\\h#1i#bar\n' | groff -T ascii -Z
x T ascii
x res 240 24 40
x init
p1
x trailer
V2640
x stop
$ printf '\\c\n.device tty: foo\\h#1i#bar\n' | groff -T ascii -Z
x T ascii
x res 240 24 40
x init
p1
V40
H0
x font 1 R
f1
s10
V40
H0
md
DFd
x X tty: foo\h#1i#bar
n40 0
x trailer
V2640
x stop

Having second thoughts about the direction in which I achieved parity here.

Maybe both `device` and `\X` should be reading their argument in copy mode.

Technically, though, that would break compatibility with AT&T troff.  However, there are no cross-formatter conventions for the format or content of device control commands in the first place, so I don't expect this to be a practical issue.

Proposal/discussion thread:

https://lists.gnu.org/archive/html/groff/2024-01/msg00060.html

My last paragraph in comment #6 was a bit mangled.

Let me try that again.

The foregoing means that I should abandon my plan to make a new fatal error in response to bug #65138 and also that the scenario in question is probably a corner case.  Generally, you won't issue device control command with a request but never even once have a partially collected line in your document.  A unit test might do that, but a practical document likely will not.

Another oddity is that you can invoke the `device` request too early, and it won't be honored, but can use `\X` anywhere you want.

$ cat EXPERIMENTS/device-control-escape-sequence.roff
\X'ps: nop'
$ ~/groff-stable/bin/groff -Z EXPERIMENTS/device-control-escape-sequence.roff
x T ps
x res 72000 1 1
x init
p1
V12000
H72000
x font 5 TR
f5
s10000
V12000
H72000
md
DFd
x X ps: nop
n12000 0
x trailer
V792000
x stop
$ cat EXPERIMENTS/device-control-request.roff
.device ps: nop
$ ~/groff-stable/bin/groff -Z EXPERIMENTS/device-control-request.roff
x T ps
x res 72000 1 1
x init
p1
x trailer
V792000
x stop

comment #2:

> There are 4 separate ways to send text to a postprocessor.

Thanks, Deri.  I've documented all of these but did not consider them this way.  This is good stuff.

> \X is the only one which attempts to "asciify" the parameters.

Yes.  My instinct it to change this so that the `device` request does the same processing as `\X`.

> There is a difference between the two pairs of commands, if you remove the .fl and rerun:-

> x T ps
> x res 72000 1 1
> x init
> p1
>  ! \(ti deri
> output \(ti deri
> tm \(ti deri
> V12000
> H72000
> x font 5 TR
> f5
> s10000
> V12000
> H72000
> md
> DFd
> x X X ~ deri
> wh2500
> V12000
> H74500
> x X device \(ti deri
> n12000 0
> x trailer
> V792000
> x stop

> .output and \! are now out of chronological order.

This is well worth studying.  I think that thanks to you we have at last identified a use case for the `fl` request in GNU troff (as opposed to AT&T nroff, where it had an application in interactive use).  I've seen macro packages use it pointlessly.

> In gropdf I used to rely on manual "asciifying" of parameters, the new gropdf does not, in order to support unicode (passed as \[uXXXX]) I need it to be passed untouched.
>
> so it can be converted to UTF-16 by gropdf itself (also applies to any special chars - \(xx \[xxx] \C'xxx' \N'nnn'). It also needs to maintain chronological order.

I think we might be in for some design changes here in the 1.24 cycle.

TANAKA Takuji's patches to grops in bug #62830 have what look to me like similar concerns regarding the production of UTF-16 by the output driver.

> I don't know whether the pdfmark macros rely on the asciifying behaviour of \X or not.

I don't either.

I'd like `device` and `\X` to be counterparts, as `output` and `\!` appear to be.

Even better would be for `sy`, `tm`, and other requests that access the standard error stream or the file system to use the same mechanism for encoding characters.  See bug #64071.

This stuff is going to require some thought.

Assigning to Deri to solicit his perspective on this ticket, about which I am conflicted.