bugGNU roff - Bugs: bug #63074, develop convention for encoding...

 
 

bug #63074: develop convention for encoding Unicode character sequences for passage to device control commands

Submitter:  Nikita Ivanov <nikitaivanov>
Submitted:  Sat 17 Sep 2022 05:01:15 PM UTC
   
 
Category:  General Severity:  1 - Wish
Item Group:  Feature change Status:  Need Info
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 16 Jan 2024 03:05:05 PM UTC, comment #30: 

Deri's right that his `stringhex` solution, and the underlying problem it solves, aren't fundamentally about how the formatter talks to the device driver (though that is ultimately a necessary step), but how macro packages can sensibly pass Unicode character sequences internally and among each other without trying, and failing, to pun between *roff string contents and *roff identifiers, a bit of code jockeying that worked fine back in the days when everything was ASCII (or at least ISO 8859).

Re-categorizing to "general" and updating Summary.

G. Branden Robinson <gbranden>
Group administrator
Tue 16 Jan 2024 02:59:17 PM UTC, comment #29: 

Self-corrections...

> And since I changed `device` to stop readings its argument (there's really only one, scanned until the end of the line, line `tm`), here is the "NEWS" entry.


That was a botch.  It should say:

And since I changed `device` to stop reading its argument (there's really only one, scanned until the end of the line, like `tm`) in copy mode, here is the "NEWS" entry.

Also, there is goofery in the synopsis of `device` and `\X` in our Texinfo manual (we need just @var{contents} and no ellipses), and I already have a TODO screenshot on my tablet to remind me to fix that.

G. Branden Robinson <gbranden>
Group administrator
Tue 16 Jan 2024 02:53:50 PM UTC, comment #28: 

Hi Deri,

comment #27:

> Yes, I want .device to pass everything exactly "as-is" just as it does now, all escapes left untouched.


Unfortunately, that is inconsistent with how \X works.


$ cat EXPERIMENTS/device-foolery.groff
.sp
.device ps: \fB
.device ps: \s0
.device ps: \-
.device ps: \[*A]
.device ps: \[u0391]
$ ~/groff-1.22.3/bin/groff -Z EXPERIMENTS/device-foolery.groff | grep '^x X'
x X ps: \fB
x X ps: \s0
x X ps: \-
x X ps: \[*A]
x X ps: \[u0391]


The foregoing output is the same in groff 1.22.4 and 1.23.0.  But...


$ cat EXPERIMENTS/backslash-X-foolery.roff
.sp
\X'ps: \fB'\
\X'ps: \s0'\
\X'ps: \-'\
\X'ps: \[*A]'\
\X'ps: \[u0391]'
$ ~/groff-1.22.3/bin/groff -Z EXPERIMENTS/backslash-X-foolery.roff | grep '^x X'
EXPERIMENTS/backslash-X-foolery.roff:4: a special character is invalid within \X
EXPERIMENTS/backslash-X-foolery.roff:5: a special character is invalid within \X
EXPERIMENTS/backslash-X-foolery.roff:6: a special character is invalid within \X
x X ps:
x X ps:
x X ps:
x X ps:
x X ps:


The foregoing output is the same in groff 1.22.4.  In groff 1.23.0, it differs.


$ ~/groff-stable/bin/groff -Z EXPERIMENTS/backslash-X-foolery.roff | grep '^x X'
x X ps:
x X ps:
x X ps: -
x X ps:
x X ps:


This was the outcome of bug #61401 (October 2021).

I'd like to get `.device` and `\X` treating their arguments the same way, hence bug #64484.  You can still pass things representing arbitrary byte sequences to the output driver--but if you want to represent them with groff escape sequences, you will need to further escape them.


$ cat EXPERIMENTS/backslash-X-escapery.roff
.sp
\X'ps: \\fB'\
\X'ps: \\s0'\
\X'ps: \\-'\
\X'ps: \\[*A]'\
\X'ps: \\[u0391]'
$ ~/groff-HEAD/bin/groff -Z EXPERIMENTS/backslash-X-escapery.roff | grep '^x X'
x X ps: \fB
x X ps: \s0
x X ps: \-
x X ps: \[*A]
x X ps: \[u0391]


And `device` request behavior is now (in Git) consistent with this.


$ cat ./EXPERIMENTS/device-escapery.groff
.sp
.device ps: \\fB
.device ps: \\s0
.device ps: \\-
.device ps: \\[*A]
.device ps: \\[u0391]
$ ~/groff-HEAD/bin/groff -Z EXPERIMENTS/device-escapery.groff | grep '^x X'
x X ps: \fB
x X ps: \s0
x X ps: \-
x X ps: \[*A]
x X ps: \[u0391]


Here is what our Texinfo manual now says (in Git) about `device` and `\X`.


5.34 Postprocessor Access
=========================

Two escape sequences and two requests enable documents to pass
information directly to an output driver or other postprocessor.  These
are useful for exercising device-specific capabilities that the 'groff'
language does not abstract or generalize; examples include the embedding
of hyperlinks and image files.  Device-specific functions are documented
in each output driver's man page, such as 'gropdf(1)', 'grops(1)', or
'grotty(1)'.

 -- Request: .device xxx ...
 -- Escape sequence: \X'''xxx ...'''
     Embed all XXX arguments into GNU 'troff' output as parameters to an
     'x X' device control command.(1)  (*note Postprocessor
     Access-Footnote-1::) The meaning and interpretation of such
     parameters is determined by the output driver or other
     postprocessor.

     The 'device' request strips an initial neutral double quote from
     CONTENTS to allow embedding of leading spaces.

     Within a device control command, the escape sequences '\&', '\)',
     '\%', and '\:' are ignored; '\<SPC>' and '\~' are converted to
     single space characters; and '\\' has its escape character
     stripped.  So that the basic Latin subset of the Unicode character
     set(2) (*note Postprocessor Access-Footnote-2::) can be reliably
     encoded in device control commands, seven special character escape
     sequences ('\-', '\[aq]', '\[dq]', '\[ga]', '\[ha]', '\[rs]', and
     '\[ti]') are mapped to basic Latin characters; see the
     'groff_char(7)' man page.  For this transformation, character
     translations and special character definitions are ignored.(3)
     (*note Postprocessor Access-Footnote-3::)

     Escape sequences other than the foregoing in device control command
     may be ignored, or produce an error.

     A device control command issued with the 'device' request will not
     be reflected in the output unless a partially collected line exists
     at least once in the top-level diversion (recall *note
     Diversions::).  When experimenting with such device controls in
     minimal documents, a 'br' request will ensure this to be the case.

     If the 'use_charnames_in_special' directive appears in the output
     device's 'DESC' file, the use of special character escape sequences
     is _not_ an error; they are simply output verbatim (with the
     exception of the seven mapped to Unicode basic Latin characters,
     discussed above).  'use_charnames_in_special' is currently employed
     only by 'grohtml'.

[.devicem and \Y snipped]


And since I changed `device` to stop readings its argument (there's really only one, scanned until the end of the line, line `tm`), here is the "NEWS" entry.


o The `device` request no longer reads its arguments in copy mode; this
  change makes it more consistent with the `\X` device control command
  escape sequence.  This request also no longer emits a self-quoted
  *roff escape character as itself, but instead as a backslash.
  (troff's input and output languges are not the same thing.)  These
  changes are to enable postprocessors to reliably interpret device
  control commands that wish to express arbitrary byte sequences.  For
  example, PDF bookmarks need to be expressed in UTF-16LE.


...but I perceive that (1) I should probably break that up into 2 items, and (2) I should add diagnostics to `encode_char_for_troff_input()` when it encounters something incomprehensible like a font selection or type size escape sequence.

This part of the discussion is pretty much independent of to-stringhex-or-not-to-stringhex (as you've been trying to tell me), so I will reply to that separately.  If you have comments on the foregoing, please direct them to bug #64484.

G. Branden Robinson <gbranden>
Group administrator
Tue 16 Jan 2024 12:44:42 AM UTC, comment #27: 

comment #25:

> I hear your expression of urgency but I don't think "stringhex" is good long-term solution to what ails us.  You are correct in comment #22 that I did not correctly apprehend at first what it was for.  I thought you developed it because we had no way to reliably transmit arbitrary byte sequences to device control commands.  But we did, sort of--it just needed to be made consistent and reliable.  That it wasn't is what my test case attempts to illustrate and what the fix to bug #64484 attempts to prove.


Yes, I want .device to pass everything exactly "as-is" just as it does now, all escapes left untouched. I don't understand your opposition to stringhex. If I remember correctly you didn't like "shouty" capitals in .SH/Sh and people generally agreed, so you added .stringup/down to enforce the case change - fair enough. Similarly people on the list have asked for pdf support for non-latin languages, and to support all the features currently supported for latin languages I need stringhex or a suitable alternative. So far none of your suggestions will solve the problem, which means that perhaps I have not explained it well enough. Let's look at a real example in the file mom-pdf.mom in the mom/examples directory, it contains:-

.HEADING 2 NAMED internal "Creating internal links"
....
.PP
Furthermore, \*[cod]NAMED\|<id>\*[codx] stores the text of the
heading for use later on when linking to it (see
.PDF_LINK internal SUFFIX ). +
If headings are being numbered, the heading number is prepended.

Which produces what is shown in the attached png file. However, if this was written by a greek person (or google translate!) it might look more like:-

.HEADING 2 NAMED εσωτερικό "Δημιουργία εσωτερικών συνδέσεων"
....
.PP
Επιπλέον, \*[cod]NAMED\|<id>\*[codx] αποθηκεύει το κείμενο του επικεφαλίδα
για χρήση αργότερα κατά τη σύνδεση με αυτήν (βλ
.PDF_LINK εσωτερικό SUFFIX ). +
Εάν οι επικεφαλίδες αριθμούνται, προηγείται ο αριθμός της επικεφαλίδας.

In either case the string at the end of the HEADING line has to be available (using the .PDF_LINK key) so that the "+" symbol can be replaced by the given text. This means that whatever is given as the NAMED <id> has to be used as part of the array which holds the heading strings and for text converted to \[uXXXX] form this would not be a legal identifier name, hence the need to munge it into something acceptable which can be reconstituted.

Your example using \A'' is not helpful since it would reject any non-latin language being used as a NAMED <id>.

> No, I accept your premise that the main driver behind "stringhex" was this:
>
> > The problem lies in the original pdfmark API, if you look at the pdfmark.pdf you will see that in the sections describing .pdfhref M and .pdfhref L which both refer to a "dest-name" and "descriptive text", it says that if a dest-name is not given the first word in the description is used as the dest-name.
>
> I appreciate your explanation.  If the problem was with the pdfmark API, then let's fix the pdfmark API.
>
> In particular, this:
>
> > if a dest-name is not given the first word in the description is used as the dest-name
>
> ...strikes me a short-sighted, especially without any validation going on.  A textual description of a hyperlink/bookmark might contain all sorts of crazy stuff.  (Like Cyrillic or CJK characters or, worse, motion or type-size or font-selection escape sequences.)


The pdfmark API was around well before I wrote gropdf, so I just had to support it.

What about embedded calls to macros (i.e. \*[macro arg]) I deal with that as well. Here's an example from mom-pdf.mom again:-

.AUTHOR "Deri James" "\*[UP .5p]and" "Peter Schaffter"

Mom's AUTHOR macro predates pdf integration so it made sense to just add:-

.pdfinfo /Author \\$*

(or similar) to the macro. Which produces in the grout file:-

x X ps:exec [/Author (Deri James, \v'-.5p'and, Peter Schaffter,) /DOCINFO pdfmark

(using the pdf.tmac on my branch) and if you look at the finished pdf you will see a pristine author attribution in the document properties.

>Assuming that it was going to be a well-behaved sequence of ASCII bytes or even that one could "sanitize" or "cln" one's way through was a hopeless notion.  That won't be practical until we have a string iterator and more conditional expressions that enable the user of an iterator to identify the type of each item in an iterated string/macro/diversion.  But if I understand you correctly, we don't need that fancy new stuff to solve the present problem, with stringhex or without.


No such assumption is made, I expect the data to be dirty, I have removed an*cln, asciify and never used sanitize. Stringhex is to make the NAMED <id> acceptable to be used as a string name. I need none of your "fancy new stuff".

> It would probably benefit me to look up Peter's documentation on mom's "HEADING" macro.  It is a bit baffling to me that one has to repeat arguments like this:
>


> .HEADING 1 NAMED Гуляйпольщина "Гуляйпольщина"
> ...
> .PDF_LINK Гуляйпольщина PREFIX ( SUFFIX ) "see: +"


This is a bad example. Perhaps the real life examples above will help.

> > Where the "+" is replaced by the contents of the string register pdf:look(Гуляйпольщина), which would actually be a string of \[uXXXX] nodes, so would generate an error. This is what stringhex is for, to hide the contents so that groff does not see it as a sequence of nodes. The ideal solution would be to allow string registers to have an attribute (say "glass") which signals that groff should never try to interpret its contents, i.e. operate as if the escape mechanism was turned off just for the contents of that register, and have a way of turning that attribute on/off or an escape which sets the attribute for the enclosed string.
>
> Right now I don't understand why we would need to elaborate a fairly fundamental *roff language data type (the string) with a "glass" attribute when, if you have a list indexed by a number or a valid identifier, you can simply define a string using a list item's index as a prefix.


A numeric id is no good since you don't know what refno has been assigned to each destination target, unless you count the HEADINGs and PDF_TARGETs from the top of the document and insert that number in a PDF_LINK. And, as I hoped you have grasped by now, it would be a bit rich to insist users must use english for their named destinations, simply to ensure a valid identifier.


> .nr refno 1
> .de DEFREF
> .  nr refno +1
> .  ds ref*id!\n[refno]!tag \\$1
> .  ds ref*id!\n[refno]!author \\$2
> .  ds ref*id!\n[refno]!desc \\$3
> .  ds ref*id!\n[refno]!year \\$4
> ..
> .DEFREF story "Dupr\[e aa]" "Best \%Story\%Book Ever" 1989


>
> That's a simplified example of how macro packages have been implementing arrays of data structures for decades, complete with idioms for "*" and "!", which are not imposed by the language in any way.  Maybe I'm missing something.


Yes. The fact that users may wish to use their own language for target destinations.

> As it happens, this bug is probably fixed, too--I simply need to come up with a convincing acceptance criterion for it.  A bit tough without adding a feature to an existing output driver.  I trust it's obvious that, with appropriate escaping, one can transmit "\000\001..\377" or "\x00..\xff" or "\[u0000]..\[u00FF]".


Transmit! Where!! Oh, you mean this bug, about the pdf outline not capturing cyrillic text, which was fixed in my branch 6 months ago, what feature and output driver are you talking about?

> I will try to make some time to reply to comment #22 more thoughtfully soon.  Leaving in "Need Info" status and assigned to myself for that reason.




Deri James <deri>
Group Member
Sun 14 Jan 2024 10:13:03 PM UTC, comment #26: 

In my seat-of-the-pants example, I committed the same crime I complained of: lack of input validation.  My point might be easier to understand if I add it in.


.nr refno 1
.de DEFREF
.  nr refno +1
.  if !\A'\\$1' \
.    ab bogus tag '\\$1' in reference definition; stick to ASCII
.  ds ref*id!\n[refno]!tag \\$1
.  ds ref*id!\n[refno]!author \\$2
.  ds ref*id!\n[refno]!desc \\$3
.  ds ref*id!\n[refno]!year \\$4
..
.DEFREF story "Dupr\[e aa]" "Best \%Story\%Book Ever" 1989
.DEFREF Гуляйпольщина "Махновщина" "весенне-летнего" 1919 \" gets rejected


G. Branden Robinson <gbranden>
Group administrator
Sun 14 Jan 2024 10:05:51 PM UTC, comment #25: 

comment #24:

> Bug #64484 is marked as fixed.


Right, but I believe there was a relationship nevertheless.

> I already have a reliable way to pass byte sequences in device control commands, .stringhex.


Okay.  But it didn't do anything about this failing test case (which admittedly didn't exist until I started to research this issue).

https://git.savannah.gnu.org/cgit/groff.git/diff/src/roff/groff/tests/device-control-special-character-handling.sh?id=974c063f0a9e1ef6c0d2cac4755a3b9d6e925b0d

Of which the salient part is the actual test input:


input='.nf
\X#bogus1: esc \%man-beast\[u1F63C]\\[u1F00] -\[aq]\[dq]\[ga]\[ha]\[rs]\[ti]#
.device bogus1: req \%man-beast\[u1F63C]\\[u1F00] -\[aq]\[dq]\[ga]\[ha]\[rs]\[ti]
.ec @
@X#bogus2: esc @%man-beast@[u1F63C]@@[u1F00] -@[aq]@[dq]@[ga]@[ha]@[rs]@[ti]#
.device bogus2: req @%man-beast@[u1F63C]@@[u1F00] -@[aq]@[dq]@[ga]@[ha]@[rs]@[ti]'


...which looks pretty noisy but tests several things.

1.  Use of \X escape sequences versus `device` requests.
2.  Use of \% escape sequences in device control commands (do they get removed?).
3.  Use of ordinary hyphens in device control commands (do they get converted to some crazy Unicode thing?).
4.  Use of special character escape sequences to represent ASCII characters in device control commands and which should therefore be passed through as ASCII.
5.  Robustness in the face of a changed roff escape character.  This did not work prior to the bug #64484 fix.

> This bug was previously named "warning messages when using special characters in TITLE or AUTHOR" and the attached cyrillic.pdf shows both the pdf title and author shown with cyrillics and no warnings. So I would say this one is dependent on bug #65098, i.e. merge the rest of my branch.


I hear your expression of urgency but I don't think "stringhex" is good long-term solution to what ails us.  You are correct in comment #22 that I did not correctly apprehend at first what it was for.  I thought you developed it because we had no way to reliably transmit arbitrary byte sequences to device control commands.  But we did, sort of--it just needed to be made consistent and reliable.  That it wasn't is what my test case attempts to illustrate and what the fix to bug #64484 attempts to prove.

No, I accept your premise that the main driver behind "stringhex" was this:

> The problem lies in the original pdfmark API, if you look at the pdfmark.pdf you will see that in the sections describing .pdfhref M and .pdfhref L which both refer to a "dest-name" and "descriptive text", it says that if a dest-name is not given the first word in the description is used as the dest-name.


I appreciate your explanation.  If the problem was with the pdfmark API, then let's fix the pdfmark API.

In particular, this:

> if a dest-name is not given the first word in the description is used as the dest-name


...strikes me a short-sighted, especially without any validation going on.  A textual description of a hyperlink/bookmark might contain all sorts of crazy stuff.  (Like Cyrillic or CJK characters or, worse, motion or type-size or font-selection escape sequences.)  Assuming that it was going to be a well-behaved sequence of ASCII bytes or even that one could "sanitize" or "cln" one's way through was a hopeless notion.  That won't be practical until we have a string iterator and more conditional expressions that enable the user of an iterator to identify the type of each item in an iterated string/macro/diversion.  But if I understand you correctly, we don't need that fancy new stuff to solve the present problem, with stringhex or without.

It would probably benefit me to look up Peter's documentation on mom's "HEADING" macro.  It is a bit baffling to me that one has to repeat arguments like this:


.HEADING 1 NAMED Гуляйпольщина "Гуляйпольщина"
...
.PDF_LINK Гуляйпольщина PREFIX ( SUFFIX ) "see: +"


> Where the "+" is replaced by the contents of the string register pdf:look(Гуляйпольщина), which would actually be a string of \[uXXXX] nodes, so would generate an error. This is what stringhex is for, to hide the contents so that groff does not see it as a sequence of nodes. The ideal solution would be to allow string registers to have an attribute (say "glass") which signals that groff should never try to interpret its contents, i.e. operate as if the escape mechanism was turned off just for the contents of that register, and have a way of turning that attribute on/off or an escape which sets the attribute for the enclosed string.


Right now I don't understand why we would need to elaborate a fairly fundamental *roff language data type (the string) with a "glass" attribute when, if you have a list indexed by a number or a valid identifier, you can simply define a string using a list item's index as a prefix.


.nr refno 1
.de DEFREF
.  nr refno +1
.  ds ref*id!\n[refno]!tag \\$1
.  ds ref*id!\n[refno]!author \\$2
.  ds ref*id!\n[refno]!desc \\$3
.  ds ref*id!\n[refno]!year \\$4
..
.DEFREF story "Dupr\[e aa]" "Best \%Story\%Book Ever" 1989


That's a simplified example of how macro packages have been implementing arrays of data structures for decades, complete with idioms for "*" and "!", which are not imposed by the language in any way.  Maybe I'm missing something.

As it happens, this bug is probably fixed, too--I simply need to come up with a convincing acceptance criterion for it.  A bit tough without adding a feature to an existing output driver.  I trust it's obvious that, with appropriate escaping, one can transmit "\000\001..\377" or "\x00..\xff" or "\[u0000]..\[u00FF]".

I will try to make some time to reply to comment #22 more thoughtfully soon.  Leaving in "Need Info" status and assigned to myself for that reason.

G. Branden Robinson <gbranden>
Group administrator
Sun 14 Jan 2024 08:59:19 PM UTC, comment #24: 

Bug #64484 is marked as fixed. I already have a reliable way to pass byte sequences in device control commands, .stringhex. This bug was previously named "warning messages when using special characters in TITLE or AUTHOR" and the attached cyrillic.pdf shows both the pdf title and author shown with cyrillics and no warnings. So I would say this one is dependent on bug #65098, i.e. merge the rest of my branch.

(file #55567)

Deri James <deri>
Group Member
Sun 14 Jan 2024 03:53:49 PM UTC, comment #23: 

Adding dependency on bug #64484, because to achieve the aim of this ticket, a reliable means of transmitting encoded byte sequences to the output device is necessary.  Not all devices need to use the same convention, though that would be nice.

I still owe Deri a reply to comment #22.

G. Branden Robinson <gbranden>
Group administrator
Tue 09 Jan 2024 09:34:14 PM UTC, comment #22: 

Whew, rather a lot to cover!

First the original "bug" was "fixed" by including -f U-T in the command.

Next it became a wish to include non-latin character in the bookmarks. This is now working on my branch, waiting for Branden's integration.

Then it became a discussion on Branden's for iterator being used as a replacement for stringhex, and using it to send arbitrary bytes in device control commands, and his recent discovery that you can already do this. My statement in 2022 (see comment #11):-

"If I dropped the .asciify from pdf.tmac it would mean all the \[uXXXX] strings would reach the post processor gropdf, which could then assemble a UTF-16 string from the hex numbers."

Which is exactly what I have done in the new pdf.tmac/gropdf.

I think Branden has not fully grasped the reason why stringhex is required. The problem lies in the original pdfmark API, if you look at the pdfmark.pdf you will see that in the sections describing .pdfhref M and .pdfhref L which both refer to a "dest-name" and "descriptive text", it says that if a dest-name is not given the first word in the description is used as the dest-name.

The macros create a string like:-

.ds pdf:look(\\*[dest-name]) descriptive text

Since descriptive text can include any groff escape this means that dest-name may also include any groff escape occurring in the first word. The reason it creates these string registers is to support mom features such as:-

.HEADING 1 NAMED Гуляйпольщина "Гуляйпольщина"
Гуляйпольщина (укр. Гуляйпольщина) или Махновщина, также Вольная
Территория — повстанческий район в Северном Приазовье в период
Гражданской войны 1918—1921 гг.
.PP
And so it goes on.
.PDF_LINK Гуляйпольщина PREFIX ( SUFFIX ) "see: +"

Where the "+" is replaced by the contents of the string register pdf:look(Гуляйпольщина), which would actually be a string of \[uXXXX] nodes, so would generate an error. This is what stringhex is for, to hide the contents so that groff does not see it as a sequence of nodes. The ideal solution would be to allow string registers to have an attribute (say "glass") which signals that groff should never try to interpret its contents, i.e. operate as if the escape mechanism was turned off just for the contents of that register, and have a way of turning that attribute on/off or an escape which sets the attribute for the enclosed string.

I don't know if this is helpful, and helps you understand why stringhex is being used.

Deri James <deri>
Group Member
Tue 09 Jan 2024 02:19:59 PM UTC, comment #21: 


comment #20:

> I'll bolster the unit tests I have pending to make sure this works okay for all output devices.


Bolstered and confirmed.

Setting ticket to "Need Info" status to solicit feedback from Deri in particular, and anyone else who wants to comment.

G. Branden Robinson <gbranden>
Group administrator
Tue 09 Jan 2024 01:42:52 PM UTC, comment #20: 

Sleeping on the problem provoked illumination, as it sometimes does.

In any other roff context, how would we smuggle something that looks like an escape sequence into a different interpretation context?

By escaping it.


$ cat EXPERIMENTS/device-control-request-with-double-escape-special-character-and-partially-collected-line.roff
.device ps: nop \\[u007E]
.br
$ groff -Z EXPERIMENTS/device-control-request-with-double-escape-special-character-and-partially-collected-line.roff
x T ps
x res 72000 1 1
x init
p1
V12000
H72000
x font 5 TR
f5
s10000
V12000
H72000
md
DFd
x X ps: nop \[u007E]
n12000 0
x trailer
V792000
x stop
$ cat EXPERIMENTS/device-control-escape-sequence-with-double-escape-special-character.roff
\X'ps: nop \\[u007E]'
$ groff -Z EXPERIMENTS/device-control-escape-sequence-with-double-escape-special-character.roff
x T ps
x res 72000 1 1
x init
p1
V12000
H72000
x font 5 TR
f5
s10000
V12000
H72000
md
DFd
x X ps: nop \[u007E]
n12000 0
x trailer
V792000
x stop


I get the same output from groff 1.22.4, 1.23.0, and Git HEAD.

This feels like a win.  Needs some discussion in documentation as the assumption of many users may be "you can't do that".  And to mention the fact that it's up to the postprocessor to interpret the syntax (and even then possibly only in the context of a specific device control command).

I'll bolster the unit tests I have pending to make sure this works okay for all output devices.

G. Branden Robinson <gbranden>
Group administrator
Tue 09 Jan 2024 02:24:27 AM UTC, comment #19: 

Tricky.  By the time `encode_char` sees the input it has already been tokenized.  This means that what is of interest for my approach is a "token" node, and that in turn means that input that looks like


\X'\[u007E]'


has already effectively been rewritten as if it were


\X'\[ti]'


...so that sucks.  (AIUI, "uniglyph.cpp" does this remapping.)

More thinking required.  A different escape notation might be required after all, but I'm not resigned to that possibility YET.

G. Branden Robinson <gbranden>
Group administrator
Tue 09 Jan 2024 01:42:09 AM UTC, comment #18: 

This is a skeleton of my proposal, an alternative to Deri's new `stringhex` request in his branch, "deri-gropdf-ng".

Originally posted in a recent duplicate bug #65137.


diff --git a/src/roff/troff/input.cpp b/src/roff/troff/input.cpp
index a0b987634..f6e5b1279 100644
--- a/src/roff/troff/input.cpp
+++ b/src/roff/troff/input.cpp
@@ -5571,7 +5571,7 @@ static node *do_non_interpreted()
   return new non_interpreted_node(mac);
 }

-static void encode_char(macro *mac, char c)
+static void encode_char_for_device_control(macro *mac, char c)
 {
   if (c == '\0') {
     if (tok.is_stretchable_space()
@@ -5600,6 +5600,13 @@ static void encode_char(macro *mac, char c)
       else if (strcmp("ti", sc) == 0)
        mac->append('~');
       else {
+       // TODO: Support '\[uXXXX]' for all devices to support
+       // transmission of arbitrary data to the output device.  It's a
+       // misnomer--this doesn't necessarily represent a Unicode code
+       // point, but this syntax beats inventing a new one for this
+       // esoteric purpose.  Whether one sends \[uAABB],
+       // \[u00AA]\[u00BB], or the latter's byte-swapped counterpart is
+       // an interface detail that the output device must specify.
        if (font::use_charnames_in_special) {
          if (sc[0] != (char)0) {
            mac->append('\\');
@@ -5612,9 +5619,14 @@ static void encode_char(macro *mac, char c)
            mac->append(']');
          }
          else
-             error("special character '%1' cannot be used within"
-                   " device control escape sequence", sc);
+           error("special character '%1' cannot be used within a"
+                 " device control escape sequence", sc);
        }
+       else
+         // TODO: Put '\[uXXXX]' support here.  Don't allow
+         // '\[uXXXX_YYYY]'.
+         error("special character '%1' cannot be used within a device"
+               " control escape sequence", sc);
       }
     }
     else if (!(tok.is_hyphen_indicator()
@@ -5668,7 +5680,7 @@ static node *do_special()
       c = '\b';
     else
       c = tok.ch();
-    encode_char(&mac, c);
+    encode_char_for_device_control(&mac, c);
   }
   return new special_node(mac);
 }


G. Branden Robinson <gbranden>
Group administrator
Thu 10 Aug 2023 02:31:12 PM UTC, comment #17: 

The new gropdf handles this by not asciifying device parameters any more.

Deri James <deri>
Group Member
Sun 26 Mar 2023 05:04:15 PM UTC, comment #16: 

Around comment #4, this bug report took a turn from what was originally reported to its current state.

> Processing your sample file with the U-TR font in font/devpdf spits out a few "can't find special character" warnings,


Assuming the messages were specifically of the form:


troff:ATTIC/makhnovshchina.groff:1: warning: special character 'u0413' not defined


...then this issue arises from a font lacking coverage for a glyph required by the document.  This came up again this past week on the groff development list.

https://lists.gnu.org/archive/html/groff/2023-03/msg00100.html

> When I process the file with your patch applied, it produces the same result (missing characters) but adds several more of the unavoidable "can't transparently output node at top level" warnings.


This issue is different, Deri covered it extensively in comment #11, and it is now the the focus of this ticket.

G. Branden Robinson <gbranden>
Group administrator
Wed 28 Sep 2022 08:15:51 AM UTC, comment #15: 


comment #8:

> Nikita,
>
> If you've done any switching between groff 1.22.4 and a recent git build (specifically, any since commit 557bc055 six months ago), the newer code disables these warnings by default, so perhaps this accounts for the difference you were seeing?


I was using the most recent groff build in both of those cases, so I don't think that it explains why the patch stopped working.

Nikita Ivanov <nikitaivanov>
Tue 27 Sep 2022 10:10:48 PM UTC, comment #14: 


comment #13:

> I read Deri's statement as a response to my comment #8, wherein I suggested that whether Nikita was seeing the warning was in response to the setting of the recently introduced GROFF_ENABLE_TRANSPARENCY_WARNINGS environment variable.  But of course Deri is correct: the warning message originally reported is not one of the ones under the control of that variable, so my comment was a red herring.


Ah, lo comprendo ahora.  Don't mind me!  😅

G. Branden Robinson <gbranden>
Group administrator
Tue 27 Sep 2022 09:35:25 PM UTC, comment #13: 

comment #12:

> comment #11:
> > The messages which started this bug: "special characters are not defined", have very little to do with the message you recently suppressed.
>
> I followed most of the rest of this message but I didn't see where anyone suppressed anything!  Can you clarify?


I read Deri's statement as a response to my comment #8, wherein I suggested that whether Nikita was seeing the warning was in response to the setting of the recently introduced GROFF_ENABLE_TRANSPARENCY_WARNINGS environment variable.  But of course Deri is correct: the warning message originally reported is not one of the ones under the control of that variable, so my comment was a red herring.

Dave <barx>
Group Member
Tue 27 Sep 2022 09:15:39 PM UTC, comment #12: 

Hi Deri,

Thank you very much for the illuminating response.

comment #11:

> The messages which started this bug: "special characters are not defined", have very little to do with the message you recently suppressed.


I followed most of the rest of this message but I didn't see where anyone suppressed anything!  Can you clarify?

[output comparison operator example snipped]

I love a good empirical test! :D

> Now to deal with why Cyrillic glyphs do not appear in the bookmark panel, but do appear in the text of the document. The text is using the embedded fonts which contain the Cyrillic glyphs mapped to appropriate code points. The bookmark panel is using whatever system font you have configured for window text. The system font will have Cyrillic glyphs but they will be using UTF code points, not the 8-bit codes available to a type 1 postscript font.


Ouch!  This almost seems like a lack of foresight or i18n on the part of PDF viewer programs...but perhaps not, as you address below.

> The pdf standard allows two encodings for strings in pdfs. We are using PDFDocEncoding which is a superset of ISO Latin 1 and does not include cyrillics. The alternative is UTF-16 (UTF-8 is not supported),


The Adobe/Microsoft axis will bedevil us forever if they get their way.

> the string must start with a BOM character, and this would allow any UTF glyph to appear in bookmarks. The reason I used the 8-bit encoding is because the groff .asciify command converts the \[UXXXX] back to ascii for me and as a bonus dropped all other escapes from the string which could not be represented as ascii. So a string such as "\fB\s'+2p'foo\s'-2p'\fP" would be converted to "foo". The only niggle was the warning message (now suppressed) each time it dropped a node such as "\fB".


Yes.  I don't know if we want to change the "asciify" request or add a "sanitize" one, but either way there should be some means for the user to ask troff to extract only the glyphs from a string/macro/diversion, i.e., with deliberate discard of everything else.  That is the behavior we need for applications like this.

I regret the name "asciify"; it implies too much.  Like, you'll get only ASCII output as opposed to only groff's representations of characters.
 

> If I dropped the .asciify from pdf.tmac it would mean all the \[uXXXX] strings would reach the post processor gropdf,


As-is?  Meaning they'd appear like 'x X blah\[uXXXX]blah'?  That's excellent, and what I already proposed!

> which could then assemble a UTF-16 string from the hex numbers. As a proof of concept I made some changes to pdf.tmac and gropdf and pdfmom -k -f U-T mom-ru.mom produced the attached pdf. Still a fair bit to do, the biggest job is to sanitise the string to remove unwanted escapes, convert any glyph producing escapes such as \C and \N back to a UTF-16 character, and convert basic latin characters to UTF-16. I suspect a deep dive into the asciify routine in groff will be helpful.


People really shouldn't have to do this in macro packages.  String processing in the roff language is incredibly tedious.  Saith I, after my experiences with an*abbreviate-page-title, an*abbreviate-inner-footer, and an*scan-for-backslash.  https://git.savannah.gnu.org/cgit/groff.git/tree/tmac/an.tmac

I've been contemplating adding a `sanitize` request for some time.

Regards,
Branden

G. Branden Robinson <gbranden>
Group administrator
Tue 27 Sep 2022 08:47:50 PM UTC, comment #11: 

The messages which started this bug: "special characters are not defined", have very little to do with the message you recently suppressed. This message is because groff starts up with the TR font, which has no Cyrillic glyphs, and if the macro package includes a .if 'text'text' statement before the font is switched to U-TR, and compares the output of the two text portions, if either "text" includes glyphs not present in the font, you will receive this warning.

You can prove this with:-


[derij@pip bug-63074]$ printf ".if \'\\[u041D]\'\\[u041D]\' .nop" | groff -Tpdf -z
troff: <standard input>:1: warning: can't find special character 'u041D'


But if you add -f U-T to the groff command there is no error.


[derij@pip bug-63074]$ printf ".if \'\\[u041D]\'\\[u041D]\' .nop" | groff -Tpdf -z -f U-T


The mom macro set has an .if statement in the .TITLE macro which is called before the .FAMILY takes effect.

Now to deal with why Cyrillic glyphs do not appear in the bookmark panel, but do appear in the text of the document. The text is using the embedded fonts which contain the Cyrillic glyphs mapped to appropriate code points. The bookmark panel is using whatever system font you have configured for window text. The system font will have Cyrillic glyphs but they will be using UTF code points, not the 8-bit codes available to a type 1 postscript font.

I suspect, although I have not attempted, that if you set your system to legacy Russian rather than the UTF variant it would be possible to get cyrillics into the bookmark panel.

The pdf standard allows two encodings for strings in pdfs. We are using PDFDocEncoding which is a superset of ISO Latin 1 and does not include cyrillics. The alternative is UTF-16 (UTF-8 is not supported), the string must start with a BOM character, and this would allow any UTF glyph to appear in bookmarks. The reason I used the 8-bit encoding is because the groff .asciify command converts the \[UXXXX] back to ascii for me and as a bonus dropped all other escapes from the string which could not be represented as ascii. So a string such as "\fB\s'+2p'foo\s'-2p'\fP" would be converted to "foo". The only niggle was the warning message (now suppressed) each time it dropped a node such as "\fB".

If I dropped the .asciify from pdf.tmac it would mean all the \[uXXXX] strings would reach the post processor gropdf, which could then assemble a UTF-16 string from the hex numbers. As a proof of concept I made some changes to pdf.tmac and gropdf and pdfmom -k -f U-T mom-ru.mom produced the attached pdf. Still a fair bit to do, the biggest job is to sanitise the string to remove unwanted escapes, convert any glyph producing escapes such as \C and \N back to a UTF-16 character, and convert basic latin characters to UTF-16. I suspect a deep dive into the asciify routine in groff will be helpful.


(file #53761)

Deri James <deri>
Group Member
Mon 26 Sep 2022 05:58:03 PM UTC, comment #10: 


> 2. The `tm`, `tm1`, `tmc`, `ab`, and `rd` requests all write to standard error.


If it makes sense to hive off this part (and `lf` in the next item, because while `lf` itself doesn't write to stderr, a UTF-8 file name will need to expressed in a manner that can be written to stderr in the event of diagnostic messages), then that can be tracked in the existing bug #62787.

> 3. The `cf`, `lf`, `nx`, `open`, `opena`, `psbb`, and `trf` requests all expect to be able to express (to standard error, in the case of `lf`) or, importantly, open files by name from the filesystem.  Right now groff doesn't have a story for being able to open UTF-8-encoded file names that use continuation bytes.

G. Branden Robinson <gbranden>
Group administrator
Mon 26 Sep 2022 05:06:49 PM UTC, comment #9: 

Thanks for re-opening this, Dave.

Peter rightly observed that this is not a problem with mom(7).

It is an absent feature in the formatter itself.

I am going to quote in slightly edited form my earlier research to the groff list on this issue.

https://lists.gnu.org/archive/html/groff/2022-09/msg00077.html


...this is our old friend "can't output node in transparent throughput".

...I recently disabled these diagnostics by default in groff Git.

Try regenerating the document with

  GROFF_ENABLE_TRANSPARENCY_WARNINGS=1

(actually, you can set the variable to anything) in your environment.

The problem, I think, is that PDF bookmark generation, like the `pdfinfo`
macro defined in _pdf.tmac_ to include document author and title
metadata, and maybe some other advanced features, relies upon use of
device control escape sequences.  That means '\X' stuff.

In device-independent output ("grout", as I term it), these become "x X"
commands, and the arguments to the escape sequence are, you'd think,
passed through as-is.

The trouble comes with the assumption people make about what "as-is"
means.

The problem is this: what if we want to represent a non-ASCII character
in the device control escape sequence?

groff's device-independent output is, up to a point, strictly ISO Basic
Latin, a property we inherited from AT&T troff.

Except in device control escape sequences.

We have the same problem with the requests that write to the standard
error stream, like `tm`.  I'm not sure that problem is worth solving;
groff's own diagnostic messages are not i18n-ed.  Even if it is worth
solving, teaching device control commands how to interpret more kinds of
"node" seems like a higher priority.

We don't have any infrastructure for handling any character encoding but
the default for input.  That's ISO Latin-1 for most platforms, but IBM
code page 1047 for OS/390 Unix (I think--no one who runs groff on such a
machine has ever spoken with me of their experiences).  And in practice
GNU troff doesn't, as far as I can tell, ever write anything but the 94
graphical code points in ASCII, spaces, and newlines to its output.

I imagine a lot of people's first instinct to fix this is to say, "just
give groff full Unicode support and enable input and output of UTF-8"!

That's a huge ask--bug #40720.

A shorter pole might be to establish a protocol for communication of
Unicode code points within device control commands.  Portability isn't
much of an issue here: as far as I know there has been no effort to
achieve interoperation of device control escape sequences among troffs.

That convention even _could_ be UTF-8, but my initial instinct is _not_
to go that way.  I like the 7-bit cleanliness of GNU troff output, and
when I've mused about solving The Big Unicode Problem I have given
strong consideration to preserving it, or enabling tricked-out UTF-8
"grout" only via an option for the kids who really like to watch their
chrome rims spin.  I realize that Heirloom and neatroff can both boast
of this, but how many people _really_ look at device-independent troff
output?  A few curious people, and the poor saps who are stuck
developing and debugging the implementations, like me.  For the latter
community, a modest and well-behaved format saves a lot of time.

Concretely, when I run the following command:

  GROFF_ENABLE_TRANSPARENCY_WARNINGS=1 ./test-groff -Z -mom -Tpdf -pet \
  -Kutf8 ../contrib/mom/examples/mon_premier_doc.mom

I get the following diagnostics familiar to all who have build groff
1.22.4 from source.

troff:../contrib/mom/examples/mon_premier_doc.mom:28: error: can't
transparently output node at top level

(The foregoing is document metadata going wrong, tripping over the "é"
in "Cicéron".)

troff:../contrib/mom/examples/mon_premier_doc.mom:13: error: can't translate
character code 233 to special character ''e' in transparent throughput
troff:../contrib/mom/examples/mon_premier_doc.mom:30: error: can't translate
character code 233 to special character ''e' in transparent throughput
troff:../contrib/mom/examples/mon_premier_doc.mom:108: error: can't translate
character code 233 to special character ''e' in transparent throughput
troff:../contrib/mom/examples/mon_premier_doc.mom:136: error: can't translate
character code 232 to special character '`e' in transparent throughput

(These are section headings of the document being made into PDF bookmarks.
The headings that happen to be in plain basic Latin have no such trouble.)

More tellingly, if I page the foregoing output with "less -R", I see
non-ASCII code points screaming out their rage in reverse video.

x X ps:exec [/Author (Cic<E9>ron,) /DOCINFO pdfmark

x X ps:exec [/Dest /pdf:bm4 /Title (1. Les diff<E9>rentes versions) /Level 2
/OUT pdfmark

x X ps:exec [/Dest /evolution /Title (2. Les <E9>volutions du Lorem) /Level 2
/OUT pdfmark

x X ps:exec [/Dest /pdf:bm8 /Title (Table des mati<E8>res) /Level 1 /OUT pdfmark

It therefore appears to me that the pdfmark extension to PostScript, or
PostScript itself, happily processes Latin-1...but that means that it
accept _only_ Latin-1, which forecloses the use of Cyrillic code points.

I'm a little concerned that we're blindly _feeding_ the device control
commands characters with the eighth bit set.  It's obviously a useful
expedient for documents like mon_premier_doc.mom.  I am curious to know
why instead of getting no text for headings and titles in the Cyrillic
PDF outline, you didn't get horrendous mojibake garbage--but plainly
Latin-1 garbage at that.

Anyway, some type of mode switching or alternative notation within the
PostScript command stream is required for us to be able to encode
Cyrillic code points.

And once we've figured out what that is, maybe we can teach GNU troff
something about it.  The answer might be to do just whatever works for
PostScript and PDF, since I assume this problem has been solved already,
but it also might mean having our own escaping protocol, which the
output drivers then interpret.


I know of three places it would make sense to support the output of UTF-8, and until I encounter a problem I see no reason not to employ the same solution for all three.

1.  We need to be able to put multibyte UTF-8 sequences into PDFs.  That means encoding them in grout as "x X" device commands.  That in turn means being able to encode them in `\X` escape sequences and `.device` requests.  `\Y` and `.devicem` may have to wait for the resolution of bug #40720, since they will require groff to be able to store UTF-8 code points internally.  Or maybe not, since we already have \[uXXXX].

2. The `tm`, `tm1`, `tmc`, `ab`, and `rd` requests all write to standard error.

3. The `cf`, `lf`, `nx`, `open`, `opena`, `psbb`, and `trf` requests all expect to be able to express (to standard error, in the case of `lf`) or, importantly, open files by name from the filesystem.  Right now groff doesn't have a story for being able to open UTF-8-encoded file names that use continuation bytes.

I can think of two approaches to take.

A. Re-use Unicode named glyph notation \[uXXXX] in these contexts.  The advantages are that we don't need to track any kind of shift state while processing them (see below), the notation will be familiar to experienced groff users, is already explained in groff_char(7) and our Texinfo manual, and its purpose is deducible by novices who haven't read the documentation.

B. We could employ a couple of C0 control characters that groff doesn't already use internally, like STX and ETX (Control+B and Control+C) to shift in and out of a "verbatim" mode where any bytes encountered between the shift characters are given as-is to the next layer of the interface.  (So, they'd appear in grout [case 1] or would end up in arguments to C library calls: fprintf(stderr, ...) [case 2, and `lf` in case 3], and fopen() or similar [the rest of case 3].

(A) has disadvantages.  One is that it's kind of an abuse of the special character/named glyph notation; the whole point of these is that they don't become formatted glyphs.  They are merely a way to encode integers.  Another problem is that there's no obvious way to adapt this to any encoding but UTF-8.  A weisenheimer can say that someone can always re-encode the output if they need to, but that remedy is not available for the file-opening case.  I take a dark view of advising groff users to write an open()-intercepting wrapper in C to be used with LD_PRELOAD.

(B) is more flexible--more accommodating of other character encodings--and seems more like the old-school Unix way to handle the problem, especially back in the days of sundry character encodings warring for supremacy, but, at least as I've sketched it, has the problem that you can't encode the ETX (Control+C) character in the "verbatim region".  Eventually, that limitation will bite someone.

I'm leaning toward (A) because the importance of all encodings other than UTF-8 is dwindling.  But there is plenty of time to wrangle over this and for brilliant new ideas to be pitched, since I see no prospect of this work being undertaken for the groff 1.23 release.

G. Branden Robinson <gbranden>
Group administrator
Sat 24 Sep 2022 08:34:49 PM UTC, comment #8: 

comment #5:

> I've tested my patch again and now it doesn't work to some
> reason (the warnings are still printed).


Nikita,

If you've done any switching between groff 1.22.4 and a recent git build (specifically, any since commit 557bc055 six months ago), the newer code disables these warnings by default, so perhaps this accounts for the difference you were seeing?

Dave <barx>
Group Member
Sat 24 Sep 2022 03:06:34 PM UTC, comment #7: 

comment #6:

> The problem is with groff itself, not mom... The same issue came
> up just a few days ago and the groff developers are aware of it.


If this is the "PDF outline not capturing Cyrillic text" thread beginning at http://lists.gnu.org/r/groff/2022-09/msg00074.html, there appears to be no existing Savannah bug for this issue, so I'm changing the scope of this one to indicate that this is a core groff issue, not a -mom one.

Dave <barx>
Group Member
Wed 21 Sep 2022 10:45:43 PM UTC, comment #6: 


comment #5:

> I've tested my patch again and now it doesn't work to some reason (the warnings are still printed). I don't understand the internals of MOM well enough to figure out what's wrong, so I think it would be better if you come up with your solution.
>
> I get the same behavior with my fonts as you get with yours: the text is rendered correctly but some warnings are still printed (which seems to happen because of the PDF outline, indeed). So I believe that using different fonts doesn't affect anything.


The problem is with groff itself, not mom, so I can't be of further help.  The same issue came up just a few days ago and the groff developers are aware of it.  From discussions on the list, it won't be addressed until after the next groff release.

If you don't need a pdf outline, you can safely ignore the warnings.

Marking this item as Closed.

Peter Schaffter <PTPi>
Group Member
Wed 21 Sep 2022 06:33:21 PM UTC, comment #5: 

I've tested my patch again and now it doesn't work to some reason (the warnings are still printed). I don't understand the internals of MOM well enough to figure out what's wrong, so I think it would be better if you come up with your solution.

I get the same behavior with my fonts as you get with yours: the text is rendered correctly but some warnings are still printed (which seems to happen because of the PDF outline, indeed). So I believe that using different fonts doesn't affect anything. If you still want to test with the TM font, it's Microsoft Times font family, installed with your script and obtained from here: http://corefonts.sourceforge.net/. I guess it's considered a proprietary font, so I'm not sure if I'm allowed to share the font files directly here.

Nikita Ivanov <nikitaivanov>
Wed 21 Sep 2022 05:47:33 PM UTC, comment #4: 


comment #3:

> comment #2:
> > It's not entirely clear what you mean by "characters that are not defined in default fonts."
>
> Oh, I mistyped there. I meant that I get warnings when using Cyrillic characters (that are not defined in default fonts).
>
> I've attached a sample file. Note that the TM font is an installed font that contains Cyrillic symbols.


Without the TM font, I can't check what's going on.

Processing your sample file with the U-TR font in font/devpdf spits out a few "can't find special character" warnings, but the text renders correctly, as nearly as I can tell.  The missing special characters are the title, and where they're missing from is the pdf outline (not the text), which presently does not accept Cyrillic.

When I process the file with your patch applied, it produces the same result (missing characters) but adds several more of the unavoidable "can't transparently output node at top level" warnings.

Can you provide copies of your TM fonts (roman and bold) so I can see if the results differ from my tests with U-TR?

Peter Schaffter <PTPi>
Group Member
Tue 20 Sep 2022 07:01:58 PM UTC, comment #3: 

comment #2:

> It's not entirely clear what you mean by "characters that are not defined in default fonts."


Oh, I mistyped there. I meant that I get warnings when using Cyrillic characters (that are not defined in default fonts).

I've attached a sample file. Note that the TM font is an installed font that contains Cyrillic symbols.

(file #53732)

Nikita Ivanov <nikitaivanov>
Tue 20 Sep 2022 04:21:16 PM UTC, comment #2: 


original submission:

> I get warnings that "special characters are not defined" when using characters that are not defined in default fonts. I've managed to fix it by adding setting font family to $FAMILY after changing an environment in the pdfmomclean macro (see the attached patch).


It's not entirely clear what you mean by "characters that are not defined in default fonts."  Please attach a sample input file demonstrating the problem.

Peter Schaffter <PTPi>
Group Member
Tue 20 Sep 2022 10:12:06 AM UTC, comment #1: 

This is Peter's dominion; adding him to cc.

Dave <barx>
Group Member
Sat 17 Sep 2022 05:01:15 PM UTC, original submission:  

I get warnings that "special characters are not defined" when using characters that are not defined in default fonts. I've managed to fix it by adding setting font family to $FAMILY after changing an environment in the pdfmomclean macro (see the attached patch).

Nikita Ivanov <nikitaivanov>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #55576:  internal.png added by deri (17KiB - image/png)
file #55567:  cyrillic.pdf added by deri (27KiB - application/pdf)
file #53761:  mom-ru.pdf added by deri (211KiB - application/pdf)
file #53732:  mom-ru.mom added by nikitaivanov (2KiB - application/octet-stream)
file #53709:  mom-fix.patch added by nikitaivanov (299B - text/x-patch)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by deri (Updated the item)
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by PTPi (Posted a comment)
  • -email is unavailable- added by barx (mom maintainer)
  • -email is unavailable- added by nikitaivanov (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 24 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-01-16 gbranden Summarydevelop convention for encoding Unicode character sequences for passage to output drivers develop convention for encoding Unicode character sequences for passage to device control commands
    2024-01-16 gbranden CategoryCore General
        Summary[troff] support construction of arbitrary byte sequences in device control commands develop convention for encoding Unicode character sequences for passage to output drivers
    2024-01-16 deri Attached File- Added internal.png, #55576
    2024-01-14 deri Attached File- Added cyrillic.pdf, #55567
    2024-01-14 gbranden Dependencies- Depends on bugs #64484
    2024-01-09 gbranden StatusIn Progress Need Info
    2024-01-09 gbranden StatusNone In Progress
        Assigned toNone gbranden
        Summary[troff] need a way to embed non-Basic Latin glyphs in device control commands [troff] support construction of arbitrary byte sequences in device control commands
    2023-08-10 gbranden StatusPostponed None
    2022-09-27 deri Attached File- Added mom-ru.pdf, #53761
    2022-09-26 gbranden Severity3 - Normal 1 - Wish
        Item GroupIncorrect behaviour Feature change
        StatusNone Postponed
        Summarywarning messages when using special characters in TITLE or AUTHOR [troff] need a way to embed non-Basic Latin glyphs in device control commands
    2022-09-24 barx CategoryMacro mom Core
        Open/ClosedClosed Open
        Summary[PATCH] [mom] warning messages when using special characters in TITLE or AUTHOR warning messages when using special characters in TITLE or AUTHOR
    2022-09-21 PTPi Open/ClosedOpen Closed
    2022-09-20 nikitaivanov Attached File- Added mom-ru.mom, #53732
    2022-09-20 barx Carbon-CopyRemoved 93119 -
    2022-09-20 barx Carbon-Copy- Added ptpi
    2022-09-17 nikitaivanov Attached File- Added mom-fix.patch, #53709

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code