bugGNU troff - Bugs: bug #63074, [troff] need a way to embed...

 
 

bug #63074: [troff] need a way to embed non-Basic Latin glyphs in device control commands

Submitter:  Nikita Ivanov <nikitaivanov>
Submitted:  Sat 17 Sep 2022 05:01:15 PM UTC
 
Category:  Core Severity:  1 - Wish
Item Group:  Feature change Status:  Postponed
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       No canned response available

 

( Jump to the original submission )

Wed 28 Sep 2022 08:15:51 AM UTC, comment #15: 

comment #8:

> Nikita,
>
> If you've done any switching between groff 1.22.4 and a recent git build (specifically, any since commit 557bc055 six months ago), the newer code disables these warnings by default, so perhaps this accounts for the difference you were seeing?


I was using the most recent groff build in both of those cases, so I don't think that it explains why the patch stopped working.  

Nikita Ivanov <nikitaivanov>
Tue 27 Sep 2022 10:10:48 PM UTC, comment #14: 

comment #13:

> I read Deri's statement as a response to my comment #8, wherein I suggested that whether Nikita was seeing the warning was in response to the setting of the recently introduced GROFF_ENABLE_TRANSPARENCY_WARNINGS environment variable.  But of course Deri is correct: the warning message originally reported is not one of the ones under the control of that variable, so my comment was a red herring.


Ah, lo comprendo ahora.  Don't mind me!  😅

G. Branden Robinson <gbranden>
Project Administrator
Tue 27 Sep 2022 09:35:25 PM UTC, comment #13: 

comment #12:

> comment #11:
> > The messages which started this bug: "special characters are not defined", have very little to do with the message you recently suppressed.
>
> I followed most of the rest of this message but I didn't see where anyone suppressed anything!  Can you clarify?


I read Deri's statement as a response to my comment #8, wherein I suggested that whether Nikita was seeing the warning was in response to the setting of the recently introduced GROFF_ENABLE_TRANSPARENCY_WARNINGS environment variable.  But of course Deri is correct: the warning message originally reported is not one of the ones under the control of that variable, so my comment was a red herring.

Dave <barx>
Project Member
Tue 27 Sep 2022 09:15:39 PM UTC, comment #12: 

Hi Deri,

Thank you very much for the illuminating response.

comment #11:

> The messages which started this bug: "special characters are not defined", have very little to do with the message you recently suppressed.


I followed most of the rest of this message but I didn't see where anyone suppressed anything!  Can you clarify?

[output comparison operator example snipped]

I love a good empirical test! :D

> Now to deal with why Cyrillic glyphs do not appear in the bookmark panel, but do appear in the text of the document. The text is using the embedded fonts which contain the Cyrillic glyphs mapped to appropriate code points. The bookmark panel is using whatever system font you have configured for window text. The system font will have Cyrillic glyphs but they will be using UTF code points, not the 8-bit codes available to a type 1 postscript font.


Ouch!  This almost seems like a lack of foresight or i18n on the part of PDF viewer programs...but perhaps not, as you address below.

> The pdf standard allows two encodings for strings in pdfs. We are using PDFDocEncoding which is a superset of ISO Latin 1 and does not include cyrillics. The alternative is UTF-16 (UTF-8 is not supported),


The Adobe/Microsoft axis will bedevil us forever if they get their way.

> the string must start with a BOM character, and this would allow any UTF glyph to appear in bookmarks. The reason I used the 8-bit encoding is because the groff .asciify command converts the \[UXXXX] back to ascii for me and as a bonus dropped all other escapes from the string which could not be represented as ascii. So a string such as "\fB\s'+2p'foo\s'-2p'\fP" would be converted to "foo". The only niggle was the warning message (now suppressed) each time it dropped a node such as "\fB".


Yes.  I don't know if we want to change the "asciify" request or add a "sanitize" one, but either way there should be some means for the user to ask troff to extract only the glyphs from a string/macro/diversion, i.e., with deliberate discard of everything else.  That is the behavior we need for applications like this.

I regret the name "asciify"; it implies too much.  Like, you'll get only ASCII output as opposed to only groff's representations of characters.

> If I dropped the .asciify from pdf.tmac it would mean all the \[uXXXX] strings would reach the post processor gropdf,


As-is?  Meaning they'd appear like 'x X blah\[uXXXX]blah'?  That's excellent, and what I already proposed!

> which could then assemble a UTF-16 string from the hex numbers. As a proof of concept I made some changes to pdf.tmac and gropdf and pdfmom -k -f U-T mom-ru.mom produced the attached pdf. Still a fair bit to do, the biggest job is to sanitise the string to remove unwanted escapes, convert any glyph producing escapes such as \C and \N back to a UTF-16 character, and convert basic latin characters to UTF-16. I suspect a deep dive into the asciify routine in groff will be helpful.


People really shouldn't have to do this in macro packages.  String processing in the roff language is incredibly tedious.  Saith I, after my experiences with an*abbreviate-page-title, an*abbreviate-inner-footer, and an*scan-for-backslash.  https://git.savannah.gnu.org/cgit/groff.git/tree/tmac/an.tmac

I've been contemplating adding a `sanitize` request for some time.

Regards,
Branden

G. Branden Robinson <gbranden>
Project Administrator
Tue 27 Sep 2022 08:47:50 PM UTC, comment #11: 

The messages which started this bug: "special characters are not defined", have very little to do with the message you recently suppressed. This message is because groff starts up with the TR font, which has no Cyrillic glyphs, and if the macro package includes a .if 'text'text' statement before the font is switched to U-TR, and compares the output of the two text portions, if either "text" includes glyphs not present in the font, you will receive this warning.

You can prove this with:-

[derij@pip bug-63074]$ printf ".if \'\\[u041D]\'\\[u041D]\' .nop" | groff -Tpdf -z
troff: <standard input>:1: warning: can't find special character 'u041D'

But if you add -f U-T to the groff command there is no error.

[derij@pip bug-63074]$ printf ".if \'\\[u041D]\'\\[u041D]\' .nop" | groff -Tpdf -z -f U-T

The mom macro set has an .if statement in the .TITLE macro which is called before the .FAMILY takes effect.

Now to deal with why Cyrillic glyphs do not appear in the bookmark panel, but do appear in the text of the document. The text is using the embedded fonts which contain the Cyrillic glyphs mapped to appropriate code points. The bookmark panel is using whatever system font you have configured for window text. The system font will have Cyrillic glyphs but they will be using UTF code points, not the 8-bit codes available to a type 1 postscript font.

I suspect, although I have not attempted, that if you set your system to legacy Russian rather than the UTF variant it would be possible to get cyrillics into the bookmark panel.

The pdf standard allows two encodings for strings in pdfs. We are using PDFDocEncoding which is a superset of ISO Latin 1 and does not include cyrillics. The alternative is UTF-16 (UTF-8 is not supported), the string must start with a BOM character, and this would allow any UTF glyph to appear in bookmarks. The reason I used the 8-bit encoding is because the groff .asciify command converts the \[UXXXX] back to ascii for me and as a bonus dropped all other escapes from the string which could not be represented as ascii. So a string such as "\fB\s'+2p'foo\s'-2p'\fP" would be converted to "foo". The only niggle was the warning message (now suppressed) each time it dropped a node such as "\fB".

If I dropped the .asciify from pdf.tmac it would mean all the \[uXXXX] strings would reach the post processor gropdf, which could then assemble a UTF-16 string from the hex numbers. As a proof of concept I made some changes to pdf.tmac and gropdf and pdfmom -k -f U-T mom-ru.mom produced the attached pdf. Still a fair bit to do, the biggest job is to sanitise the string to remove unwanted escapes, convert any glyph producing escapes such as \C and \N back to a UTF-16 character, and convert basic latin characters to UTF-16. I suspect a deep dive into the asciify routine in groff will be helpful.

(file #53761)

Deri James <deri>
Project Member
Mon 26 Sep 2022 05:58:03 PM UTC, comment #10: 

> 2. The `tm`, `tm1`, `tmc`, `ab`, and `rd` requests all write to standard error.


If it makes sense to hive off this part (and `lf` in the next item, because while `lf` itself doesn't write to stderr, a UTF-8 file name will need to expressed in a manner that can be written to stderr in the event of diagnostic messages), then that can be tracked in the existing bug #62787.

> 3. The `cf`, `lf`, `nx`, `open`, `opena`, `psbb`, and `trf` requests all expect to be able to express (to standard error, in the case of `lf`) or, importantly, open files by name from the filesystem.  Right now groff doesn't have a story for being able to open UTF-8-encoded file names that use continuation bytes.

G. Branden Robinson <gbranden>
Project Administrator
Mon 26 Sep 2022 05:06:49 PM UTC, comment #9: 

Thanks for re-opening this, Dave.

Peter rightly observed that this is not a problem with mom(7).

It is an absent feature in the formatter itself.

I am going to quote in slightly edited form my earlier research to the groff list on this issue.

https://lists.gnu.org/archive/html/groff/2022-09/msg00077.html

...this is our old friend "can't output node in transparent throughput".

...I recently disabled these diagnostics by default in groff Git.

Try regenerating the document with

  GROFF_ENABLE_TRANSPARENCY_WARNINGS=1

(actually, you can set the variable to anything) in your environment.

The problem, I think, is that PDF bookmark generation, like the `pdfinfo`
macro defined in _pdf.tmac_ to include document author and title
metadata, and maybe some other advanced features, relies upon use of
device control escape sequences.  That means '\X' stuff.

In device-independent output ("grout", as I term it), these become "x X"
commands, and the arguments to the escape sequence are, you'd think,
passed through as-is.

The trouble comes with the assumption people make about what "as-is"
means.

The problem is this: what if we want to represent a non-ASCII character
in the device control escape sequence?

groff's device-independent output is, up to a point, strictly ISO Basic
Latin, a property we inherited from AT&T troff.

Except in device control escape sequences.

We have the same problem with the requests that write to the standard
error stream, like `tm`.  I'm not sure that problem is worth solving;
groff's own diagnostic messages are not i18n-ed.  Even if it is worth
solving, teaching device control commands how to interpret more kinds of
"node" seems like a higher priority.

We don't have any infrastructure for handling any character encoding but
the default for input.  That's ISO Latin-1 for most platforms, but IBM
code page 1047 for OS/390 Unix (I think--no one who runs groff on such a
machine has ever spoken with me of their experiences).  And in practice
GNU troff doesn't, as far as I can tell, ever write anything but the 94
graphical code points in ASCII, spaces, and newlines to its output.

I imagine a lot of people's first instinct to fix this is to say, "just
give groff full Unicode support and enable input and output of UTF-8"!

That's a huge ask--bug #40720.

A shorter pole might be to establish a protocol for communication of
Unicode code points within device control commands.  Portability isn't
much of an issue here: as far as I know there has been no effort to
achieve interoperation of device control escape sequences among troffs.

That convention even _could_ be UTF-8, but my initial instinct is _not_
to go that way.  I like the 7-bit cleanliness of GNU troff output, and
when I've mused about solving The Big Unicode Problem I have given
strong consideration to preserving it, or enabling tricked-out UTF-8
"grout" only via an option for the kids who really like to watch their
chrome rims spin.  I realize that Heirloom and neatroff can both boast
of this, but how many people _really_ look at device-independent troff
output?  A few curious people, and the poor saps who are stuck
developing and debugging the implementations, like me.  For the latter
community, a modest and well-behaved format saves a lot of time.

Concretely, when I run the following command:

  GROFF_ENABLE_TRANSPARENCY_WARNINGS=1 ./test-groff -Z -mom -Tpdf -pet \
  -Kutf8 ../contrib/mom/examples/mon_premier_doc.mom

I get the following diagnostics familiar to all who have build groff
1.22.4 from source.

troff:../contrib/mom/examples/mon_premier_doc.mom:28: error: can't
transparently output node at top level

(The foregoing is document metadata going wrong, tripping over the "é"
in "Cicéron".)

troff:../contrib/mom/examples/mon_premier_doc.mom:13: error: can't translate
character code 233 to special character ''e' in transparent throughput
troff:../contrib/mom/examples/mon_premier_doc.mom:30: error: can't translate
character code 233 to special character ''e' in transparent throughput
troff:../contrib/mom/examples/mon_premier_doc.mom:108: error: can't translate
character code 233 to special character ''e' in transparent throughput
troff:../contrib/mom/examples/mon_premier_doc.mom:136: error: can't translate
character code 232 to special character '`e' in transparent throughput

(These are section headings of the document being made into PDF bookmarks.
The headings that happen to be in plain basic Latin have no such trouble.)

More tellingly, if I page the foregoing output with "less -R", I see
non-ASCII code points screaming out their rage in reverse video.

x X ps:exec [/Author (Cic<E9>ron,) /DOCINFO pdfmark

x X ps:exec [/Dest /pdf:bm4 /Title (1. Les diff<E9>rentes versions) /Level 2
/OUT pdfmark

x X ps:exec [/Dest /evolution /Title (2. Les <E9>volutions du Lorem) /Level 2
/OUT pdfmark

x X ps:exec [/Dest /pdf:bm8 /Title (Table des mati<E8>res) /Level 1 /OUT pdfmark

It therefore appears to me that the pdfmark extension to PostScript, or
PostScript itself, happily processes Latin-1...but that means that it
accept _only_ Latin-1, which forecloses the use of Cyrillic code points.

I'm a little concerned that we're blindly _feeding_ the device control
commands characters with the eighth bit set.  It's obviously a useful
expedient for documents like mon_premier_doc.mom.  I am curious to know
why instead of getting no text for headings and titles in the Cyrillic
PDF outline, you didn't get horrendous mojibake garbage--but plainly
Latin-1 garbage at that.

Anyway, some type of mode switching or alternative notation within the
PostScript command stream is required for us to be able to encode
Cyrillic code points.

And once we've figured out what that is, maybe we can teach GNU troff
something about it.  The answer might be to do just whatever works for
PostScript and PDF, since I assume this problem has been solved already,
but it also might mean having our own escaping protocol, which the
output drivers then interpret.

I know of three places it would make sense to support the output of UTF-8, and until I encounter a problem I see no reason not to employ the same solution for all three.

1.  We need to be able to put multibyte UTF-8 sequences into PDFs.  That means encoding them in grout as "x X" device commands.  That in turn means being able to encode them in `\X` escape sequences and `.device` requests.  `\Y` and `.devicem` may have to wait for the resolution of bug #40720, since they will require groff to be able to store UTF-8 code points internally.  Or maybe not, since we already have \[uXXXX].

2. The `tm`, `tm1`, `tmc`, `ab`, and `rd` requests all write to standard error.

3. The `cf`, `lf`, `nx`, `open`, `opena`, `psbb`, and `trf` requests all expect to be able to express (to standard error, in the case of `lf`) or, importantly, open files by name from the filesystem.  Right now groff doesn't have a story for being able to open UTF-8-encoded file names that use continuation bytes.

I can think of two approaches to take.

A. Re-use Unicode named glyph notation \[uXXXX] in these contexts.  The advantages are that we don't need to track any kind of shift state while processing them (see below), the notation will be familiar to experienced groff users, is already explained in groff_char(7) and our Texinfo manual, and its purpose is deducible by novices who haven't read the documentation.

B. We could employ a couple of C0 control characters that groff doesn't already use internally, like STX and ETX (Control+B and Control+C) to shift in and out of a "verbatim" mode where any bytes encountered between the shift characters are given as-is to the next layer of the interface.  (So, they'd appear in grout [case 1] or would end up in arguments to C library calls: fprintf(stderr, ...) [case 2, and `lf` in case 3], and fopen() or similar [the rest of case 3].

(A) has disadvantages.  One is that it's kind of an abuse of the special character/named glyph notation; the whole point of these is that they don't become formatted glyphs.  They are merely a way to encode integers.  Another problem is that there's no obvious way to adapt this to any encoding but UTF-8.  A weisenheimer can say that someone can always re-encode the output if they need to, but that remedy is not available for the file-opening case.  I take a dark view of advising groff users to write an open()-intercepting wrapper in C to be used with LD_PRELOAD.

(B) is more flexible--more accommodating of other character encodings--and seems more like the old-school Unix way to handle the problem, especially back in the days of sundry character encodings warring for supremacy, but, at least as I've sketched it, has the problem that you can't encode the ETX (Control+C) character in the "verbatim region".  Eventually, that limitation will bite someone.

I'm leaning toward (A) because the importance of all encodings other than UTF-8 is dwindling.  But there is plenty of time to wrangle over this and for brilliant new ideas to be pitched, since I see no prospect of this work being undertaken for the groff 1.23 release.

G. Branden Robinson <gbranden>
Project Administrator
Sat 24 Sep 2022 08:34:49 PM UTC, comment #8: 

comment #5:

> I've tested my patch again and now it doesn't work to some
> reason (the warnings are still printed).


Nikita,

If you've done any switching between groff 1.22.4 and a recent git build (specifically, any since commit 557bc055 six months ago), the newer code disables these warnings by default, so perhaps this accounts for the difference you were seeing?

Dave <barx>
Project Member
Sat 24 Sep 2022 03:06:34 PM UTC, comment #7: 

comment #6:

> The problem is with groff itself, not mom... The same issue came
> up just a few days ago and the groff developers are aware of it.


If this is the "PDF outline not capturing Cyrillic text" thread beginning at http://lists.gnu.org/r/groff/2022-09/msg00074.html, there appears to be no existing Savannah bug for this issue, so I'm changing the scope of this one to indicate that this is a core groff issue, not a -mom one.

Dave <barx>
Project Member
Wed 21 Sep 2022 10:45:43 PM UTC, comment #6: 

comment #5:

> I've tested my patch again and now it doesn't work to some reason (the warnings are still printed). I don't understand the internals of MOM well enough to figure out what's wrong, so I think it would be better if you come up with your solution.
>
> I get the same behavior with my fonts as you get with yours: the text is rendered correctly but some warnings are still printed (which seems to happen because of the PDF outline, indeed). So I believe that using different fonts doesn't affect anything.


The problem is with groff itself, not mom, so I can't be of further help.  The same issue came up just a few days ago and the groff developers are aware of it.  From discussions on the list, it won't be addressed until after the next groff release.

If you don't need a pdf outline, you can safely ignore the warnings.

Marking this item as Closed.

Peter Schaffter <PTPi>
Project Member
Wed 21 Sep 2022 06:33:21 PM UTC, comment #5: 

I've tested my patch again and now it doesn't work to some reason (the warnings are still printed). I don't understand the internals of MOM well enough to figure out what's wrong, so I think it would be better if you come up with your solution.

I get the same behavior with my fonts as you get with yours: the text is rendered correctly but some warnings are still printed (which seems to happen because of the PDF outline, indeed). So I believe that using different fonts doesn't affect anything. If you still want to test with the TM font, it's Microsoft Times font family, installed with your script and obtained from here: http://corefonts.sourceforge.net/. I guess it's considered a proprietary font, so I'm not sure if I'm allowed to share the font files directly here.

Nikita Ivanov <nikitaivanov>
Wed 21 Sep 2022 05:47:33 PM UTC, comment #4: 

comment #3:

> comment #2:
> > It's not entirely clear what you mean by "characters that are not defined in default fonts."
>
> Oh, I mistyped there. I meant that I get warnings when using Cyrillic characters (that are not defined in default fonts).
>
> I've attached a sample file. Note that the TM font is an installed font that contains Cyrillic symbols.


Without the TM font, I can't check what's going on.

Processing your sample file with the U-TR font in font/devpdf spits out a few "can't find special character" warnings, but the text renders correctly, as nearly as I can tell.  The missing special characters are the title, and where they're missing from is the pdf outline (not the text), which presently does not accept Cyrillic.

When I process the file with your patch applied, it produces the same result (missing characters) but adds several more of the unavoidable "can't transparently output node at top level" warnings.

Can you provide copies of your TM fonts (roman and bold) so I can see if the results differ from my tests with U-TR?

Peter Schaffter <PTPi>
Project Member
Tue 20 Sep 2022 07:01:58 PM UTC, comment #3: 

comment #2:

> It's not entirely clear what you mean by "characters that are not defined in default fonts."


Oh, I mistyped there. I meant that I get warnings when using Cyrillic characters (that are not defined in default fonts).

I've attached a sample file. Note that the TM font is an installed font that contains Cyrillic symbols.

(file #53732)

Nikita Ivanov <nikitaivanov>
Tue 20 Sep 2022 04:21:16 PM UTC, comment #2: 

original submission:

> I get warnings that "special characters are not defined" when using characters that are not defined in default fonts. I've managed to fix it by adding setting font family to $FAMILY after changing an environment in the pdfmomclean macro (see the attached patch).


It's not entirely clear what you mean by "characters that are not defined in default fonts."  Please attach a sample input file demonstrating the problem.

Peter Schaffter <PTPi>
Project Member
Tue 20 Sep 2022 10:12:06 AM UTC, comment #1: 

This is Peter's dominion; adding him to cc.

Dave <barx>
Project Member
Sat 17 Sep 2022 05:01:15 PM UTC, original submission:  

I get warnings that "special characters are not defined" when using characters that are not defined in default fonts. I've managed to fix it by adding setting font family to $FAMILY after changing an environment in the pdfmomclean macro (see the attached patch).

Nikita Ivanov <nikitaivanov>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #53761:  mom-ru.pdf added by deri (211KiB - application/pdf)
file #53732:  mom-ru.mom added by nikitaivanov (2KiB - application/octet-stream)
file #53709:  mom-fix.patch added by nikitaivanov (299B - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by deri (Updated the item)
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by PTPi (Posted a comment)
  • -email is unavailable- added by barx (mom maintainer)
  • -email is unavailable- added by nikitaivanov (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-09-27 deri Attached File- Added mom-ru.pdf, #53761
    2022-09-26 gbranden Severity3 - Normal 1 - Wish
        Item GroupIncorrect behaviour Feature change
        StatusNone Postponed
        Summarywarning messages when using special characters in TITLE or AUTHOR [troff] need a way to embed non-Basic Latin glyphs in device control commands
    2022-09-24 barx CategoryMacro mom Core
        Open/ClosedClosed Open
        Summary[PATCH] [mom] warning messages when using special characters in TITLE or AUTHOR warning messages when using special characters in TITLE or AUTHOR
    2022-09-21 PTPi Open/ClosedOpen Closed
    2022-09-20 nikitaivanov Attached File- Added mom-ru.mom, #53732
    2022-09-20 barx Carbon-CopyRemoved 93119 -
    2022-09-20 barx Carbon-Copy- Added ptpi
    2022-09-17 nikitaivanov Attached File- Added mom-fix.patch, #53709

    Back to the top

    Powered by Savane 3.9