bugGNU roff - Bugs: bug #66323, [gropdf] rendering differences...

 
 

bug #66323: [gropdf] rendering differences between PostScript and PDF output?

Submitter:  None
Submitted:  Fri 11 Oct 2024 01:49:25 PM UTC
   
 
Category:  Driver gropdf Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  Fixed
Privacy:  Public Assigned to:  deri
Open/Closed:  Closed Planned Release:  1.24.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 23 Oct 2024 02:31:09 PM UTC, comment #35: 

Thanks for bringing the issues to my attention, I'm glad you are now happy with the -Tpdf output. Closing as fixed.

Deri James <deri>
Group Member
Tue 22 Oct 2024 06:05:59 PM UTC, comment #34: 

Agree with your diagnosis on the на kerning, I think I'm using a mismatched font/metadata pair on purpose (you can see the difference in the fonts between our two PDFs in how <> is rendered, and the shape of the cyrillic — the metrics seem equivalent between our renders and clearly match yours more correctly than mine).
Guess I'll have to figure out how to break it in the same way. Not a groff issue.

All renders fine on HEAD (bed13e17e50e80a20e42bd396857cb55cb1d0418), can confirm all the fonts showing up with a good text layer, polyomino renders fine, colours the same, adding … to a section heading works and shows up in the bookmarks. Thanks for the -P-e + ps2pdfwr recompression trick (the full chain takes 23 seconds for me, but I don't need to embed Chinese all that often thankfully). The micron-scale isn't a material difference at all.

Can also confirm the different location of the "Contents" &c. on the TOC page. But the one in -Tpdf seems to actually match the rest of the document better? idk, this is too far in the weeds for me.

Also, the -Tps TOC bleeds colours, but the -Tpdf one doesn't (bleeding starts after va_list and va_start lines). This is consistent with my bookworm setup. Good to see -Tpdf fixes this.

Overall, sans the Contents location, I'd say this is mostly resolved (and -Tpdf behaves better), thanks.

Anonymous
Fri 18 Oct 2024 09:39:16 PM UTC, comment #33: 

comment #25:

> Thanks. By comparing what your dump does and what my attempts didn't, I managed to get an equivalent setup locally for -Tpdf as I have for -Tps (not that it didn't fight me every step of the way). As a bonus, ps2pdf(?) no longer spends minutes unpicking the chinese font.
>
> The test dataset was obtained by appending

[...]

> As I have it configured, -Tps and -Tpdf use the exact same metrics and the exact same font files.


Not quite. As I have pointed out before, in your new* pdfs, grops and gropdf are now using the same meta-data but using different embedded fonts. gropdf uses the font pointed to in the download file, whereas ghostscript uses the fonts pointed to in /usr/share/ghostscript/*/Resource/Font, which are different versions of the URW fonts, since grops does not embed the base35 fonts in the postscript it produces, so ghostscript embeds its own copy of the font. However, the difference in the glyphs in both versions is probably minimal, it is the meta-data which has the major difference, and you have sorted that out fine.

In the two attached pdfs I have got gropdf and ghostscript using the same fonts.

> The trends I see are thus:
>
> newpdf6ps6.pdf has everything, but neither -Tpdf outputs have STIX or TW-Sung-98_1 characters.


Bug in gropdf (Ingo was correct, it bit me). Change line 3393 of current gropdf to:-

    $foundry=$1 if $fontnm=~m/^(.)-/;

Fixed in HEAD, git pull.

> On -Tpdf, the на pair is overkerned. (And I'd say it's /over/kerned not just "more kerned", it definitely looks weird.)


Wrong font. Not embedded, see above.

> On -Tpdf everything's, like, inconsistently shifted around by like 1pt. This is blameless but still weird.


Wrong font. You can see this by running pdffonts:-

[derij@pip nabi (master)]$ pdffonts newpdf6ps6.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
JNMFLI+NimbusRoman-Italic            Type 1C           Custom           yes yes yes     31  0
EMWDZW+NimbusMonoPS-Regular          Type 1C           Custom           yes yes no      33  0
JNUELA+NimbusMonoPS-Bold             Type 1C           Custom           yes yes no      35  0
LTFQDS+NimbusRoman-Bold              Type 1C           WinAnsi          yes yes no      37  0
JJSGTG+NimbusRoman-Regular           Type 1C           Custom           yes yes no      39  0
GDRORT+NimbusRoman-Italic            Type 1C           WinAnsi          yes yes no      47  0
GZVNUO+NimbusRoman-Regular           Type 1C           Custom           yes yes yes     45  0
ELKFXB+STIX-Regular                  Type 1C           Custom           yes yes yes     70  0
WYEFFC+TW-Sung-98_1                  Type 1C           Custom           yes yes yes     72  0
TDCAQH+TW-Sung-98_1                  Type 1C           Custom           yes yes yes     74  0
XRBVSF+Symbola                       Type 1C           Custom           yes yes no      66  0
BDRMJB+topaz                         TrueType          WinAnsi          yes yes no      76  0
SDXHZD+DejaVuSans                    Type 1C           Custom           yes yes yes     68  0

Ghostscript has embedded all fonts (emb=yes). Notice the groff T and C fonts have Nimbus* names.

[derij@pip nabi (master)]$ pdffonts newpdf95c.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
NFOCCZ+Symbola                       Type 1            Custom           yes yes no      70  0
BUYXVJ+DejaVuSans                    Type 1            Custom           yes yes no      74  0
STIX-Regular                         Type 1            Custom           no  no  no      77  0
TW-Sung-98_1                         Type 1            Custom           no  no  no      80  0
UNSAWY+topaz                         Type 1            Custom           yes yes no      84  0
Times-Bold                           Type 1            Custom           no  no  no      87  0
Times-Italic                         Type 1            Custom           no  no  no      90  0
Times-Roman                          Type 1            Custom           no  no  no      93  0
Courier                              Type 1            Custom           no  no  no      96  0
Courier-Bold                         Type 1            Custom           no  no  no      99  0

Not all fonts embedded. T and C because no -P-e, and STIX and TW-Sun because of gropdf bug fixed above.

> groff HEAD justifies (fills? whatever the nomenclature is) the "all comments were applied" line differently so it breaks it mid-word earlier. Conversely with the polyomino link line.


The difference in the "all comments" line is a difference between 1.22.4 and 1.23.0, since if I run pdfmom -Tps and default on HEAD the word breaks in the same place. So not a difference between grops and gropdf, but a difference in groff versions.

The polyomino difference has been fixed in HEAD.

> groff HEAD -mom spaces the 2.2.1. heading. Good!
>
> I can't attach site-font/ because it blows the max file size many times over, so it's at https://lfs.nabijaczleweli.xyz/0025-groff-Tpdf-fonts/site-font.tar.zst. This is for groff HEAD on sid. This should ease testing significantly.
>
> (file #56544, file #56545, file #56546)


I have generated a grops and gropdf version using your site-font (thanks, it helps to debug) using HEAD and the fixes outlined above, called dj-newpdf6ps6 and dj-newpdf95c. The gropdf version has the following advantages, bookmark page numbers match the printed page numbers, selecting text from the pdf results in the correct unicode characters being used:-

o another paper on advice of the sponsor).
 😩😖🙄
 ⋈⋈▷
 ⟕⟖⟗
 不用烤箱教你在家做面包简单一蒸蓬松宣软
 ␤

Whereas the ghostscript version shows:-

t to another paper on advice of the sponsor).
 😩😖🙄
 ⋈⋈▷
 ⟕⟖⟗
 不用烤箱教你 在家做面包 简 单 一 蒸蓬松宣软

Which has introduced spaces in the Sung font, and ignored Topaz. Unicode characters are allowed in pdf  metadata (Author/Title) and bookmarks (HEAD only). The Contents page is automatically positioned after the title page. On a raspberrypi4b (Debian 11) -Tpdf takes just over 12 seconds and -Tps takes almost 3 1/2 minutes. The -Tps version is smaller, but if you run the -Tpdf version through ps2pdfwr as a final step (.6 second) it produces the smallest version..

Differences.

The contents page starts at different positions on the page (maybe an issue with mom).
Diffpdf shows  pixel differences which change at different zoom levels.

As an example "va_arg" on physical page 4 (page number 2) of dj-newpdf95c.pdf with its equivalent physical page 3 of dj-newpdf95c-ps.pdf, at the top of the page, after the line "Replace". Diffpdf does not like this word, so let's see how it is produced in both pdfs.

First this is the source from nab3359.mom:-

.defcolor darkgreen rgb #006400  /# sourced from ps.tmac
(0x64=100 * 257=25700)

.XCOLOR    darkgreen green

.ds va_arg   \*[green]\f(CBva_arg\fP\*[black]

.QUOTELINE 2 red
.in +1cm
.nf
\*[va_arg], \*[va_end], and \*[va_copy].  If access to the varying arguments is desired, the called funct
ion

These are the generated grout commands:-

f8
V103603                /# Vertical position from top of page.
H85038                /# Horizontal position in millipoints.
mr 0 25700 0        /# RGB colour
tva_arg
f5
mr 0 0 0
t,

Grops produces this postscript:-

 0 0.392 0 Cr/F7 10.5/NimbusMonL-Bold@0 SF(va_arg)85.038 103.603 Q 0 0 0 Cr F6(,)

 (0.392=25700/65535) rounded to 3dp.
 (85.038 103.603 = x,y in points)

The origin groff uses for positioning 0,0 is top left, but pdf origin is bottom-left, so the conversion is to subtract groff's y coordinate from the current media length (a4 document = 842.000-103.603 = 738.397).

Ghostscript reads the postscript and produces this in the dj-newpdf95c-ps.pdf.

0 0.39209 0 rg
q
10 0 0 10 0 0 cm BT
/R11 10.5 Tf
1 0 0 1 85.0379 738.287 Tm
(va_arg)Tj
ET

Both the RGB colour and x,y position are not exactly the same as requested by the postscript (0.392 v. 0.39209 and 85.038,738.397 v. 85.0379,738.287)

Gropdf reading the same grout above produces this in the dj-newpdf95c.pdf:-

1 0 0 1 85.038 738.397 Tm
0 Tc
0.000 0.392 0.000 rg
/F8 10.5 Tf
0.000 Tw [ (va_arg)] TJ

As you can see from the above postscript has requested green .329 and gs is using colour .32909.
Postscript requests x,y for "va_arg" at 85.038,738.397 and gs is using 85.0379,738.287.
gropdf, on the other hand, sets the values exactly, ghostscript does a very good job approximating the postscript. I believe it is these differences which diffpdf is highlighting, but it would be incorrect to assume it was gropdf which was inaccurate in glyph placement/colour.


(file #56553, file #56554)

Deri James <deri>
Group Member
Thu 17 Oct 2024 12:31:27 PM UTC, comment #32: 

At 2024-10-14T16:54:11-0400, Deri James wrote:

> Follow-up Comment #18, bug #66323 (group groff):
> > >   pdfmom -Kutf8 -P-e good-clean.mom > good-clean.pdf
> > > along with their respective ps and pdf files.  The family
> > > is T, not U-T, because the URW fonts are not, by default, in
> > > font/devps.  (Why?)
>
> > Deri's been asking me that for a long time and I no longer remember
> > the answer.  Something was difficult about it.  Maybe I wouldn't
> > find it so anymore.
>
> Using afmtodit to generate new groff fonts from current versions of
> the URW fonts yields a magnitude more kern pairs than our current
> stock 35 fonts, which means all documents will render differently
> after the new fonts installed (tighter text because a lot more
> kerning).


Ahh, thanks for clearing this up.  My memory was pretty muddled.

> One solution is we retire our current fonts to an oldfont-1.23.0
> directory and generate new fonts for devps, which would mean people
> could use -F to restore the old font behaviour.  As well as more
> kerning, there is significantly more glyph coverage as well, which may
> be a problem.  grops does not embed any of the 35 standard fonts in
> its postscript, it relies on the fact that all postscript printers
> would have a rom containing the 35 fonts in order for adobe to allow
> it to be called a "postscript printer".  Would these roms hold more
> than the 256 standard glyphs, unlikely if we are talking about a 30 yr
> old apple laser writer.  grops can of course embed fonts in the
> postscript, so grops could be given an -e flag (like gropdf) which
> tells it to embed all fonts.  This would require a suitable download
> file.


Most of the above prospect doesn't excite me, except for a new grops
`-e` flag that would behave in parity with gropdf's (including a new
default of embedding all fonts, as noted in bug #66342).

> Another would be to extend the foundry solution and add a -y flag to
> groff so with -yU when it processed .ft TR it would go looking for
> U-TR and .fam T would be understood as U-T.  This means either set of
> fonts can be used by the same roff doc.


In case I haven't said it before (or recently), I think this is a good
idea.  But I didn't officially record that opinion in Savannah, it
seems, so I've filed it as bug #66344.

G. Branden Robinson <gbranden>
Group administrator
Wed 16 Oct 2024 06:50:04 AM UTC, comment #31: 

Oh, since you're not posting using a savannah account, savannah might not include that footer in mail to you, since you can't meet the requirements for email replies.

Dave <barx>
Group Member
Wed 16 Oct 2024 05:26:39 AM UTC, comment #30: 

comment #29:

> nothing indicates you can mailpost here


Every email I get from the savannah bug tracker contains this footer:

To reply to this notification, you have two options:
* In the Web UI at <https://savannah.gnu.org/bugs/?66323>.
* By email, ONLY IF you sign your email with a GPG key registered
  in your Savannah account AND you include the following line
  in the reply:

Your email client might be hiding this from you.  Gmail does that to me sometimes.

> (indeed, <INVALID.NOREPLY@gnu.org> contraindicates this to me)


True, that's misleading.  It dates from before the comment-by-mail functionality was added to savannah earlier this year; they should probably change it now.

Dave <barx>
Group Member
Wed 16 Oct 2024 12:31:03 AM UTC, comment #29: 

web; nothing indicates you can mailpost here (indeed, <INVALID.NOREPLY@gnu.org> contraindicates this to me)

Anonymous
Tue 15 Oct 2024 11:41:18 PM UTC, comment #28: 

At 2024-10-15T19:36:37-0400, anonymous wrote:

> Follow-up Comment #26, bug #66323 (group groff):
>
> Savannah failed to attach newpdf94.pdf. Let's try again


Did you try to attach via the email or the Web interface?

If the former, https://savannah.nongnu.org/support/?111134 may interest
you.

Regards,
Branden

G. Branden Robinson <gbranden>
Group administrator
Tue 15 Oct 2024 11:38:33 PM UTC, comment #27: 

Not this time either. Ah well, it's all at https://lfs.nabijaczleweli.xyz/0025-groff-Tpdf-fonts/

Anonymous
Tue 15 Oct 2024 11:36:32 PM UTC, comment #26: 

Savannah failed to attach newpdf94.pdf. Let's try again

Anonymous
Tue 15 Oct 2024 11:35:18 PM UTC, comment #25: 

Thanks. By comparing what your dump does and what my attempts didn't, I managed to get an equivalent setup locally for -Tpdf as I have for -Tps (not that it didn't fight me every step of the way). As a bonus, ps2pdf(?) no longer spends minutes unpicking the chinese font.

The test dataset was obtained by appending
.br
.ft CR
\Z'Symbola'\h'4c' 😩😖🙄
.br
\Z'DejaVuSans'\h'4c' ⨝⋈▷
.br
\Z'STIX-Regular'\h'4c' ⟕⟖⟗
.br
.ft CR
\Z'TW-Sung-98_1'\h'4c' 不用烤箱,教你在家做面包,简单一蒸,蓬松宣软
.br
\Z'topaz'\h'4c' ␤
.br

This is the sum total of all the fonts I use. Might as well try them all :)

I'm attaching:
-Tps created on bookworm 1.24.4 — I'd consider this the most correct baseline — with
  sed -e 's/[😩😖🙄]/\\f[Symbola]\\z&\\fP\\ \\ /g' -e 'y/⨝/⋈/' -e 's/[⨝⋈▷]/\\f[DejaVuSans]\\z&\\fP\\ \\fC\\ \\fP/g' -e 's/[⟕⟖⟗]/\\f[STIX-Regular]\\z&\\fC\\ \\ \\fP/g' -e 's/[一-拿挀-矿砀-賿贀-鼀]/\\f[TW-Sung-98_1]\\z&\\fP\\ \\ /g' -e 's/␤/\\f[topaz]␤\\fP/' n3363.mom | pdfmom -Tps -K utf-8 -t | ps2pdf - - > newpdf6ps6.pdf

-Tpdf created on bookworm 1.24.4 with
  sed -e 's/[😩😖🙄]/\\f[Symbola]\\z&\\fP\\ \\ /g' -e 'y/⨝/⋈/' -e 's/[⨝⋈▷]/\\f[DejaVuSans]\\z&\\fP\\ \\fC\\ \\fP/g' -e 's/[⟕⟖⟗]/\\f[STIX-Regular]\\z&\\fC\\ \\ \\fP/g' -e 's/[一-拿挀-矿砀-賿贀-鼀]/\\f[TW-Sung-98_1]\\z&\\fP\\ \\ /g' -e 's/␤/\\f[topaz]␤\\fP/' n3363.mom | pdfmom  -K utf-8 -t  > newpdf94.pdf

-Tpdf created on sid with groff HEAD (ddbbe95e27fd3f2aa633d719c95e921a178fd25a) with
  sed -e 's/[😩😖🙄]/\\f[Symbola]\\z&\\fP\\ \\ /g' -e 'y/⨝/⋈/' -e 's/[⨝⋈▷]/\\f[DejaVuSans]\\z&\\fP\\ \\f(CR\\ \\fP/g' -e 's/[⟕⟖⟗]/\\f[STIX-Regular]\\z&\\f(CR\\ \\ \\fP/g' -e 's/[一-拿挀-矿砀-賿贀-鼀]/\\f[TW-Sung-98_1]\\z&\\fP\\ \\ /g' -e 's/␤/\\f[topaz]␤\\fP/' n3363.mom | pdfmom  -K utf-8 -t  > newpdf95c.pdf

As I have it configured, -Tps and -Tpdf use the exact same metrics and the exact same font files.

The trends I see are thus:

newpdf6ps6.pdf has everything, but neither -Tpdf outputs have STIX or TW-Sung-98_1 characters.

On -Tpdf, the на pair is overkerned. (And I'd say it's /over/kerned not just "more kerned", it definitely looks weird.)

On -Tpdf everything's, like, inconsistently shifted around by like 1pt. This is blameless but still weird.

groff HEAD justifies (fills? whatever the nomenclature is) the "all comments were applied" line differently so it breaks it mid-word earlier. Conversely with the polyomino link line.

groff HEAD -mom spaces the 2.2.1. heading. Good!

I can't attach site-font/ because it blows the max file size many times over, so it's at https://lfs.nabijaczleweli.xyz/0025-groff-Tpdf-fonts/site-font.tar.zst. This is for groff HEAD on sid. This should ease testing significantly.

(file #56544, file #56545, file #56546)

Anonymous
Tue 15 Oct 2024 07:32:52 PM UTC, comment #24: 

Hi наб,

I have created devps and devpdf fonts for the gsfonts you want to use, similar to the gsfonts tar on your website, but including all 35 fonts, and for devpdf. gsfonts-all.tgz contains two subdirectories, devpdf and devps, so, for testing, I suggest you create a directory and unpack the archive there. Then you can test with :-

pdfmom -Kutf8 -F. good.mom > good.pdf

And:-

pdfmom -Tps -Kutf8 -F. -P-pa4 good.mom > good-ps.pdf

Bad.mom will also work properly with the default pdf version.

If you are happy with the results you can then install the fonts in the usual place. Please feel free to use the tar on your website.

You are likely to see differences in diffpdf because ghostscript is using the fonts in the urw-base35 directory (as you confirmed below), but you wanted to use the afm files from the gsfonts directory (because it looks better). This means that the grops produced file is using the meta-data from fonts in gsfonts but the actual glyph shapes from fonts in urw-base35. Where as the gropdf produced pdfs will be using meta-data and font glyphs from the same directory (gsfonts). Both types of pdf are using the same meta-data so kerning should be identical, it is just the glyph shapes could be ever so slightly different which would be caught as a pixel difference by diffpdf.

Cheers

Deri


(file #56543)

Deri James <deri>
Group Member
Tue 15 Oct 2024 12:26:34 PM UTC, comment #23: 

Updating "Category" to gropdf.

G. Branden Robinson <gbranden>
Group administrator
Tue 15 Oct 2024 12:33:05 AM UTC, comment #22: 

AFAICT the fonts below that path haven't changed since buster, so, yes, I have the same ones there.

Anonymous
Tue 15 Oct 2024 12:26:09 AM UTC, comment #21: 

Weird.  "Assigned to" was still showing as "PTPi" for me even after two page reloads.  "History" showed that Deri reassigned it to himself.  What gives, Savannah?

G. Branden Robinson <gbranden>
Group administrator
Mon 14 Oct 2024 11:49:50 PM UTC, comment #20: 

Hi наб,

Please could you do

ls -l /usr/share/ghostscript/*/Resource/Font

On my debian bullseye this contains links to the fonts in /usr/share/fonts/type1/urw-base35 rather than the fonts in /usr/share/fonts/type1/gsfonts (which are the fonts you used with afmtodit to generate the groff fonts). This means that groff and ghostscript are accessing different versions of the fonts.

Is it the same on your system?

Deri James <deri>
Group Member
Mon 14 Oct 2024 09:03:54 PM UTC, comment #19: 

(afmtoditting the fonts already used by -Tps (/usr/share/fonts/type1/gsfonts/n0{210,220}{03,04,23,24}l.afm) is exactly what I do for full-coverage -Tps output, and indeed it kerns much better. I haven't gotten this methodology to work at all for -Tpdf output, so it's useless.)

Anonymous
Mon 14 Oct 2024 08:54:07 PM UTC, comment #18: 


> >   pdfmom -Kutf8 -P-e good-clean.mom > good-clean.pdf
> > along with their respective ps and pdf files.  The family
> > is T, not U-T, because the URW fonts are not, by default, in
> > font/devps.  (Why?)


>  Deri's been asking me that for a long time and I no longer remember the answer.  Something was difficult about it.  Maybe I wouldn't find it so anymore.


Using afmtodit to generate new groff fonts from current versions of the URW fonts yields a magnitude more kern pairs than our current stock 35 fonts, which means all documents will render differently after the new fonts installed (tighter text because a lot more kerning).

One solution is we retire our current fonts to an oldfont-1.23.0 directory and generate new fonts for devps, which would mean people could use -F to restore the old font behaviour. As well as more kerning, there is significantly more glyph coverage as well, which may be a problem. grops does not embed any of the 35 standard fonts in its postscript, it relies on the fact that all postscript printers would have a rom containing the 35 fonts in orderfor adobe to allow it to be called a "postscript printer". Would these roms hold more than the 256 standard glyphs, unlikely if we are talking about a 30 yr old apple laser writer. grops can of course embed fonts in the postscript, so grops could be given an -e flag (like gropdf) which tells it to embed all fonts. This would require a suitable download file.

Another would be to extend the foundry solution and add a -y flag to groff so with -yU when it processed .ft TR it would go looking for U-TR and .fam T would be understood as U-T. This means either set of fonts can be used by the same roff doc.

Deri James <deri>
Group Member
Mon 14 Oct 2024 07:07:14 PM UTC, comment #17: 

наб,

Please follow up to Peter's comment below; this ticket's "Need Info" status is pending on you, but we have no field datum to record that.

comment #16:

> I'm finding this issue difficult because I am not sure we're working from identical source files.  For the purposes of comparison, I'm attaching good-clean.mom, the file I worked from.  I'm also attaching the results of both
>   pdfmom -Tps -Kutf8 -P-pa4 good-clean-mom > good-clean.ps
> and
>   pdfmom -Kutf8 -P-e good-clean.mom > good-clean.pdf
> along with their respective ps and pdf files.  The family is T, not U-T, because the URW fonts are not, by default, in font/devps.  (Why?)


Deri's been asking me that for a long time and I no longer remember the answer.  Something was difficult about it.  Maybe I wouldn't find it so anymore.
 
[...]

> That said, I'm almost certain that om.tmac itself has nothing to do with the problems raised by this ticket.


We'll get this off your plate ASAP.  It is evidence of the perversity of the universe that you get a spate of mom tickets after months of peace and quiet right when you have eye surgery.  :(

Apropos of comment #13 and wait status collection issues...


make[2]: Entering directory '/home/branden/src/GIT/groff/build'
  GEN      src/devices/gropdf/gropdf.1
  GEN      man/groff_tmac.5
  GROFF    doc/groff-man-pages.pdf
  GROFF    doc/groff-man-pages.utf8.txt
an.tmac:src/devices/gropdf/gropdf.1:981: style: .IR expects at least 2 arguments, got 1
^Cmake[2]: *** Deleting file 'doc/groff-man-pages.pdf'
pdfmom: fatal error: groff died with signal 2, without core dump
make[2]: *** [Makefile:15488: doc/groff-man-pages.pdf] Error 1


Now that's what I'm talkin' 'bout!  😎

G. Branden Robinson <gbranden>
Group administrator
Mon 14 Oct 2024 06:29:44 PM UTC, comment #16: 

I'm finding this issue difficult because I am not sure we're working from identical source files.  For the purposes of comparison, I'm attaching good-clean.mom, the file I worked from.  I'm also attaching the results of both
  pdfmom -Tps -Kutf8 -P-pa4 good-clean-mom > good-clean.ps
and
  pdfmom -Kutf8 -P-e good-clean.mom > good-clean.pdf
along with their respective ps and pdf files.  The family is T, not U-T, because the URW fonts are not, by default, in font/devps.  (Why?)

The ps and pdf output look identical to me (remember: my eyes aren't completely trustworthy right now owing to surgery).  The only difference occurs on page 3, where the ps sensibly breaks the very long URL after "References" but the pdf moves the whole thing to a new line.  I have absolutely no idea why.  There is no difference in the way PDF_WWW_LINK passes information to the processor.  The .cflags directive at the top of om.tmac should guarantee that the URL can, if necessary, break at forward slashes.  Why this works for ps but not pdf is beyond me.

That said, I'm almost certain that om.tmac itself has nothing to do with the problems raised by this ticket.



(file #56530, file #56531, file #56532)

Peter Schaffter <PTPi>
Group Member
Mon 14 Oct 2024 05:31:02 PM UTC, comment #15: 

Hi Deri,

At 2024-10-14T12:00:32-0400, Deri James wrote:

> Follow-up Comment #14, bug #66323 (group groff):
>
> First a comment on a mom difference between grops and gropdf post
> 1.23.0, when both used the same method, ascii-ing the pdfmark
> parameters by calling pdfmomclean, (pdf.tmac had an eqivalent pdfclean
> routine which did a similar job). In HEAD, mom no longer uses
> pdfmomclean for pdf output the whole "dirty" string is passed to
> gropdf, whereas it is still used for grops.
>
> What this means is that gropdf more intelligently "cleans" the string
> than is possible by using asciify, in particular the \[uXXXX]
> characters are converted to UTF-16, any glyphs (\C, \N, etc) are
> converted back to ascii, and spaces are handled.


Right.  I recall feeling (and perhaps saying) that this is a good model
for grops to emulate, at least insofar as \[uXXXX] goes...\C and \N
conversion seem like gravy to me.

> comment #13:
> > Maybe; Deri can speak better to that.
>
> If you look at -Tps -Z output for good.mom the Author/Title have been
> cleaned (asciify) so unicode has been dropped, but for pdf the dirty
> string is passed and pdfinfo shows:-
>
> Title:          N????: stdarg.h wording...
> Author:         наб, seb, rCs,
> Creator:        groff version 1.23.0.2178-b25f0
> Producer:       gropdf version 1.23.0.2178-b25f0
>
> So this is definitely not an issue, except for postscript pdfmark not
> supporting unicode.


Ah, good--thanks!

> > Personally, I'm resigned to living with the resurrected "can't
> > output node in transparent throughput" diagnostic until we get all
> > the machinery required to eliminate it sorted out.
>
> No longer occurs with -Tpdf, since asciify no longer used.


Right, but 'ps' is still the default output device--or the default
default--and so I expect people still to encounter it with some
frequency and to expect an explanation with why their beautiful, perfect
documents sometimes provoke diagnostics.

> > That involves not just #62264, but bug #63074 (still a 1.24 release
> > goal) and maybe bug #64484, which like many of our tickets has
> > sprawled in scope; at this point it seems essentially a duplicate of
> > #63074 in its central purpose, but Deri and I have perhaps not
> > reached a full meeting of the minds on some auxiliary issues.  (He
> > would like to see explicit horizontal motions discarded from device
> > extension command arguments automatically by the formatter; I would
> > not [because as input validation goes, I'm an irascible,
> > white-gloved barracks inspector].
>
> I don't remember saying this, perhaps you can point me in the right
> direction.


Comment #26 in bug #64484.

https://savannah.gnu.org/bugs/?64484#comment26

> I do remember telling you I wanted 'x X' output untouched by the
> formatter,


You mean via `.output` and `\!` rather than `.device` and `\X`?

If so, yes, I remember that clearly and agree.  `.output` and `\!` are
formatter escape hatches--you get total power with total responsibility.

> just passed as a string (copy-in mode?) the same as it was in 1.23.0.


I've tried to migrate our terminology away from "copy-in mode" (because
"copy-out" mode, or "copy-" with any other preposition, either doesn't
exist or is never named thus).  I favor simply "copy mode", but yes.  In
"copy mode", the formatter is a fairly dumb robot that reads characters
from the input stream and copies them someplace else, without attempting
to convert them into lexical tokens, let alone nodes.  This is an
oversimplification because sometimes copy mode gets applied to things
that have already been tokenized or coverted to nodes, and some things
have to be tokenized because it's difficult to manage certain
non-printable code points in C/C++, especially given GNU troff's
historical support for CCSID 1047 ("EBCDIC").

And yes, that robotic copying is what we want for "transparent output".
I wish I could come up with a better name.  The word "transparency" has
been ruined by people who can't competently design interfaces.  They
were too busy maximizing synergies to leverage economies of scale and
giving 110% at the weekly office pizza party/mandatory unpaid overtime
session.

> #63074 is not required for 1.24 since all the advantages this would
> bring, are already available in pdf output now, and you have known
> that even before you started your long (oft complained of) travails
> with this self imposed wish.


Touché.

It's there but it's not as clean as it should be.  We don't have parity
between `device` and `\X` in one respect.

There's a commented-out test in
"src/roff/groff/tests/device-control-special-character-handling.sh":


#echo "checking practical bookmarking with device request" >&2
#printf "%s\n" "$output" \
#  | grep -Fqx 'x X ps:exec 7:device [/Dest /pdf:bm1 /Title (Caf\[u00E9] Hyphen-Minus and \\[u2010]) /Level 1 /OUT pdfmark' \
#  || wail


When I uncomment it and "make check":


FAIL: src/roff/groff/tests/device-control-special-character-handling.sh


...and when we run the test manually to see what went awry (or look at
the log), the following lines leap out to the trained eye:


# A more practical case, suggested by Deri James.

input='.
.ds h Caf\['"'"'e] Hyphen-Minus and \[rs]\[u2010]
...
.device ps:exec 3:device [/Dest /pdf:bm1 /Title (\*[h]) /Level 1 /OUT pdfmark
...
# Test the same thing, but with a composite special character escape
# sequence.

input='.
.ds h Caf\[e aa] Hyphen-Minus and \[rs]\[u2010]
.device ps:exec 7:device [/Dest /pdf:bm1 /Title (\*[h]) /Level 1 /OUT pdfmark


The output (in part)?


x X ps:exec 3:device [/Dest /pdf:bm1 /Title (Caf\[u00E9] Hyphen-Minus and \\[u2010]) /Level 1 /OUT pdfmark
troff:<standard input>:6: error: composite special character escape sequences not yet supported in device extension command arguments
x X ps:exec 7:device [/Dest /pdf:bm1 /Title (Caf\[e aa] Hyphen-Minus and \\[u2010]) /Level 1 /OUT pdfmark
checking practical bookmarking with device request
...FAILED


I want that fixed.  Non-orthogonality is the devil.

> > But I acknowledge that without a string iterator (#62264 again),
> > most users will simply have to endure diagnostic messages about
> > them.
>
> If using -Tps.


Acknowledged.

> > I wouldn't wish the tedium of composing things like "an.tmac"'s
> > ellipsifiers on anyone, not even those calling for me to be dragged
> > before a war crimes tribunal in The Hague for removing an
> > undocumented macro without a deprecation cycle. ;-) )
>
> And removing a line simply because you did not know what it did!!


I tested it.  But I don't think I A/B compared it to BSD ms behavior.
Maybe back then I didn't yet have them in my ever-growing archive of
historical exhibits.

> > > * "special characters aren't getting into 'grout' at all."  I take
> > > it this is a separate problem from the grops one that is now the
> > > focus of this ticket, since grops takes grout as input.
> >
> > This isn't a defect if nothing has yet taken on the responsibility
> > of encoding special characters in PDF bookmarks in PostScript output
> > in the first place.  I had assumed that was the case, in part
> > because of the surprising spelling of the device extension tags
> > exercising the "pdfmark" command word ("ps: exec"), but if these are
> > "ps: exec"s that grops will in fact never see in the first place,
> > then my observation is a potential feature request at best.
>
> Correct. PDFMARK is an extension to the postscript language which
> allows distillers to include pdf features, I am not sure if it even
> allows UTF-16 strings, although pdf detects UTF-16 with an initial
> BOM, so if ghostscript treats pdfmark input as an array of 8-bit codes
> and plomps them straight into the pdf there is at least a chance it
> may work, given the correct input.


Aha!  With (many) repeated applications, this stuff begins to soak even
into my brain.

> > That moves it back to mom, so reassigning to Peter--but I'm
> > leaving the status as "Need Info" because it's not clear to me that
> > it is actually true that any discrepancy between mom's PostScript
> > and PDF output remains; наб noted in comment 11 to bug #66322 that
> > the page size differences observed were due to a Debian patch to its
> > groff package.
>
> It is definitely not mom, this is a question of minute differences in
> the positioning of text on the page between grops/ghostscript and
> gropdf. The example shown in comment #10 (appears to be a diffpdf
> output)  is a little misleading since it is apparent the OP is testing
> his own created grops fonts (from /gsfonts using afmtodit) with the
> stock devpdf fonts (which are a copy of the original devps groff
> fonts). The gsfonts contain many more kern pairs, which affect the
> output, so its comparing apples with pears.
>
> However, I am still investigating, and when I compare apples, there
> are still differences. Initially it looks like a tiny bug in
> ghostscript which introduces .11 of a point difference in the vertical
> positioning of all text, and it specifies colours to six decimal
> places, with the 3 least significant containing garbage. I don't
> currently believe my findings, and I am currently documenting them in
> an email to the groff list in the hope I have made a mistake!!


Looking forward to reading and learning more about the stage of the
sausage factory closer to the loading dock than the section I walk.

> Since the grout produced on the same mom document by both -Tps and
> Tpdf is the same in all respects regarding text positioning the issue
> is definitely not mom.


Okay.  Well, when you reach the point you want to seize this ticket for
gropdf, just do so, or ask me to.

> > Who else needs a beverage after all that?
>
> A pint of "Old Peculiar", since you're offering - thanks.


It's a real thing!  Not quite as light-suckingly black as I feared.

G. Branden Robinson <gbranden>
Group administrator
Mon 14 Oct 2024 04:00:24 PM UTC, comment #14: 

First a comment on a mom difference between grops and gropdf post 1.23.0, when both used the same method, ascii-ing the pdfmark parameters by calling pdfmomclean, (pdf.tmac had an eqivalent pdfclean routine which did a similar job). In HEAD, mom no longer uses pdfmomclean for pdf output the whole "dirty" string is passed to gropdf, whereas it is still used for grops.

What this means is that gropdf more intelligently "cleans" the string than is possible by using asciify, in particular the \[uXXXX] characters are converted to UTF-16, any glyphs (\C, \N, etc) are converted back to ascii, and spaces are handled.

comment #13:

> comment #12:
> > * mom is not scrubbing the argument to her .AUTHOR macro.  This may be dependent on the aforementioned #62264, but some of it may be currently handleable.
>
> Maybe; Deri can speak better to that. 


If you look at -Tps -Z output for good.mom the Author/Title have been cleaned (asciify) so unicode has been dropped, but for pdf the dirty string is passed and pdfinfo shows:-

Title:          N????: stdarg.h wording...
Author:         наб, seb, rCs,
Creator:        groff version 1.23.0.2178-b25f0
Producer:       gropdf version 1.23.0.2178-b25f0

So this is definitely not an issue, except for postscript pdfmark not supporting unicode.

> Personally, I'm resigned to living with the resurrected "can't output node in transparent throughput" diagnostic until we get all the machinery required to eliminate it sorted out.


No longer occurs with -Tpdf, since asciify no longer used.

> That involves not just #62264, but bug #63074 (still a 1.24 release goal) and maybe bug #64484, which like many of our tickets has sprawled in scope; at this point it seems essentially a duplicate of #63074 in its central purpose, but Deri and I have perhaps not reached a full meeting of the minds on some auxiliary issues.  (He would like to see explicit horizontal motions discarded from device extension command arguments automatically by the formatter; I would not [because as input validation goes, I'm an irascible, white-gloved barracks inspector].


I don't remember saying this, perhaps you can point me in the right direction. I do remember telling you I wanted 'x X' output untouched by the formatter, just passed as a string (copy-in mode?) the same as it was in 1.23.0.

#63074 is not required for 1.24 since all the advantages this would bring, are already available in pdf output now, and you have known that even before you started your long (oft complained of) travails with this self imposed wish.

> But I acknowledge that without a string iterator (#62264 again), most users will simply have to endure diagnostic messages about them. 


If using -Tps.

> I wouldn't wish the tedium of composing things like "an.tmac"'s ellipsifiers on anyone, not even those calling for me to be dragged before a war crimes tribunal in The Hague for removing an undocumented macro without a deprecation cycle. ;-) )


And removing a line simply because you did not know what it did!!

>
> > * "special characters aren't getting into 'grout' at all."  I take it this is a separate problem from the grops one that is now the focus of this ticket, since grops takes grout as input.
>
> This isn't a defect if nothing has yet taken on the responsibility of encoding special characters in PDF bookmarks in PostScript output in the first place.  I had assumed that was the case, in part because of the surprising spelling of the device extension tags exercising the "pdfmark" command word ("ps: exec"), but if these are "ps: exec"s that grops will in fact never see in the first place, then my observation is a potential feature request at best.


Correct. PDFMARK is an extension to the postscript language which allows distillers to include pdf features, I am not sure if it even allows UTF-16 strings, although pdf detects UTF-16 with an initial BOM, so if ghostscript treats pdfmark input as an array of 8-bit codes and plomps them straight into the pdf there is at least a chance it may work, given the correct input.

[...]

>
> Consequently, I'm rescoping and retargeting this ticket at the first point you raised in comment #12:
>
> > Comment #10 talks about rendering differences between grops and gropdf.  These have not yet been root-caused, so they may be down to user error--but if so, this may point to a documentation deficiency.
>
> That moves it back to mom, so reassigning to Peter--but I'm leaving the status as "Need Info" because it's not clear to me that it is actually true that any discrepancy between mom's PostScript and PDF output remains; наб noted in comment 11 to bug #66322 that the page size differences observed were due to a Debian patch to its groff package.


It is definitely not mom, this is a question of minute differences in the positioning of text on the page between grops/ghostscript and gropdf. The example shown in comment #10 (appears to be a diffpdf output)  is a little misleading since it is apparent the OP is testing his own created grops fonts (from /gsfonts using afmtodit) with the stock devpdf fonts (which are a copy of the original devps groff fonts). The gsfonts contain many more kern pairs, which affect the output, so its comparing apples with pears.

However, I am still investigating, and when I compare apples, there are still differences. Initially it looks like a tiny bug in ghostscript which introduces .11 of a point difference in the vertical positioning of all text, and it specifies colours to six decimal places, with the 3 least significant containing garbage. I don't currently believe my findings, and I am currently documenting them in an email to the groff list in the hope I have made a mistake!!

>
> So, if наб can tell us, or if Peter is confident that no such differences exist, we can close this ticket as Invalid.


Since the grout produced on the same mom document by both -Tps and Tpdf is the same in all respects regarding text positioning the issue is definitely not mom.

> Who else needs a beverage after all that?


A pint of "Old Peculiar", since you're offering - thanks.

Deri James <deri>
Group Member
Mon 14 Oct 2024 11:05:06 AM UTC, comment #13: 

comment #12:

> * mom is not scrubbing the argument to her .AUTHOR macro.  This may be dependent on the aforementioned #62264, but some of it may be currently handleable.


Maybe; Deri can speak better to that.  Personally, I'm resigned to living with the resurrected "can't output node in transparent throughput" diagnostic until we get all the machinery required to eliminate it sorted out.  That involves not just #62264, but bug #63074 (still a 1.24 release goal) and maybe bug #64484, which like many of our tickets has sprawled in scope; at this point it seems essentially a duplicate of #63074 in its central purpose, but Deri and I have perhaps not reached a full meeting of the minds on some auxiliary issues.  (He would like to see explicit horizontal motions discarded from device extension command arguments automatically by the formatter; I would not [because as input validation goes, I'm an irascible, white-gloved barracks inspector].  But I acknowledge that without a string iterator (#62264 again), most users will simply have to endure diagnostic messages about them.  I wouldn't wish the tedium of composing things like "an.tmac"'s ellipsifiers on anyone, not even those calling for me to be dragged before a war crimes tribunal in The Hague for removing an undocumented macro without a deprecation cycle. ;-) )

> * pdfmom sometimes exits with a 0 even if there's been an error


This one isn't pdfmom's fault.  pdfmom does not (directly) invoke Ghostscript.


elsif ($dev eq 'ps')
{
        $waitstatus = system("groff -Tpdf -dLABEL.REFS=1 $mom -z $cmdstring 2>&1 | LC_ALL=C grep '^\\. *ds' | pdfroff -mpdfmark $mom --no-toc - $preconv $cmdstring");
        abort(autopsy($?)) unless $waitstatus == 0;
}


It's pdfroff's, which is no longer maintained as part of the groff project, and is slated for removal (bug #63827).  I see no reason to split off a ticket for it since we won't be fixing it anyway; that's up to its (external) maintainer.  It's pdfroff that launches Ghostscript, so it's pdfroff's duty to responsibly gather its wait/exit status.

pdfmom will need to be prepared for pdfroff not existing on the system, but I expect to cover that base when grepping the tree for "pdfroff" in the course of resolving #63827.

> * "special characters aren't getting into 'grout' at all."  I take it this is a separate problem from the grops one that is now the focus of this ticket, since grops takes grout as input.


This isn't a defect if nothing has yet taken on the responsibility of encoding special characters in PDF bookmarks in PostScript output in the first place.  I had assumed that was the case, in part because of the surprising spelling of the device extension tags exercising the "pdfmark" command word ("ps: exec"), but if these are "ps: exec"s that grops will in fact never see in the first place, then my observation is a potential feature request at best.

Besides, if I can land the rest of fix for #63074 (all that remains, that I know of, is temporarily constructing special character nodes for composite special character escape sequences), and if macro packages like "pdfmark.tmac" elect to use the `\X` escape sequence and/or `device` request to embed device extension commands instead of going behind the formatter's back with `\!` or `output`, the formatter will take care of the special character encoding issue for them.

grops will then need to know how to translate groff-style Unicode special character escape sequences like `\[u043D]` for emission to PostScript output.  (I suppose that means conversion to UTF-16, but no one ever tells me if that's LE or BE.)  But that, too, is already accounted for: I believe it to be part of bug #62830, also still a 1.24 release goal.

> Should any of the above be separate tickets (besides the one that already is) so that they don't get lost in the deluge?


The foregoing should strike a few off the list.

To summarize the central point, I withdraw my diagnosis in comment #9.  PostScript per se doesn't support PDF bookmarks, and the only think that injects pdfmark extension commands into GNU troff output is Keith Marshall's "pdfmark.tmac", which is no longer maintained as part of groff and thus is no longer our issue to address.

Consequently, I'm rescoping and retargeting this ticket at the first point you raised in comment #12:

> Comment #10 talks about rendering differences between grops and gropdf.  These have not yet been root-caused, so they may be down to user error--but if so, this may point to a documentation deficiency.


That moves it back to mom, so reassigning to Peter--but I'm leaving the status as "Need Info" because it's not clear to me that it is actually true that any discrepancy between mom's PostScript and PDF output remains; наб noted in comment 11 to bug #66322 that the page size differences observed were due to a Debian patch to its groff package.

So, if наб can tell us, or if Peter is confident that no such differences exist, we can close this ticket as Invalid.

Who else needs a beverage after all that?

G. Branden Robinson <gbranden>
Group administrator
Sat 12 Oct 2024 09:32:42 PM UTC, comment #12: 

Comment #9 clarifies the scope and purpose of this bug.

Subsequent comments seem to identify several others.

Comment #10 talks about rendering differences between grops and gropdf.  These have not yet been root-caused, so they may be down to user error--but if so, this may point to a documentation deficiency.  In any case, it may help focus discussion if this issue were a separate ticket.

Comment #11 raises several others:

  • scrubbing a string is ill supported in current groff (bug #62264)
  • mom is not scrubbing the argument to her .AUTHOR macro.  This may be dependent on the aforementioned #62264, but some of it may be currently handleable.
  • pdfmom sometimes exits with a 0 even if there's been an error
  • "special characters aren't getting into 'grout' at all."  I take it this is a separate problem from the grops one that is now the focus of this ticket, since grops takes grout as input.


Should any of the above be separate tickets (besides the one that already is) so that they don't get lost in the deluge?

Dave <barx>
Group Member
Sat 12 Oct 2024 05:24:21 PM UTC, comment #11: 

original submission:

> # f=bad; pdfmom -Tps -K utf-8 $f.mom > $f.ps
> troff:bad.mom:5: error: cannot write a node to device-independent output
> troff:bad.mom:5: error: cannot write a node to device-independent output
> troff:bad.mom:5: error: cannot write a node to device-independent output
> troff:bad.mom:5: error: cannot write a node to device-independent output
> troff:bad.mom:5: error: cannot write a node to device-independent output
> GPL Ghostscript 10.04.0: Unrecoverable error, exit code 1
>
> good.ps works (but see #66322), bad.ps detonates every parser I gave it to.


I get slightly different output from bleeding-edge groff Git.


$ groff --version | head -n 1
GNU groff version 1.23.0.2208-e004b
$ cksum ATTIC/bad.mom
4257237161 9848 ATTIC/bad.mom
$ (f=ATTIC/bad; pdfmom -Tps -K utf-8 $f.mom > $f.ps)
troff:ATTIC/bad.mom:5: warning: special character 'u043D' not defined
troff:ATTIC/bad.mom:5: warning: special character 'u0430' not defined
troff:ATTIC/bad.mom:5: warning: special character 'u0431' not defined
troff:ATTIC/bad.mom:5: error: cannot write a node to device-independent output
troff:ATTIC/bad.mom:5: error: cannot write a node to device-independent output
GPL Ghostscript 9.53.3: Unrecoverable error, exit code 1


Here's line 5 of the document.


.AUTHOR    "\v'-.5v'\*[UP 8p]наб, seb, rCs"


`\v` is a vertical motion escape sequence and produces a vertical motion node.  `UP` is a mom(7) string that does something similar.  And one of her own model documents shipped with groff as an example does the same `AUTHOR UP` thing.

This matter is covered at length in the so-called "demystified" mailing list thread I started earlier this year.

In a nutshell, mom(7) should be scrubbing things like vertical motions out the argument to her `AUTHOR` macro before passing that argument to a device extension escape sequence (or, nearly equivalently, a `device` request).

However, GNU troff at present makes such scrubbing much harder than it should be.  See bug #62264.

So, that part is a duplicate of a known issue.

As an aside:


GPL Ghostscript 9.53.3: Unrecoverable error, exit code 1


...but pdfmom exits with status 0, so that sucks.

I guess this commit:


commit b62f0605427caa156f23c81280f345496dd8a5cc
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Thu Sep 26 17:26:16 2024 -0500

    [pdfmom]: Handle errors and abnormal exits.


Didn't go far enough.

But what about "grout", the output of GNU troff?

Well, for that, let's peel off the pdfmom wrapper and have a look at all the device control commands in the grout.


$ groff -Tps -K utf-8 -Z ATTIC/bad.mom | grep '^x'
troff:ATTIC/bad.mom:84: warning: special character 'u043D' not defined
troff:ATTIC/bad.mom:84: warning: special character 'u0430' not defined
troff:ATTIC/bad.mom:84: warning: special character 'u0431' not defined
x T ps
x res 72000 1 1
x init
x font 5 TR
x font 5 TR
x font 6 CR
x trailer
x stop


They special characters aren't getting into "grout" at all.

Not sure if this is a duplicate of an existing ticket or not, but it would be nice to have parity between grops and gropdf in this respect.

I'll modify the document to make the `AUTHOR` call motionless and degenerately Western.


$ groff -Tps -K utf-8 -Z ATTIC/bad_american.mom | grep '^x'
troff:ATTIC/bad_american.mom:84: warning: special character 'u043D' not defined
troff:ATTIC/bad_american.mom:84: warning: special character 'u0430' not defined
troff:ATTIC/bad_american.mom:84: warning: special character 'u0431' not defined
x T ps
x res 72000 1 1
x init
x font 5 TR
x font 5 TR
x font 6 CR
x trailer
x stop


Hmm, no change.  Even if I add in the slated-for-the-chopping-block "pdfmark" macro package, there is no difference.


$ groff -mpdfmark -Tps -K utf-8 -Z ATTIC/bad_american.mom | grep '^x'
troff:ATTIC/bad_american.mom:84: warning: special character 'u043D' not defined
troff:ATTIC/bad_american.mom:84: warning: special character 'u0430' not defined
troff:ATTIC/bad_american.mom:84: warning: special character 'u0431' not defined
x T ps
x res 72000 1 1
x init
x font 5 TR
x font 5 TR
x font 6 CR
x trailer
x stop


So maybe parity is too far a reach for groff 1.24.0.

G. Branden Robinson <gbranden>
Group administrator
Sat 12 Oct 2024 02:40:50 PM UTC, comment #10: 

Well, I use a different set of fonts in -Tps (cf. "bullseye fonts" link under https://nabijaczleweli.xyz/content/blogn_t/007-groff-Tps-cyrillic-et-al.html). And so far I haven't been able to make them be real under -Tpdf — repeating the procedure with all /devpdf/ paths instead of /devps/ ones yields identical files, and installing them yields... default fonts I think? It's certainly something sans-serif /and/ inconsistent across renderers. -P-e doesn't change anything for me.

I would attribute the layout differences between stock T* and my T* to correct kerning (the post also notes this, so I'd say this is expected). Diff (of final) attached, regardless. This is most visible in the numbered headings (of course the author title line and reply-to are meaningless-as-expected because stock T* is partial; do note that, for example, the 2.1.1. header floats down a bit, as do other lines randomly. I'd attribute this to better metrics, too, probably?).

I'd like to have all my other fonts work, too. But I can't seem to make any font work, so -Tpdf is a non-starter (even though correct page numbers would be nice). Nothing interesting is logged, so I have to assume this is expected behaviour.


Anonymous
Sat 12 Oct 2024 02:34:25 PM UTC, comment #9: 

At 2024-10-11T23:01:07-0400, Peter Schaffter wrote:

> Follow-up Comment #7, bug #66323 (group groff):
> comment #2:
> > Forgot to mention, the problem ultimately could very well lie in the
> > formatter, or even in grops, but it would be helpful to me if we
> > could narrow down what snipped of mom the formatter is handling
> > badly, or what sequence of nodes in the output list, or sequence of
> > "grout" commands, is sending grops off the rails.
> >
> > mom is pretty esoteric to me so I'm hoping Peter can aim the
> > searchlight.
>
> With -Tps, the pdf outline and TOC are choking on the ellipsis
> character in a string passed to HEADING in the 'bad' file.  Remove the
> ellipsis and the file processes correctly.  Note that the same error
> occurs with direct unicode input of special characters and most of the
> special character escape sequences listed under "Text markers" in
> groff_char(7). \[ps] and \[sc] get a pass for some reason.  Whatever
> this is, it isn't a mom problem.  The arguments mom passes
> PDF_BOOKMARK from HEADING are the same for both '\*[.T]'ps' and
> '\*[.T]'pdf'.


Got it.  This sounds like the "unsanitized section heading"[1] problem,
and an item from my "transparent throughput demystification" checklist
that I hadn't taken care of yet.  Namely, even though the grops driver
can't (directly) handle hyperlinks, it needs to handle them similarly to
gropdf, for the exact reason we're seeing here.  Deri's already taken
care of the latter but grops is on my plate.

I'll re-assign and re-item group.

Thanks, Peter!

G. Branden Robinson <gbranden>
Group administrator
Sat 12 Oct 2024 12:44:20 PM UTC, comment #8: 


>  Maybe I'll reconsider it when -Tpdf learns to lay out text as well as -Tps | ps2pdf can (and when I figure out how to make it read the fonts I need), but that's neither here nor there.


The "layout" difference you are seeing may be due to ghostscript embedding the actual fonts in the pdf. You can tell groff to embed the fonts in the PDF by including the flag -P-e in the command, the attached file bad-Pe.pdf was created with:-

pdfmom -Kutf8 -P-e bad.mom > bad-Pe.pdf

(Including the changes to bad.mom outlined in comment #4 results in no warnings). You can even go one step further and run:-

ps2pdfwr bad-Pe.pdf bad-Pe-gs.pdf

Which will produce a more compact pdf.

If this does not cure the "layout" differences you mention, I'd be very grateful if you can point these issues out, since I am keen to make the gropdf output as good as grops.

You might notice the following differences in -Tpdf output:-

  • The ellipsis character can be safely used in .HEADING.
  • Cyrillic characters in bookmarks and the text are accepted (if you use the U- fonts provided with -Tpdf).
  • The TOC is located after the title page, rather than appearing at the end.
  • If you use a pdf viewer which shows page numbers in the bookmark panel, the numbers match the page numbers on each page.



(file #56509)

Deri James <deri>
Group Member
Sat 12 Oct 2024 03:01:01 AM UTC, comment #7: 


comment #2:

> Forgot to mention, the problem ultimately could very well lie in the formatter, or even in grops, but it would be helpful to me if we could narrow down what snipped of mom the formatter is handling badly, or what sequence of nodes in the output list, or sequence of "grout" commands, is sending grops off the rails.
>
> mom is pretty esoteric to me so I'm hoping Peter can aim the searchlight.


With -Tps, the pdf outline and TOC are choking on the ellipsis character in a string passed to HEADING in the 'bad' file.  Remove the ellipsis and the file processes correctly.  Note that the same error occurs with direct unicode input of special characters and most of the special character escape sequences listed under "Text markers" in groff_char(7). \[ps] and \[sc] get a pass for some reason.  Whatever this is, it isn't a mom problem.  The arguments mom passes PDF_BOOKMARK from HEADING are the same for both '\*[.T]'ps' and '\*[.T]'pdf'.

Not related to the reported bug, but...

The incorrect bottom margin and absent page number on page one come from a bug in the DRV macro. Fix will be in my next commit.

Peter Schaffter <PTPi>
Group Member
Sat 12 Oct 2024 12:28:45 AM UTC, comment #6: 

comment #4:

> The author of the pdfmark macro and pdfroff no longer contributes to groff


Bug #63827 discusses their removal from groff.

Dave <barx>
Group Member
Sat 12 Oct 2024 12:27:29 AM UTC, comment #5: 

The pipeline is well-considered (and, AFAIR, one of the ones the -mom manual blesses). Maybe I'll reconsider it when -Tpdf learns to lay out text as well as -Tps | ps2pdf can (and when I figure out how to make it read the fonts I need), but that's neither here nor there.

The one advantage of -Tpdf is that it seems to nest the PDF outline correctly (-Tps | ps2pdf yields
  Title Page:
  [title]
  \- 1.
  \- 2.
     \- 2.1.
        \- 2.1.1.
     \- 2.2.
        \- 2.2.1.
     \- 2.3.
        \- 2.3.1.
     \- 3.
     \- 4.
     \- TOC
but -Tpdf yields
  Title Page:
  [title]
  \- 1.
  \- 2.
  |  \- 2.1.
  |     \- 2.1.1.
  |  \- 2.2.
  |     \- 2.2.1.
  |  \- 2.3.
  |     \- 2.3.1.
  \- 3.
  \- 4.
  \- TOC
) but eh, minor.

The OSDN back-end seems to take minutes to reply, and the front-end 502s like half the time. I've nevertheless managed to install pdfmark.tmac (same error) and all the tmacs (same error). The result for the former is attached as "bad+hg.ps". I see a two-byte size difference (but it's all a compressed blob so who knows).

(file #56508)

Anonymous
Fri 11 Oct 2024 11:19:49 PM UTC, comment #4: 

Using -T ps with pdfmom results in a call to pdfroff with the extra parameters "-mom -mpdfmark --no-toc". This can be verified by running:-

pdfroff -Tps -mom -mpdfmark -Kutf8 bad.mom > bad.pdf

And receive the same results. The author of the pdfmark macro and pdfroff no longer contributes to groff but keeps development here:-

https://osdn.net/users/keith/pf/groff-pdfmark/scm/

If you remove the -mpdfmark flag it no longer crashes but the resultant pdf has no pdf features, it is the same as producing postscript and distilling to a pdf.

However, you may prefer to use pdfmom without the -T ps flag so that it uses groff's native pdf output. The attached file is the pdf produced using the current git HEAD and a few minor changes to bad mom to use a font with cyrillic glyphs) and remove a .bp command on line 168:-

.AUTHOR    "\f[U-TR]\v'-.5v'\*[UP 8p]наб, seb, rCs\fP"

...

.FAM      U-T

The U-T fonts are the same as the T fonts (Times-Roman) but with a greater glyph coverage.

If you compile from git please ensure configure finds the URW fonts on your system (these are the U- fonts).

(file #56507)

Deri James <deri>
Group Member
Fri 11 Oct 2024 03:43:56 PM UTC, comment #3: 

Also, pdfmom may not be crashing so much as politely passing along the rudeness that killed a program in the pipeline it runs.

And that's good thing, because until recently, a program in the pipeline could die horribly and pdfmom, emerging winsomely from the kitchen in an apron with a hot apple pie, would exit with status 0.


G. Branden Robinson <gbranden>
Group administrator
Fri 11 Oct 2024 03:39:22 PM UTC, comment #2: 

Forgot to mention, the problem ultimately could very well lie in the formatter, or even in grops, but it would be helpful to me if we could narrow down what snipped of mom the formatter is handling badly, or what sequence of nodes in the output list, or sequence of "grout" commands, is sending grops off the rails.

mom is pretty esoteric to me so I'm hoping Peter can aim the searchlight.

G. Branden Robinson <gbranden>
Group administrator
Fri 11 Oct 2024 03:36:49 PM UTC, comment #1: 

Reassigning to Peter.

G. Branden Robinson <gbranden>
Group administrator
Fri 11 Oct 2024 01:49:25 PM UTC, original submission:  

At 12169aa269341753d491a69e9adb86c58dca039a. In a clean sid chroot.
Originally found in the course of https://bugs.debian.org/1082520.

# f=good; pdfmom -Tps -K utf-8 $f.mom > $f.ps
troff:good.mom:5: error: cannot write a node to device-independent output
troff:good.mom:5: error: cannot write a node to device-independent output
troff:good.mom:5: error: cannot write a node to device-independent output
troff:good.mom:5: error: cannot write a node to device-independent output
troff:good.mom:5: error: cannot write a node to device-independent output
# f=bad; pdfmom -Tps -K utf-8 $f.mom > $f.ps
troff:bad.mom:5: error: cannot write a node to device-independent output
troff:bad.mom:5: error: cannot write a node to device-independent output
troff:bad.mom:5: error: cannot write a node to device-independent output
troff:bad.mom:5: error: cannot write a node to device-independent output
troff:bad.mom:5: error: cannot write a node to device-independent output
GPL Ghostscript 10.04.0: Unrecoverable error, exit code 1

good.ps works (but see #66322), bad.ps detonates every parser I gave it to.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #56553:  dj-newpdf95c-ps.pdf added by deri (84KiB - application/pdf)
file #56554:  dj-newpdf95c.pdf added by deri (128KiB - application/pdf)
file #56544:  newpdf6ps6.pdf added by None (78KiB - application/pdf)
file #56545:  newpdf95c.pdf added by None (37KiB - application/pdf)
file #56546:  n3363.mom added by None (14KiB - application/octet-stream)
file #56543:  gsfonts-all.tgz added by deri (702KiB - application/x-compressed-tar)
file #56530:  good-clean.mom added by PTPi (9KiB - application/octet-stream)
file #56531:  good-clean.ps added by PTPi (45KiB - application/postscript)
file #56532:  good-clean.pdf added by PTPi (75KiB - application/pdf)
file #56510:  diff.png added by None (193KiB - image/png)
file #56509:  bad-Pe.pdf added by deri (77KiB - application/pdf)
file #56508:  bad+hg.ps added by None (45KiB - application/postscript)
file #56507:  bad.pdf added by deri (62KiB - application/pdf)
file #56504:  bad.mom added by None (10KiB - application/octet-stream)
file #56505:  bad.ps added by None (45KiB - application/postscript)
file #56502:  good.mom added by None (9KiB - application/octet-stream)
file #56503:  good.ps added by None (45KiB - application/postscript)

 

Depends on the following items: None found

Items that depend on this one

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by PTPi (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by deri (Updated the item)
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by gbranden (original submitter)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 34 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-12-17 gbranden Planned ReleaseNone 1.24.0
    2024-10-23 deri StatusNeed Info Fixed
        Open/ClosedOpen Closed
    2024-10-18 deri Attached File- Added dj-newpdf95c-ps.pdf, #56553
        Attached File- Added dj-newpdf95c.pdf, #56554
    2024-10-15 None Attached File- Added newpdf6ps6.pdf, #56544
        Attached File- Added newpdf95c.pdf, #56545
        Attached File- Added n3363.mom, #56546
    2024-10-15 deri Attached File- Added gsfonts-all.tgz, #56543
    2024-10-15 gbranden CategoryMacro package mom Driver gropdf
    2024-10-14 deri Assigned toPTPi deri
        Summary[mom] rendering differences between PostScript and PDF output? [gropdf] rendering differences between PostScript and PDF output?
    2024-10-14 PTPi Attached File- Added good-clean.mom, #56530
        Attached File- Added good-clean.ps, #56531
        Attached File- Added good-clean.pdf, #56532
    2024-10-14 gbranden Summarygrops should handle special character escape sequences in certain device control commands [mom] rendering differences between PostScript and PDF output?
    2024-10-14 gbranden CategoryDriver grops Macro package mom
        Assigned togbranden PTPi
    2024-10-12 gbranden StatusNone Need Info
    2024-10-12 None Attached File- Added diff.png, #56510
    2024-10-12 gbranden Dependencies- bugs #65099 is dependent
    2024-10-12 gbranden Assigned toPTPi gbranden
        Summary[mom]: groff HEAD produces invalid postscript (that also crashes pdfmom) grops should handle special character escape sequences in certain device control commands
    2024-10-12 gbranden CategoryMacro package mom Driver grops
    2024-10-12 deri Attached File- Added bad-Pe.pdf, #56509

    Back to the top

    Powered by Savane 3.14-9aa3.
    Corresponding source code