bugGNU roff - Bugs: bug #66653, [troff] .asciify odd behaviour...

 
 

bug #66653: [troff] .asciify odd behaviour with composite characters

Submitter:  Deri James <deri>
Submitted:  Fri 10 Jan 2025 05:13:24 PM UTC
   
 
Category:  Core Severity:  3 - Normal
Item Group:  Warning/Suspicious behaviour Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.24.0
* Mandatory Fields

Post a Comment

Add a New Comment Rich Markup
   

Discussion

Jump to the original submission

Thu 06 Nov 2025 05:55:02 PM UTC, comment #33: 

comment #32:

> I know this is marked as "fixed" and I appreciate the sterling work Branden has put into asciify, but the test script in comment #21 is still causing issues. This time the composite glyph is causing the string to be truncated (if using \|) or skipped with a warning to use asciify if using \X. I attach a screen shot which shows the first bookmark entry has been truncated.

 

>


What PDF viewer is shown in the attached image?

I've opened a new ticket, bug #67680, for this problem.  Please follow-up there; I'm curious to see if you get the same output as I do, modulus the selection of special character.

> Branden suggested I split off the fruits of my work on adding pdf features to the ms macro set and documenting how to use pdf features in groff. The result is bug #67665.


Thanks!

G. Branden Robinson <gbranden>
Group administrator
Tue 04 Nov 2025 12:25:35 AM UTC, comment #32: 

I know this is marked as "fixed" and I appreciate the sterling work Branden has put into asciify, but the test script in comment #21 is still causing issues. This time the composite glyph is causing the string to be truncated (if using \|) or skipped with a warning to use asciify if using \X. I attach a screen shot which shows the first bookmark entry has been truncated.

In other news...

Branden suggested I split off the fruits of my work on adding pdf features to the ms macro set and documenting how to use pdf features in groff. The result is bug #67665.


Deri James <deri>
Group Member
Thu 18 Sep 2025 06:35:48 PM UTC, comment #31: 


commit c642f5fc7e1a274e89598aa54aa5199e4b7796d0
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Thu Sep 18 00:22:55 2025 -0500

    [troff]: Regression-test Savannah #66653.

    * src/roff/groff/tests/asciify-composite-nodes-correctly.sh: Do it.

    * src/roff/groff/groff.am (groff_TESTS): Run test.

    Also drop ersatz '(C)' copyright symbol from "groff.am".  Software
    developers have long labored under the no-longer-correct misconception
    that omitting a copyright symbol from one's notice was a fatal defect
    that effectively placed the work in the public domain.  That stopped
    being true as of 1 March 1989.[1]  Further, prior to guidance issued by
    the U.S. Copyright Office in the decades since, the use of "(C)" as a
    substitute for a copyright sign _may not have sufficed_ to prevent the
    copyright notice from being regarded as defective.  The Copyright
    Office, then and now, prefers the abbreviation "copr." when © is
    typographically unavailable.[1]  Nowadays, its advice is that "c" (note
    lowercase) is an "acceptable variant", that _may_ retain the efficacy of
    the copyright notice.  The word "copyright", spelled out in full, also
    suffices per that resource, and is already present in this notice.

    [1] https://www.copyright.gov/circs/circ03.pdf

commit ee2011b6c921b3e04ebb5b7d58383049cdd75a04
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Thu Sep 18 03:40:15 2025 -0500

    tmac/fallbacks.tmac: Fix Savannah #66653.

    * tmac/fallbacks.tmac: Define different fallbacks for accented
      non-Latin-1 Latin characters, using one ordering for nroff-mode
      devices and another for troff-mode devices.  We assume that the former
      can't constructively overstrike and the latter can.

      On troff-mode devices, it can make sense to use the `asciify` request
      to serialize special characters in device extension commands, and in
      that case we want to write the base glyph before any combining ones.

      On non-constructively overstriking devices, the last character written
      at the drawing position "wins"; we want that to be the base glyph.

    Fixes <https://savannah.gnu.org/bugs/?66653>.  Thanks to Deri James for
    pushing device extension commands to the limit, exposing this defect.


G. Branden Robinson <gbranden>
Group administrator
Thu 18 Sep 2025 06:28:58 PM UTC, comment #30: 


comment #27:

> I propose reducing the scope of this ticket to this composite character problem, and following up with the new PDF features for ms in a new ticket, since this one's approaching 30 comments.
>
> Updating ticket properties on the assumption my proposal's okay.


Not a new ticket, but bug #58946, the oldest one open against ms.

I expect to push the fix for this one soon, within the next hour.

G. Branden Robinson <gbranden>
Group administrator
Thu 18 Sep 2025 08:19:40 AM UTC, comment #29: 

I'm wrong.  It's my own damn fault.

The trouble arises from commit 81aa7b77c0, 2022-10-03.

My approach there makes sense for non-overstriking devices, but I think life will be simpler if we define different fallbacks for overstriking (troff) devices.

G. Branden Robinson <gbranden>
Group administrator
Thu 18 Sep 2025 07:59:22 AM UTC, comment #28: 

The difference between U+0106 and U+015E to groff appears to be that the former shows up in a table in "glyphuni.cpp" and the latter doesn't.


$ git grep 015E src tmac
src/libs/libgroff/uniuni.cpp:  { "015E", "20053_0327" },
src/roff/groff/tests/asciify-composite-nodes-correctly.sh:@[u0106] @[u015E]
src/roff/troff/node.cpp:// with \[u015E] -> S<ac>.
src/utils/afmtodit/afmtodit.tables:  "015E", "0053_0327",
src/utils/afmtodit/afmtodit.tables:  "Scedilla", "015E",
src/utils/hpftodit/hpuni.cpp:  { "1063", "015E", },     // Uppercase S Cedilla
$ git grep 0106 src tmac
src/libs/libgroff/glyphuni.cpp:  { "'C", "0106" },
src/libs/libgroff/uniuni.cpp:  { "0106", "20043_0301" },
src/roff/groff/tests/asciify-composite-nodes-correctly.sh:@[u0106] @[u015E]
src/roff/groff/tests/asciify-request-works.sh:A#[dq]#[e aa]#[u00E1]#[u0106]i#[fl]o#[Fl]#[Z]
src/roff/groff/tests/asciify-request-works.sh:  "character) Unicode special character 'u0106'" >&2
src/utils/afmtodit/afmtodit.tables:  "0106", "0043_0301",
src/utils/afmtodit/afmtodit.tables:  "Cacute", "0106",
src/utils/afmtodit/afmtodit.tables:  "afii10106", "0459",
src/utils/hpftodit/hpuni.cpp:  { "406", "0106", },      // Uppercase C Acute


G. Branden Robinson <gbranden>
Group administrator
Thu 18 Sep 2025 07:54:06 AM UTC, comment #27: 

An interesting wrinkle with this.

Some composite characters[1] stick the combining character first, and some don't.


$ cat src/roff/groff/tests/asciify-composite-nodes-correctly.sh
#!/bin/sh
#
# Copyright 2025 G. Branden Robinson
#
# This file is part of groff.
#
# groff is free software; you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# groff is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

groff="${abs_top_builddir:-.}/test-groff"

input='.
.ec @
.box DIV
@[u0106] @[u015E]
.br
.box
.asciify DIV
.DIV
.'

output=$(printf "%s\n" "$input" | "$groff" -T ps -a)
echo "$output"
echo "$output" | grep -qx 'C<aa> S<ac>'

# vim:set autoindent expandtab shiftwidth=2 tabstop=2 textwidth=72:
$ cd build
$ ../src/roff/groff/tests/asciify-composite-nodes-correctly.sh
<beginning of page>
C<aa> <ac>S


I don't quite have this licked, but I think I'm close.

I propose reducing the scope of this ticket to this composite character problem, and following up with the new PDF features for ms in a new ticket, since this one's approaching 30 comments.

Updating ticket properties on the assumption my proposal's okay.

[1] in the general sense--unfortunately GNU troff internally abuses this term to mean "any user-defined character, including fallback character definitions arising from macro files or contrived via "uniuni.cpp", which no user is likely to know about

G. Branden Robinson <gbranden>
Group administrator
Thu 04 Sep 2025 04:35:59 PM UTC, comment #26: 

comment #21:

> I have also included a screen shot of the pdf produced. You can see the missing characters in the document and the same diversion used in the bookmark has the base character only.


Yeah, something funny is going on there.  I'll return to that below.

> The most interesting thing to note is that the original string register \*[khant] retains the original input value when used in a bookmark (I was super-pleased when you got that working).


Thanks!

> If you want to experiment with a font you have installed, try:-


> .ft U-TR
> .sp 1i0
> .ds khant "time to meet the Şhaka Khan


> At the top of the file and include "-Kutf8" on the command line.


It also works to just use the corresponding Unicode escape sequence directly.  Either way it gets decomposed.

The thing that baffles me is that, after decomposition, the base and combining characters get reversed in order when they go to the output.  This doesn't happen with (some) other complex characters.  It does happen with U+0102.

It's a consequence of post-1.23.0 changes, though.


$ cat ATTIC/66653-7-undivert-weird-character.groff
.box DIV
\[u015E]
.br
.box
.\"pm DIV
.DIV
.asciify DIV
.\"tm *** UNDIVERTED NOW ***
.\"pm DIV
.DIV
$ ~/groff-1.23.0/bin/groff -a ATTIC/66653-7-undivert-weird-character.groff
<beginning of page>
<u0053_0327> <u0053_0327>


Here's the behavior on HEAD (Savannah as of , not my working copy).


$ ~/groff-HEAD/bin/groff -a ATTIC/66653-7-undivert-weird-character.groff
<beginning of page>
<ac>S <ac>S


Possibly this has to do with node list reversal.  If we restore the commented requests (of which the interesting ones, 1.23.0 doesn't support), we see the following.


$ ~/groff-HEAD/bin/groff -a ATTIC/66653-7-undivert-weird-character.groff
{"name": "DIV", "file name": "ATTIC\/66653-7-undivert-weird-character.groff", "starting line number": 1, "length": 5, "contents": "\u0000\u0000\u0000\u0000\n", "node list": [{"type": "line_start_node", "diversion level": 0, "is_special_node": false}, {"type": "composite_node", "diversion level": 0, "is_special_node": false, "special character": "u0053_0327", "contents": [{"type": "line_start_node", "diversion level": 0, "is_special_node": false},
{"type": "zero_width_node", "diversion level": 0, "is_special_node": false, "contains": [{"type": "glyph_node", "diversion level": 0, "is_special_node": false, "special character": "ac"}]},
{"type": "glyph_node", "diversion level": 0, "is_special_node": false, "character": "S"}]
}, {"type": "vertical_size_node", "diversion level": 0, "is_special_node": false, "vunits": -12000}, {"type": "vertical_size_node", "diversion level": 0, "is_special_node": false, "vunits": 0}]}
<beginning of page>
*** UNDIVERTED NOW ***
{"name": "DIV", "file name": "ATTIC\/66653-7-undivert-weird-character.groff", "starting line number": 7, "length": 2, "contents": "\u0000\n", "node list": [{"type": "composite_node", "diversion level": 0, "is_special_node": false, "special character": "u0053_0327", "contents": [{"type": "line_start_node", "diversion level": 0, "is_special_node": false},
{"type": "zero_width_node", "diversion level": 0, "is_special_node": false, "contains": [{"type": "glyph_node", "diversion level": 0, "is_special_node": false, "special character": "ac"}]},
{"type": "glyph_node", "diversion level": 0, "is_special_node": false, "character": "S"}]
}]}
<ac>S <ac>S


Feeding the JSONic stuff to jq(1), we get:


{
  "name": "DIV",
  "file name": "ATTIC/66653-7-undivert-weird-character.groff",
  "starting line number": 1,
  "length": 5,
  "contents": "\u0000\u0000\u0000\u0000\n",
  "node list": [
    {
      "type": "line_start_node",
      "diversion level": 0,
      "is_special_node": false
    },
    {
      "type": "composite_node",
      "diversion level": 0,
      "is_special_node": false,
      "special character": "u0053_0327",
      "contents": [
        {
          "type": "line_start_node",
          "diversion level": 0,
          "is_special_node": false
        },
        {
          "type": "zero_width_node",
          "diversion level": 0,
          "is_special_node": false,
          "contains": [
            {
              "type": "glyph_node",
              "diversion level": 0,
              "is_special_node": false,
              "special character": "ac"
            }
          ]
        },
        {
          "type": "glyph_node",
          "diversion level": 0,
          "is_special_node": false,
          "character": "S"
        }
      ]
    },
    {
      "type": "vertical_size_node",
      "diversion level": 0,
      "is_special_node": false,
      "vunits": -12000
    },
    {
      "type": "vertical_size_node",
      "diversion level": 0,
      "is_special_node": false,
      "vunits": 0
    }
  ]
}
{
  "name": "DIV",
  "file name": "ATTIC/66653-7-undivert-weird-character.groff",
  "starting line number": 7,
  "length": 2,
  "contents": "\u0000\n",
  "node list": [
    {
      "type": "composite_node",
      "diversion level": 0,
      "is_special_node": false,
      "special character": "u0053_0327",
      "contents": [
        {
          "type": "line_start_node",
          "diversion level": 0,
          "is_special_node": false
        },
        {
          "type": "zero_width_node",
          "diversion level": 0,
          "is_special_node": false,
          "contains": [
            {
              "type": "glyph_node",
              "diversion level": 0,
              "is_special_node": false,
              "special character": "ac"
            }
          ]
        },
        {
          "type": "glyph_node",
          "diversion level": 0,
          "is_special_node": false,
          "character": "S"
        }
      ]
    }
  ]
}


> This remaining problem concerning composite characters is not a show stopper,


I'd still like to defeat it.  But I may need to let it go for this week and push a bunch of other changes I have prepared, including the ones that deliberately discard dummy nodes ("transparent" and otherwise).

> it only raises its ugly head when we add pdf features to ms (me?) in 1.25.


If I can get all the within-formatter obstacles cleared out of the way, then we could enable widespread experimentation with replacements of "s.tmac", or supplements to it, that everyone with a stock groff 1.24 can play with easily.

That'd be nice to have.  But it will depend on this decomposed character issue falling to a second attempt in the near term.

G. Branden Robinson <gbranden>
Group administrator
Thu 04 Sep 2025 03:04:45 PM UTC, comment #25: 

comment #21:

> As far as I know the special character '\A' does not exist as a groff name, it is not mentioned in groff_char.


That's correct.  The only special character name with a single-character groff predefines is `\[-]`, more familiar as `\-`.

groff_char(7):

       \C'c' and \[c] are not synonyms for the ordinary character “c”,
       but request the special character named “\c”.  For example,
       “\[a]” is not “a”, but rather a special character with the
       internal glyph name (used in font description files and
       diagnostic messages) \a, which is typically undefined.  The only
       such glyph name groff predefines is the minus sign, which can
       therefore be accessed as \C'-' or \[-].


There are other "single-character" special characters, like `\_`, but they get remapped to longer identifiers at input processing.

groff_char(7):

       A few escape sequences that are not groff special characters also
       produce glyphs; these exist for syntactical or historical
       reasons.  \', \`, \-, and \_ are translated on input to the
       special character escape sequences \[aa], \[ga], \[-], and \[ul],
       respectively.  Others include \\, \. (backslash‐dot), and \e; see
       groff(7).  A small number of special characters represent glyphs
       that are not encoded in Unicode; examples include the baseline
       rule \[ru] and the Bell System logo \[bs].


> I'm dealing with groff's problems with more modern fonts in my forthcoming reply to bug #67244, which is taking an age to write as I keep getting sidetracked into writing little utility programs discovering the innards of ttf and otf fonts.


Ooh, I'm excited to see what this yields.


G. Branden Robinson <gbranden>
Group administrator
Mon 01 Sep 2025 11:53:45 PM UTC, comment #24: 

comment #23:

> > > Name: pdfms.tmac                     Size: 4KiB
> > >     <https://file.sava ... mac?file_id=57608>
> >
> > I've got it.  This is much more elaborate than the earlier one.
> >
> > I'll experiment with it at once.
>
> It looks great!  I don't see any changes to output compared to PostScript that don't appear to be deliberate (like increasing the leading in the table of contents).


Oh and of course the TOC entries are real hyperlinks now, which only underscores how much we're going to want to add support for user-driven internal hyperlink macros.  For example, the leftmost table entries in the "Document control settings" section of ms.ms could hyperlink to the subsequent section (or, with user-specifiable anchor locations, paragraph) that presents the register, string, special character.

By God, we'll get ms modernized yet.

G. Branden Robinson <gbranden>
Group administrator
Mon 01 Sep 2025 11:39:24 PM UTC, comment #23: 


> > Name: pdfms.tmac                     Size: 4KiB
> >     <https://file.sava ... mac?file_id=57608>
>
> I've got it.  This is much more elaborate than the earlier one.
>
> I'll experiment with it at once.


It looks great!  I don't see any changes to output compared to PostScript that don't appear to be deliberate (like increasing the leading in the table of contents).

G. Branden Robinson <gbranden>
Group administrator
Mon 01 Sep 2025 05:15:26 PM UTC, comment #22: 

Screenshot!!


Deri James <deri>
Group Member
Mon 01 Sep 2025 05:12:21 PM UTC, comment #21: 

Hi Branden,

You are so close. :-)

The dummy node no longer truncates the asciified text. Unicode characters are converted to \[uXXXX] form. The only issue left is that composite input characters (as decided by NFD) are not successful. If you send the asciified diversion to the document stream the characters are completely missing and a strange error is reported, if it is sent via device control only the base character is converted to \[uXXXX] (not \[uXXXX_XXXX]).

Using this:-

.ft Tinos-Regular
.sp 1i0
.ds khant "time to meet the ShǍka Khǎn
.box DIV
\*[khant]
.br
.box
.DIV
.asciify DIV
.DIV
.pdfbookmark 1 "\*[DIV]
.pdfbookmark 1 "\*[khant]

.\" in Tinos-Regular
.\" u0041_030C        722,859        2        814        uni01CD
.\" u0061_030C        443,696,10        2        815        uni01CE

Two errors are shown:-

[derij@pip build (master)]$ test-groff -Tpdf -F font -k G.trf -Z | egrep "^x|C"
troff:G.trf:10: warning: special character '\A' not defined
troff:G.trf:10: warning: special character '\a' not defined
x T pdf
x res 72000 1 1
x init
x X ps:exec [/Dest /pdf:bm1 /View [/FitH -67000 u] /DEST pdfmark
x X ps:exec [/Dest /pdf:bm1 /Title (time to meet the Sh\[u0041]ka Kh\[u0061]n) /Level 1 /OUT pdfmark
x font 40 Tinos-Regular
Cu0041_030C
Cu0061_030C
x X ps:exec [/Dest /pdf:bm2 /View [/FitH -91000 u] /DEST pdfmark
x X ps:exec [/Dest /pdf:bm2 /Title (time to meet the Sh\[u01CD]ka Kh\[u01CE]n) /Level 1 /OUT pdfmark
x trailer
x stop

I have also included a screen shot of the pdf produced. You can see the missing characters in the document and the same diversion used in the bookmark has the base character only. The most interesting thing to note is that the original string register \*[khant] retains the original input value when used in a bookmark (I was super-pleased when you got that working).

As far as I know the special character '\A' does not exist as a groff name, it is not mentioned in groff_char. In Tinos the postscript name is uni10CD, but other fonts have the postscript name as "Ahacek" (which is not in our glyph list). I'm dealing with groff's problems with more modern fonts in my forthcoming reply to bug #67244, which is taking an age to write as I keep getting sidetracked into writing little utility programs discovering the innards of ttf and otf fonts.

If you want to experiment with a font you have installed, try:-

.ft U-TR
.sp 1i0
.ds khant "time to meet the Şhaka Khan

At the top of the file and include "-Kutf8" on the command line.

This remaining problem concerning composite characters is not a show stopper, it only raises its ugly head when we add pdf features to ms (me?) in 1.25.

Cheers

Deri

Deri James <deri>
Group Member
Mon 01 Sep 2025 09:50:56 AM UTC, comment #20: 

Attaching a new diff of my working copy vs. "origin/master".

Clearing "Need Info" status.

(file #57609)

G. Branden Robinson <gbranden>
Group administrator
Mon 01 Sep 2025 09:41:40 AM UTC, comment #19: 

At 2025-09-01T05:36:51-0400, Deri James wrote:

> Follow-up Comment #18, bug #66653 (group groff):
>
> Here's the missing file. The email API is picky!


Oh, heck yeah.

If you have the patience, you might report issues to the Savannah
admins' task tracker.

https://savannah. ... up=administration

> The body of the email was accepted but the attachment was silently
> dropped! Using the web api now.

[...]

> Name: pdfms.tmac                     Size: 4KiB
>     <https://file.sava ... mac?file_id=57608>


I've got it.  This is much more elaborate than the earlier one.

I'll experiment with it at once.

G. Branden Robinson <gbranden>
Group administrator
Mon 01 Sep 2025 09:36:43 AM UTC, comment #18: 

Here's the missing file. The email API is picky! The body of the email was accepted but the attachment was silently dropped! Using the web api now.

(file #57608)

Deri James <deri>
Group Member
Mon 01 Sep 2025 01:51:43 AM UTC, comment #17: 

Quick scribble to myself:

1.  `delete this` after successfully undiverting a glyph node or composite node?

2.  What about composite nodes that really are composite?  This might scotch my ham-fisted, fixed-length "hexbuf".


$ cat ATTIC/undivert-composite-character.groff
.box DIV
time to meet the \[A ah]ga Kh\[a ah]n
.br
.box
.DIV
.asciify DIV
.DIV


To test the latter, I need to identify a non-Latin-1 composite character that is covered in the URW fonts, I think.  Otherwise the formatter knows the composite character doesn't exist and discards it on input (and throws a `char` warning).

Modern Greek text will definitely exercise this scenario.

G. Branden Robinson <gbranden>
Group administrator
Mon 01 Sep 2025 01:38:42 AM UTC, comment #16: 

comment #12:

> I am attaching the latest version, use it like this:-


> pdfmom --roff -T pdf -petk -ms -mpdfms doc/ms.ms | okular -


> You should notice a clickable TOC at the end of the pdf for ms.ms, if you add
> a judicious ".RP (no)" to the document you should see a title page and the TOC
> following (auto relocation).


I'm not seeing an attachment newer than file #57604, "deri-66653.diff".  Did it get lost along the way?
 

> I wrote the new tmac in mitigation for losing pdfmark, pdfroff and spdf,
> simply as a proof of concept. For 1.25 I think it would be tidier to integrate
> the techniques into s.tmac itself with a flag (OPMODE) which can be used to
> turn off pdf features, and you are better at writing macro files than I am, my
> choice of names veer towards "stunted"!!


As someone said, the three most vexing problems in software development are:

  • Naming things
  • Off-by-one errors
G. Branden Robinson <gbranden>
Group administrator
Mon 01 Sep 2025 12:45:50 AM UTC, comment #15: 

At 2025-08-31T19:35:24-0400, G. Branden Robinson wrote:

> "Transparent dummy nodes", which are produced by `\c`, also provoke the
> diagnostic.  I think we can discard them on asciification as well.


My next push will be discarding all of the following on undiversion
("asciification", not to be confused with "-a" output):

dummy nodes (\&)
transparent dummy nodes (\c)
tag nodes (.tag, .taga)
device extension nodes (.device, .devicem, \X, \Y)

G. Branden Robinson <gbranden>
Group administrator
Sun 31 Aug 2025 11:35:20 PM UTC, comment #14: 

At 2025-08-31T19:14:48-0400, G. Branden Robinson wrote:

> So I'll commit that to my working copy and look into the tag node
> thing.


"Transparent dummy nodes", which are produced by `\c`, also provoke the
diagnostic.  I think we can discard them on asciification as well.

G. Branden Robinson <gbranden>
Group administrator
Sun 31 Aug 2025 11:14:44 PM UTC, comment #13: 

Hi Deri,

At 2025-08-31T17:31:07-0400, Deri James wrote:

> I like it a lot. One problem outstanding, it does not like your name!
> If I use
>
> this as a test:-
>
> .box DIV
> hello \[u260E] world and G.\& \fBBranden\fP Robinson
> \#hello \[ru] world
> .br
> .box
> .
> .\"pm DIV
> .chop DIV
> .asciify DIV
> .\"pm DIV
> .DIV
> \X'pdf: ignoreme \*[DIV]'
> .pdfinfo /Title "\*[DIV]"
>
> Using:-
>
> test-groff -Tpdf -Z G.trf
>
> You will notice:-
>
> troff:G.trf:12: warning: a node is not encodable in device-independent output
> ('asciify' might help)
> x X ps:exec [/Title (hello \[u260E] world and G.) /DOCINFO pdfmark
>
> Cut off in your prime! However, if you apply:-
>
> --- ../src/roff/troff/node.cpp  2025-08-31 21:06:24.050087492 +0100
> +++ node.cpp    2025-08-31 21:08:58.421762137 +0100
> @@ -3883,7 +3883,8 @@
>
>  void node::asciify(macro *m)
>  {
> -  m->append(this);
> +  if (strcmp("dummy_node", this->type()) != 0 && strcmp("tag_node", this-
> > type()) != 0)
> +    m->append(this);
>  }


The gods of OOP would punish you severely for that!  But I like it as a
bug-fix demonstrator.  It also prompts me to try sticking a "tag node"
(the name of which itches me) into the diversion and see what blows up.

> The problem (and the error) disappear. This is probably not the best
> way to stop dummy_nodes depositing null chars into c-string (which is
> what causes the truncation), I don't know anything about C++
> inheritance, it might just require a dummy_node::asciify routine which
> does nothing!


You know more about C++ programming than you think!  I think your idea
is pretty close to what other classes do when the `asciify` operation
just isn't meaningful for them.


void line_start_node::asciify(macro *)
{
  delete this;
}


So I tried this...


$ git diff
diff --git a/src/roff/troff/node.cpp b/src/roff/troff/node.cpp
index 03bbd5421..f0a555d6b 100644
--- a/src/roff/troff/node.cpp
+++ b/src/roff/troff/node.cpp
@@ -4038,6 +4038,11 @@ void vertical_size_node::asciify(macro *)
   delete this;
 }

+void dummy_node::asciify(macro *)
+{
+  delete this;
+}
+
 breakpoint *node::get_breakpoints(hunits /* width */, int /* nspaces */,
                                  breakpoint *rest, bool /* is_inner */)
 {
diff --git a/src/roff/troff/node.h b/src/roff/troff/node.h
index a22c2084d..f79204a33 100644
--- a/src/roff/troff/node.h
+++ b/src/roff/troff/node.h
@@ -466,6 +466,7 @@ public:
 class dummy_node : public node {
 public:
   dummy_node(node * nd = 0 /* nullptr */) : node(nd) {}
+  void asciify(macro *);
   node *copy();
   bool is_same_as(node *);
   const char *type();


...on this...


.box DIV
hello, G.\& Branden Robinson
\#hello \[ru] world
.br
.box
.DIV
.\"pm DIV
.chop DIV
.asciify DIV
.\"pm DIV
.DIV
\X'pdf: ignoreme \*[DIV]'


...and got this.


$ ./build/test-groff -bww -Z ATTIC/dummy-char-in-diversion.groff
x T ps
x res 72000 1 1
x init
p1
x font 5 TR
f5
s10000
md
DFd
V12000
H72000
thello,
wh2500
tJ.
wh2500
tR.
wh2500
Clq
h4440
tBob
Crq
wh6940
tDobbs
wh2500
thello,
wh2500
tJ.
wh2500
tR.
wh2500
Clq
h4440
tBob
Crq
wh6940
tDobbs
V12000
H273940
x X pdf: ignoreme hello, J. R. \[u201C]Bob\[u201D] Dobbs
n12000 0
x trailer
V792000
x stop


So I'll commit that to my working copy and look into the tag node thing.

If that goes well, I'll push sooner rather than later because this could
be a nice milestone.

G. Branden Robinson <gbranden>
Group administrator
Sun 31 Aug 2025 10:44:48 PM UTC, comment #12: 

On Saturday, 30 August 2025 10:46:28 BST you wrote:

> Follow-up Comment #9, bug #66653 (group groff):
>
> Hi Deri,
>
> A few days ago I started digging into this at last.
>
> At 2025-01-10T12:13:28-0500, Deri James wrote:
> > Date: Fri 10 Jan 2025 05:13:24 PM UTC By: Deri James <deri>
> > First I will explain what I am trying to accomplish, before describing
> > the issue.
> >
> > With the demise of pdfmark.pdf and mspdf.tmac I realised that creating
> > a replacement would be a lot easier if we did a similar job Branden
> > did for man but this time for ms. Branden is correct that Keith's
> > original pdf work is more unixy than roffish, using commands on a
> > single line and various flags, and his mspdf.tmac provides new
> > commands which are wrappers around existing ms commands with added pdf
> > extensions. So instead of:-
> >
> > .NH 1
> > Introduction
> >
> > It supports:-
> >
> > .NH 1
> > .XN Introduction
> >
> > Apart from having to learn new commands it is also less flexible. Ms
> > allows:-
> >
> > .NH 1
> > Introduction to
> > .I groff
> >
> > Which would have to be converted to inline font changes to work with
> > .XN.
>
> Right.  I've long been uneasy with this departure from what is idiomatic
> for ms(7) (and to some extent mm(7)) documents.
>
> > It also means that existing ms documents can't magically start using
> > pdf features without considerable editing. The gold standard
> > "solution" would be if the output was pdf then pdf features are
> > automatically included, the same as Branden did with man.
>
> It won't surprise you that that's what I think.  Getting man page
> authors to adopt new habits is just shy of futile; where possible, I
> think the path to success lies in making existing features "just work".
>
> Sometimes that's a lot easier said than done; my quest to resolve this
> problem has forced me to learn a lot about deep formatter internals,
> including stuff that's never really been documented.  It also produced a
> few struggles between us to make sense of what's really going on in
> those mysterious internals.
>
> > As a proof of concept I attempted to make .NH produce a document
> > outline as well as headings. I used ms.ms as a test document,
>
> That's an ambitious guinea pig, since ms.ms exercises most of the
> package's own features.  On the bright side, if we can tame this beast,
> I expect most user documents to fall to our sword as well.
>
> > using the command
> > "test-groff -Tpdf -ms -M. -mpdfms -pet -ww ../doc/ms.ms > msdj.pdf",
> > the result is attached.
>
> Looks excellent!  The only thing I expected that I didn't get was
> hotspots in the document's table of contents at the end.  However, since
> there's a lovely navigation bar (in a sufficiently brainful PDF viewer,
> meaning NOT DROPBOX), that's an inconsequential defect.


Hi Branden,

I continued to work on pdfms.tmac but leaving this one in aspic as an example
of asciify crashing the formatter. The pdf files in comment #8 were all
produced with the latest version of pdfms.tmac, even pdfmark.ms works without
using pdfroff and the mspdf.tmac.

I am attaching the latest version, use it like this:-

pdfmom --roff -T pdf -petk -ms -mpdfms doc/ms.ms | okular -

You should notice a clickable TOC at the end of the pdf for ms.ms, if you add
a judicious ".RP (no)" to the document you should see a title page and the TOC
following (auto relocation).

I wrote the new tmac in mitigation for losing pdfmark, pdfroff and spdf,
simply as a proof of concept. For 1.25 I think it would be tidier to integrate
the techniques into s.tmac itself with a flag (OPMODE) which can be used to
turn off pdf features, and you are better at writing macro files than I am, my
choice of names veer towards "stunted"!!

> Support for hotspotted external URLs might be more important.
>
> That in turn might mean adding `UR` and `UE` macros to ms(7).  As I've
> noted elsewhere, I am not a fan of "www.tmac"'s approach.


I agree.

> > Seems to like pic.ms too.
>
> I know from hard experience that that document is good at breaking
> grohtml; may gropdf fare better.

[...]

> > Any good ideas, or better approaches, would be appreciated.
>
> I think you're on a good directional track.  We just need to dig some
> boulders out of the formatter's roadway so your changes can pass easily.


Just the dummy_node issue left, see previous email today.

> Regards,
> Branden
>
>

Deri James <deri>
Group Member
Sun 31 Aug 2025 09:31:01 PM UTC, comment #11: 

On Saturday, 30 August 2025 13:12:05 BST you wrote:

> Follow-up Comment #10, bug #66653 (group groff):
>
> Hi Deri,
>

[...]

> I think I have this licked.
>
> I'm attaching (or trying too--let's see if Savannah's email gateway can
> cope) a diff of my changes since my last push (just a couple of hours
> ago).
>
> Here it is in action:
>
>
> $ cat ATTIC/unicode-char-in-diversion.groff
> .box DIV
> hello \[u260E] world
> \#hello \[ru] world
> .br
> .box
> .DIV
> .\"pm DIV
> .chop DIV
> .asciify DIV
> .\"pm DIV
> .DIV
> \X'pdf: ignoreme \*[DIV]'
> $ ./build/test-groff -Z ATTIC/unicode-char-in-diversion.groff
> x T ps
> x res 72000 1 1
> x init
> p1
> x font 5 TR
> f5
> s10000
> md
> DFd
> V12000
> H72000
> thello
> wx font 12 ZD
> f12
> h2500
> Cu260E
> wf5
> h9690
> tw
> H111310
> torld
> wh2500
> thello
> wf12
> h2500
> Cu260E
> wf5
> h9690
> tw
> H169230
> torld
> V12000
> H185340
> x X pdf: ignoreme hello \[u260E] world
> n12000 0
> x trailer
> V792000
> x stop
>
>
> Please try this out and let me know if it is satisfactory, or isn't.
>

Hi Branden,

I like it a lot. One problem outstanding, it does not like your name! If I use
this as a test:-

.box DIV
hello \[u260E] world and G.\& \fBBranden\fP Robinson
\#hello \[ru] world
.br
.box
.
.\"pm DIV
.chop DIV
.asciify DIV
.\"pm DIV
.DIV
\X'pdf: ignoreme \*[DIV]'
.pdfinfo /Title "\*[DIV]"

Using:-

test-groff -Tpdf -Z G.trf

You will notice:-

troff:G.trf:12: warning: a node is not encodable in device-independent output
('asciify' might help)
x X ps:exec [/Title (hello \[u260E] world and G.) /DOCINFO pdfmark

Cut off in your prime! However, if you apply:-

--- ../src/roff/troff/node.cpp  2025-08-31 21:06:24.050087492 +0100
+++ node.cpp    2025-08-31 21:08:58.421762137 +0100
@@ -3883,7 +3883,8 @@

 void node::asciify(macro *m)
 {
-  m->append(this);
+  if (strcmp("dummy_node", this->type()) != 0 && strcmp("tag_node", this-
>type()) != 0)
+    m->append(this);
 }

 // XXX: Member function is identical to `composite_node::asciify()`.

The problem (and the error) disappear. This is probably not the best way to
stop dummy_nodes depositing null chars into c-string (which is what causes the
truncation), I don't know anything about C++ inheritance, it might just
require a dummy_node::asciify routine which does nothing!

Cheers

Deri

> (file #57604)
>

Deri James <deri>
Group Member
Sat 30 Aug 2025 12:11:59 PM UTC, comment #10: 

Hi Deri,

At 2025-08-30T05:46:29-0400, G. Branden Robinson wrote:

> > fine when the diversion is output as a heading but needs to be
> > converted back to text to be used for the bookmark. Traditionally
> > this was done by calling .asciify which converted glyph nodes and
> > word space nodes back to text:-
> >
> > .{type: glyph_node, character: "F", diversion level: 1},
> > {type: glyph_node, character: "i", diversion level: 1},
> > {type: glyph_node, character: "r", diversion level: 1},
> > {type: glyph_node, character: "s", diversion level: 1},
> > {type: glyph_node, character: "t", diversion level: 1},
> > {type: word_space_node, diversion level: 1},
> >
> > But, oddly, fails to convert:-
> >
> > {type: glyph_node, character: "\u260E", diversion level: 1},
> >
> > Back to \[u260E], possibly the code was written before the advent of
> > preconv.
>
> That sounds like a bug I want to fix.
>
> Let me see if it's live in Git today.

[...]

> Hmm, yup.  Looks like a bug in `asciify` all right.


I think I have this licked.

I'm attaching (or trying too--let's see if Savannah's email gateway can
cope) a diff of my changes since my last push (just a couple of hours
ago).

Here it is in action:


$ cat ATTIC/unicode-char-in-diversion.groff
.box DIV
hello \[u260E] world
\#hello \[ru] world
.br
.box
.DIV
.\"pm DIV
.chop DIV
.asciify DIV
.\"pm DIV
.DIV
\X'pdf: ignoreme \*[DIV]'
$ ./build/test-groff -Z ATTIC/unicode-char-in-diversion.groff
x T ps
x res 72000 1 1
x init
p1
x font 5 TR
f5
s10000
md
DFd
V12000
H72000
thello
wx font 12 ZD
f12
h2500
Cu260E
wf5
h9690
tw
H111310
torld
wh2500
thello
wf12
h2500
Cu260E
wf5
h9690
tw
H169230
torld
V12000
H185340
x X pdf: ignoreme hello \[u260E] world
n12000 0
x trailer
V792000
x stop


Please try this out and let me know if it is satisfactory, or isn't.


(file #57604)

G. Branden Robinson <gbranden>
Group administrator
Sat 30 Aug 2025 09:46:24 AM UTC, comment #9: 

Hi Deri,

A few days ago I started digging into this at last.

At 2025-01-10T12:13:28-0500, Deri James wrote:

> Date: Fri 10 Jan 2025 05:13:24 PM UTC By: Deri James <deri>
> First I will explain what I am trying to accomplish, before describing
> the issue.
>
> With the demise of pdfmark.pdf and mspdf.tmac I realised that creating
> a replacement would be a lot easier if we did a similar job Branden
> did for man but this time for ms. Branden is correct that Keith's
> original pdf work is more unixy than roffish, using commands on a
> single line and various flags, and his mspdf.tmac provides new
> commands which are wrappers around existing ms commands with added pdf
> extensions. So instead of:-
>
> .NH 1
> Introduction
>
> It supports:-
>
> .NH 1
> .XN Introduction
>
> Apart from having to learn new commands it is also less flexible. Ms
> allows:-
>
> .NH 1
> Introduction to
> .I groff
>
> Which would have to be converted to inline font changes to work with
> .XN.


Right.  I've long been uneasy with this departure from what is idiomatic
for ms(7) (and to some extent mm(7)) documents.

> It also means that existing ms documents can't magically start using
> pdf features without considerable editing. The gold standard
> "solution" would be if the output was pdf then pdf features are
> automatically included, the same as Branden did with man.


It won't surprise you that that's what I think.  Getting man page
authors to adopt new habits is just shy of futile; where possible, I
think the path to success lies in making existing features "just work".

Sometimes that's a lot easier said than done; my quest to resolve this
problem has forced me to learn a lot about deep formatter internals,
including stuff that's never really been documented.  It also produced a
few struggles between us to make sense of what's really going on in
those mysterious internals.

> As a proof of concept I attempted to make .NH produce a document
> outline as well as headings. I used ms.ms as a test document,


That's an ambitious guinea pig, since ms.ms exercises most of the
package's own features.  On the bright side, if we can tame this beast,
I expect most user documents to fall to our sword as well.

> using the command
> "test-groff -Tpdf -ms -M. -mpdfms -pet -ww ../doc/ms.ms > msdj.pdf",
> the result is attached.


Looks excellent!  The only thing I expected that I didn't get was
hotspots in the document's table of contents at the end.  However, since
there's a lovely navigation bar (in a sufficiently brainful PDF viewer,
meaning NOT DROPBOX), that's an inconsequential defect.

Support for hotspotted external URLs might be more important.

That in turn might mean adding `UR` and `UE` macros to ms(7).  As I've
noted elsewhere, I am not a fan of "www.tmac"'s approach.

> Seems to like pic.ms too.


I know from hard experience that that document is good at breaking
grohtml; may gropdf fare better.

> Now onto the problem with .asciify. Because I have to use a diversion
> to capture the line(s) after the .NH the diversion contains nodes,


It is necessary to use a diversion?  The way the oldest macro packages
handled this was by storing the document's input lines into a macro.
I think this approach might be fruitful because then you're guaranteed
not to have to deal with any node objects.

However, I haven't tried this and it might have downsides I don't
anticipate.

> fine when the diversion is output as a heading but needs to be
> converted back to text to be used for the bookmark. Traditionally this
> was done by calling .asciify which converted glyph nodes and word
> space nodes back to text:-
>
> .{type: glyph_node, character: "F", diversion level: 1},
> {type: glyph_node, character: "i", diversion level: 1},
> {type: glyph_node, character: "r", diversion level: 1},
> {type: glyph_node, character: "s", diversion level: 1},
> {type: glyph_node, character: "t", diversion level: 1},
> {type: word_space_node, diversion level: 1},
>
> But, oddly, fails to convert:-
>
> {type: glyph_node, character: "\u260E", diversion level: 1},
>
> Back to \[u260E], possibly the code was written before the advent of
> preconv.


That sounds like a bug I want to fix.

Let me see if it's live in Git today.


$ cat ATTIC/unicode-char-in-diversion.groff
.box DIV
hello \[u260E] world
.br
.box
.DIV
.asciify DIV
.DIV
\X'pdf: ignoreme \*[DIV]'
$ ./build/test-groff -bww -T pdf -a ATTIC/unicode-char-in-diversion.groff
<beginning of page>
troff: backtrace: 'ATTIC/unicode-char-in-diversion.groff':6: string 'DIV'
troff: backtrace: file 'ATTIC/unicode-char-in-diversion.groff':8
troff:ATTIC/unicode-char-in-diversion.groff:8: warning: a node is not encodable in device-independent output
troff: backtrace: 'ATTIC/unicode-char-in-diversion.groff':6: string 'DIV'
troff: backtrace: file 'ATTIC/unicode-char-in-diversion.groff':8
troff:ATTIC/unicode-char-in-diversion.groff:8: warning: missing closing delimiter in device extension escape sequence; expected character "'", got a newline
hello <u260E> world hello <u260E> world '


Hmm, yup.  Looks like a bug in `asciify` all right.

While exploring the issues you've raised here, I've discovered that
`chop` is not exactly fully armed and operational.  It only "kind of"
works, but satisfies the one use case that motivated it; see bug #67453.

Solving it properly looks like a heavy lift.  GNU troff has yet more
bespoke container types for macro/string/diversion contents, `char_list`
and `node_list`, and I lack the courage to refactor these away before
the 1.24 release.

I will not be the least surprised if the issues arising thence are
entangled with bug #62264 and bug #64004.

> Now, the issues this code uncovered.
>
> 1. Near the top of the pdfms.tmac there are redefinitions of .B and
> .BI which replace the ones which are generated when s.tmac is loaded.
> If you comment out the definitions and produce the pdf there are no
> errors/warnings but some overview entries are truncated. You can see
> this in the two subsidiary entries to "Legacy Features" which both get
> truncated to one word by asciify if the s.tmac version of .I is used,
> but not if using the redefined version. One difference is that the
> s.tmac versions introduce italic correction escapes which may be what
> is upsetting asciify.


I think you're right.

> 2. The shw macro just calls pdfbookmark, so largely redundant, why not
> call pdfbookmark directly instead of shw. If you replace shw with a
> direct call to pdfbookmark, groff coredumps with an assert failure. I
> believe this is the same as a bug number we already have, but the
> reproducer started working after removal of asciify from pdf.tmac. So
> I have also attached a minimum example (it.trf) which reproduces the
> coredump if you use pdfms.tmac with the direct call to pdfbookmark
> rather than the indirection though shw.


I still need to look into this.

> This can all be put on the back burner until after the current release
> I'd just appreciate an affirmation that we agree this is the best way
> forward in replacing mspdf.tmac. I started on .NH because I think if
> we have a workable solution for that I think it is likely all pdf
> features can be slotted in without a separate api, which is what Keith
> used.
>
> Any good ideas, or better approaches, would be appreciated.


I think you're on a good directional track.  We just need to dig some
boulders out of the formatter's roadway so your changes can pass easily.

Regards,
Branden

G. Branden Robinson <gbranden>
Group administrator
Sun 19 Jan 2025 05:58:20 PM UTC, comment #8: 

The attached patch fixes the first problem (truncation of the result of asciifying) and may be a fix for the elusive crash (it certainly stops the crashing with this reproducer). In addition I have brought the result of asciify into line with Branden's new \X handling, i.e. any glyph which has not got an ascii code is represented in its \[uXXXX] form.

Technical Note

In order to extract "text" back from nodes in a diversion (or box) asciify should only be interested in glyph_nodes and word_space_nodes, since these are the nodes created from text in the first place. Unfortunately when trying to asciify other types of node, intead of text chars being added, a '\000' is added instead (which results in truncation when treated as a string. This is why the ms versions of .I caused truncation since the the italic correction also includes a dummy_node, which is one of nodes which adds '\000', the other I discovered was tag_node, there may well be others. The \& is another escape which can add a dummy_node so "G.\& Branden Robinson" (as a co-author) got truncated to just "G."

This patch prevents these two node types interfering with asciify.

One strange thing I observed in the truncated macros is that the cl structure which contains members ptr, head, tail and len, was in a case where the diversion started with a tag_node (devtag TL) so the string appears empty (starts with '\000') and ptr, head and tail all point to the same address but len held a positive number equal to the number of characters added to the macro by asciify. This may be expected behaviour, len could be the number of bytes of allocated memory, but I just wondered what would happen if this was subsequently .chop(ped). So this might be the reason for the assert crash dump at the end of the run, but I would not bank on it, it is a slippery fellow.

The patch can probably be improved, I'd appreciate Branden's insight.

My "proof of concept" work on adding pdf features to the ms macros is coming along, the attached pdfs are examples. Both djms.pdf and djpic.pdf are run with the current ms source documents (with just the addition of ".RP no" adding a separate title page so the TOC can be auto relocated to page after the title) against my current version of pdfms.tmac (still got a few wrinkles to sort out so it is not attached - yet). When finished I am hoping Branden wants to incorporate this work into s.tmac, probably after 1.24, with suitable .if '\*[.T]'pdf' protections. The pdfmark2.pdf is using the latest pdfmark.ms before it was removed from git and requires "pdfmom --roff -ms" and my new pdfms.tmac to work. Things to notice are author and title correct if you run pdfinfo and a clickable TOC. All of these needed the patch applied to work properly.

(file #56781, file #56782, file #56783, file #56784)

Deri James <deri>
Group Member
Tue 14 Jan 2025 05:58:41 PM UTC, comment #7: 

comment #5:

> You may remember a "wish" to allow Dingbats glyphs to be accessed
> using \[uXXXX] notation rather than just \N'nnn'.


Right.  Bug #63018, committed in August.  I really need to run a current build.

Dave <barx>
Group Member
Mon 13 Jan 2025 06:29:45 PM UTC, comment #6: 

comment #5:

> To see the second issue you have to run it with ms.ms,


D'oh!  I'm illiterate.  That is indeed the filename in the command you pasted in comment #2.  So much for me "trying to mimic your test as closely as possible"!

Dave <barx>
Group Member
Mon 13 Jan 2025 12:31:13 PM UTC, comment #5: 

The warning:-

troff:it.trf:7: warning: special character 'u260E' not defined

Occurs if I run 1.23.0. You may remember a "wish" to allow Dingbats glyphs to be accessed using \[uXXXX] notation rather than just \N'nnn'. I believe that is now in current master.

The document it.trf does not trigger the second problem (core dump) unfortunately, however, it does illustrate the first problem when the ms versions of B and BI are used the overview entry is truncated to a single word. To see the second issue you have to run it with ms.ms, it's not even triggered by pic.ms, but at least we have a "reproducer" again now.

Deri James <deri>
Group Member
Mon 13 Jan 2025 09:47:10 AM UTC, comment #4: 

comment #2:

> The two problems can only be seen by making an amendment to pdfms.tmac.


I'm still unable to provoke the assertion failure, whether I make both changes to pdfms.tmac, or only the second one.

$ cat pdfms.tmac
.nr PDFHREF.VIEW.LEADING  10.0p
.tm DJ loaded
.de I
\fI\\$1\fP
..
.de BI
\f(BI\\$1\fP
..
.am @NH
.  nr nh*run 1
.  box @NHpdf
.  dt 1v @NH-end
..
.de shw
.  pdfbookmark \\n[nh*hl] "\\$*"
..
.de @NH-end
.  fl
.  box
.  nr nh*run 0
.  chop @NHpdf
\&\\*[@NHpdf]
.  asciify @NHpdf
.  pdfbookmark \\n[nh*hl] "\\*[@NHpdf]"
.\" .shw \\*[@NHpdf]
..
$ groff -Tpdf -Kutf8 -ms -M. -mpdfms -pet -ww it.trf | wc
DJ loaded
troff:it.trf:7: warning: special character 'u260E' not defined
    171    2017   16989
$ groff-latest -Tpdf -Kutf8 -ms -M. -mpdfms -pet -ww it.trf | wc
DJ loaded
troff:it.trf:7: warning: special character 'u260E' not defined
     82     231    2167
$ sed 3,8d pdfms.tmac > pdfms2.tmac
$ groff -Tpdf -Kutf8 -ms -M. -mpdfms2 -pet -ww it.trf | wc
DJ loaded
troff:it.trf:7: warning: special character 'u260E' not defined
    171    2014   16983
$ groff-latest -Tpdf -Kutf8 -ms -M. -mpdfms2 -pet -ww it.trf | wc
DJ loaded
troff:it.trf:7: warning: special character 'u260E' not defined
     80     227    2164

My it.trf is unchanged from the one attached to this ticket.

The -pet options seem unnecessary for this input file (and indeed, I get the same results with or without them), but as you supplied them, I did too to try to mimic your test as closely as possible.

Dave <barx>
Group Member
Mon 13 Jan 2025 09:02:22 AM UTC, comment #3: 

comment #2:

> testing with any other ms documents you have would be appreciated


I have none, sorry, or I'd try it on them.

> The warning you get running it.trf is because it is a UTF-8
> document, so it needs preconv as well.


Ah, yes, I should have picked up on that.

However, invoking preconv merely changes it to a different warning.  (My August groff build generates a substantially smaller PDF than groff 1.23, but both give the same warning.)

$ groff -Tpdf -Kutf8 -ms -M. -mpdfms it.trf | wc
DJ loaded
troff:it.trf:7: warning: special character 'u260E' not defined
    171    2017   16989
$ groff-latest -Tpdf -Kutf8 -ms -M. -mpdfms it.trf | wc
DJ loaded
troff:it.trf:7: warning: special character 'u260E' not defined
     82     231    2167


> I can't remember the previous bug number which may be the same
> issue for the core dump.


With the full assertion text now at hand, I surmise you refer to bug #65322.

Dave <barx>
Group Member
Sun 12 Jan 2025 02:15:45 PM UTC, comment #2: 

Sorry, I was not very clear. The pdfms.tmac I sent works fine (at least with ms.ms and pic.ms, testing with any other ms documents you have would be appreciated). The two problems can only be seen by making an amendment to pdfms.tmac. The first problem can be revealed by commenting out the definitions of the I and BI near the top of the file. You will see the two sub entries to "Legacy Features" are truncated to a single word. The second problem can be seen by changing:-

.\" .  pdfbookmark \\n[nh*hl] "\\*[@NHpdf]"
.shw \\*[@NHpdf]

To:-

.  pdfbookmark \\n[nh*hl] "\\*[@NHpdf]"
.\" .shw \\*[@NHpdf]

Near the bottom of the file. You should then see:-

[derij@pip build (master)]$ test-groff -Tpdf -ms -M. -mpdfms -pet -ww ../doc/ms.ms | okular -
DJ loaded
troff: ../src/roff/troff/input.cpp:642: static int input_stack::finish_get(node**): Assertion `level == 0' failed.
/home/derij/groff-git/groff/build/groff: error: troff: Aborted (core dumped)

You get the same result if you run 1.23.0.

The warning you get running it.trf is because it is a UTF-8 document, so it needs preconv as well. I can't remember the previous bug number which may be the same issue for the core dump. I think it was a mom document which got postponed when it no longer caused the error after I removed the call to asciify from pdf.tmac.

Deri James <deri>
Group Member
Sat 11 Jan 2025 10:13:10 PM UTC, comment #1: 

original submission:

> I have also attached a minimum example (it.trf) which
> reproduces the coredump if you use pdfms.tmac with the direct
> call to pdfbookmark rather than the indirection though shw.


I can't reproduce this using the two attached files and a groff I built about 5 months ago from then-latest git source.  Is this something that requires bleeding-edge groff, or am I doing something wrong?  In particular, I'm not sure whether the above-quoted sentence means I should modify something in pdfms.tmac, or whether the version of the file as attached here should produce the core dump.

$ ls -gG pdfms.tmac it.trf
-rw------- 1  79 Jan 10 12:59 it.trf
-rw------- 1 341 Jan 10 13:01 pdfms.tmac
$ groff-latest --version | head -1
GNU groff version 1.23.0.1624-4d251-dirty
$ groff-latest -Tpdf -ms -M. -mpdfms it.trf | wc
DJ loaded
troff:it.trf:11: error: cannot translate character code 226 to special character '^a' in device-independent output
     80     226    2223

I get the same results using groff 1.23 (except without the warning, unless I set GROFF_ENABLE_TRANSPARENCY_WARNINGS), so if you're seeing the core dump on 1.23, then we're using different methodologies.

Dave <barx>
Group Member
Fri 10 Jan 2025 05:13:24 PM UTC, original submission:  

First I will explain what I am trying to accomplish, before describing the issue.

With the demise of pdfmark.pdf and mspdf.tmac I realised that creating a replacement would be a lot easier if we did a similar job Branden did for man but this time for ms. Branden is correct that Keith's original pdf work is more unixy than roffish, using commands on a single line and various flags, and his mspdf.tmac provides new commands which are wrappers around existing ms commands with added pdf extensions. So instead of:-

.NH 1
Introduction

It supports:-

.NH 1
.XN Introduction

Apart from having to learn new commands it is also less flexible. Ms allows:-

.NH 1
Introduction to
.I groff

Which would have to be converted to inline font changes to work with .XN. It also means that existing ms documents can't magically start using pdf features without considerable editing. The gold standard "solution" would be if the output was pdf then pdf features are automatically included, the same as Branden did with man.

As a proof of concept I attempted to make .NH produce a document outline as well as headings. I used ms.ms as a test document, using the command "test-groff -Tpdf -ms -M. -mpdfms -pet -ww ../doc/ms.ms > msdj.pdf", the result is attached. Seems to like pic.ms too.

Now onto the problem with .asciify. Because I have to use a diversion to capture the line(s) after the .NH the diversion contains nodes, fine when the diversion is output as a heading but needs to be converted back to text to be used for the bookmark. Traditionally this was done by calling .asciify which converted glyph nodes and word space nodes back to text:-

.{type: glyph_node, character: "F", diversion level: 1},
{type: glyph_node, character: "i", diversion level: 1},
{type: glyph_node, character: "r", diversion level: 1},
{type: glyph_node, character: "s", diversion level: 1},
{type: glyph_node, character: "t", diversion level: 1},
{type: word_space_node, diversion level: 1},

But, oddly, fails to convert:-

{type: glyph_node, character: "\u260E", diversion level: 1},

Back to \[u260E], possibly the code was written before the advent of preconv.

Now, the issues this code uncovered.

1. Near the top of the pdfms.tmac there are redefinitions of .B and .BI which replace the ones which are generated when s.tmac is loaded. If you comment out the definitions and produce the pdf there are no errors/warnings but some overview entries are truncated. You can see this in the two subsidiary entries to "Legacy Features" which both get truncated to one word by asciify if the s.tmac version of .I is used, but not if using the redefined version. One difference is that the s.tmac versions introduce italic correction escapes which may be what is upsetting asciify.

2. The shw macro just calls pdfbookmark, so largely redundant, why not call pdfbookmark directly instead of shw. If you replace shw with a direct call to pdfbookmark, groff coredumps with an assert failure. I believe this is the same as a bug number we already have, but the reproducer started working after removal of asciify from pdf.tmac. So I have also attached a minimum example (it.trf) which reproduces the coredump if you use pdfms.tmac with the direct call to pdfbookmark rather than the indirection though shw.

This can all be put on the back burner until after the current release I'd just appreciate an affirmation that we agree this is the best way forward in replacing mspdf.tmac. I started on .NH because I think if we have a workable solution for that I think it is likely all pdf features can be slotted in without a separate api, which is what Keith used.

Any good ideas, or better approaches, would be appreciated.


Deri James <deri>
Group Member

 

Attached Files

Attached Files
file #57609:  deri-66653-sep01.diff added by gbranden (24KiB - text/x-patch)
file #57608:  pdfms.tmac added by deri (3.7KiB - application/octet-stream)
file #57604:  deri-66653.diff added by gbranden (10KiB - text/x-diff)
file #56781:  asciify.patch added by deri (856B - text/x-patch)
file #56782:  djpic.pdf added by deri (103KiB - application/pdf)
file #56783:  djms.pdf added by deri (81KiB - application/pdf)
file #56784:  pdfmark2.pdf added by deri (69KiB - application/pdf)
file #56763:  pdfms.tmac added by deri (341B - application/octet-stream)
file #56764:  it.trf added by deri (79B - application/octet-stream)
file #56765:  msdj.pdf added by deri (75KiB - application/pdf)

(Note: upload size limit is set to 4.0MiB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Dependencies

Digest:
   bug dependencies.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by gbranden (Updated the item)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by deri (Submitted the item)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    History

    Follow 27 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2025-11-04 deri Attached File- Added Screenshot_20251103_231513.png, #57769
    2025-09-18 gbranden StatusIn Progress Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.24.0
    2025-09-18 gbranden Summary[troff] .asciify odd behaviour [troff] .asciify odd behaviour with composite characters
    2025-09-18 gbranden StatusNeed Info In Progress
        Assigned toderi gbranden
        Summary[PATCH] [troff] .asciify odd behaviour / adding pdf features to ms [troff] .asciify odd behaviour
    2025-09-13 gbranden Dependencies- Depends on bugs #67509
    2025-09-04 gbranden Summary[PATCH] [troff].asciify odd behaviour / adding pdf features to ms [PATCH] [troff] .asciify odd behaviour / adding pdf features to ms
    2025-09-01 deri Attached File- Added Screenshot_20250901_165033.png, #57613
    2025-09-01 deri StatusNone Need Info
    2025-09-01 gbranden Attached File- Added deri-66653-sep01.diff, #57609
        StatusNeed Info None
    2025-09-01 deri Attached File- Added pdfms.tmac, #57608
    2025-08-30 gbranden Attached File- Added deri-66653.diff, #57604
    2025-07-06 gbranden Summary[patch][troff].asciify odd behaviour / adding pdf features to ms [PATCH] [troff].asciify odd behaviour / adding pdf features to ms
    2025-01-19 deri Attached File- Added asciify.patch, #56781
        Attached File- Added djpic.pdf, #56782
        Attached File- Added djms.pdf, #56783
        Attached File- Added pdfmark2.pdf, #56784
        StatusNone Need Info
        Assigned toNone deri
        Summary.asciify odd behaviour / adding pdf features to ms [patch][troff].asciify odd behaviour / adding pdf features to ms
    2025-01-10 deri Attached File- Added pdfms.tmac, #56763

    Back to the top

    Powered by Savane 3.16-8b18.
    Corresponding source code