Hi Deri,

comment #32:

> Again a long reply will be required since you've again extended the scope bringing in heirloom, dwb, libdriver. whereas i tried to limit the scope to examine whether gropdfs grout parser was cnformant to our current docs. You may consider our current documentation inadequate, i have no issue with it, and if you alter it and libdriver

You may have missed this island in my ocean of text:

> > Fortunately, I see no need to alter libdriver in any way

And this remains the case, despite my findings in comment #30.

> to do what you want, then gropdf will no longer be conformant, because you have changed the api it relied omn.

I am telling you that gropdf is not behaving congruently with grops today.  I don't need to change anything to make that statement true.  I mention libdriver only because that is factually where the troff output parser for all of groff's other output drivers lives.

I checked the behavior of Heirloom and DWB only to see if they might shed any light on interpretations of CSTR #54.  If you're like me, you can treat them as suggestive but not dispositive, particularly given their bad behavior as shown in comment #30.

So what remains is this:

> i tried to limit the scope to examine whether gropdfs grout parser was cnformant to our current docs. You may consider our current documentation inadequate, i have no issue with it,

Then you should be able to resolve the dilemma I raised in comment #29--a dilemma which arose from two competing claims of the current documentation, which I therefore do not regard as sufficient to characterize an API.

> > does a command with one fixed-length argument get followed by a line break or not?

If it does, then this groff 1.23.0 (or 1.22.4) output demands explanation.

$ printf '\\o@abc@\n' | groff -Tascii -Z | sed -n '12p'
cacbccH24

If it doesn't, then this sentence from our docs demands explanation.

> > > in 'gtroff''s intermediate output, every command with
> > > at least one argument is followed by a line break,

If you can tell me how the foregoing facts are not in tension, I'd appreciate hearing it.  If you cannot, then I submit to you that you are putting too much stock in our documentation, and that the implementation of libdriver, which has seen little change since 2006, should be given more weight.

> Im afraikd it will be sometime before i can give f ull reply, computer gone kaput,

That sucks.  I'm sorry to hear it.  :(

> new one built (son under  guidance), but teething issues so not quite customised yet. hand typing so evben slower than noprmal. :-(

Forgot the attachment, as usual.



(file #55039)

comment #27:

> First I'd like to try to reduce the scope of this discussion, since it seems to have grown in multiple directions.

Sure.
 

> Am I correct in the assumption that the grout files for any given input would not be identical when produced by different roff implementations?

Yes.  Here's some simple input given to Heirloom Doctools troff and then GNU troff.

$ printf -- '.nf\na b\n-\\-\n' | ./bin/troff -Tps
x T ps
x res 72000 1 1
x init
V0
p1
x font 1 R /home/branden/heirloom/lib/doctools/font/devps/R.afm 4
x font 2 I /home/branden/heirloom/lib/doctools/font/devps/I.afm 4
x font 3 B /home/branden/heirloom/lib/doctools/font/devps/B.afm 4
x font 4 BI /home/branden/heirloom/lib/doctools/font/devps/BI.afm 4
x font 5 CW /home/branden/heirloom/lib/doctools/font/devps/CW.afm 4
x font 6 H /home/branden/heirloom/lib/doctools/font/devps/H.afm 4
x font 7 HB /home/branden/heirloom/lib/doctools/font/devps/HB.afm 4
x font 8 HX /home/branden/heirloom/lib/doctools/font/devps/HX.afm 4
x font 9 S1 /home/branden/heirloom/lib/doctools/font/devps/S1.afm 516
x font 10 S /home/branden/heirloom/lib/doctools/font/devps/S.afm 1028
s10
f1
x X LC_CTYPE en_US.UTF-8
H72000
V12000
ca
wh7770cb
n12000 0
H72000
V24000
c-
h3330C\-
n12000 0
x trailer
V792000
x stop
$ printf -- '.nf\na b\n-\\-\n' | troff -Tps
x T ps
x res 72000 1 1
x init
p1
x font 5 TR
f5
s10000
V12000
H72000
md
DFd
ta
wh2500
tb
n12000 0
V24000
H72000
t-
C\-
h5640
n12000 0
x trailer
V792000
x stop

In the Heirloom output, I find the line

wh7770cb

noteworthy for a reason I'll return to.

> I assume this is true, else the output drivers from different implementations would be interchangeable. I don't believe they are.

I have a feeling that Kernighan might have been reaching for this, but didn't quite nail down the syntax tightly enough to strictly permit it.  However, I could be retrojecting my thoughts in 2023 onto his in 1980, when (a) there was no other device-independent troff implementation, (b) one was not going to appear for nearly a decade, and (c) given the challenges he described in CSTR #97, he might have thought it unlikely that another would ever be created.  As further speculation, (d) in 1980 the eventual divestiture of the AT&T monopoly was not yet seen as inevitable, and as long as that hope held, there wasn't much need for anyone to reimplement device-independent troff since it could be had for low or zero cost.  If anyone held that belief, it was the most swiftly overturned of these.
 

> This means, whilst they may all be based on the information in cstr#54, each roff has developed its own private API between the formatter and output drivers. For this reason the decision on whether this is a change to the groff version of the API, has to be confined to what is contained in the groff documentation.

Or we can accept that groff's documentation doesn't adequately describe its implementation, which I believe I just demonstrated in bug #63544.
 

> Empirical observation shows that groff uses a simple rule of one operation per line

No.  Glyph output and horizontal motions are frequently mixed when the `tcommand` directive is not present.  If we remove it from font/devps/DESC, we get this.

$ printf -- '.nf\na b\n-\\-\n' | ./build/test-groff -T ps -Z | grep cawh
cawh6940

And it is ubiquitous when the "obsolete" (to use Bernd's term) output command is used.  (I would term it "legacy".)

$ printf -- '.nf\na b\n-\\-\n' | groff -T X100 -Z | grep caw
caw10bh7

Further, (recalling the point I promised to return to above) Heirloom breaks this rule in yet another respect, putting a `c` command after an `h` command, whose argument is not fixed-width (the integers it uses are not zero-padded on the left) with no separation at all.

> and using a single space to avoid a "clashing between the command code and the arguments without the space.", even though Kernighan states that it is permitted to use newlines as well as a space and tab for this purpose, none of our drivers support this.

I believe I have empirically refuted this claim in bug #63544, comment #3.

> The w command is not an operation, it is just a marker for a paddable word space so following with a newline is against our own documentation:-

I find it unnecessary to bifurcate the class of "output commands" into "operations" and "non-operations".  It certainly isn't required to explain present (and, I have to guess, decades-long outstanding) behavior of libdriver.

> in 'gtroff''s intermediate output, every command with
> at least one argument is followed by a line break,

This is demonstrably false, as I showed above.

> thus providing
> excellent readability.

...and that's an unnecessary sales pitch.

> The w command has no arguments so under this rule it should not have a following new line.

I think you have extrapolated an invalid rule from vague and inaccurate documentation.

> As regards having a space after the w command, our documentation says:-

> The 'gtroff' output parser, however, is smart about whitespace by making it
> maximally optional.

> Which I take to mean it only uses a space to avoid the "clashes" mentioned above,

This is another sales pitch, and vague besides.

> and it further says:-

> Commands and arguments with a known, fixed length need
> not be separated by syntactical space.

It does say that.  Unfortunately this claim contradicts one you already quoted.

> in 'gtroff''s intermediate output, every command with
> at least one argument is followed by a line break,

According to the above, does a command with at one fixed-length argument get followed by a line break or not?

I don't think that question is answerable without relying on an additional information channel, like reading the source code or experimenting.

You may be beginning to see why I am critical of this documentation.

> The w command is fixed length, so to satisfy "maximally optional" no space is used.

As noted previously, I cannot elicit any clear semantics from the modifier "maximally" here.
 

> I never said that white space cannot follow a w command, but if we change to include white space after it then that goes against how the groff version of the API has been documented for many years.

That, I agree with.  Our documentation in this area is inaccurate.  Please understand that I am exercising restraint by not saying more.

> I believe the change Branden has on his private branch is to output a new line after a w command, but this bug concerns white space after the w. Contrary to cstr#54, which classes space/tab/newline as the same, groff does not allow newline to be used as the white space between a command and its arguments (this difference is not documented).

That's a good interesting point and one I want to explore with further testing.  I see no reason groff output drivers shouldn't accept a newline thus, given the clarity of CSTR #54's wording on the subject.

> If groff 1.23++ is going to use w followed by a new line, none of the proposed patches is optimal. A loop is no longer required, since no further commands will be on that line. There is no point in producing code to cater for situations which will not arise. Gropdf is written to parse grout output from groff, if that output is altered so that it no longer complies with our own documentation and gropdf fails to handle it then it is not a bug, but a change in the API and should involve a change request and at least a wider discussion than just us three.

I propose for gropdf to accept the same inputs grops does and to interpret them in a compatible way.  That is all.
 

> Apparently, the reason for wanting to make this change is to "generate "grout" that is more easily lexically analyzed". Citing posix shell's poor lexical processing capabilities, I don't see what difference wh2500 and w\nh2500 makes. If that's what you want, would a simple filter like this help:-

> [derij@pip busgrap]$ perl -pe 's/(.)(.*)/$1\n$2/ if m/^w/; s/^(.)(\S.*)/$1 $2/mg' zfile
> x T ps
> x res 72000 1 1
> x init
> p 1
> x font 5 TR
> f 5
> s 10000
> V 12000
> H 72000
> m d
> D Fd
> t Deri
> w
> h 2500
> D l 100000 0
> n 12000 0
> x trailer
> V 792000
> x stop

I don't think writing such a tool is desirable.  It "feels" too small to be a shippable tool--not supportive of its own weight in terms of making it a proper command with `--help` and `--version` and a man page, or the effort of coming up with a good name for it; too long to expect anyone to type it; and too obscure for it to make it into many people's shell start-up files as a function.

In my opinion, GNU troff should simply produce output that is easy for humans to read in the first place.

> In fact I'd be very happy to write a proper grout tool with multiple output options (pretty print, markup, XML). Markup could look like:-

> x T ps                        # for grops
> x res 72000 1 1
> x init
> p1                      # Page 1
> x font 5 TR
> f5                      # Times-Roman
> s10000                  # ps 10
> V12000                  # V 1/6th in
> H72000                  # H 1in
> md                      # Default text colour: Black
> DFd                     # Default fill colour: Black
> tDeri
> wh2500                  # Word Space: 2.5p
> Dl 100000 0             # Line from x,y to x1,y1
> n12000 0                # New Line
> x trailer
> V792000
> x stop

> Using some of the code from gropdf which keeps track of current position, XML output could tag the x,y position of every element.

That would indeed be useful--a "grout annotator" if you will.  But I think it's outside the scope of this ticket.

> So it is unnecessary to change the format of grout to achieve what you say you want.

As noted above and in bug #63544, I don't have to.  We just need to correct the documentation and align gropdf with the other output drivers here.
 

> The danger in changing the current grout format is we do not know what tools have been written which parse our current grout format, didn't someone write a parser which output html/javascript, how do we know our changes won't affect them.

If they accept what grops, grotty, grodvi, grohtml, and so forth do, then they'll be fine.  They do risk disruption if they relied upon our badly composed documentation in this area.  If I don't treat CSTR #54 as scripture as some of our mailing list subscribers do, I'm surely not going to bring a higher level of reverence to our own, particularly where it has demonstrable problems.

> Given that there are non-intrusive methods to achieve the result you want, I hope your hankerings can be satisfactorily assuaged.

I have to reject your conclusion here as ill-premised.  Fortunately, I see no need to alter libdriver in any way (pending the "newlines everywhere" research).  My tasks are to (1) revise our erroneous and unclear documentation and (2) assemble a patch for gropdf that you're willing to accept, assuming you lack the time or desire to do so yourself.

> AOB
>
> Many people have praised Branden for his contributions to the documentation,

I don't think you need two hands to count them.  :P

> as I do, it just felt wrong to see open criticism of a fellow contributers use of english. I am more than happy for Branden to make our documentation more "pellucid", but I think it is  nicer to do it without denigrating previous efforts which were made with the best intentions.

I have tried (I do not claim always successfully) to critique the code, not the person.  As I understand it, this is an aspect of egoless programming.  You may have observed Alex Colomar expressing a pretty low opinion of the `is_family_valid()` function on the groff list recently.  He may not have known at the time that I had written it.  While I was taken aback at first, I did not get upset with him, either on the mailing list or privately.  On the contrary, I largely agreed with his assessment; the code's form arose from an unfortunate constraint problem we have with our decision (which I guess I'm the main person driving) to stick to ISO C++98.

The same goes for documentation.  We all put a bit of ourselves into our written words, but the text is not the author.  Ingo Schwarze is another groff contributor who pulls no punches when expressing opinions of code.  But I don't remember seeing him engage in personal attacks.  I expect the same latitude when evaluating documentation, but I am also prepared to endure criticism of my own product.  All 3 of us can likely remember Ralph Corderoy's withering assessments of my emails (particularly their length).  He seemed to have difficulty believing that I could write concisely.  Whether he ever actually read any of the documentation I have written for groff (when it wasn't pitched to the mailing list first--a tiny proportion), I don't know--he never offered any evidence of having done so.  Ralph's unrelentingly negative attitude about groff and my work on it (in contrast to other *roffs) irritated me but I didn't, and don't, let that stop me from considering and crediting such contributions as he makes.  In other words, I can work with him.  If Bernd should return, I would expect to be able to work with him, too, and I hope he'd reciprocate that.

> The latest incarnation of gropdf (in the deri-gropdf-ng git branch, give it a go :-)) is now 80 lines short of 5000 lines.

So far your work is getting rave reviews.  I'm envious!  :D

 First I'd like to try to reduce the scope of this discussion, since it seems to have grown in multiple directions.

Am I correct in the assumption that the grout files for any given input would not be identical when produced by different roff implementations? I assume this is true, else the output drivers from different implementations would be interchangeable. I don't believe they are.

This means, whilst they may all be based on the information in cstr#54, each roff has developed its own private API between the formatter and output drivers. For this reason the decision on whether this is a change to the groff version of the API, has to be confined to what is contained in the groff documentation.

Empirical observation shows that groff uses a simple rule of one operation per line and using a single space to avoid a "clashing between the command code and the arguments without the space.", even though Kernighan states that it is permitted to use newlines as well as a space and tab for this purpose, none of our drivers support this.

The w command is not an operation, it is just a marker for a paddable word space so following with a newline is against our own documentation:-

in 'gtroff''s intermediate output, every command with
at least one argument is followed by a line break, thus providing
excellent readability.

The w command has no arguments so under this rule it should not have a following new line.

As regards having a space after the w command, our documentation says:-

The 'gtroff' output parser, however, is smart about whitespace by making it
maximally optional.

Which I take to mean it only uses a space to avoid the "clashes" mentioned above, and it further says:-

Commands and arguments with a known, fixed length need
not be separated by syntactical space.

The w command is fixed length, so to satisfy "maximally optional" no space is used.

I never said that white space cannot follow a w command, but if we change to include white space after it then that goes against how the groff version of the API has been documented for many years.

I believe the change Branden has on his private branch is to output a new line after a w command, but this bug concerns white space after the w. Contrary to cstr#54, which classes space/tab/newline as the same, groff does not allow newline to be used as the white space between a command and its arguments (this difference is not documented). If groff 1.23++ is going to use w followed by a new line, none of the proposed patches is optimal. A loop is no longer required, since no further commands will be on that line. There is no point in producing code to cater for situations which will not arise. Gropdf is written to parse grout output from groff, if that output is altered so that it no longer complies with our own documentation and gropdf fails to handle it then it is not a bug, but a change in the API and should involve a change request and at least a wider discussion than just us three.

Apparently, the reason for wanting to make this change is to "generate "grout" that is more easily lexically analyzed". Citing posix shell's poor lexical processing capabilities, I don't see what difference wh2500 and w\nh2500 makes. If that's what you want, would a simple filter like this help:-

[derij@pip busgrap]$ perl -pe 's/(.)(.*)/$1\n$2/ if m/^w/; s/^(.)(\S.*)/$1 $2/mg' zfile
x T ps
x res 72000 1 1
x init
p 1
x font 5 TR
f 5
s 10000
V 12000
H 72000
m d
D Fd
t Deri
w
h 2500
D l 100000 0
n 12000 0
x trailer
V 792000
x stop

In fact I'd be very happy to write a proper grout tool with multiple output options (pretty print, markup, XML). Markup could look like:-

x T ps                        # for grops
x res 72000 1 1
x init
p1                      # Page 1
x font 5 TR
f5                      # Times-Roman
s10000                  # ps 10
V12000                  # V 1/6th in
H72000                  # H 1in
md                      # Default text colour: Black
DFd                     # Default fill colour: Black
tDeri
wh2500                  # Word Space: 2.5p
Dl 100000 0             # Line from x,y to x1,y1
n12000 0                # New Line
x trailer
V792000
x stop

Using some of the code from gropdf which keeps track of current position, XML output could tag the x,y position of every element.

So it is unnecessary to change the format of grout to achieve what you say you want.

The danger in changing the current grout format is we do not know what tools have been written which parse our current grout format, didn't someone write a parser which output html/javascript, how do we know our changes won't affect them. Given that there are non-intrusive methods to achieve the result you want, I hope your hankerings can be satisfactorily assuaged.

AOB

Many people have praised Branden for his contributions to the documentation, as I do, it just felt wrong to see open criticism of a fellow contributers use of english. I am more than happy for Branden to make our documentation more "pellucid", but I think it is  nicer to do it without denigrating previous efforts which were made with the best intentions.

Dave, I don't use dasher for coding, just a keyboard, which is slower for input, but better for navigating around the document, cut and paste, controlling indentation and such like. For a maths exam at school they gave me an amanuensis, a very nice retired lady who spent the whole time doing a birdseye view of what her ideal home would look like. Gradually adding more and more detail with a multitude of coloured pencils (each had to be sharpened before use) right down to drawing individual roses in a regular pattern on the living room blue carpet - I was fascinated! Unfortunately she did not know how to draw an integral sign, so that took some explaining! This was before personal computers, oh for groff and eqn then.

The latest incarnation of gropdf (in the deri-gropdf-ng git branch, give it a go :-)) is now 80 lines short of 5000 lines.

comment #23:

> Please can you give me a few days to answer this. It will take me a lot of hours.

Certainly.  A few points to ease your response burden.  To make an analogy to legal filings, you don't have to reply to the entire brief.

• My comment #22 was addressed to Dave, not you; please don't feel under any expectation to present a detailed response to it.

• I did express my interpretations of some of your words in that message, but if those interpretations are incorrect I won't be bothered if you address only those.

• My comment #4 and comment #19 are the code changes I'm specifically seeking your feedback on as gropdf author.  If you can give me an ACK or NAK on each of them, that would be the most expeditious course from a strict development perspective.

• I also welcome your ACK or NAK on Dave's comment #20.

• Time is not of the essence anyway.  This code lives in the working copy of a private branch on my dev box, and not in the public groff repo.  Even when I eventually push it, a private branch is where it will stay for a while as I figure out how I want to stage my changes back onto master, starting with known bug fixes and other uncontroversial changes.

• When Bertrand tags 1.23.0 I will be preoccupied with matters other than this ticket for an indefinite amount of time.

comment #24:

> For your delectation I recorded me writing the last message, to give you a rough idea how long it will take.
>
> Try:-
>
>   vlc http://chuzzlewit.co.uk/screen.mp4
>
> I'd classify it as a PG rather than a U. :-)

Fortunately for you I lived in Australia for long enough to become familiar with the U.K. rating system (as well as Oz's).  :)  (My own vernacular tends toward black-label entertainment; I have to be a good boy on FSF-hosted infrastructure.)

That's a lot of work.  :-|  I also used to live in Indiana; one of its former governors, Roger Branigin, famously said, "I never argue with a man who buys ink by the barrel."  It may be your misfortune to have a debate opponent who moves it by the supertanker.

Consider me chagrined.

Please can you give me a few days to answer this. It will take me a lot of hours.

comment #17:

> Where we deviate from CSTR #54, we say so.

A fair point, but it only addresses my second clause about CSTR #54.  The more relevant question is, do we claim CSTR #54 as a specification for groff, aside from such noted deviations?  (Maybe we do; you know the groff docs better than I do.)  Because if groff's own docs don't address this whitespace in grout, and groff code in its history has never placed whitespace here, one could argue this constitutes an API change within the groff ecosystem.

For the record, I'm not making an argument about this either way, just trying to see it from multiple sides.

> I think our output drivers should behave consistently,

True, and they do behave consistently given the grout that groff has produced to date.  My reference to Postel's law was intended to point out that libgroff being coded to liberally accept "grout" from other troffs (generically, "rout"?) doesn't necessarily indicate an intention on groff's part to support producing such grout.

> and if someone were to convert PostScript generated by grops
> to PDF and PDF generated by gropdf to PostScript, one should
> not be able to tell which output driver originally produced the
> document (ignoring comments within the files announcing the fact).

I agree with that, but I'm not sure I follow your logic, as the above seems to be about the output of the two postprocessors, whereas the API question is about their input.

Anyway, this debate is academic: ultimately, I favor making grout more usable by other tools (and more readable to humans), and whitespace is important for that, regardless of whether one characterizes it as an API change.

Here's an updated version of my proposed patch for this ticket.  It assumes the one from comment #4 is already applied, and will need manual tweaking otherwise.

diff --git a/ChangeLog b/ChangeLog
index 373fab6df..f0a12b807 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,15 @@
+2023-06-28  G. Branden Robinson <g.branden.robinson@gmail.com>
+
+       * src/devices/gropdf/gropdf.pl: Accept newlines after 'w'
+       commands as our libdriver-using output drivers do.  ("Blanks,
+       tabs, and newlines may occur as separators in the input, and are
+       mandatory to separate constructions that would otherwise be
+       confused." --CSTR #54 (1992))  Also, when debugging, write any
+       'w' commands encountered as comments to the output stream, as is
+       done with other commands.
+
+       Fixes <https://savannah.gnu.org/bugs/?64360>.
+
 2023-06-27  G. Branden Robinson <g.branden.robinson@gmail.com>

        * src/devices/gropdf/gropdf.pl: Fix code style nits.  Rename
diff --git a/src/devices/gropdf/gropdf.pl b/src/devices/gropdf/gropdf.pl
index 074a725c5..e86644da5 100644
--- a/src/devices/gropdf/gropdf.pl
+++ b/src/devices/gropdf/gropdf.pl
@@ -369,16 +369,17 @@ while (<>)
        my $cmd=substr($_,0,1);
        next if $cmd eq '#';    # just a comment
        my $lin=substr($_,1);
+       $lin=~s/^\s*//;

-       while ($cmd eq 'w')
+       # 'w' is the only zero-argument command.
+       while ($lin and $cmd eq 'w')
        {
+           $stream.="\% w\n" if $debug;
+           $w_flg=1 if $t_flg;
            $cmd=substr($lin,0,1);
            $lin=substr($lin,1);
-           $w_flg=1 if $t_flg;
+           $lin=~s/^\s*//;
        }
-
-       $lin=~s/^\s+//;
-#              $lin=~s/\s#.*?$//;      # remove comment
        $stream.="\% $_\n" if $debug;

        do_x($lin),next if ($cmd eq 'x');

I'm also attaching a tar archive of test cases I used to develop and check the logic.  It's a set of 6 specimens of GNU troff output (hand-modified in 5 cases to exercise the latitude afforded by CSTR #54), and the patched-gropdf-generated PDFs, all of which produce correct output as far as I can tell.

Annotating with "[PATCH]".  Assigning to Deri.  Is this an acceptable change?

(file #54892)

comment #16:

> comment #15:
> > I strongly object to your characterization of this as a
> > proposed API change.  It is not.  It honors the letter and
> > spirit of CSTR #54,
>
> If the layout of groff's output language hasn't changed in 30+ years, there's arguably a de facto API separate from the published one.  For that matter, groff didn't publish CSTR #54 and deviates from it in various ways.  So whether it is or isn't an API change may depend on which angle you're looking at it from.

Where we deviate from CSTR #54, we say so.  (And we don't, very much.  More often, we document errata in it.)

> > and groff's other output drivers are not tripped up by the change,
>
> A good dose of Postel's law is often advisable.

I think this ticket is entirely in the same spirit as bug #63757, bug #62950, bug #62934, bug #62923, bug #62346, bug #62294, bug #61908, and bug #51568, all of which contrasted gropdf's behavior with grops (or, in one case, could have done).

I think our output drivers should behave consistently, and if someone were to convert PostScript generated by grops to PDF and PDF generated by gropdf to PostScript, one should not be able to tell which output driver originally produced the document (ignoring comments within the files announcing the fact).

I am baffled as to why achieving parity and consistency in this respect is being tarred with the "API change" brush.

comment #14:

> Exactly, it is not a word gap its a horizontal move.

It seems we are in agreement.  But that is not an argument against this ticket.
 

> Please can you explain the practical benefit for this change.

Please review bug #63544, which I linked to earlier.  I want to make it easier for a wider variety of tools to parse GNU troff output, namely including shell scripts, which (under POSIX shell, at rate) have fairly clumsy character-based lexical processing capabilities.

> Perhaps on the list given that it is a change the current api between groff and the output drivers.

We can discuss it on the list but I strongly object to your characterization of this as a proposed API change.  It is not.  It honors the letter and spirit of CSTR #54, and groff's other output drivers are not tripped up by the change, which suggests that it this troff output is well within James Clark's expectations.

Here is an example of your output from comment #12 after this change.

$ echo "A Berty" | groff -Tpdf -fU-T -Z
x T pdf
x res 72000 1 1
x init
p1
x font 5 U-TR
f5
s10000
V12000
H72000
md
DFd
tA
w
h2500
tBer
h230
tty
n12000 0
x trailer
V792000
x stop

Since Perl can easily do multi-line matching (though you certainly don't have to use it), I am not understanding what about the above you find objectionable.

Thanks.  Mmmm, crunchy.

But the presence of an 'h' command after 't' (or 'N' or 'C' or a legacy move-and-print command) does not necessarily imply kerning.

It indicate motion to the next tab stop.

$ printf 'A\tB\n' | ./build/test-groff -Z
x T ps
x res 72000 1 1
x init
p1
x font 5 TR
f5
s10000
V12000
H72000
md
DFd
tA
h28780
tB
n12000 0
x trailer
V792000
x stop

comment #10:

> Yes, I have seen multiple "w", but never one followed by a space.

I've never seen multiple 'w' commands in sequence without some other command in between.

I can't remember if I've seen 'w' followed by a space.
 

> I thought commands and arguments with a known fixed length (such as the w command) aren't separated by syntactical space. Well, that was true until now.

As far as I know that's never been true, but it seems that AT&T troff didn't often avail itself of optional white space.  CSTR #97 suggests that there was some pressure to make the formatter's output as low in byte count as possible.
 

> Previously, before you added /n,

Well, it's not exactly live code where anyone can get to it but me.

> a w is always followed by another command, which is what my code expects.

It certainly should always be followed by another command, even if it's "x trailer".  The question is whether it can be followed by white space, and CSTR #54 seems unequivocal about that.

You can see an example of what I think is a Kernighan parser at the following.

https://github.com/n-t-roff/DWB3.3/blob/master/postscript/dpost/dpost.c#L723

> I can't see a reason for the change, it is actually helpful, when looking at troff output, to quickly distinguish word spaces whnnn and kerning hnnn.

Can you show me a concrete example of what you mean?

> I have not seen your discussion on changing this on the list, I did see your suggestion about adding a word mark to \~ but not changing the syntax of the file produced by troff.

It's an older ticket, from December.

https://savannah.gnu.org/bugs/?63544

Yes, I have seen multiple "w", but never one followed by a space, can you work out why?

I thought commands and arguments with a known fixed length (such as the w command) aren't separated by syntactical space. Well, that was true until now.

comment #5:

>
> comment #3:
> > Just move this line:-
> >
> >         $lin=~s/^\s+//;
> >
> > to before the while loop.
>
> Oh.  This is what I came up with.  Is it wrong?

I guess it is.  I didn't understand you while loop.  You're prepared for 'wwwwwww'.

comment #3:

> Just move this line:-
>
>         $lin=~s/^\s+//;
>
> to before the while loop.

Oh.  This is what I came up with.  Is it wrong?

diff --git a/src/devices/gropdf/gropdf.pl b/src/devices/gropdf/gropdf.pl
index 074a725c5..200221ce4 100644
--- a/src/devices/gropdf/gropdf.pl
+++ b/src/devices/gropdf/gropdf.pl
@@ -370,14 +370,14 @@ while (<>)
        next if $cmd eq '#';    # just a comment
        my $lin=substr($_,1);

-       while ($cmd eq 'w')
+       # 'w' is the only zero-argument command.
+       if ($cmd eq 'w')
        {
+           $w_flg=1 if $t_flg;
+           $lin=~s/^\s*//;
            $cmd=substr($lin,0,1);
            $lin=substr($lin,1);
-           $w_flg=1 if $t_flg;
        }
-
-       $lin=~s/^\s+//;
 #              $lin=~s/\s#.*?$//;      # remove comment
        $stream.="\% $_\n" if $debug;

 

> How did you get groff to plant the space?

diff --git a/src/roff/troff/node.cpp b/src/roff/troff/node.cpp
index 152dfda1b..341edce1f 100644
--- a/src/roff/troff/node.cpp
+++ b/src/roff/troff/node.cpp
@@ -5828,6 +5828,7 @@ int word_space_node::is_tag()
 void unbreakable_space_node::tprint(troff_output_file *out)
 {
   out->fill_color(col);
+  out->word_marker();
   if (is_html) {
     // we emit the space width as a negative glyph index
     out->flush_tbuf();

Just move this line:-

        $lin=~s/^\s+//;

to before the while loop.

How did you get groff to plant the space?

I propose the following patch.

diff --git a/src/devices/gropdf/gropdf.pl b/src/devices/gropdf/gropdf.pl
index c65a1051f..6ed7e4e57 100644
--- a/src/devices/gropdf/gropdf.pl
+++ b/src/devices/gropdf/gropdf.pl
@@ -370,14 +370,17 @@ while (<>)
        next if $cmd eq '#';    # just a comment
        my $lin=substr($_,1);

-       while ($cmd eq 'w')
+       if ($lin)
        {
-           $cmd=substr($lin,0,1);
-           $lin=substr($lin,1);
-           $w_flg=1 if $gotT;
-       }
+           while ($cmd eq 'w')
+           {
+               $cmd=substr($lin,0,1);
+               $lin=substr($lin,1);
+               $w_flg=1 if $gotT;
+           }

        $lin=~s/^\s+//;
+       }
 #              $lin=~s/\s#.*?$//;      # remove comment
        $stream.="\% $_\n" if $debug;

On my private branch, with the change to 'w' output, 4 automated tests fail and gropdf (Perl, really) screams thousands of lines of error output while constructing groff-man-pages.pdf.

With this patch, all 180 tests pass (or XFAIL) and no extra error output is produced.

But it may not be the most elegant fix, as I have only a worm's-eye view of gropdf's parser.

Assigning to Deri and putting into "Need info" status for feedback.

Is this patch okay or does it have problems?