bugGNU roff - Bugs: bug #62825, page header info should correspond...

 
 

bug #62825: page header info should correspond to last section on the page

Submitter:  None
Submitted:  Wed 27 Jul 2022 08:18:06 PM UTC
   
 
Category:  Macro - others/general Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 26 Sep 2022 05:16:34 PM UTC, comment #12: 

Unassigning from self.  I don't think I'm going to "git this kilt" for groff 1.23.  Properly considered, the scope of this ticket is all of our full-service macro packages except for man(7) and mdoc(7).

But if someone wants to swoop in with patches and comprehensive test coverage...

G. Branden Robinson <gbranden>
Group administrator
Mon 08 Aug 2022 05:04:23 AM UTC, comment #11: 

That's a good workaround.  A couple of conceptually simple refinements could reduce some of the tedium of the process.

First, instead of redefining $h throughout the document, define it once at the top, but include in it logic to handle the special-needs pages, by using stacked .ie/.el statements as an ersatz "switch" statement.  (An example of this was very recently added to the groff manual, in commit 41477bb4.)  Here is how you could generate the same output you got in your comment #10 example:


.hx
.ll 6.5i
.de $h
.      ie \\n%=2 .tl 'Chapter \\n(ch'Section 1.2'Page \\n%'
.el \{.ie \\n%=3 .tl 'Chapter \\n(ch'Section 1.3'Page \\n%'
.el              .tl 'Chapter \\n(ch'Section \\*($n'Page \\n%'
.\}
.hl
..
.+c ""
.sh 2 "Section 1.1"
.lp
This is the first section.
.bp
.sh 2 "Section 1.2"
.lp
This is the second section.
.bp
.sh 2 "Section 1.3"
.lp
This is the third section.


Another fairly straightforward refinement would be to automate your step 2: -me providing plug-in macros that are called at the top and bottom of every page simplifies this immensely.  And if you're not using any footers (as in your example output), and thus don't care about preserving -me's default definition of .$f, it's even more straightforward.  You can send this data to stderr (using groff's .tm request), so that it doesn't interfere with your normal output.  Your first iteration of the document could look something like this:


.hx
.ll 6.5i
.de $H
.nr chapter_number_at_page_top \\n(ch
.ds section_string_at_page_top \\*($n
..
.de $f
.if !\\n[chapter_number_at_page_top]=\\n(ch .tm Page \\n% starts with chapter \\n[chapter_number_at_page_top] and ends with chapter \\n(ch
.if !'\\*[section_string_at_page_top]'\\*($n' .tm Page \\n% starts with section \\*[section_string_at_page_top] and ends with section \\*($n
..
.de $h
.tl 'Chapter \\n(ch'Section \\*($n'Page \\n%'
.hl
..
.+c ""
.sh 2 "Section 1.1"
.lp
This is the first section.
.bp
.sh 2 "Section 1.2"
.lp
This is the second section.
.bp
.sh 2 "Section 1.3"
.lp
This is the third section.


Then on the first pass, you don't even care about the output, only what appears on stderr:


$ nroff -me section_test_first_pass.me > /dev/null
Page 2 starts with section \&1.1 and ends with section \&1.2
Page 3 starts with section \&1.2 and ends with section \&1.3
$


This makes it simple to populate the .ie block in your $h macro.

If you want to get extra fancy, you could even craft your .tm statements to output lines in .ie format that would then only need to be pasted back into the code to give you your second pass.  Once you're at that stage, you could write a driver script to run the first pass, capture stderr, put that back into the document in the right place, and groff the revised document.  Implementation of this is left as an exercise to the reader. (-:  Whether it's worth the development and debugging time to write such a script depends on how many documents you have and how long they are.

Dave <barx>
Group Member
Fri 29 Jul 2022 06:35:05 PM UTC, comment #10: 

comment #7:

> It sounds to me like this was just a straight-up bug.  My guess is that people simply haven't been putting section information in their headers because of this behavior, they worked around it, or they got lucky.


I think they might have worked around it, possibly in the same way I just figured out: using hard-coded values. I did it like this:

  1. Typeset the document with your custom "generic" $h macro definition to be used for all pages.
  2. Go through the output to find the "problem" pages with headers showing the incorrect section number/title.
  3. On the page before each "problem" page and after the last section creation on that page, re-define $h with a hard-coded section number/title.
  4. Right after the "problem" page is created, restore the "generic" $h definition.
  5. Repeat this for each "problem" page.


I tried this out on my example and it worked (the first page of a chapter doesn't matter to me because I always suppress the header there anyway):


.hx
.ll 6.5i
.de $h
.tl 'Chapter \\n(ch'Section \\*($n'Page \\n%'
.hl
..
.+c ""
.sh 2 "Section 1.1"
.lp
This is the first section.
.rm $h
.de $h
.tl 'Chapter \\n(ch'Section 1.2'Page \\n%'
.hl
..
.bp
.sh 2 "Section 1.2"
.rm $h
.de $h
.tl 'Chapter \\n(ch'Section \\*($n'Page \\n%'
.hl
..
.lp
This is the second section.
.rm $h
.de $h
.tl 'Chapter \\n(ch'Section 1.3'Page \\n%'
.hl
..
.bp
.sh 2 "Section 1.3"
.lp
This is the third section.

The output now has correct headers:


$ nroff -me section_test_fix.roff | cat -s

                           CHAPTER  1

1.1.  Section 1.1

This is the first section.

Chapter 1                  Section 1.2                     Page 2
_________________________________________________________________

1.2.  Section 1.2

This is the second section.

Chapter 1                  Section 1.3                     Page 3
_________________________________________________________________

1.3.  Section 1.3

This is the third section.


I also tried it in my larger document where I first saw the problem, and it worked there, too (with a more complicated $h definition). It's brute force and ugly, and tedious, since it has to be done for every section change (excluding new chapters). But it works. This is for the me macros, but the workaround for ms is similar.

Brian Kernighan is still around (and typesetting new books in troff), so I'm tempted to email him and ask if this is how he did it in the K&R C book. Maybe I was giving the old troff too much credit. :)

comment #7:

> An LX?  Running 2.6?  We might have this bug fixed before it finishes booting.  And we don't work fast.


Haven't booted up the LX yet. I'd rather wait until the winter when the heat from the massive Sun monitor could warm my entire house. I'll still try it, but not a priority since I found a workaround that will tide me over until a true fix is released. I appreciate you and Dave working on this. It's great that groff will likely be the first troff implementation to fix this decades-old bug that seems to go back to the old UNIX troff from the 1970s.

Anonymous
Fri 29 Jul 2022 02:28:16 AM UTC, comment #9: 

comment #7:

> One of these was the same as in the pile of books I cited
> recently to undermine Ingo's claim that the overwhelming
> majority of publishers in all fields set headers as mandoc(1)
> does.


(Off topic for this bug, but I read his claim more that mandoc was coded to set headers following (his idea of) publishing tradition.  Maybe I'm just feeling extra sympathetic towards him, having myself made a brazenly incorrect claim about headers earlier in this bug report.)

> >  There is the very real problem of breaking backward
> > compatibility even if it does arguably improve output.
>
> It sounds to me like this was just a straight-up bug.


I agree, but that doesn't address all compatibility concerns.  For instance, one of Eric Allman's cited use cases (recently removed from the -me manual (commit 3e281469)) for redefining $h was providing multi-line headers, which -me does not inherently support.  But you have to know how much space to leave for the header before outputting any page content.

It is certainly possible to give the user a way to control this for new -me documents.  But any historical ones with a redefined $h that changed the header's height will render very badly after this bug fix, because -me will simply have no way to know how much vertical space that $h will take up.

This isn't an argument against making the fix, just a caution that back compatibility needs some careful attention.  And I'm familiar enough with -me to recognize some of its potential pitfalls; in other packages I'd have no idea what issues may await.

> My guess is that people simply haven't been putting section
> information in their headers because of this behavior, they
> worked around it, or they got lucky.


Or they failed to check existing practice altogether and assumed, as I initially did, that the header should reflect the content at the beginning rather than the end of the page.

Dave <barx>
Group Member
Thu 28 Jul 2022 11:47:15 PM UTC, comment #8: 

Whoops, I didn't finish editing my reply.

comment #7:

> I reckon what we can do is leave the existing header trap macro names in place, but change them to just mark the vertical location with `mk`.  The footer trap can set its own mark, `rt` to the header, call the macro that actually formats the header, then `rt` to its own place and proceed normally.


[Dave said:]

> > I'll let Branden weigh in here.
>
> I am tempted to call this feature "reflex headers", but unfortunately for me (if I get it working), it doesn't really need a name at all because it will be how headers were supposed to work all along.
>   
> >  There is the very real problem of breaking backward compatibility even if it does arguably improve output.


Pffft.  I have no fear.  Just call me "MR" progress.

G. Branden Robinson <gbranden>
Group administrator
Thu 28 Jul 2022 11:44:12 PM UTC, comment #7: 


comment #6:

> comment #5:
> > Hmm, every book I've ever seen with section numbers/titles in
> > the header uses the info for the last section on the page.
>
> You are right; I spoke from ignorance.  I opened an O'Reilly book at random, and it does indeed populate its headers as you say.


I too have to concur with our anonymous submitter.  I checked the all-too-short pile of books I have handy.  About half didn't bother to report section numbers or titles in headers (or footers) at all, but those that did, published by Springer, MIT, and Prentice-Hall, consistently rendered them as that person argues.

One of these was the same as in the pile of books I cited recently to undermine Ingo's claim that the overwhelming majority of publishers in all fields set headers as mandoc(1) does.

(I'll just let the audacity of that proclamation hang in the air for a moment.)

> Given that, at a minimum, it seems the -me and -ms documentation ought to at least mention that the macro packages do not behave as might be expected.
>
> However, since the problem is (theoretically) fixable -- by deferring the printing of the header until the end-of-page trap is reached -- arguably the packages should do this by default.


I reckon what we can do is leave the existing header trap macro names in place, but change them to just mark the vertical location with `mk`.  The footer trap can set its own mark, `rt` to the header, call the macro that actually formats the header, then `rt` to its own place and proceed normally.  I'll let Branden weigh in here.

I am tempted to call this feature "reflex headers", but unfortunately for me (if I get it working), it doesn't really need a name at all because it will be how headers were supposed to work all along.
  

>  There is the very real problem of breaking backward compatibility even if it does arguably improve output.


It sounds to me like this was just a straight-up bug.  My guess is that people simply haven't been putting section information in their headers because of this behavior, they worked around it, or they got lucky.

By coincidence I've been looking at mm(7) today.  If we rope Peter Schaffter in, we can maybe fix all of our full-service, general-purpose macro packages in one fell swoop.

Fortunately man pages don't have this problem.  man(7) and mdoc(7) users who want to buy this trouble for themselves are welcome to do so.
 

> > This is tempting me to boot up my ancient Sun SPARCstation LX
> > running Solaris 2.6, to see if its proprietary version of
> > troff has the same issue.


An LX?  Running 2.6?  We might have this bug fixed before it finishes booting.  And we don't work fast.

(More seriously, while I love the vintage hardware, my wallet and my storage unit bill have persuaded me of the value of emulation.)
 

> I'd be very surprised if the -me macros behaved any differently in this regard: unlike the rest of groff, they weren't reverse engineered in new code, but used Eric Allman's historical implementation (albeit customized for groff).  But if you run this experiment and find otherwise, please do report back here.


I can perhaps confirm a little more easily.  Heirloom Doctools me(7) (vintage 2019 or so) has precisely the same problem.


$ ./bin/nroff -me ../62825.me | ul | cat -s

Chapter 0                 Section                     Page 1
_________________________________________________________________

                         CHAPTER  1

1.1.  Section 1.1

This is the first section.

Chapter 1               Section 1.1                   Page 2
_________________________________________________________________

1.2.  Section 1.2

This is the second section.

Chapter 1               Section 1.2                   Page 3
_________________________________________________________________

1.3.  Section 1.3

This is the third section.


G. Branden Robinson <gbranden>
Group administrator
Thu 28 Jul 2022 11:21:30 PM UTC, comment #6: 

comment #5:

> Hmm, every book I've ever seen with section numbers/titles in
> the header uses the info for the last section on the page.


You are right; I spoke from ignorance.  I opened an O'Reilly book at random, and it does indeed populate its headers as you say.

Given that, at a minimum, it seems the -me and -ms documentation ought to at least mention that the macro packages do not behave as might be expected.

However, since the problem is (theoretically) fixable -- by deferring the printing of the header until the end-of-page trap is reached -- arguably the packages should do this by default.  I'll let Branden weigh in here.  There is the very real problem of breaking backward compatibility even if it does arguably improve output.

> Since K&R and Stevens were able to get the correct section
> info in their headers, the old troff must have had a way of
> doing it. They used the ms macros.


They may have very well written their own macros to handle certain tasks such as this, though.  One can easily replace macro definitions in a standard macro package by simply redefining them in one's own document.

> This is tempting me to boot up my ancient Sun SPARCstation LX
> running Solaris 2.6, to see if its proprietary version of
> troff has the same issue.


I'd be very surprised if the -me macros behaved any differently in this regard: unlike the rest of groff, they weren't reverse engineered in new code, but used Eric Allman's historical implementation (albeit customized for groff).  But if you run this experiment and find otherwise, please do report back here.

Dave <barx>
Group Member
Thu 28 Jul 2022 10:22:27 PM UTC, comment #5: 

comment #4:

> Consider a page where the top half wraps up section 1.4, then section 2.1 begins, and has a paragraph or two on the lower half of the page.  Here you want your page header to indicate section 1.4, since that's the section in effect when the page begins, but the above-described system would put section 2.1 in the header.


Hmm, every book I've ever seen with section numbers/titles in the header uses the info for the last section on the page. So in your scenario they would use 2.1, not 1.4, in the header. I think this is pretty much the standard convention in publishing, including for FSF publications. Do you have any examples of books that don't do that?

comment #4:

> Your situation is an edge case, where a page break happens to coincide with a section change.  In this case, -me does not look ahead to see that the very first thing to happen is the beginning of a new section, so it prints what looks like an incorrect section number in the header.  But technically, this is the section that was in effect when the page began, and it is only over the course of the page that it changed.


Many books have sections that start on a new page, so I'm not sure it's such an edge case. Those books with the section number or title in the header show the new section info in the header when the section starts the page.

For example, two of the famous books typeset in troff (the K&R C book and Stevens' APUE book) do just that. I'm now curious how they did it! In fact, in K&R (2nd ed) there are examples of a new section starting a page (p.139), a single section starting in the middle of a page (p.13), and a page with two new sections in the page (p.41). In all cases the header shows the last section number on that page. APUE does it similarly, though with section titles instead of numbers. Same for every book I've ever seen.

Since K&R and Stevens were able to get the correct section info in their headers, the old troff must have had a way of doing it. They used the ms macros. I've verified that in groff an ms version of my me example has the same problem. This is tempting me to boot up my ancient Sun SPARCstation LX running Solaris 2.6, to see if its proprietary version of troff has the same issue.

Anonymous
Thu 28 Jul 2022 06:40:56 PM UTC, comment #4: 

comment #3:

> So it seems that groff does a single pass through each page from
> top to bottom: start a new page, then print the header, then
> print the body, then print any footnotes, then lastly print the
> footer. It then starts the next page.

...

> So is that how groff works? Or does groff ever "go back", upward
> to return to the header for additional formatting?


This is generally how troff-family formatters work.  In groff, it extends to everything from the way running text is formatted to the way tables of contents are handled.

To expand on these points, if you're unfamiliar with some of groff's basics:

  • Paragraphs are formatted line by line, meaning that once a line is set, it isn't modified even if a slightly different adjustment would give better adjustment results on a later line; this drawback has been remedied in some other modern troff variants, and is a much desired change for groff (bug #40716).


  • Since the page numbers are not known until all the relevant pages have been processed, troff cannot generate a table of contents indicating the page numbers until all the relevant pages have been processed.  One of two workarounds is generally used for this: a two-pass system, or having groff output the table of contents at the end, and a postprocessing tool reordering the pages to move it to the beginning.


The particular issue you're noting is less entrenched: although -me does in fact process a page in the manner you surmise, it could have implemented it differently: troff does give you the ability to move up the page, so the macro package could leave a gap at the top for the header, process the entire page, then, at the bottom, fill in the footer, move up the page, and fill in the header.

But in general, you wouldn't want this, because that would give you incorrect headers in some cases.  Consider a page where the top half wraps up section 1.4, then section 2.1 begins, and has a paragraph or two on the lower half of the page.  Here you want your page header to indicate section 1.4, since that's the section in effect when the page begins, but the above-described system would put section 2.1 in the header.

Your situation is an edge case, where a page break happens to coincide with a section change.  In this case, -me does not look ahead to see that the very first thing to happen is the beginning of a new section, so it prints what looks like an incorrect section number in the header.  But technically, this is the section that was in effect when the page began, and it is only over the course of the page that it changed.

Dave <barx>
Group Member
Thu 28 Jul 2022 01:01:57 AM UTC, comment #3: 

This seems to be about how and when groff prints the headers on each page. Notice that the only thing that's correct on each page of the output is the page number. After page 1 the chapter number in the header is then correct as well. However, at the start of page 2 when it's time to print the header, it seems groff doesn't know yet that the section number has changed. My guess is that maybe groff prints to a page from the top down without ever going back up to possibly reformat the header.

For example, suppose you put the same information in the footer:


.de $f
.hl
.tl 'Chapter \\n(ch'Section \\*($n'Page \\n%'
..
.$c
.sh 2 "Section 1.1"
.lp
This is the first section.
.bp
.sh 2 "Section 1.2"
.lp
This is the second section.
.bp
.sh 2 "Section 1.3"
.lp
This is the third section.


Then all the information in the footer is correct, even on the first page:


$ nroff -ww -me section_test2.roff | cat -s
troff: section_test.roff:5: warning: macro '$C' not defined
troff: section_test.roff:6: warning: number register '$1' not defined

                         CHAPTER  1

1.1.  Section 1.1

This is the first section.

_________________________________________________________________

Chapter 1                Section 1.1                  Page 1

1.2.  Section 1.2

This is the second section.

_________________________________________________________________

Chapter 1                Section 1.2                  Page 2

1.3.  Section 1.3

This is the third section.

_________________________________________________________________

Chapter 1                Section 1.3                  Page 3


So it seems that groff does a single pass through each page from top to bottom: start a new page, then print the header, then print the body, then print any footnotes, then lastly print the footer. It then starts the next page.

It makes sense that the footers would be correct, because by that time groff would know the current chapter and section, from the $c and .sh commands that were already executed in the body, after the header was created when the page began.

So is that how groff works? Or does groff ever "go back", upward to return to the header for additional formatting?

Anonymous
Wed 27 Jul 2022 08:33:30 PM UTC, comment #2: 

Sorry, I misunderstood the report in haste.

This is indeed a different issue.

I'm not sure what assumptions can be made about me(7)'s behavior when you take over the `$c` macro for chapter headings but never use the documented mechanism for advancing the chapter number, which is to call the `+c` macro.

I'll try to come up with what I think of as an "idiomatic" version of your example (after consulting meref.me) and see if I can identify any points of confusion, or missing documentation.

G. Branden Robinson <gbranden>
Group administrator
Wed 27 Jul 2022 08:24:03 PM UTC, comment #1: 

This is already been filed and a fix is pending.


commit 8d63b606569646b90122956880bebc85d2d225ff
Author:     G. Branden Robinson <g.branden.robinson@gmail.com>
AuthorDate: Mon Jul 25 13:43:23 2022 -0500
Commit:     G. Branden Robinson <g.branden.robinson@gmail.com>
CommitDate: Mon Jul 25 13:43:23 2022 -0500

    tmac/e.tmac: Fix Savannah #62812.

    * tmac/e.tmac: Initialize section number registers. Fixes warnings when
      starting section numbering at a depth greater than 1.

    Fixes <https://savannah.gnu.org/bugs/?62812>.

diff --git a/ChangeLog b/ChangeLog
index 7b8a50953..2e300bf42 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2022-07-25  G. Branden Robinson <g.branden.robinson@gmail.com>
+
+       * tmac/e.tmac: Initialize section number registers. Fixes
+       warnings when starting section numbering at a depth greater than
+       1.
+
+       Fixes <https://savannah.gnu.org/bugs/?62812>.
+
 2022-07-23  G. Branden Robinson <g.branden.robinson@gmail.com>

        * m4/groff.m4 (GROFF_PAGE): Use 'grep -q' instead of redirecting
diff --git a/tmac/e.tmac b/tmac/e.tmac
index 605c32018..14ea47593 100644
--- a/tmac/e.tmac
+++ b/tmac/e.tmac
@@ -2116,6 +2116,12 @@
 .
 .@R po\"               \" simulated page offset
 .@R $0\"               \" section depth
+.@R $1\"               \" section numbers a.b.c.d.e.f
+.@R $2\"
+.@R $3\"
+.@R $4\"
+.@R $5\"
+.@R $6\"
 .@R $i\"               \" paragraph base indent
 .@R $p\"               \" numbered paragraph number
 .\" [Before groff 1.06] the groff -me macros treated the $r and $R


Closing as duplicate of bug #62812.

G. Branden Robinson <gbranden>
Group administrator
Wed 27 Jul 2022 08:18:06 PM UTC, original submission:  

When re-defining the $h macro for a custom header when using -me, putting the section number in the header with \*($n results in the number being off by one (e.g. 1.1 instead of 1.2). This can be seen by running groff -me on this example:


.de $h
.tl 'Chapter \\n(ch'Section \\*($n'Page \\n%'
.hl
..
.$c
.sh 2 "Section 1.1"
.lp
This is the first section.
.bp
.sh 2 "Section 1.2"
.lp
This is the second section.
.bp
.sh 2 "Section 1.3"
.lp
This is the third section.


Here's the output when running nroff -ww -me:


$ nroff -ww -me section_test.roff | cat -s
troff: section_test.roff:5: warning: number register 'ch' not defined
troff: section_test.roff:5: warning: macro '$n' not defined
troff: section_test.roff:5: warning: macro '$C' not defined
troff: section_test.roff:6: warning: number register '$1' not defined

Chapter 0                 Section                     Page 1
_________________________________________________________________

                         CHAPTER  1

1.1.  Section 1.1

This is the first section.

Chapter 1                Section 1.1                  Page 2
_________________________________________________________________

1.2.  Section 1.2

This is the second section.

Chapter 1                Section 1.2                  Page 3
_________________________________________________________________

1.3.  Section 1.3

This is the third section.


I don't mind the section number (and chapter number) being off on the first page, as I always suppress the header anyhow on the first page of a new chapter. But the wrong section numbers in the headers of subsequent pages is an issue. If there's something I'm doing wrong, or if there's a simple fix/workaround I could use, then that would be good to know. Otherwise I guess it's a bug.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by gbranden (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-09-26 gbranden Assigned togbranden None
    2022-07-28 gbranden SummaryIncorrect section number using \*($n in a header page header info should correspond to last section on the page
    2022-07-28 gbranden CategoryMacro me Macro - others/general
    2022-07-27 gbranden StatusDuplicate None
        Open/ClosedClosed Open
    2022-07-27 gbranden StatusNone Duplicate
        Assigned toNone gbranden
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code