bugGNU roff - Bugs: bug #66187, [troff] permit control over...

 
 

bug #66187: [troff] permit control over flushing of output file state

Submitter:  G. Branden Robinson <gbranden>
Submitted:  Sun 08 Sep 2024 05:16:45 AM UTC
   
 
Category:  Core Severity:  1 - Wish
Item Group:  Feature change Status:  Postponed
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 27 Sep 2024 05:00:26 AM UTC, comment #5: 

Postponing.  I don't this gates anything we're trying to get done for groff 1.24.

G. Branden Robinson <gbranden>
Group administrator
Wed 11 Sep 2024 03:02:48 PM UTC, comment #4: 

This is one stubborn son of a gun.

I'm kicking the formatter pretty hard, and it's just not letting go of the "DFr".  It's holding tight onto it and keeping the black page background.

This may be an unfortunate consequence of its design.  The font, fill color, and stroke color all belong to things.  Glyphs or geometric objects.

I'll keep banging my head on this.  Failing that, the best alternative I have is to make `BOXSTART` use the escape hatch and blast a 'DFr' command to the output itself.  Really really gross.  Also there's the problem of retrieving the RGB color channel values of the selected color so as to write them out.  SUPER gross.

Here's the current state of my hackery for dark amusement.


diff --git a/contrib/sboxes/sboxes.tmac b/contrib/sboxes/sboxes.tmac
index 63c92636a..54636756e 100644
--- a/contrib/sboxes/sboxes.tmac
+++ b/contrib/sboxes/sboxes.tmac
@@ -75,7 +75,7 @@ .  de pdfbackground
 ..
 .
 .de BOXSTART
-.  fl
+.  nop \c \" Force the document to start.
 .  nr bx*stack \\n[bx*stack]+1u
 .  nr bx*shad 0
 .  nr bx*outl 0
@@ -85,6 +85,10 @@ .de BOXSTART
 .  while \\n[.$] \{\
 .    ie 'SHADED'\\$1' \{\
 .       nop \\M[\\$2]\c
+.\" We want to force a fill color setting command ("D F") onto the
+.\" output; the formatter cannot tell that we're using it, because it is
+.\" associated with no glyph or geometric object.
+'       fl 1
 .       nr bx*shad 1
 .       as bx*type "fill\"
 .       shift 2
diff --git a/src/roff/troff/div.cpp b/src/roff/troff/div.cpp
index 0f97c52fa..bed40e027 100644
--- a/src/roff/troff/div.cpp
+++ b/src/roff/troff/div.cpp
@@ -923,12 +923,16 @@ void output_saved_vertical_space()

 static void flush_request()
 {
-  while (!tok.is_newline() && !tok.is_eof())
-    tok.next();
+  bool flush_fill_color = false;
+  int n = 0;
+  if (has_arg() && get_integer(&n))
+    flush_fill_color = (n > 0);
   if (want_break)
     curenv->do_break();
-  if (the_output != 0 /* nullptr */)
-    the_output->flush();
+  if (the_output != 0 /* nullptr */) {
+    debug("GBR: the_output->flush(%1)", flush_fill_color);
+    the_output->flush(flush_fill_color);
+  }
   tok.next();
 }

diff --git a/src/roff/troff/node.cpp b/src/roff/troff/node.cpp
index 74b66672a..f60826017 100644
--- a/src/roff/troff/node.cpp
+++ b/src/roff/troff/node.cpp
@@ -738,6 +738,7 @@ public:
   real_output_file();
   ~real_output_file();
   void flush();
+  void flush(bool);
   void transparent_char(unsigned char);
   void print_line(hunits x, vunits y, node *n, vunits before, vunits after, hunits width);
   void begin_page(int pageno, vunits page_length);
@@ -755,6 +756,8 @@ public:
   void really_transparent_char(unsigned char);
   void really_print_line(hunits x, vunits y, node *n, vunits, vunits, hunits width);
   void really_begin_page(int pageno, vunits page_length);
+  // XXX: Should a flush of the fill color work even while output is
+  // suppressed?
 };

 class ascii_output_file : public real_output_file {
@@ -819,14 +822,14 @@ public:
   troff_output_file();
   ~troff_output_file();
   void flush();
+  void flush(bool);
   void trailer(vunits page_length);
   void put_char(charinfo *, tfont *, color *, color *);
   void put_char_width(charinfo *, tfont *, color *, color *, hunits, hunits);
   void right(hunits);
   void down(vunits);
   void moveto(hunits, vunits);
-  void start_device_extension(color * /* fcol */,
-                             bool /* omit_command_prefix */ = false);
+  void start_device_extension(bool /* omit_command_prefix */);
   void start_device_extension();
   void write_device_extension_char(unsigned char c);
   void end_device_extension();
@@ -842,7 +845,7 @@ public:
   void determine_line_limits (char code, hvpair *point, int npoints);
   void check_charinfo(tfont *tf, charinfo *ci);
   void stroke_color(color *c);
-  void fill_color(color *c);
+  void fill_color(color * /* c */, bool /* forcing */ = false);
   int get_hpos() { return hpos; }
   int get_vpos() { return vpos; }
   void add_to_tag_list(string s);
@@ -885,16 +888,13 @@ inline void troff_output_file::put(unsigned int i)
   put_string(ui_to_a(i), fp);
 }

-void troff_output_file::start_device_extension(color *fcol,
-                                              bool omit_command_prefix)
+void troff_output_file::start_device_extension(bool omit_command_prefix)
 {
   flush_tbuf();
 #if 0
   set_font(tf);
   stroke_color(gcol);
-#endif
-  fill_color(fcol);
-#if 0
+  fill_color(current_fill_color);
   do_motion();
 #endif
   if (!omit_command_prefix)
@@ -1266,10 +1266,19 @@ void troff_output_file::set_font(tfont *tf)

 // fill_color calls 'flush_tbuf' and 'do_motion' if necessary.

-void troff_output_file::fill_color(color *col)
+void troff_output_file::fill_color(color *col, bool forcing)
 {
-  if ((0 /* nullptr */ == col) || current_fill_color == col)
+  debug("GBR: troff_output_file::fill_color(ptr, %1)", forcing);
+  if (0 /* nullptr */ == col)
     return;
+  debug("GBR: color is not nullptr");
+  if (current_fill_color == col) {
+    debug("GBR: ...redundantly");
+    if (forcing)
+      debug("GBR: ...but FORCING IT");
+    else
+      return;
+  }
   current_fill_color = col;
   if (!want_color_output)
     return;
@@ -1633,10 +1642,22 @@ troff_output_file::troff_output_file()

 void troff_output_file::flush()
 {
+  debug("GBR: troff_output_file::flush()");
   flush_tbuf();
   real_output_file::flush();
 }

+void troff_output_file::flush(bool force_fill_color)
+{
+  debug("GBR: troff_output_file::flush(%1)", force_fill_color);
+  flush_tbuf();
+  if (force_fill_color) {
+    fill_color(current_fill_color, true /* forcing */);
+    put("# GBR WAS HERE\n");
+  }
+  real_output_file::flush();
+}
+
 /* output_file */

 output_file *the_output = 0 /* nullptr */;
@@ -3970,7 +3991,7 @@ node *device_extension_node::copy()

 void device_extension_node::tprint_start(troff_output_file *out)
 {
-  out->start_device_extension(fcol, lacks_command_prefix);
+  out->start_device_extension(lacks_command_prefix);
 }

 void device_extension_node::tprint_char(troff_output_file *out,
diff --git a/src/roff/troff/node.h b/src/roff/troff/node.h
index 078bf37bf..62bfa80a8 100644
--- a/src/roff/troff/node.h
+++ b/src/roff/troff/node.h
@@ -542,6 +542,7 @@ public:

 class device_extension_node : public node {
   macro mac;
+  // Should a device extension node _have_ the next 3 properties?  --GBR
   tfont *tf;
   color *gcol;
   color *fcol;
@@ -564,6 +565,8 @@ public:
   bool is_tag();
   int ends_sentence();
   tfont *get_tfont();
+  void mark_fcol_dirty();
+  // TODO: mark_gcol_dirty, mark_tf_dirty, mark_pos_dirty
 };

 class suppress_node : public node {
@@ -667,7 +670,8 @@ public:
   bool is_dying;
   virtual ~output_file();
   virtual void trailer(vunits);
-  virtual void flush() = 0;
+  void flush() {};
+  void flush(bool) {};
   virtual void transparent_char(unsigned char) = 0;
   virtual void print_line(hunits x, vunits y, node *n,
                          vunits before, vunits after,


G. Branden Robinson <gbranden>
Group administrator
Mon 09 Sep 2024 01:30:56 AM UTC, comment #3: 

At 2024-09-08T20:45:53-0400, Deri James wrote:

> Follow-up Comment #2, bug #66187 (group groff):
>
> You may remember a detailed breakdown I gave you, a few years ago, on
> the differences between .device and .output (\!).


Only vaguely.  :(  That would have been a good thing to get into the
manual.  I don't think I understood it well enough at the time to
explain it in my own words.  Even now I feel myself at the ragged edge
of my comprehension.  Only once I can get the formatter to dance to
whatever tune I call with no surprises do I achieve a sense of mastery.

> The main difference
> is shown by this:-
>
> Here is some text
> .device Mark here
> Marked
> .device To here
> .output Where does this go?
> Ending with this.
>
> Run with -Z and notice that .device follows the text flow, so I can
> bracket the word Marked" with device controls, whereas .output does
> what it says on the tin (Emit string directly to the gtroff
> intermediate output (subject to copy mode interpretation)).


Right.  I mean to preserve that property of `.output` and `\!`.

> Any text being built up on the current line is not flushed so the
> .output string appears before partial line is flushed.


Well, so far the changes in my working copy have
"device_extension_node"s (renamed from "special_node"s) no longer
unconditionally calling `do_motion()` when they get "tprint"ed.

I took your example and ran it through groff -T utf8 -Z with 1.23.0 and
my working copy.


$ cat /tmp/branden/deri-device.groff
Here is some text
.device Mark here
Marked
.device To here
.output Where does this go?
Ending with this.
$ diff -u deri-device-*
--- deri-device-1.23.0.grout    2024-09-08 19:59:18.705999501 -0500
+++ deri-device-GBR-wc.grout    2024-09-08 19:59:33.437928813 -0500
@@ -18,14 +18,14 @@
 wh24
 ttext
 wh24
+x X Mark here
 V40
 H432
-x X Mark here
 tMarked
 wh24
+x X To here
 V40
 H600
-x X To here
 tEnding
 wh24
 twith


The placement of the output commands generated by `device` shift
relative to some absolute positioning commands.  Does this make a
difference in actual output?


$ ~/groff-stable/bin/groff -T utf8 /tmp/branden/deri-device.groff > deri-device-1.23.0.out
grotty:<standard input>:5: warning: unrecognized command 'W'
grotty:<standard input>:24: error: X command without 'tty:' tag ignored
grotty:<standard input>:29: error: X command without 'tty:' tag ignored
$ tgu /tmp/branden/deri-device.groff > deri-device-GBR-wc.out
grotty:<standard input>:5: warning: unrecognized command 'W'
grotty:<standard input>:22: error: X command with 'Mark' tag ignored; expected 'tty'
grotty:<standard input>:27: error: X command with 'To' tag ignored; expected 'tty'
$ cmp deri-device-*.out && echo SAME
SAME


Does this offer any reassurance?  Could we use an example (that we can
then turn into an automated test) that will illustrate how my change is
breaking something?

> This is why the markstart/markend instructions use .device. Both
> methods are useful in different situations, but with suitable control
> of the various flushing methods it should be possible to retain the
> current behaviour just using .device.


That's my hope.  I'm not trying to reduce the author's control of the
page contents.

> I do have a slight worry in that I know to you this is incorrect
> behaviour,


I don't see anything wrong with the exhibit above as rendered by 1.23.0
or my working copy.  A pair of absolute motion commands (that I consider
somewhat noisy, but not incorrect) shifts in location, but not in a way
that alters rendering.  Rendering is what decides correctness to me.

> but to everyone else who has relied on this behaviour, it is a feature
> change.


I need to see some behavior changing in order to agree with you.

If I see that, then I agree, I've gotta raise the flag in the NEWS file.
Or abandon the change.

I was specifically concerned that killing the `do_motion()` calls would
screw up OSC 8 bookmark boundaries, but that didn't happen.  The change
I needed to make to expectations in tmac/tests/an_MR-works.sh was
minimal (3 output commands moved by 2 lines), and firing up
gnome-terminal revealed hyperlinked man page references cruising along
just like before.  No boundary changes.

> We can easily change our macros to avoid a regression, but we have
> given no warning of this big change in behaviour,


I wonder what tool you use to measure "big"--I get the feeling the
number of people who write output drivers for groff is extremely small.
Even the number who write macros that produce device extension commands
is not much larger.

Get used to the idea that you're in an elite class.

> so people may find their macros are no longer working the same.


I agree that behavior changes should be documented!  But judging by the
reception to 1.23.0, there is a paucity of people outside the groff
mailing list itself who will notice anything that doesn't affect the
rendering of a man page in Teletype Model 37-compatible form.

That is not a situation I celebrate.

Regards,
Branden

G. Branden Robinson <gbranden>
Group administrator
Mon 09 Sep 2024 12:45:50 AM UTC, comment #2: 

You may remember a detailed breakdown I gave you, a few years ago, on the differences between .device and .output (\!). The main difference is shown by this:-

Here is some text
.device Mark here
Marked
.device To here
.output Where does this go?
Ending with this.

Run with -Z and notice that .device follows the text flow, so I can bracket the word Marked" with device controls, whereas .output does what it says on the tin (Emit string directly to the gtroff intermediate output (subject to copy mode interpre-
tation)). Any text being built up on the current line is not flushed so the .output string appears before partial line is flushed.

This is why the markstart/markend instructions use .device. Both methods are useful in different situations, but with suitable control of the various flushing methods it should be possible to retain the current behaviour just using .device. I do have a slight worry in that I know to you this is incorrect behaviour, but to everyone else who has relied on this behaviour, it is a feature change. We can easily change our macros to avoid a regression, but we have given no warning of this big change in behaviour, so people may find their macros are no longer working the same.

Deri James <deri>
Group Member
Sun 08 Sep 2024 11:52:37 PM UTC, comment #1: 

Experimentation reveals that I can go this far with disabling the mandatory state changes in these functions in "node.cpp" without triggering any regressions.



diff --git a/src/roff/troff/node.cpp b/src/roff/troff/node.cpp
index 0a7b8bffe..4082a3eef 100644
--- a/src/roff/troff/node.cpp
+++ b/src/roff/troff/node.cpp
@@ -902,7 +902,6 @@ void troff_output_file::start_device_extension(tfont *tf, color *gcol,
 void troff_output_file::start_device_extension()
 {
   flush_tbuf();
-  do_motion();
   put("x X ");
 }


And then...


diff --git a/src/roff/troff/node.cpp b/src/roff/troff/node.cpp
index 4082a3eef..e61a075bc 100644
--- a/src/roff/troff/node.cpp
+++ b/src/roff/troff/node.cpp
@@ -825,8 +825,7 @@ public:
   void right(hunits);
   void down(vunits);
   void moveto(hunits, vunits);
-  void start_device_extension(tfont * /* tf */,
-                             color * /* gcol */, color * /* fcol */,
+  void start_device_extension(color * /* fcol */,
                              bool /* omit_command_prefix */ = false);
   void start_device_extension();
   void write_device_extension_char(unsigned char c);
@@ -886,15 +885,18 @@ inline void troff_output_file::put(unsigned int i)
   put_string(ui_to_a(i), fp);
 }

-void troff_output_file::start_device_extension(tfont *tf, color *gcol,
-                                              color *fcol,
+void troff_output_file::start_device_extension(color *fcol,
                                               bool omit_command_prefix)
 {
+  flush_tbuf();
+#if 0
   set_font(tf);
   stroke_color(gcol);
+#endif
   fill_color(fcol);
-  flush_tbuf();
+#if 0
   do_motion();
+#endif
   if (!omit_command_prefix)
     put("x X ");
 }
@@ -3957,7 +3959,7 @@ node *device_extension_node::copy()

 void device_extension_node::tprint_start(troff_output_file *out)
 {
-  out->start_device_extension(tf, gcol, fcol, lacks_command_prefix);
+  out->start_device_extension(fcol, lacks_command_prefix);
 }

 void device_extension_node::tprint_char(troff_output_file *out,


Reading carefully, we see that only `fill_color()` is left active.  If I disable that, too, the page background in "msboxes.pdf" turns black as Deri and I discussed in bug #64484.

So, minimally, we just need `fl` to accept an argument that causes the fill color command to be reƫmitted.

And then, I think, change the `pdfbackground` macro to use such an argumentful `fl` request.

G. Branden Robinson <gbranden>
Group administrator
Sun 08 Sep 2024 05:16:45 AM UTC, original submission:  

As recently discussed on the mailing list, when troff features that wrap device extension commands (meaning: the `\X` escape sequence and `device` request) run, they end up calling the following function.


void troff_output_file::start_special(tfont *tf, color *gcol,
                                      color *fcol,
                                      bool omit_command_prefix)
{
  set_font(tf);
  stroke_color(gcol);
  fill_color(fcol);
  flush_tbuf();
  do_motion();
  if (!omit_command_prefix)
    put("x X ");
}


This function does a lot.

It updates the state of the selected font, stroke and fill colors, and drawing position.  These all map to device-independent output commands ('f', 'm', 'D F', and 'H' and 'V', respectively).

A device extension command may influence none, some, or all of these.

Hence all the function arguments, right?

Well, no.  There's no means in the groff language (nor of Kernighan troff before it) of expressing which of these should be updated before or after a device extension command.

So, basically, this function resets everything.

For one thing, that's inelegant, and leads to pointless grout commands appearing in the output stream.

For another, and much worse, this means you can't, from the groff language, tell the formatter that your device extension command has, say, done something that has "dirtied" the state of the stroke and/or fill colors, and that these should be restored, say after the device extension command (not before).

So what I'd like to do is extend the little-used `fl` request to accept arguments (CSTR #54 `fl` accepts none).

These arguments would encode each of the foregoing data:

  • font selection
  • stroke color selection
  • fill color selection
  • horizontal drawing position
  • vertical drawing position


Consequently I propose a new idiom for use of `\X` and `device`: bracket these formatting directives with `fl` requests if and as necessary.

If a device extension command does nothing to affect the state of page output operations--whimsically we might imagine causing a message like "HELLO, DAVE" to appear on a printer's LCD display--then just like now it need not invoke any `fl` requests.

Importantly, none of these arguments is parameterized.  The idea is not to replace formatter operations but to ensure synchrony between formatter state, representation of that state in device-independent output, and the state of the device.

So when you "flush" any of these data, the parameter written to the device-independent output is what the formatter thinks it is at that moment.  If your device extension command really does do something like change the stroke color, you'll need to encode that in your input alongside the `\X` or `device` instruction.

I'll be updating this ticket with a demonstrator involving Deri's "sboxes" package, which is what brought this issue to my attention.

G. Branden Robinson <gbranden>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by deri (Posted a comment)
  • -email is unavailable- added by gbranden (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-10-17 gbranden Assigned togbranden None
    2024-09-27 gbranden StatusIn Progress Postponed
    2024-09-11 gbranden StatusNone In Progress
        Assigned toNone gbranden
    2024-09-08 gbranden Dependencies- bugs #63074 is dependent

    Back to the top

    Powered by Savane 3.14-3b9d.
    Corresponding source code