bugGNU troff - Bugs: bug #63332, recent fallbacks.tmac change...

 
 

bug #63332: recent fallbacks.tmac change degrades ASCII output

Submitter:  Dave <barx>
Submitted:  Mon 07 Nov 2022 03:53:09 PM UTC
 
Category:  Macro - others/general Severity:  2 - Minor
Item Group:  Rendering/Cosmetics Status:  Postponed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       No canned response available

 

( Jump to the original submission )

Sun 11 Dec 2022 10:51:12 PM UTC, comment #11: 

With commit 230ad718 providing a solution to the ".if t" problem raised in comment #4, this bug's mitigating commit (comment #9) should now be able to be selectively applied to only terminal devices.  That is, lines that that commit commented out should be able to be placed behind ".if \n[fallbacks*troff-mode]" tests instead.

Dave <barx>
Project Member
Thu 01 Dec 2022 09:20:39 AM UTC, comment #10: 

Replying to the rest of comment #5, as promised.  Some of it may be moot; I'm not sure how far the miscommunication extended that was resolved in comment #8, so I'll try not to belabor points that may no longer be relevant.  Feel free to ask for clarification if I skim over anything too much.

> So here's my new patch.


Now applied, so we're good here.

> I'm dubious about tty-char.tmac's definition of \(rn.


Looking at it now, I agree with this--but it's also been unchanged for over 20 years.  (Maybe readers just have low expectations for equations rendered on the terminal.)

> Well, character definitions aren't coupled in any way, as far
> as I know, to the "troff/nroff mode" bit in the formatter.


Right, not by default, but I was talking about the explicit coupling provided by the definitions sequestered behind tests for that mode.

> > I presume this is because tmac/troffrc loads fallbacks.tmac
> > before loading the device-specific .tmac file.
>
> Right.  The device description file doesn't say whether it's a
> "troff" or "nroff" device; instead a request does that.


Understood; my point is that this request is invoked after fallbacks.tmac is processed, mooting those aforementioned tests.

> Post-1.23 it might be worth looking into loading fallbacks.tmac
> later, and maybe combining it with tty-char.tmac in some way.


OK.  But for 1.23 itself, without further mitigation, a few characters will render badly on terminals (e.g., as in comment #7).

Then again, their 1.22.4 rendering was a warning message and a complete omission of the characters, so this is, if not a step forward, at least not one backward either.

> But more important is understanding how `character_exists()`
> really works, or I predict only more frustration.


Yes, and this is fine to defer.

> It seems even baby steps toward Unicode are fraught with peril.


Babies do teeter and fall a lot.

Dave <barx>
Project Member
Wed 30 Nov 2022 12:30:59 PM UTC, comment #9: 

Parking this issue until after the groff 1.23.0 release as it is now "mitigated".

commit 563fd71746331f5feafe87a468d0326e8bfc8786
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sun Nov 27 03:30:38 2022 -0600

    tmac/fallbacks.tmac: Mitigate Savannah #63332.

    * tmac/fallbacks.tmac: Comment out four of the fallback character
      definitions for (preconv'd) Unicode character input added to address
      Savannah #58930 (corresponding to \[dg], \[dd], \[%0], and \[rn]; they
      are failing for not yet understood reasons involving "macros" attached
      to groff character info structs, and the semantics of what it means
      for a character to "exist" per the 'c' conditional operator.  Fixes
      regression when using "tty-char.tmac" (which nroff loads by default).
      Problem introduced by me in commit 132182bd71, 23 October.  The hope
      is to resolve this issue post-groff 1.23.0.  Thanks to Dave Kemper for
      the report and discussion in Savannah #63332.

G. Branden Robinson <gbranden>
Project Administrator
Mon 28 Nov 2022 04:52:57 AM UTC, comment #8: 

Hi Dave,

comment #7:

> I'll reply more fully later, but I want to address one point now:
>
> comment #5:
> > > This means the other "t" conditionals in fallbacks.tmac also
> > > don't do what was intended.
> >
> > I don't think that's true.
>
> It's easy enough to demonstrate: pick a definition where     > Weirdly, even adding the line ".if t .tm troff mode" to that file prints "troff mode" on stderr when groff is run with -Tascii.  I presume this is because tmac/troffrc loads fallbacks.tmac before loading the device-specific .tmac file.


Right.  The device description file doesn't say whether it's a "troff" or "nroff" device; instead a request does that.

    > This means the other "t" conditionals in fallbacks.tmac also don't do what was intended.

fallbacks.tmac makes a distinction based on the n/t mode, such as:

> .ie t .fchar \[u2012] \v'-.3m'\l'\w"\0"u\[ru]'\v'+.3m'\" figure dash
> .el   .fchar \[u2012] \-

> $ echo '3\[u2012]4' | groff -Tascii | cat -s
> 3_4
>

> Were this working as designed, we'd see a hyphen in the output rather than an underscore.


Ah, I see what you mean.  We were both saying true things but somehow I managed to infer an incorrect one from you.  On re-reading, I'm not sure how.

> > > Weirdly, even adding the line ".if t .tm troff mode" to that file prints "troff mode" on stderr when groff is run with -Tascii.  I presume this is because tmac/troffrc loads fallbacks.tmac before loading the device-specific .tmac file.


> > Right.  The device description file doesn't say whether it's a "troff" or "nroff" device; instead a request does that.


> > > This means the other "t" conditionals in fallbacks.tmac also don't do what was intended.


I concur.

G. Branden Robinson <gbranden>
Project Administrator
Sun 27 Nov 2022 10:38:41 AM UTC, comment #7: 

I'll reply more fully later, but I want to address one point now:

comment #5:

> > This means the other "t" conditionals in fallbacks.tmac also
> > don't do what was intended.
>
> I don't think that's true.


It's easy enough to demonstrate: pick a definition where fallbacks.tmac makes a distinction based on the n/t mode, such as:

.ie t .fchar \[u2012] \v'-.3m'\l'\w"\0"u\[ru]'\v'+.3m'\" figure dash
.el   .fchar \[u2012] \-

$ echo '3\[u2012]4' | groff -Tascii | cat -s
3_4

Were this working as designed, we'd see a hyphen in the output rather than an underscore.

Dave <barx>
Project Member
Sun 27 Nov 2022 09:17:48 AM UTC, comment #6: 

To answer my own question, yes there is a terminfo capability for that.

   Basic Capabilities
       ...  If the terminal overstrikes (rather than
       clearing a position when a character is struck over) then it
       should have the os capability.  If the terminal is a printing
       terminal, with no soft copy unit, give it both hc and os.  (os
       applies to storage scope terminals, such as TEKTRONIX 4010
       series, as well as hard copy and APL terminals.)  ...

Mooha.  Muahaha.  Muahahahahahahaha.

This is the tasty cookie that awaits me when I can finally get 1.23.0 out...

G. Branden Robinson <gbranden>
Project Administrator
Sun 27 Nov 2022 09:10:19 AM UTC, comment #5: 

comment #4:

> I confirm that your patch solves the \(dd and \(dg problem.


Cool, thanks!

I'm going to rearrange the discussion to front-load the more immediate concerns, then return to our drawing-room mystery later in this reply.

> comment #3:
> > None of this explains why most of the other new fallbacks
> > don't have the same problem.
>
> I didn't actually test most of the others, but maybe you have.  It sounds like you tested at least \(bu.


Yeah, and a few others, but I wasn't comprehensive and I now see I needed to be.

The per mille sign and root en extender are also trouble.  (And also both not in ISO Latin-1--so that hypothesis is not yet overturned.  But it's weakening: neither the em and en dashes nor the directional quotation marks are.)

$ cat EXPERIMENTS/fallbacks.groff
.ta 1i
.nf
en      \(en
em      \(em
oq      \(oq
cq      \(cq
lq      \(lq
rq      \(rq
dg      \(dg
dd      \(dd
bu      \(bu
pc      \(pc
%0      \(%0
fm      \(fm
sd      \(sd
fo      \(fo
fc      \(fc
rn      \(rn
f/      \(f/
ti      \(ti
$ ./build/test-groff -Tascii -mtty-char EXPERIMENTS/fallbacks.groff | cat -s
troff:EXPERIMENTS/fallbacks.groff:13: warning: special character '%0' not defined
troff:EXPERIMENTS/fallbacks.groff:18: warning: special character 'rn' not defined
en        -
em        --
oq        `
cq        '
lq        "
rq        "
dg        <*>
dd        <**>
bu        o
pc        .
%0
fm        '
sd        "
fo        <
fc        >
rn
f/        /
ti        ~

So here's my new patch.

diff --git a/tmac/fallbacks.tmac b/tmac/fallbacks.tmac
index 633dbc711..9ba22c0f5 100644
--- a/tmac/fallbacks.tmac
+++ b/tmac/fallbacks.tmac
@@ -167,20 +167,21 @@
 .fchar \[u2019] \[cq]\" right single quotation mark
 .fchar \[u201C] \[lq]\" left double quotation mark
 .fchar \[u201D] \[rq]\" right double quotation mark
-.fchar \[u2020] \[dg]\" dagger
-.fchar \[u2021] \[dd]\" double dagger
+.\" XXX: The next two are troublesome; see Savannah #63332.
+.\"fchar \[u2020] \[dg]\" dagger
+.\"fchar \[u2021] \[dd]\" double dagger
 .fchar \[u2022] \[bu]\" bullet
 .fchar \[u2024] .\" one dot leader
 .fchar \[u2025] .\|.\" two dot leader
 .fchar \[u2026] .\|.\|.\" horizontal ellipsis
 .fchar \[u2027] \[pc]\" hyphenation point
-.fchar \[u2030] \[%0]\" per mille sign
+.\"fchar \[u2030] \[%0]\" per mille sign \" Savannah #63332 again
 .fchar \[u2032] \[fm]\" prime
 .fchar \[u2033] \[sd]\" double prime
 .fchar \[u2039] \[fo]\" left single chevron
 .fchar \[u203A] \[fc]\" right single chevron
 .if t .fchar \[u203D] \o'?!'\" interrobang
-.fchar \[u203E] \[rn]\" overline
+.\"fchar \[u203E] \[rn]\" overline \" Savannah #63332 again
 .fchar \[u2044] \[f/]\" fraction slash
 .fchar \[u2052] %\" commercial minus sign
 .fchar \[u2053] \[ti]\" swung dash

I'm dubious about tty-char.tmac's definition of \(rn.  It would be fine for overstriking terminals, none of which has a device definition in groff.  I wonder if there's a terminfo capability for that?

> > So if that patch is enough to fix the regression, I'd like to
> > apply it, un-tag this for 1.23.0, and move on to other things,
> > returning to it when I have more time and understand
> > character/glyph resolution better.
>
> Sounds good to me.


Cool.  I'll go ahead with the above.

> I was convinced that a more surgical ".if t" on those lines would also do the trick--but it doesn't.


Well, character definitions aren't coupled in any way, as far as I know, to the "troff/nroff mode" bit in the formatter.  So it doesn't surprise me (unlike Paul Eggert's claims today about the braindeadness of Solaris's troff, which shocked even my low expectations of that implementation).

> Weirdly, even adding the line ".if t .tm troff mode" to that file prints "troff mode" on stderr when groff is run with -Tascii.  I presume this is because tmac/troffrc loads fallbacks.tmac before loading the device-specific .tmac file.


Right.  The device description file doesn't say whether it's a "troff" or "nroff" device; instead a request does that.

> This means the other "t" conditionals in fallbacks.tmac also don't do what was intended.


I don't think that's true.  There is no alternative to "troff mode" and "nroff mode", you're either in one or the other, and the default is troff mode.  Observe:

$ echo ".if t .tm troff mode" | troff -R
troff mode
$ echo ".if t .tm troff mode" | troff -R -Tascii
troff mode

Still, I wonder about the timing of composite and fallback character setup.  Post-1.23 it might be worth looking into loading fallbacks.tmac later, and maybe combining it with tty-char.tmac in some way.

But more important is understanding how `character_exists()` really works, or I predict only more frustration.

One data point did pop up when I turned on backtraces (-b).

troff: backtrace: '.../groff/build/../tmac/fallbacks.tmac':178: composite character '%0'
troff: backtrace: file 'EXPERIMENTS/fallbacks.groff':13
troff:EXPERIMENTS/fallbacks.groff:13: warning: special character '%0' not defined
troff: backtrace: '.../groff/build/../tmac/fallbacks.tmac':184: composite character 'rn'
troff: backtrace: file 'EXPERIMENTS/fallbacks.groff':18
troff:EXPERIMENTS/fallbacks.groff:18: warning: special character 'rn' not defined

Oh, "composite character", eh?  Is that what the formatter thinks that is (like \[e aa])?

My nose twitches.  But I've resolved to chase that scent later.

It seems even baby steps toward Unicode are fraught with peril.

G. Branden Robinson <gbranden>
Project Administrator
Sun 27 Nov 2022 02:39:13 AM UTC, comment #4: 

I confirm that your patch solves the \(dd and \(dg problem.

I was convinced that a more surgical ".if t" on those lines would also do the trick--but it doesn't.

Weirdly, even adding the line ".if t .tm troff mode" to that file prints "troff mode" on stderr when groff is run with -Tascii.  I presume this is because tmac/troffrc loads fallbacks.tmac before loading the device-specific .tmac file.  This means the other "t" conditionals in fallbacks.tmac also don't do what was intended.

comment #3:

> None of this explains why most of the other new fallbacks
> don't have the same problem.


I didn't actually test most of the others, but maybe you have.  It sounds like you tested at least \(bu.

> So if that patch is enough to fix the regression, I'd like to
> apply it, un-tag this for 1.23.0, and move on to other things,
> returning to it when I have more time and understand
> character/glyph resolution better.


Sounds good to me.

Dave <barx>
Project Member
Sun 27 Nov 2022 12:00:44 AM UTC, comment #3: 

Hi Dave,

This one is challenging my understanding, and I am anxious about it causing further slippage of the already monumentally slipped 1.23.0.rc2.

First, I have a proposal for knocking this off the 1.23.0 criticality path.  It seems to un-degrade the \(dd and \(dg characters.

diff --git a/tmac/fallbacks.tmac b/tmac/fallbacks.tmac
index 633dbc711..39a2e8072 100644
--- a/tmac/fallbacks.tmac
+++ b/tmac/fallbacks.tmac
@@ -167,8 +167,9 @@
 .fchar \[u2019] \[cq]\" right single quotation mark
 .fchar \[u201C] \[lq]\" left double quotation mark
 .fchar \[u201D] \[rq]\" right double quotation mark
-.fchar \[u2020] \[dg]\" dagger
-.fchar \[u2021] \[dd]\" double dagger
+.\" XXX: The next two are troublesome; see Savannah #63332.
+.\"fchar \[u2020] \[dg]\" dagger
+.\"fchar \[u2021] \[dd]\" double dagger
 .fchar \[u2022] \[bu]\" bullet
 .fchar \[u2024] .\" one dot leader
 .fchar \[u2025] .\|.\" two dot leader

Please confirm on your installation.

The terrible news is that I don't know why this works.

Here's a brief brain dump for when I can get back to this.

  • node.cpp:character_exists() returns true (early) if ci->get_macro() returns a non-null pointer.  I think this means that it's been defined with a `char` or related request.  The RHS is termed a "macro" internally even if that's a bit confusing for ordinary users.  I think our Texinfo manual also has some casual language about character definitions being "wrapped up in a macro".
  • I think the above is why we're getting false positives on our .if c tests.
  • This is still a valid thing to want to do.
  • However maybe we need a new ".if g" conditional operator that really does check to see if a glyph is available in the current font.  An honest-to-God "can you draw this glyph?" query.
  • None of this explains why most of the other new fallbacks don't have the same problem.  Like \(bu.  The best guess I have for now is that because the bullet is in ISO 8859-1 (the daggers aren't), some special logic path is followed that doesn't hose things up.

So if that patch is enough to fix the regression, I'd like to apply it, un-tag this for 1.23.0, and move on to other things, returning to it when I have more time and understand character/glyph resolution better.  Even according to our existing docs it's something like a 7-step procedure.

G. Branden Robinson <gbranden>
Project Administrator
Fri 25 Nov 2022 09:35:56 PM UTC, comment #2: 

comment #1:

> This seems like a bug in the formatter.


I agree.  It is definitely a deviation from how Heirloom troff processes the same input.

Dave <barx>
Project Member
Fri 25 Nov 2022 08:53:54 PM UTC, comment #1: 

It appears that having a special character on the RHS of an `fchar` request marks it as defined for the purpose of the `c` conditional expression, even though it's not really defined.

This seems like a bug in the formatter.  I ran it with startup files disabled to illustrate this.

Evidence:

$ cat EXPERIMENTS/char.groff
.fchar \[u2020] \[dg]\" dagger
.if c \[u2020] .tm u2020 is defined (1)
.if c \[dg] .tm dg is defined (1)
.\"char \[dg] <*>
.if c \[u2020] .tm u2020 is defined (2)
.if c \[dg] .tm dg is defined (2)
.nf
u2020: \[u2020]
dg: \[dg]

$ ./build/troff -Tascii -F build/font -F font -R EXPERIMENTS/char.groff
u2020 is defined (1)
dg is defined (1)
u2020 is defined (2)
dg is defined (2)
x T ascii
x res 240 24 40
x init
p1
./build/troff:EXPERIMENTS/char.groff:8: warning: special character 'dg' not defined
x font 1 R
f1
s10
V40
H240
md
DFd
tu2020:
wh24
n40 0
V80
H240
tdg:
wh24
n40 0
x trailer
V2640
x stop

G. Branden Robinson <gbranden>
Project Administrator
Mon 07 Nov 2022 03:53:09 PM UTC, original submission:  

The changes that commit 132182bd made to tmac/fallbacks.tmac degrade ASCII output in a way I can't explain.

First I run a version of groff without this commit:

$ echo 'I \[dg] you' | groff -Tascii -mtty-char | cat -s
I <*> you

This is the expected output: tmac/tty-char.tmac defines \[dg] as output above.

But applying the commit 132182bd7 changes to fallbacks.tmac makes this go awry:

$ echo 'I \[dg] you' | groff -Tascii -mtty-char | cat -s
troff:<standard input>:1: warning: special character 'dg' not defined
I  you

I don't understand this, because while the updated fallbacks.tmac defines a new character as \[dg], it does (or should do) nothing to change the definition of \[dg] itself, or to prevent \[dg] from being subsequently defined.  The same is true of \[dd], which shows the same effect.

(This ticket reproduces comment 27 of the closed bug #58930.)

Dave <barx>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Updated the item)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-11-30 gbranden StatusNone Postponed
    2022-11-30 gbranden StatusIn Progress None
        Planned Release1.23.0 None
    2022-11-27 gbranden StatusNeed Info In Progress
    2022-11-27 gbranden StatusConfirmed Need Info
    2022-11-25 gbranden StatusNone Confirmed
    2022-11-07 gbranden Assigned toNone gbranden
        Planned ReleaseNone 1.23.0

    Back to the top

    Powered by Savane 3.9