bugGNU roff - Bugs: bug #63354, [tmac]: refine fallbacks.tmac

 
 

bug #63354: [tmac]: refine fallbacks.tmac

Submitter:  Dave <barx>
Submitted:  Sat 12 Nov 2022 11:06:51 PM UTC
   
 
Category:  Macro package - others/general Severity:  2 - Minor
Item Group:  Rendering/Cosmetics Status:  Need Info
Privacy:  Public Assigned to:  barx
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 04 Aug 2024 08:31:07 AM UTC, comment #44: 

comment #43:

> I evidently put more effort into my analogy than my basic message.


I like seeing someone with their priorities in order.

> Can you catch me up on what we've agreed upon in this 40+ message sprawl?


Certainly.  I gave myself an action plan in comment #28, then postponed working on it until the condition of comment #31 was met.  Once it is, I'll wade through all this bug's mostly notes-to-self and turn them into a patch to fallbacks.tmac.

I could have ignored the self-imposed comment #31 condition and blazed on ahead anyway, but in the subsequent dozen-odd comments, Bjarni and I have honed some of the details of this bug's pending patch, so I consider the holding time well spent.

Dave <barx>
Group Member
Sun 04 Aug 2024 06:59:16 AM UTC, comment #43: 


comment #41:

> comment #40:
> > I suggest that it doesn't.
>
> Hence "arguably."
>
> > That caveat seems to hoist on us on the very petard that
> > separates the marauding `\:` Turk from the `\&` ground soaked
> > by the bloodthirsty Romanians.
>
> I don't think a case can be made for \&.  That doesn't permit a break, and I hope if we can agree on nothing else, we can agree that a character named "BREAK PERMITTED HERE" ought to permit a break.


Sorry, I did in fact mean `\%` rather than `\&`.  I evidently put more effort into my analogy than my basic message.

>
> So the only reasonable choices are \: or \%--each permitting a break, one adding a hyphen if a break occurs there, one not.
>
> The Unicode standard v15.1 is light--to the point of malfeasance--on the meanings of most characters in the C1 control block.  (http://www.unicode.org/charts/PDF/U0080.pdf)  The Unicoderati, it seems, would rather you avoid these legacy code points altogether and use their newer, more well-defined counterparts.  And that's the advice I'd give to anyone composing new Unicode text.
>
> But what of legacy documents?  This is an area where groff ought to shine.  And what the Unicode standard is furtive about, we can deduce from the content of the overall C1 Controls and Latin-1 Supplement block.
>
> Specifically, this block contains the character U+00AD SOFT HYPHEN, the function of which Unicode is more forthcoming about: "an invisible format character indicating a possible hyphenation location".  This, in roffese, is \%.
>
> So why would another character in the same block also translate to \%?  It seems clear the original intent of U+0082 BREAK PERMITTED HERE is to define a potential break point that doesn't get hyphenated--the \:, in roffese.


Okay.  I'm persuaded well enough on this point.

Can you to catch me up on what we've agreed upon in this 40+ message sprawl?

G. Branden Robinson <gbranden>
Group administrator
Sun 04 Aug 2024 05:27:39 AM UTC, comment #42: 

Standard ECMA-48 (http://www.ecma-international.org/wp-content/uploads/ECMA-48_5th_edition_june_1991.pdf) has more verbiage but is no more explicit about this.  Its description of BPH: BREAK PERMITTED HERE (see section 8.3.4) is, "BPH is used to indicate a point where a line break may occur when text is formatted. BPH may occur between two graphic characters, either or both of which may be SPACE."

Setting aside the quibble about whether a space qualifies as a "graphic character" (I suppose it does in the worldview of this document, where the other type is a control character), I'd say this answers by inference what it doesn't answer explicitly.  No one in their right mind would add a hyphen when breaking two spaces.  The clear, if unstated, intent of BPH is to specify a possible break point that doesn't get hyphenated.

Dave <barx>
Group Member
Sun 04 Aug 2024 05:07:35 AM UTC, comment #41: 

comment #40:

> I suggest that it doesn't.


Hence "arguably."

> That caveat seems to hoist on us on the very petard that
> separates the marauding `\:` Turk from the `\&` ground soaked
> by the bloodthirsty Romanians.


I don't think a case can be made for \&.  That doesn't permit a break, and I hope if we can agree on nothing else, we can agree that a character named "BREAK PERMITTED HERE" ought to permit a break.

So the only reasonable choices are \: or \%--each permitting a break, one adding a hyphen if a break occurs there, one not.

The Unicode standard v15.1 is light--to the point of malfeasance--on the meanings of most characters in the C1 control block.  (http://www.unicode.org/charts/PDF/U0080.pdf)  The Unicoderati, it seems, would rather you avoid these legacy code points altogether and use their newer, more well-defined counterparts.  And that's the advice I'd give to anyone composing new Unicode text.

But what of legacy documents?  This is an area where groff ought to shine.  And what the Unicode standard is furtive about, we can deduce from the content of the overall C1 Controls and Latin-1 Supplement block.

Specifically, this block contains the character U+00AD SOFT HYPHEN, the function of which Unicode is more forthcoming about: "an invisible format character indicating a possible hyphenation location".  This, in roffese, is \%.

So why would another character in the same block also translate to \%?  It seems clear the original intent of U+0082 BREAK PERMITTED HERE is to define a potential break point that doesn't get hyphenated--the \:, in roffese.

Dave <barx>
Group Member
Sat 03 Aug 2024 11:00:08 PM UTC, comment #40: 

comment #39:

> comment #5:
> > Unicode's U+200B ZERO WIDTH SPACE has the semantics of groff's \: escape.
>
> Arguably, so does U+0082 BREAK PERMITTED HERE.


I suggest that it doesn't.

Wikipedia says:

> Follows a graphic character where a line break is permitted.
> *Roughly equivalent to a soft hyphen or zero-width space except
> it does not define what is printed at the line break*.


(emphasis added)

That caveat seems to hoist on us on the very petard that separates the marauding `\:` Turk from the `\&` ground soaked by the bloodthirsty Romanians.

G. Branden Robinson <gbranden>
Group administrator
Sat 03 Aug 2024 05:16:20 PM UTC, comment #39: 

comment #5:

> Unicode's U+200B ZERO WIDTH SPACE has the semantics of groff's \: escape.


Arguably, so does U+0082 BREAK PERMITTED HERE.

Dave <barx>
Group Member
Fri 19 Apr 2024 10:09:43 AM UTC, comment #38: 

comment #36:

> I still see no way around the first,


Ha, because I'm still overthinking it.  Groff offers a shorthand for \h^\w'0'u^ -- and that is \0.  Thus the comment #36 suggestion reduces to:

.fchar \[u2012] \o'-\0'

This also eliminates the need for the conditional for nroff/troff output: this fallback works in either environment.

(It seems like it shouldn't work in nroff with the overstruck characters ordered as above: as bug #63028 readers are aware, it is the last character in an overstruck sequence that nroff outputs.  But nroff seems to make an exception for space characters:

$ echo "\o'abcd'efg" | nroff | cat -s
defg

$ echo "\o'abc\0'efg" | nroff | cat -s
cefg

$ echo "\o'abc\ 'efg" | nroff | cat -s
cefg

I'll add this to the list of things that #63028 should document.)

Dave <barx>
Group Member
Mon 15 Apr 2024 07:16:30 AM UTC, comment #37: 

(Of course, for fallback usage the .char in comment #35 and comment #36 should be .fchar.)

Dave <barx>
Group Member
Mon 15 Apr 2024 07:04:35 AM UTC, comment #36: 

Maybe comment #34 and comment #35 are overthinking this.

The requirement for U+2012 is that it occupy the same horizontal space as a figure.  There is no requirement on the width of the mark itself.

So what if the fallback simply uses the font's hyphen centered within a figure's width?

.  char \[u2012] \o\# Overstrike...
'-\#                  a hyphen and...
\h\#                  a horizontal space...
^\w'0'u^'\"           the width of a zero.

This removes all considerations of generating a horizontal line at the same height as the font's existing hyphens and dashes.  It solves the second and third of the "potential implementation issues" comment #35 raises.  (I still see no way around the first, but the current fallbacks.tmac \[u2012] already has this drawback.)

It might even be simple enough that the internal commenting is overkill:

.  \" Overstrike a hyphen and a horizontal space the width of a zero.
.  char \[u2012] \o'-\h^\w'0'u^'


Dave <barx>
Group Member
Fri 08 Mar 2024 02:30:21 AM UTC, comment #35: 

This looks great!

Here's the definition from the sample file, broken down and commented, and trivially refactored (to avoid performing the same \w twice when the value of the first is already stored in a register):

.  char \[u2012] \R#wzero \w'0'u#\#      set width of a figure, setting \n[rst]
\R#hzero \\n[rst]u/2u#\#                 set horizontal midpoint of a figure
\R#lfiguredash \\n[wzero]-(2*\w'\^'u)#\# set dash length
\^\
\v'-\\n[hzero]u'\#                       adjust horizontal position
\D'l \\n[lfiguredash]u 0m'\#             draw dash
\^\
\v'\\n[hzero]u'\"                        restore horizontal position

The glyph is exactly the right width, and the spacing between it and the surrounding figures seems ideal.

As the fourth line of output shows (the one that puts the created figure dash on the same line as stock dashes and hyphens), it's a little too high in the line.  This is true across all the font families that ship with groff; dashes and hyphens are typically set below half-height so they interact well with lowercase letters.

This is easy to adjust by tweaking the "rst / 2" calculation.  The tricky part is finding an adjustment that works in all families.  The best compromise I could find was \\n[rst]u*10/27u (groff's integer arithmetic prevents simply using a value between 2 and 3 in the denominator), which set the figure dash a bit too high for a couple families, a bit too low for a couple others, and just about right for several.  But unless the dashes are right next to each other--which shouldn't happen in real-world usage--the difference won't be noticeable.

Alternately, .fschar could be used to tailor the dash for each groff font.  But there would still need to be a generic fallback for any user-supplied fonts, and the metrics for this would still be speculative.

In most families (all stock groff families but C, P, and T), the generated dash is also skinnier than the font-supplied ones, but for a fallback definition, I propose to not worry about this level of detail.  Alternately, as above, .fschar could be used, with the same caveat.

Notably, the existing \[u2012] fallback in fallbacks.tmac already has a hard-coded height and thickness that don't exactly match every family; this proposed replacement improves on it by giving it the correct width, allowing figures on adjacent lines to align properly.

A couple of potential implementation issues, ordered from most to least important:

  • This character's internal use of \w will clobber any values of its side-effect registers (st, rst, ssc, etc.) that the user may have been relying on.  (Also true of the existing \[u2012] fallback.)
  • The three named registers (wzero, hzero, lfiguredash) will clobber any registers by those names the user is using.  (These names could be tweaked to make collisions less likely, at the minor expense of code readability.)
  • Unlike .nr, the .rr request has no corresponding escape sequence, so it's impossible to clean up the registers.


I think making available a working U+2012 is a big enough gain to offset these drawbacks, especially given the hope that over time, more fonts (perhaps someday, even those that ship with groff) will include U+2012, and this fallback will see less use.

Anyone have thoughts on any of the above?

Dave <barx>
Group Member
Tue 05 Mar 2024 11:14:34 PM UTC, comment #34: 

  The register "lfiguredash" can be incorporated into the
definition string.

  There is no extra output with a correct definition of the
character "figuredash".

  An example is in the attachment.


(file #55788)

Bjarni Ingi Gislason <bjarniig>
Sun 03 Mar 2024 11:18:08 PM UTC, comment #33: 

Bjarni's proposed \[u2012] definition in comment #26 has two problems.

  • It relies on a register (lfiguredash) that calculates the width of a zero and a hair space based on the font size at the time the register is defined.  This value will thus be incorrect at any other font size used later in the document.
  • It emits an extraneous apostrophe.  This input:


.nr lfiguredash \w'0'u-(2*\w'\^'u)
.char \[u2012] \^\v'-0.3m'\D'l \n[lfiguredash]u 0m'\v'0.3m\^'
.rm lfiguredash
.
1\[u2012]2\[u2012]3

produces the attached typeset output.


Dave <barx>
Group Member
Fri 15 Sep 2023 11:18:41 PM UTC, comment #32: 

comment #31:

> In starting to work on these patches, I realized I had already made a patch to fallbacks.tmac that is still pending (see most recent comment of bug #63332), and any patches I could make to address this bug would invalidate that patch, or vice versa.  So I'll wait till that patch is applied, rejected, or kicked back to me for refinement, before doing any more fallbacks.tmac hacking, to avoid stomping on myself.


Adding dependency to reflect blockage.

G. Branden Robinson <gbranden>
Group administrator
Fri 15 Sep 2023 06:17:24 AM UTC, comment #31: 

comment #28:

> comment #27:
> > [colorful metaphor excised]
> > I feel I must simply solicit a patch from you at this point.
> [...]
> So at least two patches will follow.


In starting to work on these patches, I realized I had already made a patch to fallbacks.tmac that is still pending (see most recent comment of bug #63332), and any patches I could make to address this bug would invalidate that patch, or vice versa.  So I'll wait till that patch is applied, rejected, or kicked back to me for refinement, before doing any more fallbacks.tmac hacking, to avoid stomping on myself.

Dave <barx>
Group Member
Wed 06 Sep 2023 03:45:02 AM UTC, comment #30: 

To help support mission creep, I'm adding one item to the comment #28 to-do list.

13. \[u2008]: refine for accuracy

U+2008 PUNCTUATION SPACE is generally defined as a space the width of a font's period or comma.  Currently fallbacks.tmac aliases it to groff's hair space, \^.  But examining the typeset output of the below groff input reveals that in all the fonts that ship with groff, the period is closer in width to groff's thin space (\|) than to its hair space.

.nf
.
.de show-spaces
.fam \\$1
Family \\$1
x\|x (thin space)
x.x
x\^x (hair space)
..
.
.show-spaces A
.show-spaces BM
.show-spaces C
.show-spaces H
.show-spaces N
.show-spaces P
.show-spaces T

(The letter "x" and the period should have no kerning between them in either order, though I didn't verify this in the font files.)

In some font families, the thin space and the period are nearly the same width; in others, the thin space is narrower.  In no case is the thin space too wide.  Thus a space narrower than the thin space isn't appropriate for a punctuation space.

So as a first cut, substituting \| for \^ in the \[u2008] definition in fallbacks.tmac is an improvement.

Can we do better?  Groff has the ability to figure out the width of a period, and make a horizontal movement to match it.  Add this line to the show-spaces macro above:

x\h'\w"."u'x (punctuation space)

This creates a space the exact width of the font's period character, the definition of a punctuation space.

But this won't work in an .fchar, whose RHS is evaluated upon definition rather than interpolation.  Maybe some Tadziu-level wizardry can make it work.  But even then, there are other potential pitfalls to consider.

  • Some specialty fonts may lack a period character.
  • \w has side effects (setting numerous registers) that may interfere with a user's own use of these registers, since a user specifying a punctuation space wouldn't necessarily (and shouldn't have to) know that its implementation involves \w.


Anyway, for a fallback, the plain old \| is a good approximation of the space needed, so when I get around to making that requested patch, that's what I plan to use.

Dave <barx>
Group Member
Fri 25 Aug 2023 03:40:22 AM UTC, comment #29: 

comment #16:

> N.B., groff -a output does not "chase" fallback character
> definitions of the sort we use in "fallback.tmac".


This is a "known" (to anyone who has the entirety of the groff bug tracker memorized) issue, bug #55799.

Dave <barx>
Group Member
Fri 25 Aug 2023 03:32:57 AM UTC, comment #28: 

comment #27:

> [colorful metaphor excised]
> I feel I must simply solicit a patch from you at this point.


OK.  As a preliminary step (for my own benefit, or anyone else's who decides to take this and run with it), I'll update the to-do list from comment #15.

First I'll summarize what's already been done.  Items below marked solved or ameliorated refer to changes made in the commit cited in comment #20.

  1. reorder file: no action taken
  2. consistently comment file: no action taken
  3. \[u200B]: ameliorated: awaiting fix for bug #58958
  4. \[u2010]: mostly solved, but see comment #25
  5. \[u2011]: ameliorated: awaiting fix for bug #63360
  6. \[u2012]: ameliorated (using comment #18 patch), but has issue explained in comment #22.
  7. \[u2016]: solved
  8. \[u2018], \[u2019], \[u201C], \[u201D]: ameliorated: awaiting fix for bug #59932
  9. \[u2026]: proposed action not taken: see comment #19.
  10. \[u202F]: no action taken
  11. \[u203D]: no action taken
  12. \[u2052]: no action taken


So for the purposes of this ticket, we can cross off solved items, and ones ameliorated but awaiting fixes for other tickets.  This creates the new to-do list for this ticket below.  (I've retained the slots for the done items so the numbering won't change, to stave off potential confusion if subsequent comments refer to item numbers.)

  1. reorder file (original submission)
  2. consistently comment file (original submission)
  3. done
  4. \[u2010]: remove misleading line (comment #25)
  5. done
  6. \[u2012]: fix comment #22 issue (comment #23, comment #26)
  7. done
  8. done
  9. \[u2026]: potentially remove (comment #19, comment #24, comment #27)
  10. \[u202F]: add a fallback (comment #4)
  11. \[u203D]: give an nroff fallback (comment #9, as amended by comment #12)
  12. \[u2052]: refine definition for robustness (original submission)

These are not all of equal importance, nor should they all be done in the same commit (1, 2, and 4, for instance, are only refactors, so should be done separately from actual code changes).  So at least two patches will follow.

Dave <barx>
Group Member
Thu 29 Dec 2022 02:16:48 AM UTC, comment #27: 

Hi Dave,

comment #24:

> > And if you're right for troff devices that Symbol is always there
> > and it always defines this glyph, it can't degrade typography.
>
> This forgets the lesson I had to learn the hard way (see comment #3) that .fchar definitions take precedence over special fonts.


Ah, yes.  I seem to keep forgetting that, too.  I have visions of a `pchar` request that takes a character as an argument and makes the formatter report how it resolves the damn thing.  Making people read through the eight-step procedure documented in section 5.19.4 of our Texinfo manual--which I haven't reviewed for correctness, incidentally--is simply sadistic.

Nevertheless `fchar` seems to be the right request, rather than `schar`, because a hyphen will nearly(?) always be something that is rendered adjacent to glyphs from a text (potentially styled) font.

>  So as things stand, the definition in fallbacks.tmac will always hide the Symbol font's u2026, degrading typography when using fonts that lack their own u2026.


(U+2026 is the ellipsis.)

But does that happen?  Do people ship text fonts that lack ellipses?

The PostScript Language Reference manual, 3e, suggests that text fonts are reliable in providing ellipses.  The Symbol font seems to have one too (pp. 775-778).

>  And this hiding seems to happen even with the lowest-precedence member of the .*char family, .schar: given the following input, groff uses the "schar" definition rather than falling back to the u2026 in Symbol:
>
> .schar \[u2026] FOO
> .ft ZD
> \[u2026]


Agreed.  That much at least is consistent with §5.19.4.  If I add a `special` request, troff locates "S"'s ellipsis again.


$ cat EXPERIMENTS/zapf-hyphen.groff
.\" troff -R
.special S SS ZD
.schar \[u2026] FOO
A long time ago in a galaxy far, far away\[u2026]
$ troff -R EXPERIMENTS/zapf-hyphen.groff | grep -C3 u2026
tw
H226060
tay
Cu2026
h10000
n12000 0
x trailer

 

> > You might however be forgetting grodvi and grolbp, which don't
> > offer the PS Symbol font.
>
> Indeed I was.  And offhand I'm not sure how to accommodate those without also degrading PostScript.


Possibly the device tmac files should be making `special` requests, though I'm a little leery of this since it may be contrary to the request's initial purpose.  There is already a CSTR #54 procedure for searching special fonts.

This is not a question I feel can be resolved for groff 1.23.0.
 

> > On nroff devices, users of "ascii" and "latin1" devices
> > actually come out ahead of "utf8" users.
>
> Luckily, we can easily test for terminal devices and give them different fallbacks.


I'm not inclined to change things further in this respect at present; I may think a half-width character cell ellipsis is ugly as hell, but I'm uneasy with going out of my way to "protect" the user from it.
 

> For grodvi and grolbp, all I can do is observe that (a) terminal, PostScript and PDF are likely more important output formats for a vast majority of users, and (b) grodvi and grolbp had no u2026 fallback in prior releases, so lacking one in 1.23 isn't a regression.


I have no real argument with this, I am simply trying to think in  maintainerly, comprehensive way to serve all of groff's users, even those that use exotic output devices.

> Thus I'm starting to think the best solution is to make the u2026 fallback terminal-only for 1.23.  But if there's a way to keep the PostScript/PDF behavior as it was in 1.22.4 but also improve other non-terminal output, I'm open to that.


comment #25:

> If fallbacks.tmac gets tweaked before 1.23 (or perhaps even before rc2), one more trivial change I'd recommend.  Two lines in the file currently read:


> .\" \[u2010] is always defined thanks to uniglyph.cpp.
> .\"fchar \[u2010] -\:\" hyphen


> The first of these lines is worthwhile, in case someone comes along later and is tempted to add a \[u2010] definition.  But the second seems misleading: point (b) of comment #8 illustrates that even were the uniglyph.cpp definition removed or circumvented, the fchar here wouldn't do what was intended.  So it's a little dubious to put it in a shipped .tmac file, even in commented form, as it implies to anyone reading the code that the construction could work.
>
> So my preference would be to see the second line removed entirely.  But this is certainly not a blocker, since it's inactive code.


There are so many loose threads tugged on by this baby step toward Unicode support, as if a toddler's leg is wrapped with several strands of dental floss, each connected distally to a different precariously balanced piece of fine china that I feel I must simply solicit a patch from you at this point.

I haven't lost my spirit to fight with these issues in general, but I feel extremely constrained from whacking at them within the parameters I've imposed on myself for getting to a 1.23.0 release.

G. Branden Robinson <gbranden>
Group administrator
Thu 29 Dec 2022 02:14:09 AM UTC, comment #26: 

  About the character "figure dash".

  The total width of a figure dash must be the same as that of a digit,
so (including "air")

.if !c \[u2012] \{\
.  nr lfiguredash \w'0'u-(2*\w'\^'u)
.  char \[u2012] \^\v'-0.3m'\D'l \n[lfiguredash]u 0m'\v'0.3m\^'
.  rm lfiguredash
.\}

  And with a synonym (PostScript name)

.tr \[figuredash]\[u2012]

Bjarni Ingi Gislason <bjarniig>
Thu 29 Dec 2022 12:53:36 AM UTC, comment #25: 

If fallbacks.tmac gets tweaked before 1.23 (or perhaps even before rc2), one more trivial change I'd recommend.  Two lines in the file currently read:

.\" \[u2010] is always defined thanks to uniglyph.cpp.
.\"fchar \[u2010] -\:\" hyphen

The first of these lines is worthwhile, in case someone comes along later and is tempted to add a \[u2010] definition.  But the second seems misleading: point (b) of comment #8 illustrates that even were the uniglyph.cpp definition removed or circumvented, the fchar here wouldn't do what was intended.  So it's a little dubious to put it in a shipped .tmac file, even in commented form, as it implies to anyone reading the code that the construction could work.

So my preference would be to see the second line removed entirely.  But this is certainly not a blocker, since it's inactive code.

Dave <barx>
Group Member
Wed 28 Dec 2022 11:39:54 PM UTC, comment #24: 

Concerning U+2026:

comment #19:

> The (PostScript) Symbol font won't be there on nroff devices.


A valid point that I hadn't considered.

> And if you're right for troff devices that Symbol is always there
> and it always defines this glyph, it can't degrade typography.


This forgets the lesson I had to learn the hard way (see comment #3) that .fchar definitions take precedence over special fonts.  So as things stand, the definition in fallbacks.tmac will always hide the Symbol font's u2026, degrading typography when using fonts that lack their own u2026.  And this hiding seems to happen even with the lowest-precedence member of the .*char family, .schar: given the following input, groff uses the "schar" definition rather than falling back to the u2026 in Symbol:

.schar \[u2026] FOO
.ft ZD
\[u2026]

> You might however be forgetting grodvi and grolbp, which don't
> offer the PS Symbol font.


Indeed I was.  And offhand I'm not sure how to accommodate those without also degrading PostScript.

> On nroff devices, users of "ascii" and "latin1" devices
> actually come out ahead of "utf8" users.


Luckily, we can easily test for terminal devices and give them different fallbacks.

For grodvi and grolbp, all I can do is observe that (a) terminal, PostScript and PDF are likely more important output formats for a vast majority of users, and (b) grodvi and grolbp had no u2026 fallback in prior releases, so lacking one in 1.23 isn't a regression.

Thus I'm starting to think the best solution is to make the u2026 fallback terminal-only for 1.23.  But if there's a way to keep the PostScript/PDF behavior as it was in 1.22.4 but also improve other non-terminal output, I'm open to that.

Dave <barx>
Group Member
Wed 28 Dec 2022 10:18:47 PM UTC, comment #23: 

[U+2012 thoughts continued]

http://en.wikipedia.org/wiki/Dash#Figure_dash suggests an en dash as a figure dash replacement.  To be conservative (in case the user's font has an en dash wider than its figures), we might also define a glyph that is a hyphen centered within a figure space.  This produces decent results with the sample snippet from comment #22:

.fchar \[u2012] \o'\0-'

In Times, even using \[en] in place of the - here feels too crowded, though less so than the raised \[ru].  The hyphen falls more on the too-short side, but I contend this is the better side to fall on; this is, after all, only a fallback glyph, for fonts lacking a U+2012 designed to interact pleasingly with its figures.

(The order of the glyphs in the \o escape shouldn't matter, since this definition isn't used for terminal devices, but it is crafted to degrade well in non-overstriking situations.)

Dave <barx>
Group Member
Wed 28 Dec 2022 08:04:36 PM UTC, comment #22: 

comment #18:

> Re: #6, U+2012, I think the glyph just needs a little elbow room.
>
> Try this.


That improves the aesthetics--but also torpedoes an essential property of this glyph, that its width be the same as that of a digit.  Observe how these digits no longer align vertically:

printf '7\[u2012]5 6\[u2012]0\n.br\n735 690\n' | groff

Dave <barx>
Group Member
Wed 28 Dec 2022 09:31:59 AM UTC, comment #21: 

Step back down to minor severity (not normal) after its temporary promotion for release-critical items.

G. Branden Robinson <gbranden>
Group administrator
Tue 27 Dec 2022 09:25:36 AM UTC, comment #20: 

The 1.23.0-critical(?) stuff is pushed.

Dropping to normal severity.


commit fa1553c33ecfe925cc0c0eac41a30903ae040987
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Dec 26 12:44:32 2022 -0600

    [tmac]: Revise "fallback" character definitions.

    ...in several respects for Unicode characters from General Punctuation
    block.

    * tmac/fallbacks.tmac: Comment out fallbacks for U+200B, U+2010, U+2011,
      and U+201[89CD], due to ineffectuality or apparent groff bugs.
      Redefine U+2012 fallback to include hair space (\^) around synthetic
      figure dash.  Redefine U+2016 to use `\[ba]` special characters
      instead of ordinary `|` characters to dodge possible character
      translations by user.

    Fixes part of <https://savannah.gnu.org/bugs/?63354>.  Thanks to Dave
    Kemper for the (multifarious) discussion.


G. Branden Robinson <gbranden>
Group administrator
Mon 26 Dec 2022 06:45:12 AM UTC, comment #19: 

comment #15:

> 0 permanently remove \[u2026] (comment #6)


> 0 Should be done before 1.23: only degrades typography without any upside in real-world usage, since Symbol will always be there.


The (PostScript) Symbol font won't be there on nroff devices.  And if you're right for troff devices that Symbol is always there and it always defines this glyph, it can't degrade typography.

You might however be forgetting grodvi and grolbp, which don't offer the PS Symbol font.  grolbp in fact doesn't offer any special fonts at all.  (Technically, they're all special.  ¯\_(ツ)_/¯ )

On nroff devices, users of "ascii" and "latin1" devices actually come out ahead of "utf8" users.


$ printf 'foo\\[u2026]bar\n' | tgu | cat -s
foo…bar
$ printf 'foo\\[u2026]bar\n' | tg -Tascii | cat -s
foo...bar
$ printf 'foo\\[u2026]bar\n' | tg -Tlatin1 | cat -s
foo...bar


So I'm inclined to keep this one.

G. Branden Robinson <gbranden>
Group administrator
Mon 26 Dec 2022 06:32:43 AM UTC, comment #18: 

Re: #6, U+2012, I think the glyph just needs a little elbow room.

Try this.


-.  fchar \[u2012] \v'-.3m'\l'\w"\0"u\[ru]'\v'+.3m'\" figure dash
+.  fchar \[u2012] \^\v'-.3m'\l'\w"\0"u'\v'+.3m'\^\" figure dash


G. Branden Robinson <gbranden>
Group administrator
Mon 26 Dec 2022 06:19:59 AM UTC, comment #17: 

Dave says there are 1.23.0-critical issues here, so kicking this up to Important until they are taken care of.

G. Branden Robinson <gbranden>
Group administrator
Mon 26 Dec 2022 06:19:30 AM UTC, comment #16: 

The explanation for the first half of comment #8 appears to be this.

https://git.savannah.gnu.org/cgit/groff.git/tree/src/libs/libgroff/uniglyph.cpp?id=59efbfc364daee11af9207a3fc79675b6f5bfc25#n334

Normally, I get this:


$ printf '.fchar \\[u2010] BOGOSITY\nfoo\\[u2010]bar\n.pl \\n[nl]u\n' | ./build/test-groff -Tps -a
<beginning of page>
foo<hy>bar


But if I change it to something obviously wrong, I get this:


$ printf '.fchar \\[u2010] BOGOSITY\nfoo\\[u2010]bar\n.pl \\n[nl]u\n' | ./build/test-groff -Tps -a
<beginning of page>
foo<sl>bar


And if I comment it out, I get this:


$ printf '.fchar \\[u2010] BOGOSITY\nfoo\\[u2010]bar\n.pl \\n[nl]u\n' | ./build/test-groff -Tps -a
<beginning of page>
foo<u2010>bar


N.B., groff -a output does not "chase" fallback character definitions of the sort we use in "fallback.tmac".

G. Branden Robinson <gbranden>
Group administrator
Sat 24 Dec 2022 06:05:07 AM UTC, comment #15: 

Yeah, this ticket is chockablock with to-do items, but they're sprinkled across several comments and interspersed with a couple of blind alleys.  So here's a checklist of tasks.

  1. reorder file (original submission)
  2. consistently comment file (original submission)
  3. comment out \[u200B] until bug #58958 resolved (comment #5)
  4. permanently remove \[u2010] (comment #8)
  5. potentially defer \[u2011] until bug #63360 resolved (comment #10)
  6. fix \[u2012] (original submission; correction in comment #13)
  7. refine \[u2016] definition for robustness (original submission)
  8. comment out \[u2018], \[u2019], \[u201C], and \[u201D] until bug #59932 resolved (comment #11)
  9. permanently remove \[u2026] (comment #6)
  10. add a \[u202F] fallback (comment #4)
  11. give \[u203D] an nroff fallback (comment #9, as amended by comment #12)
  12. refine \[u2052] definition for robustness (original submission)


Every one of these involves an alteration to commit 132182bd, which was added since 1.22.4, so for 1.23 purposes, the pertinent question is: which changes make things better as-is, and which make things worse, since the last release?

Obviously, "better" and "worse" are subjective.  But I'm gonna go ahead and pretend they aren't, and further pretend that I'm the final arbiter, and pass judgment on every item above.

  1. Refactor only.  Irrelevant to 1.23.
  2. Refactor only.  Irrelevant to 1.23.
  3. Should be done before 1.23, since the the current definition is broken, and groff has a long-established escape for doing this.
  4. Not important to 1.23 (though OTOH removing the line is trivial so why not).
  5. Should be done before 1.23: while it's a shame to not be able to support U+2011, the subpar rendering is worth addressing before it's unleashed onto the world.
  6. The fix to this is (probably) not trivial, but the current rendering is ugly, so this should be commented out before 1.23.
  7. Should be done before 1.23: fix is simple and avoids plausible real-world problems.
  8. Should be done before 1.23: these four have long-established groff aliases that do the right thing, while these fallbacks would degrade typography.
  9. Should be done before 1.23: only degrades typography without any upside in real-world usage, since Symbol will always be there.
  10. Not important to 1.23.
  11. Not important to 1.23.
  12. Probably not important to 1.23, mostly because there's no easy fix.  The potential pitfall to the user is a seemingly inexplicable character substitution, but U+2052 is obscure enough that I don't see much practical risk.


These opinions are correct and incontrovertible.  I have spoken.

Dave <barx>
Group Member
Tue 20 Dec 2022 03:47:09 AM UTC, comment #14: 

Hi Dave,

Is any part of this important to get into 1.23.0?  I fixed the only Savannah ticket that bore that as the "Planned Release", but there are many details in this one.

G. Branden Robinson <gbranden>
Group administrator
Mon 19 Dec 2022 08:42:16 PM UTC, comment #13: 

original submission:

> The figure dash (\[u2016]) as currently designed works well with
> some figures but not others, in at least the TR font.  Consider
> the PostScript output of:
>
> echo '7\[u2012]5 6\[u2012]0' | groff


As the command implies, the figure dash is \[u2012], not \[u2016] as stated in the text above.

Dave <barx>
Group Member
Sun 11 Dec 2022 10:28:39 PM UTC, comment #12: 

Commit 230ad718 has mooted everything after the "HOWEVER" in comment #9.

Dave <barx>
Group Member
Wed 07 Dec 2022 09:48:44 PM UTC, comment #11: 

The point comment #10 made about \[u2011] equally applies to \[u2018] (left single quotation mark), \[u2019] (right single quotation mark), \[u201C] (left double quotation mark), and \[u201D] (right double quotation mark), all common characters that are frequently kerned next to certain uppercase letters.

I hate to take all these Unicode representations out of circulation until the thornier underlying problems can be solved.  However, they were in no previously released groff anyway, and at least with these four (unlike \[u2011]), groff has decades-established local aliases (\[oq], \[cq], \[lq], and \[rq]) that work correctly in all respects, so retaining the recently added, incorrectly kerned variants is of dubious value.

Dave <barx>
Group Member
Sun 04 Dec 2022 02:50:56 PM UTC, comment #10: 

And now... we should talk about U+2011, the character that has spilled the most ink over in bug #58930.

In a nutshell, the problem is that U+2011 is, by definition, visually identical to an ordinary hyphen (U+2010), a character which is frequently kerned next to several uppercase letters.  But as currently implemented, the U+2011 will never be kerned, because groff does not kern defined characters (bug #59932).  Bug #63360 documents the U+2011 problem, but points out that solving it is likely not simple, and thus can't be expected soon.

So the question is how to handle this in the meantime.  Much of the discussion in 58930 resolved to "punt on this issue and see if the users complain," which I agreed was reasonable in several edge cases.

I'm less comfortable with it in the U+2011 case, since the miskerning there (a) is much commoner in ordinary text, and (b) may not be something a user will complain about, per se, but that will subtly degrade the type quality--something the groff user might not even notice but a reader of that user's output might (from which we're unlikely to ever get a report back).

If the groff team decides this problem is worth addressing, the only reasonable short-term solution, as far as I can see, is to follow bug #63332's lead and comment out the \[u2011] line, marking it as requiring more R&D than can be presently dedicated to it.  (The corresponding comment should point to bug #63360, which is dedicated to this issue.)

Dave <barx>
Group Member
Thu 01 Dec 2022 09:43:43 AM UTC, comment #9: 

A possible refinement to U+203D:

The current definition is

.if t .fchar \[u203D] \o'?!'\" interrobang

This (theoretically) leaves the character undefined on the terminal--which it has been until recently, so not a huge deal--but as long as we're defining fallbacks anyway, the string "!?" is closest in spirit achievable without overstriking.  Thus the line could be:

.ie t .fchar \[u203D] \o'?!'\" interrobang
.el .fchar \[u203D] !?

HOWEVER.  Because of the ordering issue discussed in bug #63332, the ".if t" lines are not working correctly in nroff mode anyway, meaning that the character as currently defined is resulting in "!" being output on terminals.

I feel that, if we can only use one character to represent the interrobang, the "?" is a better choice than the "!" (as the "?" can change the meaning of a phrase, while the "!" typically only changes its level of emphasis).  So as long as fallbacks.tmac can't test what mode it's in, I'd recommend changing the order of the current definition, so that it reads:

.fchar \[u203D] \o'!?'\" interrobang

This won't change the typesetting at all, but will (IMHO) improve the terminal output a bit.

Dave <barx>
Group Member
Mon 21 Nov 2022 04:17:45 PM UTC, comment #8: 

The plot keeps thickening.  The \[u2010] definition should also be removed because (a) it doesn't seem to be invoked, and (b) it wouldn't work if it were.

(a): None of groff's fonts define a u2010 character.  But apparently earlier in the process, groff intercepts the u2010 and turns it into a hyphen, so the .fchar definition is never seen.  This can be demonstrated by changing the line in fallbacks.tmac to read, for example:

.fchar \[u2010] GARBAGE

Even with this change, groff outputs a hyphen for \[u2010]:

$ echo 'Boo\[u2010]yah.' | groff -Tascii | cat -s
Boo-yah.

The same results are in PostScript output, even though font TR does not contain a u2010 character.  So nothing is ever using this .fchar definition.

(b): Now let's avoid the u2010 name altogether and define a new character.

$ cat booyah
.fchar \[myhyphen] -\:
.ll 1p
Boo-yah.
Boo\[myhyphen]yah.
$ groff -Tascii booyah | cat -s
troff:booyah:3: warning [p 1, 0.0i]: cannot break line
troff:booyah:3: warning [p 1, 0.2i]: cannot break line
troff:booyah:4: warning [p 1, 0.3i]: cannot break line
Boo-
yah.
Boo-yah.


This looks like our old friend bug #58958 coming out to play.  So even if the u2010 definition in fallbacks.tmac were getting called, it wouldn't do what you wanted anyway.

Upshot: this line should be removed.  Because of point (a) above, it's not actually doing any harm, but ideally the shipped tmac files should be free of dead code.

Dave <barx>
Group Member
Mon 21 Nov 2022 04:18:13 AM UTC, comment #7: 

And in case anyone was concerned, U+2026 is the only \[u....] glyph in Symbol that fallbacks.tmac obscures:

$ egrep '^u[0-9A-F]{2}' font/devps/S | cut -f1,5
u2026   ellipsis
u2320   integraltp
u23AE   integralex
u2321   integralbt


Dave <barx>
Group Member
Mon 21 Nov 2022 12:29:57 AM UTC, comment #6: 

original submission:

> Every general-purpose font groff ships includes this character.
> Further, the Symbol font (which also includes it) being a default
> special font means that even users using a locally installed font that
> lacks a U+2026 will get it for free without doing any extra work.
>
> All this might argue for removing this definition from fallbacks.tmac
> entirely.


My realization in comment #3 that an .fchar definition supersedes a special-font glyph only reinforces the above: if the user's current font lacks a \[u2026], it is better to fall back to the well-formed one in Symbol than to the more problematic one defined in fallbacks.tmac.  Further, the fact that it's nigh impossible to remove Symbol as a special font means that a user would have to resort to an "ugly hack" of the type Branden describes in bug #63366 in order to ever be lacking a \[u2026]--and once someone is in this power-user territory, I think it's fine if they're working without a net.

Dave <barx>
Group Member
Sun 20 Nov 2022 08:04:54 PM UTC, comment #5: 

Another addendum:

Unicode's U+200B ZERO WIDTH SPACE has the semantics of groff's \: escape.  However, the current definition for \[u200B] in fallbacks.tmac (\h'0') does not have U+200B's essential breaking property:

$ cat u200B-test
.ll 1p
ab\:cd
ab\h'0'cd
$ nroff -Ww u200B-test | cat -s
ab
cd
abcd

And sadly, while bug #58958 is unresolved, we can't do the obvious ".fchar \[u200B] \:".  (.tr likewise doesn't work, though one wouldn't expect it to, since its definitions are supposed to be glyphs, which \: isn't.)  I have no other suggestions for how to tie this functionality to \[u200B].

So perhaps the best thing here is to comment out the \[u200B] definition, with an additional comment saying it's awaiting a fix to #58958.  In the meantime, \[u200B] will be unrecognized, but that's already the status quo (in 1.22.4, and in git until less than a month ago), and is less misleading than implementing it in a noncompliant way.  (I'm not sure how, or whether, \h'0' effectively differs from \&.)

Dave <barx>
Group Member
Wed 16 Nov 2022 11:40:23 AM UTC, comment #4: 

As an addendum to the list of fallbacks.tmac changes set out in the original submission, one more fallback character could be added to the file, one mentioned all the way back in the original submission of the beleaguered bug #58930: \[u202F] (NARROW NO-BREAK SPACE).

Groff has two nonbreaking thin spaces, \| and \^.  It is perhaps unclear which of these \[u202F] should map to, but either one would be an improvement over its current mapping to the warning "can't find special character `u202F'".  All the narrow spaces in groff are already nonbreaking, so nothing extra need be done for that.

Dave <barx>
Group Member
Tue 15 Nov 2022 07:25:04 PM UTC, comment #3: 

Um, you know that last comment where I said I was wrong?  Well, turns out I was wrong, and I was actually right.  (So, also wrong.)

What I missed is that whatever special fonts are in play have no bearing on the situation, because the clearly documented order of glyph-searching places .fchar definitions before special fonts.  So as long as the \[u2026] .fchar in fallbacks.tmac is in force, whatever is happening in special-font-land is irrelevant.

This is verifiable in practice.  Using the same input file from comment #2, this command run using the latest groff code produces this output:

$ groff -Z ellipsis_demo | sed -n '/font 11 S/,/x trailer/p'
x font 11 S
f11
V24000
H72000
t.
h1666
t.
h1666
t..
h1666
t.
h1666
t..
h1666
t.
h1666
t.
n12000 0
x trailer

Then I commented out the \[u2026] definition in fallbacks.tmac and ran it again.

$ groff -Z ellipsis_demo | sed -n '/font 11 S/,/x trailer/p'
x font 11 S
f11
V24000
H72000
Cu2026
h10000
Cu2026
h10000
Cu2026
h10000
n12000 0
x trailer

Here it's "easy" (if you're adept at reading intermediate output) to see that in the former case, groff is constructing the ellipsis from spaced periods, while in the latter, it's calling upon a character named u2026, which it's pulling from the current font, which is S.

This also explains why I was led astray in comment #2: I took the mere loading of font S as evidence that groff must be using that font's u2026--when in fact it was using that font's period, as a special-font fallback because the last requested font (ZD) also lacks a period glyph.

So please disregard comment #2.  The final example in the original submission is fine as-is, though the ".special" line is irrelevant (which is fine, since it doesn't do anything anyway: bug #63366 remains a valid point about special-font handling) and can be removed.

Dave <barx>
Group Member
Mon 14 Nov 2022 05:34:13 PM UTC, comment #2: 

One of my examples in the original submission doesn't do what I thought it did.

$ cat ellipsis_demo
.nf
Switch to a font without u2026, and remove the Symbol fallback.
.ft ZD
.special
\[u2026]\[u2026]\[u2026]
$ groff -Z ellipsis_demo | fgrep font
x font 5 TR
tfont
x font 11 S

I had intended for the ".special" line to zero out the list of special fonts, based on this sentence in the "Special Fonts" section of the manual: "Previous calls to 'special' or 'fspecial' are overwritten; without arguments, the particular list of special fonts is set to empty."

But the grout output shows that the Symbol font is still being invoked.  (And, notably, ZD is not, since after the request to switch to ZD, the input file asks only for one glyph, which ZD doesn't have.)

I opened bug #63366 to report the documentation issue.  As that bug report points out, there doesn't seem to be a way for a document to remove Symbol as a special font.  So to get this example to show what I intended it to show, I may have to remove the S font description file from the system.  Further research is required.

Dave <barx>
Group Member
Sun 13 Nov 2022 01:05:57 AM UTC, comment #1: 

Ah, so U+2024 and U+2025 (the one- and two-dot leader characters) aren't intended for modern use; they exist only for compatibility with Xerox Character Code Standard 1980.

http://scripts.sil.org/CharStories_2024

I reckon I don't see any harm in leaving their fallback definitions in tmac/fallbacks.tmac, but I sure wouldn't sweat the details of how they render.

Dave <barx>
Group Member
Sat 12 Nov 2022 11:06:51 PM UTC, original submission:  

A few of the fallback definitions in tmac/fallbacks.tmac could be refined for robustness, typographic quality, or code consistency.




The file organization reflects its assemblage over time: it begins with two blocks of \[u....] characters (one for diacritics, one for roman numerals), goes into some mnemonic Latin-* fallbacks, then goes back to \[u....] definitions (these added in (commit 132182bd, fixing bug #58930).  To me it makes more sense to have all the Unicode definitions together, though perhaps there's some logic to the current arrangement I haven't discerned.

Also, above some of the blocks is a simple comment giving the basic purpose of the block, while others lack this.




As comment 23 of bug #58930 points out in two other cases, the user is likelier to .tr simple ASCII characters than escapes representing those characters.  Most of the Unicode characters defined in the final block do use escapes in their definitions.  Some of those that don't are probably OK because users are unlikely to .tr characters extremely common in text such as the period and the hyphen.  But two definitions might benefit from guarding against this:

  • \[u2016], currently defined as two "|"s, might be more robustly defined as two "\[ba]"s.
  • \[u2052] is defined as "%".  This is trickier to change, because bug #63334 precludes the use of the alternate \[u0025], and groff provides no other way to specify this character.  So we either have to live with this small risk, or make this bug dependent on 63334.





The figure dash (\[u2016]) as currently designed works well with some figures but not others, in at least the TR font.  Consider the PostScript output of:

echo '7\[u2012]5 6\[u2012]0' | groff

In the second pair of figures, the dash touches the figure on either side; ideally there should be a bit of space between the dash and any adjacent figure.  (This will have to amount to shortening the dash, since the space taken up by the entire glyph must remain constant.)




Over in bug #58930 I brought up the spacing between adjacent \[u2026]s.  I quote your reply there (in comment 24):

> I'm tempted to punt on this one, too.  Possibly no serious
> font for typesetting even needs to encode these characters,


If you mean that most fonts will already have a U+2026 glyph, and therefore this fallback will rarely be used, I agree.  (If you mean something else, can you please clarify?)  Every general-purpose font groff ships includes this character.  Further, the Symbol font (which also includes it) being a default special font means that even users using a locally installed font that lacks a U+2026 will get it for free without doing any extra work.

All this might argue for removing this definition from fallbacks.tmac entirely.

> and in groff, if you want a well-typeset leader, a fundamental
> formatter feature (Control+A) will give you one of whatever
> length you like.


True, but a leader is not the only situation where one might use adjacent ellipses.  Someone writing dialogue might decide that "Um\[u2026]\[u2026]" conveys a greater degree of uncertainty than simply "Um\[u2026]".  This will look fine with Times's or Symbol's ellipsis, but the fallbacks.tmac one, should it ever be pressed into service, will not fare so well:

.nf
Baseline demonstration.
\[u2026]\[u2026]\[u2026]
Switch to a font without u2026, and remove the Symbol fallback.
.ft ZD
.special
\[u2026]\[u2026]\[u2026]

I see no problems with the \[u2024] definition, being a single character.  The \[u2025] one will exhibit the problem shown above.  But I don't know in what situations one even uses U+2025, so I'd agree that punting in this case seems the best option.

Dave <barx>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #55788:  prof.figure.dash added by bjarniig (698B - application/octet-stream)
file #55773:  figuredash.png added by barx (514B - image/png)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by bjarniig (Posted a comment)
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-08-04 gbranden Assigned toNone barx
        SummaryRefine fallbacks.tmac [tmac]: refine fallbacks.tmac
    2024-03-05 bjarniig Attached File- Added prof.figure.dash, #55788
    2024-03-03 barx Attached File- Added figuredash.png, #55773
    2023-09-15 gbranden Dependencies- Depends on bugs #63332
    2022-12-29 gbranden StatusNone Need Info
    2022-12-28 gbranden Severity3 - Normal 2 - Minor
    2022-12-27 gbranden Severity4 - Important 3 - Normal
    2022-12-26 gbranden Severity2 - Minor 4 - Important

    Back to the top

    Powered by Savane 3.14-79a4.
    Corresponding source code