comment #41:

> comment #40:
> > I suggest that it doesn't.
>
> Hence "arguably."
>
> > That caveat seems to hoist on us on the very petard that
> > separates the marauding `\:` Turk from the `\&` ground soaked
> > by the bloodthirsty Romanians.
>
> I don't think a case can be made for \&.  That doesn't permit a break, and I hope if we can agree on nothing else, we can agree that a character named "BREAK PERMITTED HERE" ought to permit a break.

Sorry, I did in fact mean `\%` rather than `\&`.  I evidently put more effort into my analogy than my basic message.

>
> So the only reasonable choices are \: or \%--each permitting a break, one adding a hyphen if a break occurs there, one not.
>
> The Unicode standard v15.1 is light--to the point of malfeasance--on the meanings of most characters in the C1 control block.  (http://www.unicode.org/charts/PDF/U0080.pdf)  The Unicoderati, it seems, would rather you avoid these legacy code points altogether and use their newer, more well-defined counterparts.  And that's the advice I'd give to anyone composing new Unicode text.
>
> But what of legacy documents?  This is an area where groff ought to shine.  And what the Unicode standard is furtive about, we can deduce from the content of the overall C1 Controls and Latin-1 Supplement block.
>
> Specifically, this block contains the character U+00AD SOFT HYPHEN, the function of which Unicode is more forthcoming about: "an invisible format character indicating a possible hyphenation location".  This, in roffese, is \%.
>
> So why would another character in the same block also translate to \%?  It seems clear the original intent of U+0082 BREAK PERMITTED HERE is to define a potential break point that doesn't get hyphenated--the \:, in roffese.

Okay.  I'm persuaded well enough on this point.

Can you to catch me up on what we've agreed upon in this 40+ message sprawl?

comment #40:

> I suggest that it doesn't.

Hence "arguably."

> That caveat seems to hoist on us on the very petard that
> separates the marauding `\:` Turk from the `\&` ground soaked
> by the bloodthirsty Romanians.

I don't think a case can be made for \&.  That doesn't permit a break, and I hope if we can agree on nothing else, we can agree that a character named "BREAK PERMITTED HERE" ought to permit a break.

So the only reasonable choices are \: or \%--each permitting a break, one adding a hyphen if a break occurs there, one not.

The Unicode standard v15.1 is light--to the point of malfeasance--on the meanings of most characters in the C1 control block.  (http://www.unicode.org/charts/PDF/U0080.pdf)  The Unicoderati, it seems, would rather you avoid these legacy code points altogether and use their newer, more well-defined counterparts.  And that's the advice I'd give to anyone composing new Unicode text.

But what of legacy documents?  This is an area where groff ought to shine.  And what the Unicode standard is furtive about, we can deduce from the content of the overall C1 Controls and Latin-1 Supplement block.

Specifically, this block contains the character U+00AD SOFT HYPHEN, the function of which Unicode is more forthcoming about: "an invisible format character indicating a possible hyphenation location".  This, in roffese, is \%.

So why would another character in the same block also translate to \%?  It seems clear the original intent of U+0082 BREAK PERMITTED HERE is to define a potential break point that doesn't get hyphenated--the \:, in roffese.

comment #5:

> Unicode's U+200B ZERO WIDTH SPACE has the semantics of groff's \: escape.

Arguably, so does U+0082 BREAK PERMITTED HERE.

(Of course, for fallback usage the .char in comment #35 and comment #36 should be .fchar.)

This looks great!

Here's the definition from the sample file, broken down and commented, and trivially refactored (to avoid performing the same \w twice when the value of the first is already stored in a register):

.  char \[u2012] \R#wzero \w'0'u#\#      set width of a figure, setting \n[rst]
\R#hzero \\n[rst]u/2u#\#                 set horizontal midpoint of a figure
\R#lfiguredash \\n[wzero]-(2*\w'\^'u)#\# set dash length
\^\
\v'-\\n[hzero]u'\#                       adjust horizontal position
\D'l \\n[lfiguredash]u 0m'\#             draw dash
\^\
\v'\\n[hzero]u'\"                        restore horizontal position

The glyph is exactly the right width, and the spacing between it and the surrounding figures seems ideal.

As the fourth line of output shows (the one that puts the created figure dash on the same line as stock dashes and hyphens), it's a little too high in the line.  This is true across all the font families that ship with groff; dashes and hyphens are typically set below half-height so they interact well with lowercase letters.

This is easy to adjust by tweaking the "rst / 2" calculation.  The tricky part is finding an adjustment that works in all families.  The best compromise I could find was \\n[rst]u*10/27u (groff's integer arithmetic prevents simply using a value between 2 and 3 in the denominator), which set the figure dash a bit too high for a couple families, a bit too low for a couple others, and just about right for several.  But unless the dashes are right next to each other--which shouldn't happen in real-world usage--the difference won't be noticeable.

Alternately, .fschar could be used to tailor the dash for each groff font.  But there would still need to be a generic fallback for any user-supplied fonts, and the metrics for this would still be speculative.

In most families (all stock groff families but C, P, and T), the generated dash is also skinnier than the font-supplied ones, but for a fallback definition, I propose to not worry about this level of detail.  Alternately, as above, .fschar could be used, with the same caveat.

Notably, the existing \[u2012] fallback in fallbacks.tmac already has a hard-coded height and thickness that don't exactly match every family; this proposed replacement improves on it by giving it the correct width, allowing figures on adjacent lines to align properly.

A couple of potential implementation issues, ordered from most to least important:

• This character's internal use of \w will clobber any values of its side-effect registers (st, rst, ssc, etc.) that the user may have been relying on.  (Also true of the existing \[u2012] fallback.)
• The three named registers (wzero, hzero, lfiguredash) will clobber any registers by those names the user is using.  (These names could be tweaked to make collisions less likely, at the minor expense of code readability.)
• Unlike .nr, the .rr request has no corresponding escape sequence, so it's impossible to clean up the registers.

I think making available a working U+2012 is a big enough gain to offset these drawbacks, especially given the hope that over time, more fonts (perhaps someday, even those that ship with groff) will include U+2012, and this fallback will see less use.

Anyone have thoughts on any of the above?

Bjarni's proposed \[u2012] definition in comment #26 has two problems.

• It relies on a register (lfiguredash) that calculates the width of a zero and a hair space based on the font size at the time the register is defined.  This value will thus be incorrect at any other font size used later in the document.
• It emits an extraneous apostrophe.  This input:

.nr lfiguredash \w'0'u-(2*\w'\^'u)
.char \[u2012] \^\v'-0.3m'\D'l \n[lfiguredash]u 0m'\v'0.3m\^'
.rm lfiguredash
.
1\[u2012]2\[u2012]3

produces the attached typeset output.

comment #28:

> comment #27:
> > [colorful metaphor excised]
> > I feel I must simply solicit a patch from you at this point.
> [...]
> So at least two patches will follow.

In starting to work on these patches, I realized I had already made a patch to fallbacks.tmac that is still pending (see most recent comment of bug #63332), and any patches I could make to address this bug would invalidate that patch, or vice versa.  So I'll wait till that patch is applied, rejected, or kicked back to me for refinement, before doing any more fallbacks.tmac hacking, to avoid stomping on myself.

comment #16:

> N.B., groff -a output does not "chase" fallback character
> definitions of the sort we use in "fallback.tmac".

This is a "known" (to anyone who has the entirety of the groff bug tracker memorized) issue, bug #55799.

Hi Dave,

comment #24:

> > And if you're right for troff devices that Symbol is always there
> > and it always defines this glyph, it can't degrade typography.
>
> This forgets the lesson I had to learn the hard way (see comment #3) that .fchar definitions take precedence over special fonts.

Ah, yes.  I seem to keep forgetting that, too.  I have visions of a `pchar` request that takes a character as an argument and makes the formatter report how it resolves the damn thing.  Making people read through the eight-step procedure documented in section 5.19.4 of our Texinfo manual--which I haven't reviewed for correctness, incidentally--is simply sadistic.

Nevertheless `fchar` seems to be the right request, rather than `schar`, because a hyphen will nearly(?) always be something that is rendered adjacent to glyphs from a text (potentially styled) font.

>  So as things stand, the definition in fallbacks.tmac will always hide the Symbol font's u2026, degrading typography when using fonts that lack their own u2026.

(U+2026 is the ellipsis.)

But does that happen?  Do people ship text fonts that lack ellipses?

The PostScript Language Reference manual, 3e, suggests that text fonts are reliable in providing ellipses.  The Symbol font seems to have one too (pp. 775-778).

>  And this hiding seems to happen even with the lowest-precedence member of the .*char family, .schar: given the following input, groff uses the "schar" definition rather than falling back to the u2026 in Symbol:
>
> .schar \[u2026] FOO
> .ft ZD
> \[u2026]

Agreed.  That much at least is consistent with §5.19.4.  If I add a `special` request, troff locates "S"'s ellipsis again.

$ cat EXPERIMENTS/zapf-hyphen.groff
.\" troff -R
.special S SS ZD
.schar \[u2026] FOO
A long time ago in a galaxy far, far away\[u2026]
$ troff -R EXPERIMENTS/zapf-hyphen.groff | grep -C3 u2026
tw
H226060
tay
Cu2026
h10000
n12000 0
x trailer

> > You might however be forgetting grodvi and grolbp, which don't
> > offer the PS Symbol font.
>
> Indeed I was.  And offhand I'm not sure how to accommodate those without also degrading PostScript.

Possibly the device tmac files should be making `special` requests, though I'm a little leery of this since it may be contrary to the request's initial purpose.  There is already a CSTR #54 procedure for searching special fonts.

This is not a question I feel can be resolved for groff 1.23.0.
 

> > On nroff devices, users of "ascii" and "latin1" devices
> > actually come out ahead of "utf8" users.
>
> Luckily, we can easily test for terminal devices and give them different fallbacks.

I'm not inclined to change things further in this respect at present; I may think a half-width character cell ellipsis is ugly as hell, but I'm uneasy with going out of my way to "protect" the user from it.
 

> For grodvi and grolbp, all I can do is observe that (a) terminal, PostScript and PDF are likely more important output formats for a vast majority of users, and (b) grodvi and grolbp had no u2026 fallback in prior releases, so lacking one in 1.23 isn't a regression.

I have no real argument with this, I am simply trying to think in  maintainerly, comprehensive way to serve all of groff's users, even those that use exotic output devices.

> Thus I'm starting to think the best solution is to make the u2026 fallback terminal-only for 1.23.  But if there's a way to keep the PostScript/PDF behavior as it was in 1.22.4 but also improve other non-terminal output, I'm open to that.

comment #25:

> If fallbacks.tmac gets tweaked before 1.23 (or perhaps even before rc2), one more trivial change I'd recommend.  Two lines in the file currently read:

> .\" \[u2010] is always defined thanks to uniglyph.cpp.
> .\"fchar \[u2010] -\:\" hyphen

> The first of these lines is worthwhile, in case someone comes along later and is tempted to add a \[u2010] definition.  But the second seems misleading: point (b) of comment #8 illustrates that even were the uniglyph.cpp definition removed or circumvented, the fchar here wouldn't do what was intended.  So it's a little dubious to put it in a shipped .tmac file, even in commented form, as it implies to anyone reading the code that the construction could work.
>
> So my preference would be to see the second line removed entirely.  But this is certainly not a blocker, since it's inactive code.

There are so many loose threads tugged on by this baby step toward Unicode support, as if a toddler's leg is wrapped with several strands of dental floss, each connected distally to a different precariously balanced piece of fine china that I feel I must simply solicit a patch from you at this point.

I haven't lost my spirit to fight with these issues in general, but I feel extremely constrained from whacking at them within the parameters I've imposed on myself for getting to a 1.23.0 release.

If fallbacks.tmac gets tweaked before 1.23 (or perhaps even before rc2), one more trivial change I'd recommend.  Two lines in the file currently read:

.\" \[u2010] is always defined thanks to uniglyph.cpp.
.\"fchar \[u2010] -\:\" hyphen

The first of these lines is worthwhile, in case someone comes along later and is tempted to add a \[u2010] definition.  But the second seems misleading: point (b) of comment #8 illustrates that even were the uniglyph.cpp definition removed or circumvented, the fchar here wouldn't do what was intended.  So it's a little dubious to put it in a shipped .tmac file, even in commented form, as it implies to anyone reading the code that the construction could work.

So my preference would be to see the second line removed entirely.  But this is certainly not a blocker, since it's inactive code.

[U+2012 thoughts continued]

http://en.wikipedia.org/wiki/Dash#Figure_dash suggests an en dash as a figure dash replacement.  To be conservative (in case the user's font has an en dash wider than its figures), we might also define a glyph that is a hyphen centered within a figure space.  This produces decent results with the sample snippet from comment #22:

.fchar \[u2012] \o'\0-'

In Times, even using \[en] in place of the - here feels too crowded, though less so than the raised \[ru].  The hyphen falls more on the too-short side, but I contend this is the better side to fall on; this is, after all, only a fallback glyph, for fonts lacking a U+2012 designed to interact pleasingly with its figures.

(The order of the glyphs in the \o escape shouldn't matter, since this definition isn't used for terminal devices, but it is crafted to degrade well in non-overstriking situations.)

Step back down to minor severity (not normal) after its temporary promotion for release-critical items.

comment #15:

> 0 permanently remove \[u2026] (comment #6)

> 0 Should be done before 1.23: only degrades typography without any upside in real-world usage, since Symbol will always be there.

The (PostScript) Symbol font won't be there on nroff devices.  And if you're right for troff devices that Symbol is always there and it always defines this glyph, it can't degrade typography.

You might however be forgetting grodvi and grolbp, which don't offer the PS Symbol font.  grolbp in fact doesn't offer any special fonts at all.  (Technically, they're all special.  ¯\_(ツ)_/¯ )

On nroff devices, users of "ascii" and "latin1" devices actually come out ahead of "utf8" users.

$ printf 'foo\\[u2026]bar\n' | tgu | cat -s
foo…bar
$ printf 'foo\\[u2026]bar\n' | tg -Tascii | cat -s
foo...bar
$ printf 'foo\\[u2026]bar\n' | tg -Tlatin1 | cat -s
foo...bar

So I'm inclined to keep this one.

Dave says there are 1.23.0-critical issues here, so kicking this up to Important until they are taken care of.

Yeah, this ticket is chockablock with to-do items, but they're sprinkled across several comments and interspersed with a couple of blind alleys.  So here's a checklist of tasks.

1. reorder file (original submission)
2. consistently comment file (original submission)
3. comment out \[u200B] until bug #58958 resolved (comment #5)
4. permanently remove \[u2010] (comment #8)
5. potentially defer \[u2011] until bug #63360 resolved (comment #10)
6. fix \[u2012] (original submission; correction in comment #13)
7. refine \[u2016] definition for robustness (original submission)
8. comment out \[u2018], \[u2019], \[u201C], and \[u201D] until bug #59932 resolved (comment #11)
9. permanently remove \[u2026] (comment #6)
10. add a \[u202F] fallback (comment #4)
11. give \[u203D] an nroff fallback (comment #9, as amended by comment #12)
12. refine \[u2052] definition for robustness (original submission)

Every one of these involves an alteration to commit 132182bd, which was added since 1.22.4, so for 1.23 purposes, the pertinent question is: which changes make things better as-is, and which make things worse, since the last release?

Obviously, "better" and "worse" are subjective.  But I'm gonna go ahead and pretend they aren't, and further pretend that I'm the final arbiter, and pass judgment on every item above.

1. Refactor only.  Irrelevant to 1.23.
2. Refactor only.  Irrelevant to 1.23.
3. Should be done before 1.23, since the the current definition is broken, and groff has a long-established escape for doing this.
4. Not important to 1.23 (though OTOH removing the line is trivial so why not).
5. Should be done before 1.23: while it's a shame to not be able to support U+2011, the subpar rendering is worth addressing before it's unleashed onto the world.
6. The fix to this is (probably) not trivial, but the current rendering is ugly, so this should be commented out before 1.23.
7. Should be done before 1.23: fix is simple and avoids plausible real-world problems.
8. Should be done before 1.23: these four have long-established groff aliases that do the right thing, while these fallbacks would degrade typography.
9. Should be done before 1.23: only degrades typography without any upside in real-world usage, since Symbol will always be there.
10. Not important to 1.23.
11. Not important to 1.23.
12. Probably not important to 1.23, mostly because there's no easy fix.  The potential pitfall to the user is a seemingly inexplicable character substitution, but U+2052 is obscure enough that I don't see much practical risk.

These opinions are correct and incontrovertible.  I have spoken.

original submission:

> The figure dash (\[u2016]) as currently designed works well with
> some figures but not others, in at least the TR font.  Consider
> the PostScript output of:
>
> echo '7\[u2012]5 6\[u2012]0' | groff

As the command implies, the figure dash is \[u2012], not \[u2016] as stated in the text above.

The point comment #10 made about \[u2011] equally applies to \[u2018] (left single quotation mark), \[u2019] (right single quotation mark), \[u201C] (left double quotation mark), and \[u201D] (right double quotation mark), all common characters that are frequently kerned next to certain uppercase letters.

I hate to take all these Unicode representations out of circulation until the thornier underlying problems can be solved.  However, they were in no previously released groff anyway, and at least with these four (unlike \[u2011]), groff has decades-established local aliases (\[oq], \[cq], \[lq], and \[rq]) that work correctly in all respects, so retaining the recently added, incorrectly kerned variants is of dubious value.

A possible refinement to U+203D:

The current definition is

.if t .fchar \[u203D] \o'?!'\" interrobang

This (theoretically) leaves the character undefined on the terminal--which it has been until recently, so not a huge deal--but as long as we're defining fallbacks anyway, the string "!?" is closest in spirit achievable without overstriking.  Thus the line could be:

.ie t .fchar \[u203D] \o'?!'\" interrobang
.el .fchar \[u203D] !?

HOWEVER.  Because of the ordering issue discussed in bug #63332, the ".if t" lines are not working correctly in nroff mode anyway, meaning that the character as currently defined is resulting in "!" being output on terminals.

I feel that, if we can only use one character to represent the interrobang, the "?" is a better choice than the "!" (as the "?" can change the meaning of a phrase, while the "!" typically only changes its level of emphasis).  So as long as fallbacks.tmac can't test what mode it's in, I'd recommend changing the order of the current definition, so that it reads:

.fchar \[u203D] \o'!?'\" interrobang

This won't change the typesetting at all, but will (IMHO) improve the terminal output a bit.

And in case anyone was concerned, U+2026 is the only \[u....] glyph in Symbol that fallbacks.tmac obscures:

$ egrep '^u[0-9A-F]{2}' font/devps/S | cut -f1,5
u2026   ellipsis
u2320   integraltp
u23AE   integralex
u2321   integralbt

Another addendum:

Unicode's U+200B ZERO WIDTH SPACE has the semantics of groff's \: escape.  However, the current definition for \[u200B] in fallbacks.tmac (\h'0') does not have U+200B's essential breaking property:

$ cat u200B-test
.ll 1p
ab\:cd
ab\h'0'cd
$ nroff -Ww u200B-test | cat -s
ab
cd
abcd

And sadly, while bug #58958 is unresolved, we can't do the obvious ".fchar \[u200B] \:".  (.tr likewise doesn't work, though one wouldn't expect it to, since its definitions are supposed to be glyphs, which \: isn't.)  I have no other suggestions for how to tie this functionality to \[u200B].

So perhaps the best thing here is to comment out the \[u200B] definition, with an additional comment saying it's awaiting a fix to #58958.  In the meantime, \[u200B] will be unrecognized, but that's already the status quo (in 1.22.4, and in git until less than a month ago), and is less misleading than implementing it in a noncompliant way.  (I'm not sure how, or whether, \h'0' effectively differs from \&.)

Um, you know that last comment where I said I was wrong?  Well, turns out I was wrong, and I was actually right.  (So, also wrong.)

What I missed is that whatever special fonts are in play have no bearing on the situation, because the clearly documented order of glyph-searching places .fchar definitions before special fonts.  So as long as the \[u2026] .fchar in fallbacks.tmac is in force, whatever is happening in special-font-land is irrelevant.

This is verifiable in practice.  Using the same input file from comment #2, this command run using the latest groff code produces this output:

$ groff -Z ellipsis_demo | sed -n '/font 11 S/,/x trailer/p'
x font 11 S
f11
V24000
H72000
t.
h1666
t.
h1666
t..
h1666
t.
h1666
t..
h1666
t.
h1666
t.
n12000 0
x trailer

Then I commented out the \[u2026] definition in fallbacks.tmac and ran it again.

$ groff -Z ellipsis_demo | sed -n '/font 11 S/,/x trailer/p'
x font 11 S
f11
V24000
H72000
Cu2026
h10000
Cu2026
h10000
Cu2026
h10000
n12000 0
x trailer

Here it's "easy" (if you're adept at reading intermediate output) to see that in the former case, groff is constructing the ellipsis from spaced periods, while in the latter, it's calling upon a character named u2026, which it's pulling from the current font, which is S.

This also explains why I was led astray in comment #2: I took the mere loading of font S as evidence that groff must be using that font's u2026--when in fact it was using that font's period, as a special-font fallback because the last requested font (ZD) also lacks a period glyph.

So please disregard comment #2.  The final example in the original submission is fine as-is, though the ".special" line is irrelevant (which is fine, since it doesn't do anything anyway: bug #63366 remains a valid point about special-font handling) and can be removed.

Ah, so U+2024 and U+2025 (the one- and two-dot leader characters) aren't intended for modern use; they exist only for compatibility with Xerox Character Code Standard 1980.

http://scripts.sil.org/CharStories_2024

I reckon I don't see any harm in leaving their fallback definitions in tmac/fallbacks.tmac, but I sure wouldn't sweat the details of how they render.