comment #28:
(concerning U+2016)

> I defined it as "\[ba]\[ba]" rather than "||", which I did
> because I had already translated | into something else.
> fallbacks.tmac might consider doing the same

Among its other concerns, bug #63354 now pursues this.

comment #26:

> My aforementioned footnote macro defined an ersatz U+2016
> character largely as you do here,

I say "largely" because I defined it as "\[ba]\[ba]" rather than "||", which I did because I had already translated | into something else.  fallbacks.tmac might consider doing the same (as I suggested (in comment #23) and you agreed with (in comment #24) for ` and ', for similar reasons).

comment #24:

> > > +.fchar \[u2016] || \" double vertical line (matrix norm)
> >
> > This one presents a kerning issue: if two U+2016s are set next to each other, they should have a little space between them.
>
> You've studied kerning a lot more than I have.

I don't know about that in general.  I happen to have in this instance, in implementing a macro to support the CMOS sequence of footnote symbols for tables (specified in section 3.79 in the 17th edition), which includes U+2016 fifth in the sequence of six symbols, and specifies doubling the symbols for the seventh and beyond.  So in the unfortunate table with eleven or more footnotes, two U+2016s will appear next to each other.  Arguably, for footnote identification, it matters little whether the reader sees a solid block of |||| rather than two groups of ||, but the latter I think has the aesthetic edge, and might even turn out to be helpful in the truly unfortunate table where three or four of them appear next to each other: "|| || ||" seems easier to distinguish from "|| || || ||" than "||||||" from "||||||||" (though at this point we're getting into absurdly unrealistic territory where there's no room for any table on the page with all those footnotes, and maybe a table isn't the best way to present this information after all).

> I'm not sure that kerns are even applied to fallback character definitions.

Not only aren't they, there's really no mechanism for them to be, since the only place groff allows defining kernpairs is in the font description file, which by definition cannot include fallback characters.  There's no user-level way to add kernpairs (see bug #44244).

My aforementioned footnote macro defined an ersatz U+2016 character largely as you do here, and included logic to add a thin space between adjacent ones, which was the only way to handle this (probably only theoretical) scenario because of this limitation.

> Or is that what you're proposing to work around?

At this stage I'm not proposing anything, just identifying potential problems.

> I am proposing to punt on this issue and see if the users complain, basically.

For an edge case like this, that's reasonable.

comment #23:

> A few nits about specific definitions:
>
> comment #21:
> > +.fchar \[u2000] \h'1n' \" en quad
> > +.fchar \[u2001] \h'1m' \" em quad
> > +.fchar \[u2002] \h'1n' \" en space
> > +.fchar \[u2003] \h'1m' \" em space
>
> As the "quad" and "space" forms are canonically equivalent (see http://www.unicode.org/mail-arch/unicode-ml/y2003-m04/0316.html), might it be better DRYwise to define one in terms of the other?

Indeed so; thanks for the reference.  Since we otherwise don't use the term "quad" (despite its impressive typographical pedigree), I think I'll alias the quads to the spaces.  Not like this should make much, if any, practical difference.

>
> > +.fchar \[u2016] || \" double vertical line (matrix norm)
>
> This one presents a kerning issue: if two U+2016s are set next to each other, they should have a little space between them.

You've studied kerning a lot more than I have.  I'm not sure that kerns are even applied to fallback character definitions.  Or is that what you're proposing to work around?

> (In typeset output I've found \| sufficient to make ersatz U+2016s defined as two U+007Cs not look like an unbroken row of bars.  In terminal (or any monospace) output, if the font has no U+2016, a full space becomes the only way to distinguish two U+2016s defined as above from four U+007Cs.)

I'm not sure we can achieve unambiguous typography on typesetters, let alone terminals.  More precisely, I am not sure that one can reliably infer input characters from rendered glyphs in the general case.  For one thing, fonts aren't under our control at all.  Secondly, the Unicode glyph confusability issue suggests that we couldn't overcome this problem even if we did have such control.

I am proposing to punt on this issue and see if the users complain, basically.
 

> > +.fchar \[u2018] ` \" left single quotation mark
> > +.fchar \[u2019] ' \" right single quotation mark
>
> Defining these as \[oq] and \[cq] seems more semantically meaningful (and less prone to failure, e.g. if the user has ".tr"ed ` to something else).

Quite right.  I'll do this.

> > +.fchar \[u201C] \[lq] \" left single quotation mark
> > +.fchar \[u201D] \[rq] \" right single quotation mark
>
> These definitions are fine, but the comments say "single" where they mean "double."

Whoops!  Thanks again.  Will fix.

> > +.fchar \[u2025] .\|. \" two dot leader
> > +.fchar \[u2026] .\|.\|. \" horizontal ellipsis
>
> With the internal space between the dots, consecutive versions of these will also typeset poorly.

I'm tempted to punt on this one, too.  Possibly no serious font for typesetting even needs to encode these characters, and in groff, if you want a well-typeset leader, a fundamental formatter feature (Control+A) will give you one of whatever length you like.

> > +.fchar \[u203D] \z?! \" interrobang
>
> Per bug #62983, this one might be sequestered behind an ".if t".

Another fair point.

This exercise has made me wonder if we could use a warning category (or some mechanism) to inform the user when fallbacks are used.

• The expanded Unicode coverage is great!
• fallbacks.tmac does seem like a better place for these definitions.
• I'm interested in your thoughts on the trade-off presented in comment #11, because it has bearing on some of your other proposed definitions here besides \[u2011]:
  • Other characters, such as single quotation marks, tend to be kerned in the font files, and thus would not be when using these aliases.
  • The various spaces implemented with \h'' will be nonbreaking.  In many cases, that's probably OK or even desired.  But in cases where it isn't, might it be better to warn the user that the character is unrecognized, rather than accept it but potentially typeset it in a subpar way?

Ah.  Yes.  I read that.  I just failed to put two and two together.  I'll try to keep up.

comment #17:

> Setting Severity to "Blocker" so that I don't forget about this.

You since unBlockered it, so, did you decide to forget about it after all, or did you come to a conclusion?

And while I have your attention:

Before 1.23.0, and potentially before rc2, there should be a decision about the question from comment #10.  Leaving the commit in place is a valid choice; I just want to make sure it is a deliberate choice rather than an oversight.

Per reasoning in comment #11, I still lean toward reverting the commit: I think it's better to give the user a warning and reject the character, than to handle it on some devices suboptimally.  After 1.23.0 we can revisit the issue and maybe come at it with fresh ideas.

Hi Dave,

comment #13:

> "The input sequence '\[u00A0]' is syntactically valid...but like '\[uFFFF]' and '\[u0000]', it's not meaningful"
>
> This is true of the current implementation but less true conceptually: U+0000 and U+FFFF are not meaningful input characters to groff, but U+00A0 is, and users ideally ought to be able to specify the character as \[u00A0].

I guess what we need to do here is be more clear whether \[u...] escape sequences are intended to represent input characters, or desired output glyphs.  The distinction comes into sharp relief here.

If the former, then you're right--\[u...] is a way of getting around groff's narrow-character input interpretation.

But if the latter, then there are many other Unicode code points that don't represent things we can ask a font to draw for us.  In groff, with two exceptions, if an ordinary or special character doesn't correspond to something that "puts ink on the page", it isn't a glyph.  In device-independent output, spaces of all kinds are represented with horizontal or vertical motions, not glyph-writing commands.

The two exceptions are only partial ones: \| and \^.  One might interpret them as special character escape sequences like \- or \_, but they aren't.  They become horizontal motions.  It is true that these two sequences can be defined as if they were special characters in font description files to customize their widths (groff_font(5) discusses this).  With my present understanding, I think this is a bit of a bodge, and as far as I've seen, no one has ever taken advantage of this configurability.  (Possibly because *roff font description files have been regarded as a bit esoteric.)  This approach does not generalize well to the many additional space code points in Unicode.  U+2001, "EM QUAD", is just one example.

Maybe there's a good reason to have \^ and \| customizable in this way--I don't feel like I have a command of the history of this topic.

If you or anybody has some knowledge to bring to light here, I'd appreciate it!

>
> But this is an edge case I don't intend to pursue.  Users who want to stick to pure-ASCII input have the escape sequence \~ to specify the nonbreaking space, so don't need the alternate spelling \[u00A0].

The test case in comment #9 is probably the best way to see the kerning deficiency.

The test cases presented in the 86b99bdbf58c8dd1a4036f4004a6d8518a5b8357 commit message don't show the kerning issue, because font TR has no "a-" kernpair.  But it does have a "Y-" one, so changing these test cases' .ds lines to use the letter Y before the hyphen or the "\[u2011]" ought to show the difference in kerning.  And it does... but then the second test case bumps up against bug #57448, and thus fails to show the difference in breaking behavior.

Hi Dave,

Something I'm not clear on.  Do you think 86b99bdbf58c8dd1a4036f4004a6d8518a5b8357 should be reverted?

comment #2:

> Unicode considers U+2009 THIN SPACE and U+200A HAIR SPACE breakable...
> Groff... does not offer breaking versions of these spaces, and the only
> reason to add them would be strict compliance with a Unicode property
> that probably no one who uses those code points actually wants

I believe my reasoning here was inaccurate.  Although Unicode allows breaking at a thin space or hair space, it does not require it,* so groff declining to treat these as break points does not violate Unicode compliance at all.  Thus I now propose that U+2009 THIN SPACE be mapped to groff's (nonbreaking) \|, and U+200A HAIR SPACE to groff's (nonbreaking) \^.

* The gory details: Unicode line breaking is covered in "Unicode Standard Annex #14: Unicode Line Breaking Algorithm" (http://www.unicode.org/reports/tr14/tr14-45.html), whose introductory section makes its scope clear: "Given an input text, [this algorithm] produces a set of positions called 'break opportunities' that are appropriate points to begin a new line. The selection of actual line break positions from the set of break opportunities is not covered by the Unicode Line Breaking Algorithm, but is in the domain of higher level software."  Groff declining to break at points that Unicode specifies as "break opportunities" is perfectly in line with this.

comment #5:

> On further investigation, it appears in fact to be 0% accurate.  See bug #58962.

groff_char(7) is full of problems with accuracy.

It's on my (s)hit list.  I recently fixed up the introductory material but it needs a lot more work.

comment #2:

> "\~" and "\ " shouldn't be equivalent; they're documented as behaving differently.

No, not suggesting they should, just lamenting the total disjunctivity of the set.

>
> The input string "\[u00A0]" being equivalent to neither of these is exactly the problem this plank of this bug report is looking to solve.
>
> It's only the character NO-BREAK SPACE in its Latin-1 form, which groff accepts as direct input, that groff recognizes and interprets as a nonbreaking space.  groff_char(7) (which I only now thought to check) says it maps to \~.  But that appears to be less than 100% accurate:
>

> $ LC_CTYPE=en_US.iso88591 printf ".if '\u00A0'\~' .tm equal\n" | groff
> $

>
> But the upshot is, however groff interprets a Latin-1 A0, it really ought to interpret the form of that character emitted by preconv, \[u00A0], identically.

Yes, I think I agree here.  I can't think of a more appropriate mapping for it.

> So the only one I didn't cover was U+2007 FIGURE SPACE, which should map to groff's (already nonbreaking) \0.

Might as well sweep that one into this report, then.  Once the "where" to fix this has been determined, the incremental effort to handle that one will probably be tiny.

> > there are bunch of others (hair space, thin space, ideographic space,
> > ...) but I don't know what their breaking semantics are in Unicode.
>
> Irrational, IMO.  Unicode considers U+2009 THIN SPACE and
> U+200A HAIR SPACE breakable, for no good reason that I can see.  Groff (quite sensibly, since the concept is sort of absurd) does not offer breaking versions of these spaces, and the only reason to add them would be strict compliance with a Unicode property that probably no one who uses those code points actually wants: I can't think of a single real-world use case for a breaking thin space (though perhaps this is merely a failure of my imagination).

Well, I can't think of one either.
 

> This is all another can of worms I intentionally didn't address in what I intended to be a simple change.

Hah.  This is Sparta^Wgroff!  Complexity rapidly ramifies.

> > 4. A non-breaking hyphen would then be something that looks
> > like \[hy] but doesn't actually break?
>
> Yes.
>
> > You can just use the character as-is in input.
>
> Ah, I guess you used -Tutf8 output, where that does work.  (Somehow your groff command got stripped from your comment.)

The "somehow" was me not thinking to include it.

> All other output formats (notably -Tps and -Tpdf) produce "warning: can't find special character 'u2011'".

Okay, yes, that seems like another mapping issue.

And it appears to be a one-liner fix (morally).

tmac/pdf.tmac sources tmac/ps.tmac so the fix only has to be made in one place.

I did have to goose the loop count in the test up to 100.

$ ./build/test-groff -Tps ./EXPERIMENTS/non-breaking-hyphen.groff  >|/tmp/2011.ps
troff: warning [p 1, 0.0i]: can't break line
$ ./build/test-groff -Tpdf ./EXPERIMENTS/non-breaking-hyphen.groff  >|/tmp/2011.pdf
troff: warning [p 1, 0.0i]: can't break line
$ cat ./EXPERIMENTS/non-breaking-hyphen.groff
.pl 1v
.ds a a\[u2011]
.nr b 100 -1
.while \n+b \*a\c
$ git di tmac/ps.tmac
diff --git a/tmac/ps.tmac b/tmac/ps.tmac
index 18928765..860919e1 100644
--- a/tmac/ps.tmac
+++ b/tmac/ps.tmac
@@ -28,6 +28,9 @@
 .
 .cflags 8 \[an]
 .
+\# non-breaking hyphen
+.fchar \[u2011] -
+.
 .char \[radicalex] \h'-\w'\[sr]'u'\[radicalex]\h'\w'\[sr]'u'
 .fchar \[sqrtex] \[radicalex]
 .char \[mo] \h'.08m'\[mo]\h'-.08m'

comment #1:

> 1. "U+00A0 NO-BREAK SPACE
>
> None of these are equivalent to the others. :-/

"\~" and "\ " shouldn't be equivalent; they're documented as behaving differently.

The input string "\[u00A0]" being equivalent to neither of these is exactly the problem this plank of this bug report is looking to solve.

It's only the character NO-BREAK SPACE in its Latin-1 form, which groff accepts as direct input, that groff recognizes and interprets as a nonbreaking space.  groff_char(7) (which I only now thought to check) says it maps to \~.  But that appears to be less than 100% accurate:

$ LC_CTYPE=en_US.iso88591 printf ".if '\u00A0'\~' .tm equal\n" | groff
$

But the upshot is, however groff interprets a Latin-1 A0, it really ought to interpret the form of that character emitted by preconv, \[u00A0], identically.

> 2. The behavior of \: when used as the RHS of a .char request
> does indeed seem a bit strange.

Yeah, I really need to open a separate bug report for this, because it's unrelated to everything else here.

> 3. Narrow no-break space.  Have you named all of the non-breaking
> spaces in Unicode in this ticket?

No.  I was intentionally trying to keep it simple and minimal.  But it turns out there are only three:

http://en.wikipedia.org/wiki/Whitespace_character#Unicode

So the only one I didn't cover was U+2007 FIGURE SPACE, which should map to groff's (already nonbreaking) \0.

> there are bunch of others (hair space, thin space, ideographic space,
> ...) but I don't know what their breaking semantics are in Unicode.

Irrational, IMO.  Unicode considers U+2009 THIN SPACE and
U+200A HAIR SPACE breakable, for no good reason that I can see.  Groff (quite sensibly, since the concept is sort of absurd) does not offer breaking versions of these spaces, and the only reason to add them would be strict compliance with a Unicode property that probably no one who uses those code points actually wants: I can't think of a single real-world use case for a breaking thin space (though perhaps this is merely a failure of my imagination).

This is all another can of worms I intentionally didn't address in what I intended to be a simple change.

> 4. A non-breaking hyphen would then be something that looks
> like \[hy] but doesn't actually break?

Yes.

> You can just use the character as-is in input.

Ah, I guess you used -Tutf8 output, where that does work.  (Somehow your groff command got stripped from your comment.)  All other output formats (notably -Tps and -Tpdf) produce "warning: can't find special character 'u2011'".

One small change that would improve groff's Unicode support would be to recognize Unicode versions of things groff already knows how to do.

Four examples:

U+00A0 NO-BREAK SPACE

This character is in the Latin-1 character set, which groff recognizes, and when groff's input is in Latin-1 encoding, it correctly handles this character (though I'm not certain whether it interprets it as "\~" or "\ ").

But if the input is some other encoding, preconv converts the character into the string "\[u00A0]", which groff does not recognize.  In macro space, a simple

.char \[u00A0] \~

is enough to take care of this; presumably the equivalent mechanism to make the code handle it internally is just as simple.

U+200B ZERO WIDTH SPACE

This is another character implemented in an existing groff escape (\:) but unrecognized as "\[u200B]".

In this case, the simple, obvious, elegant solution that worked above:

.char \[u200B] \:

stupidly, irritatingly, and undocumentedly doesn't work.  (.char being unable to map something to an escape, or at least to this particular escape, is another bug--either in the implementation, or the lack of documentation of the restriction--for another day.)

U+202F NARROW NO-BREAK SPACE

Groff has two nonbreaking thin spaces, \| and \^.  It is perhaps unclear which of these groff should map "\[u202F]" to, but either one would be an improvement over its current mapping to the warning "can't find special character `u202F'".

U+2011 NON-BREAKING HYPHEN

I deem this change "extra credit" as it's the least likely to be easily implementable, groff syntax having no direct correlate.  Groff can only (via \%) make an entire "word" (sequence of non-whitespace, including hyphens) unbreakable, but has no easy way to support a mix of breaking and nonbreaking hyphens in the same word, such as making the first hyphen of "jack-in-the-box" nonbreaking but the other two breakable.  (This can be done with a mix of \% and \: escapes, as "\%jack-in-\:the-\:box" -- or even, taking advantage of the bug/quirk Branden discovered, as "\%jack-in-\:the-box" -- but this is not obvious.)  So it's possible, but convoluted, to represent "jack\[u2011]in-the-box" in groff syntax; whether this means it's equally convoluted in the underlying code, or whether the code actually does have the concept of a nonbreaking hyphen but just doesn't expose a direct representation of it to user space, I cannot guess.