bugGNU roff - Bugs: bug #58930, take baby steps toward Unicode

 
 

bug #58930: take baby steps toward Unicode

Submitter:  Dave <barx>
Submitted:  Mon 10 Aug 2020 02:56:06 PM UTC
   
 
Category:  Core Severity:  3 - Normal
Item Group:  Feature change Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.23.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 14 Nov 2022 03:49:19 AM UTC, comment #33: 

This bug has now spawned its fifth spinoff: I opened bug #63360 to document the concern raised here in comment #9.

Dave <barx>
Group Member
Sat 12 Nov 2022 11:11:53 PM UTC, comment #32: 

comment #28:
(concerning U+2016)

> I defined it as "\[ba]\[ba]" rather than "||", which I did
> because I had already translated | into something else.
> fallbacks.tmac might consider doing the same


Among its other concerns, bug #63354 now pursues this.

Dave <barx>
Group Member
Mon 07 Nov 2022 08:24:17 PM UTC, comment #31: 

Well, it also seemed cleaner to give that particular problem its own space rather than grafting any ensuing discussion about it onto a bug that already had 30 comments.

Dave <barx>
Group Member
Mon 07 Nov 2022 04:32:12 PM UTC, comment #30: 

Thanks, Dave.  I hadn't forgotten, but there was a risk that I would.  :)

G. Branden Robinson <gbranden>
Group administrator
Mon 07 Nov 2022 03:54:39 PM UTC, comment #29: 

comment #27:

> This can be a new bug report if that makes more sense.


Now bug #63332.

Dave <barx>
Group Member
Mon 31 Oct 2022 06:13:49 AM UTC, comment #28: 

comment #26:

> My aforementioned footnote macro defined an ersatz U+2016
> character largely as you do here,


I say "largely" because I defined it as "\[ba]\[ba]" rather than "||", which I did because I had already translated | into something else.  fallbacks.tmac might consider doing the same (as I suggested (in comment #23) and you agreed with (in comment #24) for ` and ', for similar reasons).

Dave <barx>
Group Member
Wed 26 Oct 2022 03:19:57 AM UTC, comment #27: 

Hmm, the fallbacks.tmac changes in the resolving commit are having an effect on ASCII output I can't (yet) explain.

If I run a groff without this commit:

$ echo 'I \[dg] you' | groff-latest -Tascii -mtty-char | cat -s
I <*> you

This is the expected output: tty-char.tmac defines \[dg] as output above.

But applying the commit 132182bd7 changes to fallbacks.tmac makes this go awry:

$ echo 'I \[dg] you' | groff-latest -Tascii -mtty-char | cat -s
troff:<standard input>:1: warning: special character 'dg' not defined
I  you

I don't understand this, because while the updated fallbacks.tmac defines a new character as \[dg], it does (or should do) nothing to change the definition of \[dg] itself.  The same is true of \[dd], which shows the same effect.

This can be a new bug report if that makes more sense.

Dave <barx>
Group Member
Tue 25 Oct 2022 05:11:57 PM UTC, comment #26: 

comment #24:

> > > +.fchar \[u2016] || \" double vertical line (matrix norm)
> >
> > This one presents a kerning issue: if two U+2016s are set next to each other, they should have a little space between them.
>
> You've studied kerning a lot more than I have.


I don't know about that in general.  I happen to have in this instance, in implementing a macro to support the CMOS sequence of footnote symbols for tables (specified in section 3.79 in the 17th edition), which includes U+2016 fifth in the sequence of six symbols, and specifies doubling the symbols for the seventh and beyond.  So in the unfortunate table with eleven or more footnotes, two U+2016s will appear next to each other.  Arguably, for footnote identification, it matters little whether the reader sees a solid block of |||| rather than two groups of ||, but the latter I think has the aesthetic edge, and might even turn out to be helpful in the truly unfortunate table where three or four of them appear next to each other: "|| || ||" seems easier to distinguish from "|| || || ||" than "||||||" from "||||||||" (though at this point we're getting into absurdly unrealistic territory where there's no room for any table on the page with all those footnotes, and maybe a table isn't the best way to present this information after all).

> I'm not sure that kerns are even applied to fallback character definitions.


Not only aren't they, there's really no mechanism for them to be, since the only place groff allows defining kernpairs is in the font description file, which by definition cannot include fallback characters.  There's no user-level way to add kernpairs (see bug #44244).

My aforementioned footnote macro defined an ersatz U+2016 character largely as you do here, and included logic to add a thin space between adjacent ones, which was the only way to handle this (probably only theoretical) scenario because of this limitation.

> Or is that what you're proposing to work around?


At this stage I'm not proposing anything, just identifying potential problems.

> I am proposing to punt on this issue and see if the users complain, basically.


For an edge case like this, that's reasonable.

Dave <barx>
Group Member
Mon 24 Oct 2022 02:30:03 AM UTC, comment #25: 


commit 132182bd714a7b327bae1627831836cd1484522b
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sun Oct 23 14:54:45 2022 -0500

    [tmac]: Fix Savannah #58930.

    * tmac/fallbacks.tmac: Define fallback characters for many code points
      from the Unicode General Punctuation Block.  This is to ease migration
      from other documentation formats; in many cases there are more
      idiomatic *roff ways of getting results that typeset better and are
      amenable to fine tuning.  For instance, Unicode defines spaces and
      dashes of various discrete widths; when typesetting with *roff, you
      can have a space or dash (among other things) of any length achievable
      by the output device.
    * tmac/ps.tmac: Migrate non-breaking hyphen fallback character
      definition from here to the foregoing.

    Fixes <https://savannah.gnu.org/?58930>.  Thanks to Dave Kemper for the
    report and discussion.


G. Branden Robinson <gbranden>
Group administrator
Sun 23 Oct 2022 07:16:31 PM UTC, comment #24: 


comment #23:

> A few nits about specific definitions:
>
> comment #21:
> > +.fchar \[u2000] \h'1n' \" en quad
> > +.fchar \[u2001] \h'1m' \" em quad
> > +.fchar \[u2002] \h'1n' \" en space
> > +.fchar \[u2003] \h'1m' \" em space
>
> As the "quad" and "space" forms are canonically equivalent (see http://www.unicode.org/mail-arch/unicode-ml/y2003-m04/0316.html), might it be better DRYwise to define one in terms of the other?


Indeed so; thanks for the reference.  Since we otherwise don't use the term "quad" (despite its impressive typographical pedigree), I think I'll alias the quads to the spaces.  Not like this should make much, if any, practical difference.

>
> > +.fchar \[u2016] || \" double vertical line (matrix norm)
>
> This one presents a kerning issue: if two U+2016s are set next to each other, they should have a little space between them.


You've studied kerning a lot more than I have.  I'm not sure that kerns are even applied to fallback character definitions.  Or is that what you're proposing to work around?

> (In typeset output I've found \| sufficient to make ersatz U+2016s defined as two U+007Cs not look like an unbroken row of bars.  In terminal (or any monospace) output, if the font has no U+2016, a full space becomes the only way to distinguish two U+2016s defined as above from four U+007Cs.)


I'm not sure we can achieve unambiguous typography on typesetters, let alone terminals.  More precisely, I am not sure that one can reliably infer input characters from rendered glyphs in the general case.  For one thing, fonts aren't under our control at all.  Secondly, the Unicode glyph confusability issue suggests that we couldn't overcome this problem even if we did have such control.

I am proposing to punt on this issue and see if the users complain, basically.
 

> > +.fchar \[u2018] ` \" left single quotation mark
> > +.fchar \[u2019] ' \" right single quotation mark
>
> Defining these as \[oq] and \[cq] seems more semantically meaningful (and less prone to failure, e.g. if the user has ".tr"ed ` to something else).


Quite right.  I'll do this.

> > +.fchar \[u201C] \[lq] \" left single quotation mark
> > +.fchar \[u201D] \[rq] \" right single quotation mark
>
> These definitions are fine, but the comments say "single" where they mean "double."


Whoops!  Thanks again.  Will fix.

> > +.fchar \[u2025] .\|. \" two dot leader
> > +.fchar \[u2026] .\|.\|. \" horizontal ellipsis
>
> With the internal space between the dots, consecutive versions of these will also typeset poorly.


I'm tempted to punt on this one, too.  Possibly no serious font for typesetting even needs to encode these characters, and in groff, if you want a well-typeset leader, a fundamental formatter feature (Control+A) will give you one of whatever length you like.

> > +.fchar \[u203D] \z?! \" interrobang
>
> Per bug #62983, this one might be sequestered behind an ".if t".


Another fair point.

This exercise has made me wonder if we could use a warning category (or some mechanism) to inform the user when fallbacks are used.

G. Branden Robinson <gbranden>
Group administrator
Sat 15 Oct 2022 03:16:44 AM UTC, comment #23: 

A few nits about specific definitions:

comment #21:

> +.fchar \[u2000] \h'1n' \" en quad
> +.fchar \[u2001] \h'1m' \" em quad
> +.fchar \[u2002] \h'1n' \" en space
> +.fchar \[u2003] \h'1m' \" em space


As the "quad" and "space" forms are canonically equivalent (see http://www.unicode.org/mail-arch/unicode-ml/y2003-m04/0316.html), might it be better DRYwise to define one in terms of the other?

> +.fchar \[u2016] || \" double vertical line (matrix norm)


This one presents a kerning issue: if two U+2016s are set next to each other, they should have a little space between them.  (In typeset output I've found \| sufficient to make ersatz U+2016s defined as two U+007Cs not look like an unbroken row of bars.  In terminal (or any monospace) output, if the font has no U+2016, a full space becomes the only way to distinguish two U+2016s defined as above from four U+007Cs.)

> +.fchar \[u2018] ` \" left single quotation mark
> +.fchar \[u2019] ' \" right single quotation mark


Defining these as \[oq] and \[cq] seems more semantically meaningful (and less prone to failure, e.g. if the user has ".tr"ed ` to something else).

> +.fchar \[u201C] \[lq] \" left single quotation mark
> +.fchar \[u201D] \[rq] \" right single quotation mark


These definitions are fine, but the comments say "single" where they mean "double."

> +.fchar \[u2025] .\|. \" two dot leader
> +.fchar \[u2026] .\|.\|. \" horizontal ellipsis


With the internal space between the dots, consecutive versions of these will also typeset poorly.

> +.fchar \[u203D] \z?! \" interrobang


Per bug #62983, this one might be sequestered behind an ".if t".

Dave <barx>
Group Member
Sat 15 Oct 2022 01:45:37 AM UTC, comment #22: 
  • The expanded Unicode coverage is great!
  • fallbacks.tmac does seem like a better place for these definitions.
  • I'm interested in your thoughts on the trade-off presented in comment #11, because it has bearing on some of your other proposed definitions here besides \[u2011]:
    • Other characters, such as single quotation marks, tend to be kerned in the font files, and thus would not be when using these aliases.
    • The various spaces implemented with \h'' will be nonbreaking.  In many cases, that's probably OK or even desired.  But in cases where it isn't, might it be better to warn the user that the character is unrecognized, rather than accept it but potentially typeset it in a subpar way?
Dave <barx>
Group Member
Thu 13 Oct 2022 06:05:39 PM UTC, comment #21: 

Dave, what do you think of this?


diff --git a/tmac/fallbacks.tmac b/tmac/fallbacks.tmac
index 428aaa2b7..02ac998a2 100644
--- a/tmac/fallbacks.tmac
+++ b/tmac/fallbacks.tmac
@@ -142,6 +142,49 @@
 .fchar \[oe] oe
 .fchar \[:Y] \z\[ad]Y
 .
+.fchar \[u2000] \h'1n' \" en quad
+.fchar \[u2001] \h'1m' \" em quad
+.fchar \[u2002] \h'1n' \" en space
+.fchar \[u2003] \h'1m' \" em space
+.fchar \[u2004] \h'(1/3u)m' \" three-per-em space
+.fchar \[u2005] \h'(1/4u)m' \" four-per-em space
+.fchar \[u2006] \h'(1/6u)m' \" six-per-em space
+.fchar \[u2007] \0 \" figure space
+.fchar \[u2008] \^ \" puncutation space
+.fchar \[u2009] \| \" thin space
+.fchar \[u200A] \^ \" hair space
+.fchar \[u200B] \h'0' \" zero-width space
+.fchar \[u2010] -\: \" hyphen
+.fchar \[u2011] - \" non-breaking hyphen (won't break w/o .hcode or \:)
+.ie t .fchar \[u2012] \v'-.3m'\l'\0\[ru]'\v'+.3m' \" figure dash
+.el   .fchar \[u2012] \-
+.fchar \[u2013] \[en] \" en dash
+.fchar \[u2014] \[em] \" em dash
+.fchar \[u2015] \[em] \" horizontal bar (quotation dash)
+.fchar \[u2016] || \" double vertical line (matrix norm)
+.if t .fchar \[u2017] \Z'\[ul]'\v'+.1m'\[ul]\v'-.1m' \" double low line
+.fchar \[u2018] ` \" left single quotation mark
+.fchar \[u2019] ' \" right single quotation mark
+.fchar \[u201C] \[lq] \" left single quotation mark
+.fchar \[u201D] \[rq] \" right single quotation mark
+.fchar \[u2020] \[dg] \" dagger
+.fchar \[u2021] \[dd] \" double dagger
+.fchar \[u2022] \[bu] \" bullet
+.fchar \[u2024] . \" one dot leader
+.fchar \[u2025] .\|. \" two dot leader
+.fchar \[u2026] .\|.\|. \" horizontal ellipsis
+.fchar \[u2027] \[pc] \" hyphenation point
+.fchar \[u2030] \[%0] \" per mille sign
+.fchar \[u2032] \[fm] \" prime
+.fchar \[u2033] \[sd] \" double prime
+.fchar \[u2039] \[fo] \" left single chevron
+.fchar \[u203A] \[fc] \" right single chevron
+.fchar \[u203D] \z?! \" interrobang
+.fchar \[u203E] \[rn] \" overline
+.fchar \[u2044] \[f/] \" fraction slash
+.fchar \[u2052] % \" commercial minus sign
+.fchar \[u2053] \[ti] \" swung dash
+.
 .cp \n[*groff_fallbacks_tmac_C]
 .do rr *groff_fallbacks_tmac_C
 .
diff --git a/tmac/ps.tmac b/tmac/ps.tmac
index 0f512b42a..9e59c571b 100644
--- a/tmac/ps.tmac
+++ b/tmac/ps.tmac
@@ -28,9 +28,6 @@
 .
 .cflags 8 \[an]
 .
-\# non-breaking hyphen
-.fchar \[u2011] -
-.
 .char \[radicalex] \h'-\w'\[sr]'u'\[radicalex]\h'\w'\[sr]'u'
 .fchar \[sqrtex] \[radicalex]
 .char \[mo] \h'.08m'\[mo]\h'-.08m'


This doesn't help with kerning one little bit, I acknowledge.  But that is a much thornier problem, and this one seems responsive to the "Unicode baby steps" topic of this ticket.

G. Branden Robinson <gbranden>
Group administrator
Sat 08 Oct 2022 05:59:15 PM UTC, comment #20: 

Ah.  Yes.  I read that.  I just failed to put two and two together.  I'll try to keep up.

Dave <barx>
Group Member
Sat 08 Oct 2022 09:11:56 AM UTC, comment #19: 

comment #18:

> comment #17:
> > Setting Severity to "Blocker" so that I don't forget about this.
>
> You since unBlockered it, so, did you decide to forget about it after all, or did you come to a conclusion?


Neither.  I have an easier way to track "release goals" now.  (Well, I always had one, but now I have an easier way to share them with other people than passing a mile-long URL around in email.)

A person can use the "Backlog" query form I created recently and set "Planned Release" to 1.23.0.

I made some noise about this on the groff development list this month.  :)

G. Branden Robinson <gbranden>
Group administrator
Sat 08 Oct 2022 08:35:10 AM UTC, comment #18: 

comment #17:

> Setting Severity to "Blocker" so that I don't forget about this.


You since unBlockered it, so, did you decide to forget about it after all, or did you come to a conclusion?

Dave <barx>
Group Member
Thu 02 Jun 2022 12:56:45 AM UTC, comment #17: 


comment #16:

> And while I have your attention:
>
> Before 1.23.0, and potentially before rc2, there should be a decision about the question from comment #10.  Leaving the commit in place is a valid choice; I just want to make sure it is a deliberate choice rather than an oversight.
>
> Per reasoning in comment #11, I still lean toward reverting the commit: I think it's better to give the user a warning and reject the character, than to handle it on some devices suboptimally.  After 1.23.0 we can revisit the issue and maybe come at it with fresh ideas.


Setting Severity to "Blocker" so that I don't forget about this.

G. Branden Robinson <gbranden>
Group administrator
Mon 30 May 2022 12:04:21 AM UTC, comment #16: 

And while I have your attention:

Before 1.23.0, and potentially before rc2, there should be a decision about the question from comment #10.  Leaving the commit in place is a valid choice; I just want to make sure it is a deliberate choice rather than an oversight.

Per reasoning in comment #11, I still lean toward reverting the commit: I think it's better to give the user a warning and reject the character, than to handle it on some devices suboptimally.  After 1.23.0 we can revisit the issue and maybe come at it with fresh ideas.

Dave <barx>
Group Member
Sun 29 May 2022 11:00:18 PM UTC, comment #15: 

comment #14:

> I guess what we need to do here is be more clear whether
> \[u...] escape sequences are intended to represent input
> characters, or desired output glyphs.


Fair point.  I don't have an opinion either way.

But the answer has bearing on how the U+202F NARROW NO-BREAK SPACE part of the original submission should be handled.  It could be a change only to preconv, paralleling the fix for bug #62300, making preconv convert a U+202F on input to one of groff's thin-space escapes on output.  Or groff itself could recognize \[u202F], which is what preconv currently outputs for this character.

$ LC_CTYPE=en_US.utf8 printf '\u202F\n' | preconv -eutf8
.lf 1 -
\[u202F]


> Maybe there's a good reason to have \^ and \| customizable
> in this way--I don't feel like I have a command of the history
> of this topic.


I don't know the history or the reasoning either, but, as you allude, there is at least a justification for it in those two cases, where Unicode does not specify a width.

Dave <barx>
Group Member
Sat 28 May 2022 06:52:58 PM UTC, comment #14: 

Hi Dave,

comment #13:

> "The input sequence '\[u00A0]' is syntactically valid...but like '\[uFFFF]' and '\[u0000]', it's not meaningful"
>
> This is true of the current implementation but less true conceptually: U+0000 and U+FFFF are not meaningful input characters to groff, but U+00A0 is, and users ideally ought to be able to specify the character as \[u00A0].


I guess what we need to do here is be more clear whether \[u...] escape sequences are intended to represent input characters, or desired output glyphs.  The distinction comes into sharp relief here.

If the former, then you're right--\[u...] is a way of getting around groff's narrow-character input interpretation.

But if the latter, then there are many other Unicode code points that don't represent things we can ask a font to draw for us.  In groff, with two exceptions, if an ordinary or special character doesn't correspond to something that "puts ink on the page", it isn't a glyph.  In device-independent output, spaces of all kinds are represented with horizontal or vertical motions, not glyph-writing commands.

The two exceptions are only partial ones: \| and \^.  One might interpret them as special character escape sequences like \- or \_, but they aren't.  They become horizontal motions.  It is true that these two sequences can be defined as if they were special characters in font description files to customize their widths (groff_font(5) discusses this).  With my present understanding, I think this is a bit of a bodge, and as far as I've seen, no one has ever taken advantage of this configurability.  (Possibly because *roff font description files have been regarded as a bit esoteric.)  This approach does not generalize well to the many additional space code points in Unicode.  U+2001, "EM QUAD", is just one example.

Maybe there's a good reason to have \^ and \| customizable in this way--I don't feel like I have a command of the history of this topic.

If you or anybody has some knowledge to bring to light here, I'd appreciate it!


>
> But this is an edge case I don't intend to pursue.  Users who want to stick to pure-ASCII input have the escape sequence \~ to specify the nonbreaking space, so don't need the alternate spelling \[u00A0].

G. Branden Robinson <gbranden>
Group administrator
Sat 28 May 2022 05:35:55 PM UTC, comment #13: 

original submission:

> But if the input is some other encoding, preconv converts
> the character into the string "\[u00A0]", which groff does
> not recognize.


The resolved bug #62300 has fixed preconv to emit "\~" rather than "\[u00A0]" for a U+00A0 input character.

In preconv 1.22.4:

$ echo -e '\xA0' | preconv -elatin1
.lf 1 -
\[u00A0]

In preconv built from the latest code:

$ echo -e '\xA0' | preconv -elatin1
.lf 1 -
\~


So I think we can mark this part as resolved, despite one remaining issue 62300 points out in its comment 2:

"The input sequence '\[u00A0]' is syntactically valid...but like '\[uFFFF]' and '\[u0000]', it's not meaningful"

This is true of the current implementation but less true conceptually: U+0000 and U+FFFF are not meaningful input characters to groff, but U+00A0 is, and users ideally ought to be able to specify the character as \[u00A0].

But this is an edge case I don't intend to pursue.  Users who want to stick to pure-ASCII input have the escape sequence \~ to specify the nonbreaking space, so don't need the alternate spelling \[u00A0].

Dave <barx>
Group Member
Mon 12 Oct 2020 10:28:35 AM UTC, comment #12: 

The test case in comment #9 is probably the best way to see the kerning deficiency.

The test cases presented in the 86b99bdbf58c8dd1a4036f4004a6d8518a5b8357 commit message don't show the kerning issue, because font TR has no "a-" kernpair.  But it does have a "Y-" one, so changing these test cases' .ds lines to use the letter Y before the hyphen or the "\[u2011]" ought to show the difference in kerning.  And it does... but then the second test case bumps up against bug #57448, and thus fails to show the difference in breaking behavior.

Dave <barx>
Group Member
Mon 12 Oct 2020 09:28:18 AM UTC, comment #11: 

It's a judgement call, certainly.  Is a change that allows groff to handle U+2011 on some output devices but incorrectly kerns it better or worse than having groff emit a warning and discard the character?  In the latter case, the user has a clear sign that his input is not understood.  In the former, the input is accepted for PostScript/PDF but produces subpar typography and gives the user no sign that anything is amiss (though it would at least not be the only miskerned character in Unicode's General Punctuation block: see bug #58897).

My perfectionist tendencies argue for either implementing a solution that is 100% correct (as far as testing can demonstrate anyway) or bluntly telling the user "sorry, don't know how to do that."  But an 80%-correct solution is not without merit.

Dave <barx>
Group Member
Thu 08 Oct 2020 12:04:23 PM UTC, comment #10: 

Hi Dave,

Something I'm not clear on.  Do you think 86b99bdbf58c8dd1a4036f4004a6d8518a5b8357 should be reverted?

G. Branden Robinson <gbranden>
Group administrator
Wed 23 Sep 2020 03:57:35 AM UTC, comment #9: 

comment #4:

> And it appears to be a one-liner fix (morally).


Turns out another reason it's not quite that simple is this caveat from the info manual (which is documented, so it can't be a bug):

"Only the current font is checked for... kerns; neither special fonts nor entities defined with the char request (and its siblings) are taken into account."

Thus this one-line solution results in incorrect kerning with any characters adjacent to the \[u2011].


.nf
.ps 64
.vs 64
.sp
.fchar \[u2011] -
MAY\[u2011]DAY
MAY-DAY


Dave <barx>
Group Member
Thu 20 Aug 2020 05:23:18 AM UTC, comment #8: 

comment #2:

> Unicode considers U+2009 THIN SPACE and U+200A HAIR SPACE breakable...
> Groff... does not offer breaking versions of these spaces, and the only
> reason to add them would be strict compliance with a Unicode property
> that probably no one who uses those code points actually wants


I believe my reasoning here was inaccurate.  Although Unicode allows breaking at a thin space or hair space, it does not require it,* so groff declining to treat these as break points does not violate Unicode compliance at all.  Thus I now propose that U+2009 THIN SPACE be mapped to groff's (nonbreaking) \|, and U+200A HAIR SPACE to groff's (nonbreaking) \^.

* The gory details: Unicode line breaking is covered in "Unicode Standard Annex #14: Unicode Line Breaking Algorithm" (http://www.unicode.org/reports/tr14/tr14-45.html), whose introductory section makes its scope clear: "Given an input text, [this algorithm] produces a set of positions called 'break opportunities' that are appropriate points to begin a new line. The selection of actual line break positions from the set of break opportunities is not covered by the Unicode Line Breaking Algorithm, but is in the domain of higher level software."  Groff declining to break at points that Unicode specifies as "break opportunities" is perfectly in line with this.

Dave <barx>
Group Member
Sun 16 Aug 2020 12:53:18 AM UTC, comment #7: 

comment #4:

>  just lamenting the total disjunctivity of the set.


That two of the three, intended to serve different purposes, are disjunct seems more laudable than lamentable.  But I'm not here to police your feelings.

> I can't think of a more appropriate mapping for it.


Well, if there were a more appropriate mapping for \[u00A0], that mapping should also apply to the Latin-1 A0.  They're the same character, just with different input representations.

Speaking more generally, for a Latin-1 input file, "groff latin1.txt" and "groff -Klatin1 latin1.txt" should produce identical output.  Presently, for this character they do not.

> Might as well sweep that one into this report, then.


As long as it doesn't change the billing, I won't complain about you doing more work than I asked for.

> tmac/pdf.tmac sources tmac/ps.tmac so the fix only has to be made in one place.


I should have said "notably but not limited to -Tps and -Tpdf."  Fixing this in the device-specific tmac file then requires duplicating that fix for at least -Tascii, -Tlatin1, and the various -TX* devices, and I couldn't even begin to guess about the more obscure legacy devices.

On the one hand, I get that \[u2011] is a character, and characters are mapped to glyphs, and glyphs reside in fonts, and fonts are device-specific, so some device-specific code seems a reasonable place to handle it.

But zooming out, the semantics of U+2011 NON-BREAKING HYPHEN are not device-specific; as an output glyph, it is always identical (as you note) to \[hy], or \[u2010].  What separates them is its behavior--and this should be the same across all devices, suggesting it should be handled in a device-independent section of the code.

I mean, I don't want to back-seat drive, and tell you your very simple solution, which covers most output formats most people care about, isn't good enough--except I guess I do, because that's kind of what I'm doing.

Dave <barx>
Group Member
Sat 15 Aug 2020 05:46:43 PM UTC, comment #6: 


comment #5:

> On further investigation, it appears in fact to be 0% accurate.  See bug #58962.


groff_char(7) is full of problems with accuracy.

It's on my (s)hit list.  I recently fixed up the introductory material but it needs a lot more work.

G. Branden Robinson <gbranden>
Group administrator
Sat 15 Aug 2020 05:38:01 PM UTC, comment #5: 

comment #2:

> groff_char(7) (which I only now thought to check) says it
> maps to \~.  But that appears to be less than 100% accurate:


On further investigation, it appears in fact to be 0% accurate.  See bug #58962.

Dave <barx>
Group Member
Sat 15 Aug 2020 04:05:39 AM UTC, comment #4: 

comment #2:

> "\~" and "\ " shouldn't be equivalent; they're documented as behaving differently.


No, not suggesting they should, just lamenting the total disjunctivity of the set.

>
> The input string "\[u00A0]" being equivalent to neither of these is exactly the problem this plank of this bug report is looking to solve.
>
> It's only the character NO-BREAK SPACE in its Latin-1 form, which groff accepts as direct input, that groff recognizes and interprets as a nonbreaking space.  groff_char(7) (which I only now thought to check) says it maps to \~.  But that appears to be less than 100% accurate:
>


> $ LC_CTYPE=en_US.iso88591 printf ".if '\u00A0'\~' .tm equal\n" | groff
> $


>
> But the upshot is, however groff interprets a Latin-1 A0, it really ought to interpret the form of that character emitted by preconv, \[u00A0], identically.


Yes, I think I agree here.  I can't think of a more appropriate mapping for it.

> So the only one I didn't cover was U+2007 FIGURE SPACE, which should map to groff's (already nonbreaking) \0.


Might as well sweep that one into this report, then.  Once the "where" to fix this has been determined, the incremental effort to handle that one will probably be tiny.

> > there are bunch of others (hair space, thin space, ideographic space,
> > ...) but I don't know what their breaking semantics are in Unicode.
>
> Irrational, IMO.  Unicode considers U+2009 THIN SPACE and
> U+200A HAIR SPACE breakable, for no good reason that I can see.  Groff (quite sensibly, since the concept is sort of absurd) does not offer breaking versions of these spaces, and the only reason to add them would be strict compliance with a Unicode property that probably no one who uses those code points actually wants: I can't think of a single real-world use case for a breaking thin space (though perhaps this is merely a failure of my imagination).


Well, I can't think of one either.
 

> This is all another can of worms I intentionally didn't address in what I intended to be a simple change.


Hah.  This is Sparta^Wgroff!  Complexity rapidly ramifies.

> > 4. A non-breaking hyphen would then be something that looks
> > like \[hy] but doesn't actually break?
>
> Yes.
>
> > You can just use the character as-is in input.
>
> Ah, I guess you used -Tutf8 output, where that does work.  (Somehow your groff command got stripped from your comment.)


The "somehow" was me not thinking to include it.

> All other output formats (notably -Tps and -Tpdf) produce "warning: can't find special character 'u2011'".


Okay, yes, that seems like another mapping issue.

And it appears to be a one-liner fix (morally).

tmac/pdf.tmac sources tmac/ps.tmac so the fix only has to be made in one place.

I did have to goose the loop count in the test up to 100.


$ ./build/test-groff -Tps ./EXPERIMENTS/non-breaking-hyphen.groff  >|/tmp/2011.ps
troff: warning [p 1, 0.0i]: can't break line
$ ./build/test-groff -Tpdf ./EXPERIMENTS/non-breaking-hyphen.groff  >|/tmp/2011.pdf
troff: warning [p 1, 0.0i]: can't break line
$ cat ./EXPERIMENTS/non-breaking-hyphen.groff
.pl 1v
.ds a a\[u2011]
.nr b 100 -1
.while \n+b \*a\c
$ git di tmac/ps.tmac
diff --git a/tmac/ps.tmac b/tmac/ps.tmac
index 18928765..860919e1 100644
--- a/tmac/ps.tmac
+++ b/tmac/ps.tmac
@@ -28,6 +28,9 @@
 .
 .cflags 8 \[an]
 .
+\# non-breaking hyphen
+.fchar \[u2011] -
+.
 .char \[radicalex] \h'-\w'\[sr]'u'\[radicalex]\h'\w'\[sr]'u'
 .fchar \[sqrtex] \[radicalex]
 .char \[mo] \h'.08m'\[mo]\h'-.08m'


G. Branden Robinson <gbranden>
Group administrator
Sat 15 Aug 2020 03:29:05 AM UTC, comment #3: 

comment #1:

> 2. The behavior of \: when used as the RHS of a .char request
> does indeed seem a bit strange.


Now its very own bug!  Bug #58958.

Dave <barx>
Group Member
Sat 15 Aug 2020 03:04:15 AM UTC, comment #2: 

comment #1:

> 1. "U+00A0 NO-BREAK SPACE
>
> None of these are equivalent to the others. :-/


"\~" and "\ " shouldn't be equivalent; they're documented as behaving differently.

The input string "\[u00A0]" being equivalent to neither of these is exactly the problem this plank of this bug report is looking to solve.

It's only the character NO-BREAK SPACE in its Latin-1 form, which groff accepts as direct input, that groff recognizes and interprets as a nonbreaking space.  groff_char(7) (which I only now thought to check) says it maps to \~.  But that appears to be less than 100% accurate:


$ LC_CTYPE=en_US.iso88591 printf ".if '\u00A0'\~' .tm equal\n" | groff
$


But the upshot is, however groff interprets a Latin-1 A0, it really ought to interpret the form of that character emitted by preconv, \[u00A0], identically.

> 2. The behavior of \: when used as the RHS of a .char request
> does indeed seem a bit strange.


Yeah, I really need to open a separate bug report for this, because it's unrelated to everything else here.

> 3. Narrow no-break space.  Have you named all of the non-breaking
> spaces in Unicode in this ticket?


No.  I was intentionally trying to keep it simple and minimal.  But it turns out there are only three:

http://en.wikipedia.org/wiki/Whitespace_character#Unicode

So the only one I didn't cover was U+2007 FIGURE SPACE, which should map to groff's (already nonbreaking) \0.

> there are bunch of others (hair space, thin space, ideographic space,
> ...) but I don't know what their breaking semantics are in Unicode.


Irrational, IMO.  Unicode considers U+2009 THIN SPACE and
U+200A HAIR SPACE breakable, for no good reason that I can see.  Groff (quite sensibly, since the concept is sort of absurd) does not offer breaking versions of these spaces, and the only reason to add them would be strict compliance with a Unicode property that probably no one who uses those code points actually wants: I can't think of a single real-world use case for a breaking thin space (though perhaps this is merely a failure of my imagination).

This is all another can of worms I intentionally didn't address in what I intended to be a simple change.

> 4. A non-breaking hyphen would then be something that looks
> like \[hy] but doesn't actually break?


Yes.

> You can just use the character as-is in input.


Ah, I guess you used -Tutf8 output, where that does work.  (Somehow your groff command got stripped from your comment.)  All other output formats (notably -Tps and -Tpdf) produce "warning: can't find special character 'u2011'".

Dave <barx>
Group Member
Fri 14 Aug 2020 10:00:02 AM UTC, comment #1: 

It's a little demoralizing that even these baby steps seem fraught with complication.

1. "U+00A0 NO-BREAK SPACE

This character is in the Latin-1 character set, which groff recognizes, and when groff's input is in Latin-1 encoding, it correctly handles this character (though I'm not certain whether it interprets it as "\~" or "\ ")."

None of the above, it seems:


$ cat EXPERIMENTS/spaces.groff
.pl 1v
.if '\ '\ ' \eSP = \eSP
.if '\ '\~' \eSP = \e\[ti]
.if '\ '\[u00A0]' \eSP = \e[u00A0]
.br
.if '\~'\ ' \e\[ti] = \eSP
.if '\~'\~' \e\[ti] = \e\[ti]
.if '\~'\[u00A0]' \e\[ti] = \e[u00A0]
.br
.if '\[u00A0]'\ ' \e[u00A0] = \eSP
.if '\[u00A0]'\~' \e[u00A0] = \e\[ti]
.if '\[u00A0]'\[u00A0]' \e[u00A0] = \e[u00A0]
$ ./build/test-groff -Tutf8
\SP = \SP
\~ = \~
\[u00A0] = \[u00A0]


None of these are equivalent to the others. :-/

2. The behavior of \: when used as the RHS of a .char request does indeed seem a bit strange.  It looks like the transform is just not happening:


.pl 1v
.char \[u200B] \:
.ds a \[u200B]
.length i \*a
\ni
8

.pl 1v
.ds a \[u200B]
.length i \*a
\ni
8

.pl 1v
.char a b
.ds a a
\*a
b


That unchanged length of 8, the exact character count of "\[u2000B]" is highly suspicious to me.

3. Narrow no-break space.  Have you named all of the non-breaking spaces in Unicode in this ticket?  I know there are bunch of others (hair space, thin space, ideographic space, ...) but I don't know what their breaking semantics are in Unicode.

4. A non-breaking hyphen would then be something that looks like \[hy] but doesn't actually break?  I don't know that this is actually the hardest of the tasks on this list.  You can just use the character as-is in input.  groff doesn't know it's a hyphen, and no hyphenation patterns include it, so it never gets a break after it.


$ cat EXPERIMENTS/non-breaking-hyphen.groff
.pl 1v
.ds a a\[u2011]
.nr b 50 -1
.while \n+b \*a\c

troff: warning [p 1, 0.0i]: can't break line
a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑


Let me know what you think of these findings.

G. Branden Robinson <gbranden>
Group administrator
Mon 10 Aug 2020 02:56:06 PM UTC, original submission:  

One small change that would improve groff's Unicode support would be to recognize Unicode versions of things groff already knows how to do.

Four examples:

U+00A0 NO-BREAK SPACE


This character is in the Latin-1 character set, which groff recognizes, and when groff's input is in Latin-1 encoding, it correctly handles this character (though I'm not certain whether it interprets it as "\~" or "\ ").

But if the input is some other encoding, preconv converts the character into the string "\[u00A0]", which groff does not recognize.  In macro space, a simple

.char \[u00A0] \~

is enough to take care of this; presumably the equivalent mechanism to make the code handle it internally is just as simple.

U+200B ZERO WIDTH SPACE


This is another character implemented in an existing groff escape (\:) but unrecognized as "\[u200B]".

In this case, the simple, obvious, elegant solution that worked above:

.char \[u200B] \:

stupidly, irritatingly, and undocumentedly doesn't work.  (.char being unable to map something to an escape, or at least to this particular escape, is another bug--either in the implementation, or the lack of documentation of the restriction--for another day.)

U+202F NARROW NO-BREAK SPACE


Groff has two nonbreaking thin spaces, \| and \^.  It is perhaps unclear which of these groff should map "\[u202F]" to, but either one would be an improvement over its current mapping to the warning "can't find special character `u202F'".

U+2011 NON-BREAKING HYPHEN


I deem this change "extra credit" as it's the least likely to be easily implementable, groff syntax having no direct correlate.  Groff can only (via \%) make an entire "word" (sequence of non-whitespace, including hyphens) unbreakable, but has no easy way to support a mix of breaking and nonbreaking hyphens in the same word, such as making the first hyphen of "jack-in-the-box" nonbreaking but the other two breakable.  (This can be done with a mix of \% and \: escapes, as "\%jack-in-\:the-\:box" -- or even, taking advantage of the bug/quirk Branden discovered, as "\%jack-in-\:the-box" -- but this is not obvious.)  So it's possible, but convoluted, to represent "jack\[u2011]in-the-box" in groff syntax; whether this means it's equally convoluted in the underlying code, or whether the code actually does have the concept of a nonbreaking hyphen but just doesn't expose a direct representation of it to user space, I cannot guess.

Dave <barx>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-10-24 gbranden StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2022-10-23 gbranden StatusNeed Info In Progress
    2022-09-26 gbranden Severity5 - Blocker 3 - Normal
        Planned ReleaseNone 1.23.0
    2022-06-02 gbranden Severity3 - Normal 5 - Blocker
    2020-08-14 gbranden StatusNone Need Info
        Assigned toNone gbranden

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code