bugGNU troff - Bugs: bug #58930, take baby steps toward Unicode

 
 

bug #58930: take baby steps toward Unicode

Submitter:  Dave <barx>
Submitted:  Mon 10 Aug 2020 02:56:06 PM UTC
 
Category:  Core Severity:  3 - Normal
Item Group:  Feature change Status:  Need Info
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Open Planned Release:  1.23.0
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       No canned response available

 

( Jump to the original submission )

Thu 02 Jun 2022 12:56:45 AM UTC, comment #17: 

comment #16:

> And while I have your attention:
>
> Before 1.23.0, and potentially before rc2, there should be a decision about the question from comment #10.  Leaving the commit in place is a valid choice; I just want to make sure it is a deliberate choice rather than an oversight.
>
> Per reasoning in comment #11, I still lean toward reverting the commit: I think it's better to give the user a warning and reject the character, than to handle it on some devices suboptimally.  After 1.23.0 we can revisit the issue and maybe come at it with fresh ideas.


Setting Severity to "Blocker" so that I don't forget about this.

G. Branden Robinson <gbranden>
Project Administrator
Mon 30 May 2022 12:04:21 AM UTC, comment #16: 

And while I have your attention:

Before 1.23.0, and potentially before rc2, there should be a decision about the question from comment #10.  Leaving the commit in place is a valid choice; I just want to make sure it is a deliberate choice rather than an oversight.

Per reasoning in comment #11, I still lean toward reverting the commit: I think it's better to give the user a warning and reject the character, than to handle it on some devices suboptimally.  After 1.23.0 we can revisit the issue and maybe come at it with fresh ideas.

Dave <barx>
Project Member
Sun 29 May 2022 11:00:18 PM UTC, comment #15: 

comment #14:

> I guess what we need to do here is be more clear whether
> \[u...] escape sequences are intended to represent input
> characters, or desired output glyphs.


Fair point.  I don't have an opinion either way.

But the answer has bearing on how the U+202F NARROW NO-BREAK SPACE part of the original submission should be handled.  It could be a change only to preconv, paralleling the fix for bug #62300, making preconv convert a U+202F on input to one of groff's thin-space escapes on output.  Or groff itself could recognize \[u202F], which is what preconv currently outputs for this character.

$ LC_CTYPE=en_US.utf8 printf '\u202F\n' | preconv -eutf8
.lf 1 -
\[u202F]

> Maybe there's a good reason to have \^ and \| customizable
> in this way--I don't feel like I have a command of the history
> of this topic.


I don't know the history or the reasoning either, but, as you allude, there is at least a justification for it in those two cases, where Unicode does not specify a width.

Dave <barx>
Project Member
Sat 28 May 2022 06:52:58 PM UTC, comment #14: 

Hi Dave,

comment #13:

> "The input sequence '\[u00A0]' is syntactically valid...but like '\[uFFFF]' and '\[u0000]', it's not meaningful"
>
> This is true of the current implementation but less true conceptually: U+0000 and U+FFFF are not meaningful input characters to groff, but U+00A0 is, and users ideally ought to be able to specify the character as \[u00A0].


I guess what we need to do here is be more clear whether \[u...] escape sequences are intended to represent input characters, or desired output glyphs.  The distinction comes into sharp relief here.

If the former, then you're right--\[u...] is a way of getting around groff's narrow-character input interpretation.

But if the latter, then there are many other Unicode code points that don't represent things we can ask a font to draw for us.  In groff, with two exceptions, if an ordinary or special character doesn't correspond to something that "puts ink on the page", it isn't a glyph.  In device-independent output, spaces of all kinds are represented with horizontal or vertical motions, not glyph-writing commands.

The two exceptions are only partial ones: \| and \^.  One might interpret them as special character escape sequences like \- or \_, but they aren't.  They become horizontal motions.  It is true that these two sequences can be defined as if they were special characters in font description files to customize their widths (groff_font(5) discusses this).  With my present understanding, I think this is a bit of a bodge, and as far as I've seen, no one has ever taken advantage of this configurability.  (Possibly because *roff font description files have been regarded as a bit esoteric.)  This approach does not generalize well to the many additional space code points in Unicode.  U+2001, "EM QUAD", is just one example.

Maybe there's a good reason to have \^ and \| customizable in this way--I don't feel like I have a command of the history of this topic.

If you or anybody has some knowledge to bring to light here, I'd appreciate it!

>
> But this is an edge case I don't intend to pursue.  Users who want to stick to pure-ASCII input have the escape sequence \~ to specify the nonbreaking space, so don't need the alternate spelling \[u00A0].

G. Branden Robinson <gbranden>
Project Administrator
Sat 28 May 2022 05:35:55 PM UTC, comment #13: 

original submission:

> But if the input is some other encoding, preconv converts
> the character into the string "\[u00A0]", which groff does
> not recognize.


The resolved bug #62300 has fixed preconv to emit "\~" rather than "\[u00A0]" for a U+00A0 input character.

In preconv 1.22.4:

$ echo -e '\xA0' | preconv -elatin1
.lf 1 -
\[u00A0]

In preconv built from the latest code:

$ echo -e '\xA0' | preconv -elatin1
.lf 1 -
\~

So I think we can mark this part as resolved, despite one remaining issue 62300 points out in its comment 2:

"The input sequence '\[u00A0]' is syntactically valid...but like '\[uFFFF]' and '\[u0000]', it's not meaningful"

This is true of the current implementation but less true conceptually: U+0000 and U+FFFF are not meaningful input characters to groff, but U+00A0 is, and users ideally ought to be able to specify the character as \[u00A0].

But this is an edge case I don't intend to pursue.  Users who want to stick to pure-ASCII input have the escape sequence \~ to specify the nonbreaking space, so don't need the alternate spelling \[u00A0].

Dave <barx>
Project Member
Mon 12 Oct 2020 10:28:35 AM UTC, comment #12: 

The test case in comment #9 is probably the best way to see the kerning deficiency.

The test cases presented in the 86b99bdbf58c8dd1a4036f4004a6d8518a5b8357 commit message don't show the kerning issue, because font TR has no "a-" kernpair.  But it does have a "Y-" one, so changing these test cases' .ds lines to use the letter Y before the hyphen or the "\[u2011]" ought to show the difference in kerning.  And it does... but then the second test case bumps up against bug #57448, and thus fails to show the difference in breaking behavior.

Dave <barx>
Project Member
Mon 12 Oct 2020 09:28:18 AM UTC, comment #11: 

It's a judgement call, certainly.  Is a change that allows groff to handle U+2011 on some output devices but incorrectly kerns it better or worse than having groff emit a warning and discard the character?  In the latter case, the user has a clear sign that his input is not understood.  In the former, the input is accepted for PostScript/PDF but produces subpar typography and gives the user no sign that anything is amiss (though it would at least not be the only miskerned character in Unicode's General Punctuation block: see bug #58897).

My perfectionist tendencies argue for either implementing a solution that is 100% correct (as far as testing can demonstrate anyway) or bluntly telling the user "sorry, don't know how to do that."  But an 80%-correct solution is not without merit.

Dave <barx>
Project Member
Thu 08 Oct 2020 12:04:23 PM UTC, comment #10: 

Hi Dave,

Something I'm not clear on.  Do you think 86b99bdbf58c8dd1a4036f4004a6d8518a5b8357 should be reverted?

G. Branden Robinson <gbranden>
Project Administrator
Wed 23 Sep 2020 03:57:35 AM UTC, comment #9: 

comment #4:

> And it appears to be a one-liner fix (morally).


Turns out another reason it's not quite that simple is this caveat from the info manual (which is documented, so it can't be a bug):

"Only the current font is checked for... kerns; neither special fonts nor entities defined with the char request (and its siblings) are taken into account."

Thus this one-line solution results in incorrect kerning with any characters adjacent to the \[u2011].

.nf
.ps 64
.vs 64
.sp
.fchar \[u2011] -
MAY\[u2011]DAY
MAY-DAY

Dave <barx>
Project Member
Thu 20 Aug 2020 05:23:18 AM UTC, comment #8: 

comment #2:

> Unicode considers U+2009 THIN SPACE and U+200A HAIR SPACE breakable...
> Groff... does not offer breaking versions of these spaces, and the only
> reason to add them would be strict compliance with a Unicode property
> that probably no one who uses those code points actually wants


I believe my reasoning here was inaccurate.  Although Unicode allows breaking at a thin space or hair space, it does not require it,* so groff declining to treat these as break points does not violate Unicode compliance at all.  Thus I now propose that U+2009 THIN SPACE be mapped to groff's (nonbreaking) \|, and U+200A HAIR SPACE to groff's (nonbreaking) \^.

* The gory details: Unicode line breaking is covered in "Unicode Standard Annex #14: Unicode Line Breaking Algorithm" (http://www.unicode.org/reports/tr14/tr14-45.html), whose introductory section makes its scope clear: "Given an input text, [this algorithm] produces a set of positions called 'break opportunities' that are appropriate points to begin a new line. The selection of actual line break positions from the set of break opportunities is not covered by the Unicode Line Breaking Algorithm, but is in the domain of higher level software."  Groff declining to break at points that Unicode specifies as "break opportunities" is perfectly in line with this.

Dave <barx>
Project Member
Sun 16 Aug 2020 12:53:18 AM UTC, comment #7: 

comment #4:

>  just lamenting the total disjunctivity of the set.


That two of the three, intended to serve different purposes, are disjunct seems more laudable than lamentable.  But I'm not here to police your feelings.

> I can't think of a more appropriate mapping for it.


Well, if there were a more appropriate mapping for \[u00A0], that mapping should also apply to the Latin-1 A0.  They're the same character, just with different input representations.

Speaking more generally, for a Latin-1 input file, "groff latin1.txt" and "groff -Klatin1 latin1.txt" should produce identical output.  Presently, for this character they do not.

> Might as well sweep that one into this report, then.


As long as it doesn't change the billing, I won't complain about you doing more work than I asked for.

> tmac/pdf.tmac sources tmac/ps.tmac so the fix only has to be made in one place.


I should have said "notably but not limited to -Tps and -Tpdf."  Fixing this in the device-specific tmac file then requires duplicating that fix for at least -Tascii, -Tlatin1, and the various -TX* devices, and I couldn't even begin to guess about the more obscure legacy devices.

On the one hand, I get that \[u2011] is a character, and characters are mapped to glyphs, and glyphs reside in fonts, and fonts are device-specific, so some device-specific code seems a reasonable place to handle it.

But zooming out, the semantics of U+2011 NON-BREAKING HYPHEN are not device-specific; as an output glyph, it is always identical (as you note) to \[hy], or \[u2010].  What separates them is its behavior--and this should be the same across all devices, suggesting it should be handled in a device-independent section of the code.

I mean, I don't want to back-seat drive, and tell you your very simple solution, which covers most output formats most people care about, isn't good enough--except I guess I do, because that's kind of what I'm doing.

Dave <barx>
Project Member
Sat 15 Aug 2020 05:46:43 PM UTC, comment #6: 

comment #5:

> On further investigation, it appears in fact to be 0% accurate.  See bug #58962.


groff_char(7) is full of problems with accuracy.

It's on my (s)hit list.  I recently fixed up the introductory material but it needs a lot more work.

G. Branden Robinson <gbranden>
Project Administrator
Sat 15 Aug 2020 05:38:01 PM UTC, comment #5: 

comment #2:

> groff_char(7) (which I only now thought to check) says it
> maps to \~.  But that appears to be less than 100% accurate:


On further investigation, it appears in fact to be 0% accurate.  See bug #58962.

Dave <barx>
Project Member
Sat 15 Aug 2020 04:05:39 AM UTC, comment #4: 

comment #2:

> "\~" and "\ " shouldn't be equivalent; they're documented as behaving differently.


No, not suggesting they should, just lamenting the total disjunctivity of the set.

>
> The input string "\[u00A0]" being equivalent to neither of these is exactly the problem this plank of this bug report is looking to solve.
>
> It's only the character NO-BREAK SPACE in its Latin-1 form, which groff accepts as direct input, that groff recognizes and interprets as a nonbreaking space.  groff_char(7) (which I only now thought to check) says it maps to \~.  But that appears to be less than 100% accurate:
>

> $ LC_CTYPE=en_US.iso88591 printf ".if '\u00A0'\~' .tm equal\n" | groff
> $

>
> But the upshot is, however groff interprets a Latin-1 A0, it really ought to interpret the form of that character emitted by preconv, \[u00A0], identically.


Yes, I think I agree here.  I can't think of a more appropriate mapping for it.

> So the only one I didn't cover was U+2007 FIGURE SPACE, which should map to groff's (already nonbreaking) \0.


Might as well sweep that one into this report, then.  Once the "where" to fix this has been determined, the incremental effort to handle that one will probably be tiny.

> > there are bunch of others (hair space, thin space, ideographic space,
> > ...) but I don't know what their breaking semantics are in Unicode.
>
> Irrational, IMO.  Unicode considers U+2009 THIN SPACE and
> U+200A HAIR SPACE breakable, for no good reason that I can see.  Groff (quite sensibly, since the concept is sort of absurd) does not offer breaking versions of these spaces, and the only reason to add them would be strict compliance with a Unicode property that probably no one who uses those code points actually wants: I can't think of a single real-world use case for a breaking thin space (though perhaps this is merely a failure of my imagination).


Well, I can't think of one either.

> This is all another can of worms I intentionally didn't address in what I intended to be a simple change.


Hah.  This is Sparta^Wgroff!  Complexity rapidly ramifies.

> > 4. A non-breaking hyphen would then be something that looks
> > like \[hy] but doesn't actually break?
>
> Yes.
>
> > You can just use the character as-is in input.
>
> Ah, I guess you used -Tutf8 output, where that does work.  (Somehow your groff command got stripped from your comment.)


The "somehow" was me not thinking to include it.

> All other output formats (notably -Tps and -Tpdf) produce "warning: can't find special character 'u2011'".


Okay, yes, that seems like another mapping issue.

And it appears to be a one-liner fix (morally).

tmac/pdf.tmac sources tmac/ps.tmac so the fix only has to be made in one place.

I did have to goose the loop count in the test up to 100.

$ ./build/test-groff -Tps ./EXPERIMENTS/non-breaking-hyphen.groff  >|/tmp/2011.ps
troff: warning [p 1, 0.0i]: can't break line
$ ./build/test-groff -Tpdf ./EXPERIMENTS/non-breaking-hyphen.groff  >|/tmp/2011.pdf
troff: warning [p 1, 0.0i]: can't break line
$ cat ./EXPERIMENTS/non-breaking-hyphen.groff
.pl 1v
.ds a a\[u2011]
.nr b 100 -1
.while \n+b \*a\c
$ git di tmac/ps.tmac
diff --git a/tmac/ps.tmac b/tmac/ps.tmac
index 18928765..860919e1 100644
--- a/tmac/ps.tmac
+++ b/tmac/ps.tmac
@@ -28,6 +28,9 @@
 .
 .cflags 8 \[an]
 .
+\# non-breaking hyphen
+.fchar \[u2011] -
+.
 .char \[radicalex] \h'-\w'\[sr]'u'\[radicalex]\h'\w'\[sr]'u'
 .fchar \[sqrtex] \[radicalex]
 .char \[mo] \h'.08m'\[mo]\h'-.08m'

G. Branden Robinson <gbranden>
Project Administrator
Sat 15 Aug 2020 03:29:05 AM UTC, comment #3: 

comment #1:

> 2. The behavior of \: when used as the RHS of a .char request
> does indeed seem a bit strange.


Now its very own bug!  Bug #58958.

Dave <barx>
Project Member
Sat 15 Aug 2020 03:04:15 AM UTC, comment #2: 

comment #1:

> 1. "U+00A0 NO-BREAK SPACE
>
> None of these are equivalent to the others. :-/


"\~" and "\ " shouldn't be equivalent; they're documented as behaving differently.

The input string "\[u00A0]" being equivalent to neither of these is exactly the problem this plank of this bug report is looking to solve.

It's only the character NO-BREAK SPACE in its Latin-1 form, which groff accepts as direct input, that groff recognizes and interprets as a nonbreaking space.  groff_char(7) (which I only now thought to check) says it maps to \~.  But that appears to be less than 100% accurate:

$ LC_CTYPE=en_US.iso88591 printf ".if '\u00A0'\~' .tm equal\n" | groff
$

But the upshot is, however groff interprets a Latin-1 A0, it really ought to interpret the form of that character emitted by preconv, \[u00A0], identically.

> 2. The behavior of \: when used as the RHS of a .char request
> does indeed seem a bit strange.


Yeah, I really need to open a separate bug report for this, because it's unrelated to everything else here.

> 3. Narrow no-break space.  Have you named all of the non-breaking
> spaces in Unicode in this ticket?


No.  I was intentionally trying to keep it simple and minimal.  But it turns out there are only three:

http://en.wikipedia.org/wiki/Whitespace_character#Unicode

So the only one I didn't cover was U+2007 FIGURE SPACE, which should map to groff's (already nonbreaking) \0.

> there are bunch of others (hair space, thin space, ideographic space,
> ...) but I don't know what their breaking semantics are in Unicode.


Irrational, IMO.  Unicode considers U+2009 THIN SPACE and
U+200A HAIR SPACE breakable, for no good reason that I can see.  Groff (quite sensibly, since the concept is sort of absurd) does not offer breaking versions of these spaces, and the only reason to add them would be strict compliance with a Unicode property that probably no one who uses those code points actually wants: I can't think of a single real-world use case for a breaking thin space (though perhaps this is merely a failure of my imagination).

This is all another can of worms I intentionally didn't address in what I intended to be a simple change.

> 4. A non-breaking hyphen would then be something that looks
> like \[hy] but doesn't actually break?


Yes.

> You can just use the character as-is in input.


Ah, I guess you used -Tutf8 output, where that does work.  (Somehow your groff command got stripped from your comment.)  All other output formats (notably -Tps and -Tpdf) produce "warning: can't find special character 'u2011'".

Dave <barx>
Project Member
Fri 14 Aug 2020 10:00:02 AM UTC, comment #1: 

It's a little demoralizing that even these baby steps seem fraught with complication.

1. "U+00A0 NO-BREAK SPACE

This character is in the Latin-1 character set, which groff recognizes, and when groff's input is in Latin-1 encoding, it correctly handles this character (though I'm not certain whether it interprets it as "\~" or "\ ")."

None of the above, it seems:

$ cat EXPERIMENTS/spaces.groff
.pl 1v
.if '\ '\ ' \eSP = \eSP
.if '\ '\~' \eSP = \e\[ti]
.if '\ '\[u00A0]' \eSP = \e[u00A0]
.br
.if '\~'\ ' \e\[ti] = \eSP
.if '\~'\~' \e\[ti] = \e\[ti]
.if '\~'\[u00A0]' \e\[ti] = \e[u00A0]
.br
.if '\[u00A0]'\ ' \e[u00A0] = \eSP
.if '\[u00A0]'\~' \e[u00A0] = \e\[ti]
.if '\[u00A0]'\[u00A0]' \e[u00A0] = \e[u00A0]
$ ./build/test-groff -Tutf8
\SP = \SP
\~ = \~
\[u00A0] = \[u00A0]

None of these are equivalent to the others. :-/

2. The behavior of \: when used as the RHS of a .char request does indeed seem a bit strange.  It looks like the transform is just not happening:

.pl 1v
.char \[u200B] \:
.ds a \[u200B]
.length i \*a
\ni
8

.pl 1v
.ds a \[u200B]
.length i \*a
\ni
8

.pl 1v
.char a b
.ds a a
\*a
b

That unchanged length of 8, the exact character count of "\[u2000B]" is highly suspicious to me.

3. Narrow no-break space.  Have you named all of the non-breaking spaces in Unicode in this ticket?  I know there are bunch of others (hair space, thin space, ideographic space, ...) but I don't know what their breaking semantics are in Unicode.

4. A non-breaking hyphen would then be something that looks like \[hy] but doesn't actually break?  I don't know that this is actually the hardest of the tasks on this list.  You can just use the character as-is in input.  groff doesn't know it's a hyphen, and no hyphenation patterns include it, so it never gets a break after it.

$ cat EXPERIMENTS/non-breaking-hyphen.groff
.pl 1v
.ds a a\[u2011]
.nr b 50 -1
.while \n+b \*a\c

troff: warning [p 1, 0.0i]: can't break line
a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑

Let me know what you think of these findings.

G. Branden Robinson <gbranden>
Project Administrator
Mon 10 Aug 2020 02:56:06 PM UTC, original submission:  

One small change that would improve groff's Unicode support would be to recognize Unicode versions of things groff already knows how to do.

Four examples:

U+00A0 NO-BREAK SPACE

This character is in the Latin-1 character set, which groff recognizes, and when groff's input is in Latin-1 encoding, it correctly handles this character (though I'm not certain whether it interprets it as "\~" or "\ ").

But if the input is some other encoding, preconv converts the character into the string "\[u00A0]", which groff does not recognize.  In macro space, a simple

.char \[u00A0] \~

is enough to take care of this; presumably the equivalent mechanism to make the code handle it internally is just as simple.

U+200B ZERO WIDTH SPACE

This is another character implemented in an existing groff escape (\:) but unrecognized as "\[u200B]".

In this case, the simple, obvious, elegant solution that worked above:

.char \[u200B] \:

stupidly, irritatingly, and undocumentedly doesn't work.  (.char being unable to map something to an escape, or at least to this particular escape, is another bug--either in the implementation, or the lack of documentation of the restriction--for another day.)

U+202F NARROW NO-BREAK SPACE

Groff has two nonbreaking thin spaces, \| and \^.  It is perhaps unclear which of these groff should map "\[u202F]" to, but either one would be an improvement over its current mapping to the warning "can't find special character `u202F'".

U+2011 NON-BREAKING HYPHEN

I deem this change "extra credit" as it's the least likely to be easily implementable, groff syntax having no direct correlate.  Groff can only (via \%) make an entire "word" (sequence of non-whitespace, including hyphens) unbreakable, but has no easy way to support a mix of breaking and nonbreaking hyphens in the same word, such as making the first hyphen of "jack-in-the-box" nonbreaking but the other two breakable.  (This can be done with a mix of \% and \: escapes, as "\%jack-in-\:the-\:box" -- or even, taking advantage of the bug/quirk Branden discovered, as "\%jack-in-\:the-box" -- but this is not obvious.)  So it's possible, but convoluted, to represent "jack\[u2011]in-the-box" in groff syntax; whether this means it's equally convoluted in the underlying code, or whether the code actually does have the concept of a nonbreaking hyphen but just doesn't expose a direct representation of it to user space, I cannot guess.

Dave <barx>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-09-26 gbranden Severity5 - Blocker 3 - Normal
        Planned ReleaseNone 1.23.0
    2022-06-02 gbranden Severity3 - Normal 5 - Blocker
    2020-08-14 gbranden StatusNone Need Info
        Assigned toNone gbranden

    Back to the top


    Powered by Savane 3.9