bugGNU troff - Bugs: bug #58930, take baby steps toward Unicode

 
 

bug #58930: take baby steps toward Unicode

Submitted by:  Dave <barx>
Submitted on:  Mon 10 Aug 2020 02:56:06 PM UTC  
 
Category:  Core Severity:  3 - Normal
Item Group:  New feature Status:  Need Info
Privacy:  Public Assigned to:  G. Branden Robinson <gbranden>
Open/Closed:  Open Planned Release:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Mon 12 Oct 2020 10:28:35 AM UTC, comment #12: 

The test case in comment #9 is probably the best way to see the kerning deficiency.

The test cases presented in the 86b99bdbf58c8dd1a4036f4004a6d8518a5b8357 commit message don't show the kerning issue, because font TR has no "a-" kernpair.  But it does have a "Y-" one, so changing these test cases' .ds lines to use the letter Y before the hyphen or the "\[u2011]" ought to show the difference in kerning.  And it does... but then the second test case bumps up against bug #57448, and thus fails to show the difference in breaking behavior.

Dave <barx>
Project Member
Mon 12 Oct 2020 09:28:18 AM UTC, comment #11: 

It's a judgement call, certainly.  Is a change that allows groff to handle U+2011 on some output devices but incorrectly kerns it better or worse than having groff emit a warning and discard the character?  In the latter case, the user has a clear sign that his input is not understood.  In the former, the input is accepted for PostScript/PDF but produces subpar typography and gives the user no sign that anything is amiss (though it would at least not be the only miskerned character in Unicode's General Punctuation block: see bug #58897).

My perfectionist tendencies argue for either implementing a solution that is 100% correct (as far as testing can demonstrate anyway) or bluntly telling the user "sorry, don't know how to do that."  But an 80%-correct solution is not without merit.

Dave <barx>
Project Member
Thu 08 Oct 2020 12:04:23 PM UTC, comment #10: 

Hi Dave,

Something I'm not clear on.  Do you think 86b99bdbf58c8dd1a4036f4004a6d8518a5b8357 should be reverted?

G. Branden Robinson <gbranden>
Project AdministratorIn charge of this item.
Wed 23 Sep 2020 03:57:35 AM UTC, comment #9: 

comment #4:

> And it appears to be a one-liner fix (morally).


Turns out another reason it's not quite that simple is this caveat from the info manual (which is documented, so it can't be a bug):

"Only the current font is checked for... kerns; neither special fonts nor entities defined with the char request (and its siblings) are taken into account."

Thus this one-line solution results in incorrect kerning with any characters adjacent to the \[u2011].

.nf
.ps 64
.vs 64
.sp
.fchar \[u2011] -
MAY\[u2011]DAY
MAY-DAY

Dave <barx>
Project Member
Thu 20 Aug 2020 05:23:18 AM UTC, comment #8: 

comment #2:

> Unicode considers U+2009 THIN SPACE and U+200A HAIR SPACE breakable...
> Groff... does not offer breaking versions of these spaces, and the only
> reason to add them would be strict compliance with a Unicode property
> that probably no one who uses those code points actually wants


I believe my reasoning here was inaccurate.  Although Unicode allows breaking at a thin space or hair space, it does not require it,* so groff declining to treat these as break points does not violate Unicode compliance at all.  Thus I now propose that U+2009 THIN SPACE be mapped to groff's (nonbreaking) \|, and U+200A HAIR SPACE to groff's (nonbreaking) \^.

* The gory details: Unicode line breaking is covered in "Unicode Standard Annex #14: Unicode Line Breaking Algorithm" (http://www.unicode.org/reports/tr14/tr14-45.html), whose introductory section makes its scope clear: "Given an input text, [this algorithm] produces a set of positions called 'break opportunities' that are appropriate points to begin a new line. The selection of actual line break positions from the set of break opportunities is not covered by the Unicode Line Breaking Algorithm, but is in the domain of higher level software."  Groff declining to break at points that Unicode specifies as "break opportunities" is perfectly in line with this.

Dave <barx>
Project Member
Sun 16 Aug 2020 12:53:18 AM UTC, comment #7: 

comment #4:

>  just lamenting the total disjunctivity of the set.


That two of the three, intended to serve different purposes, are disjunct seems more laudable than lamentable.  But I'm not here to police your feelings.

> I can't think of a more appropriate mapping for it.


Well, if there were a more appropriate mapping for \[u00A0], that mapping should also apply to the Latin-1 A0.  They're the same character, just with different input representations.

Speaking more generally, for a Latin-1 input file, "groff latin1.txt" and "groff -Klatin1 latin1.txt" should produce identical output.  Presently, for this character they do not.

> Might as well sweep that one into this report, then.


As long as it doesn't change the billing, I won't complain about you doing more work than I asked for.

> tmac/pdf.tmac sources tmac/ps.tmac so the fix only has to be made in one place.


I should have said "notably but not limited to -Tps and -Tpdf."  Fixing this in the device-specific tmac file then requires duplicating that fix for at least -Tascii, -Tlatin1, and the various -TX* devices, and I couldn't even begin to guess about the more obscure legacy devices.

On the one hand, I get that \[u2011] is a character, and characters are mapped to glyphs, and glyphs reside in fonts, and fonts are device-specific, so some device-specific code seems a reasonable place to handle it.

But zooming out, the semantics of U+2011 NON-BREAKING HYPHEN are not device-specific; as an output glyph, it is always identical (as you note) to \[hy], or \[u2010].  What separates them is its behavior--and this should be the same across all devices, suggesting it should be handled in a device-independent section of the code.

I mean, I don't want to back-seat drive, and tell you your very simple solution, which covers most output formats most people care about, isn't good enough--except I guess I do, because that's kind of what I'm doing.

Dave <barx>
Project Member
Sat 15 Aug 2020 05:46:43 PM UTC, comment #6: 

comment #5:

> On further investigation, it appears in fact to be 0% accurate.  See bug #58962.


groff_char(7) is full of problems with accuracy.

It's on my (s)hit list.  I recently fixed up the introductory material but it needs a lot more work.

G. Branden Robinson <gbranden>
Project AdministratorIn charge of this item.
Sat 15 Aug 2020 05:38:01 PM UTC, comment #5: 

comment #2:

> groff_char(7) (which I only now thought to check) says it
> maps to \~.  But that appears to be less than 100% accurate:


On further investigation, it appears in fact to be 0% accurate.  See bug #58962.

Dave <barx>
Project Member
Sat 15 Aug 2020 04:05:39 AM UTC, comment #4: 

comment #2:

> "\~" and "\ " shouldn't be equivalent; they're documented as behaving differently.


No, not suggesting they should, just lamenting the total disjunctivity of the set.

>
> The input string "\[u00A0]" being equivalent to neither of these is exactly the problem this plank of this bug report is looking to solve.
>
> It's only the character NO-BREAK SPACE in its Latin-1 form, which groff accepts as direct input, that groff recognizes and interprets as a nonbreaking space.  groff_char(7) (which I only now thought to check) says it maps to \~.  But that appears to be less than 100% accurate:
>

> $ LC_CTYPE=en_US.iso88591 printf ".if '\u00A0'\~' .tm equal\n" | groff
> $

>
> But the upshot is, however groff interprets a Latin-1 A0, it really ought to interpret the form of that character emitted by preconv, \[u00A0], identically.


Yes, I think I agree here.  I can't think of a more appropriate mapping for it.

> So the only one I didn't cover was U+2007 FIGURE SPACE, which should map to groff's (already nonbreaking) \0.


Might as well sweep that one into this report, then.  Once the "where" to fix this has been determined, the incremental effort to handle that one will probably be tiny.

> > there are bunch of others (hair space, thin space, ideographic space,
> > ...) but I don't know what their breaking semantics are in Unicode.
>
> Irrational, IMO.  Unicode considers U+2009 THIN SPACE and
> U+200A HAIR SPACE breakable, for no good reason that I can see.  Groff (quite sensibly, since the concept is sort of absurd) does not offer breaking versions of these spaces, and the only reason to add them would be strict compliance with a Unicode property that probably no one who uses those code points actually wants: I can't think of a single real-world use case for a breaking thin space (though perhaps this is merely a failure of my imagination).


Well, I can't think of one either.

> This is all another can of worms I intentionally didn't address in what I intended to be a simple change.


Hah.  This is Sparta^Wgroff!  Complexity rapidly ramifies.

> > 4. A non-breaking hyphen would then be something that looks
> > like \[hy] but doesn't actually break?
>
> Yes.
>
> > You can just use the character as-is in input.
>
> Ah, I guess you used -Tutf8 output, where that does work.  (Somehow your groff command got stripped from your comment.)


The "somehow" was me not thinking to include it.

> All other output formats (notably -Tps and -Tpdf) produce "warning: can't find special character 'u2011'".


Okay, yes, that seems like another mapping issue.

And it appears to be a one-liner fix (morally).

tmac/pdf.tmac sources tmac/ps.tmac so the fix only has to be made in one place.

I did have to goose the loop count in the test up to 100.

$ ./build/test-groff -Tps ./EXPERIMENTS/non-breaking-hyphen.groff  >|/tmp/2011.ps
troff: warning [p 1, 0.0i]: can't break line
$ ./build/test-groff -Tpdf ./EXPERIMENTS/non-breaking-hyphen.groff  >|/tmp/2011.pdf
troff: warning [p 1, 0.0i]: can't break line
$ cat ./EXPERIMENTS/non-breaking-hyphen.groff
.pl 1v
.ds a a\[u2011]
.nr b 100 -1
.while \n+b \*a\c
$ git di tmac/ps.tmac
diff --git a/tmac/ps.tmac b/tmac/ps.tmac
index 18928765..860919e1 100644
--- a/tmac/ps.tmac
+++ b/tmac/ps.tmac
@@ -28,6 +28,9 @@
.
.cflags 8 \[an]
.
+\# non-breaking hyphen
+.fchar \[u2011] -
+.
.char \[radicalex] \h'-\w'\[sr]'u'\[radicalex]\h'\w'\[sr]'u'
.fchar \[sqrtex] \[radicalex]
.char \[mo] \h'.08m'\[mo]\h'-.08m'

G. Branden Robinson <gbranden>
Project AdministratorIn charge of this item.
Sat 15 Aug 2020 03:29:05 AM UTC, comment #3: 

comment #1:

> 2. The behavior of \: when used as the RHS of a .char request
> does indeed seem a bit strange.


Now its very own bug!  Bug #58958.

Dave <barx>
Project Member
Sat 15 Aug 2020 03:04:15 AM UTC, comment #2: 

comment #1:

> 1. "U+00A0 NO-BREAK SPACE
>
> None of these are equivalent to the others. :-/


"\~" and "\ " shouldn't be equivalent; they're documented as behaving differently.

The input string "\[u00A0]" being equivalent to neither of these is exactly the problem this plank of this bug report is looking to solve.

It's only the character NO-BREAK SPACE in its Latin-1 form, which groff accepts as direct input, that groff recognizes and interprets as a nonbreaking space.  groff_char(7) (which I only now thought to check) says it maps to \~.  But that appears to be less than 100% accurate:

$ LC_CTYPE=en_US.iso88591 printf ".if '\u00A0'\~' .tm equal\n" | groff
$

But the upshot is, however groff interprets a Latin-1 A0, it really ought to interpret the form of that character emitted by preconv, \[u00A0], identically.

> 2. The behavior of \: when used as the RHS of a .char request
> does indeed seem a bit strange.


Yeah, I really need to open a separate bug report for this, because it's unrelated to everything else here.

> 3. Narrow no-break space.  Have you named all of the non-breaking
> spaces in Unicode in this ticket?


No.  I was intentionally trying to keep it simple and minimal.  But it turns out there are only three:

http://en.wikipedia.org/wiki/Whitespace_character#Unicode

So the only one I didn't cover was U+2007 FIGURE SPACE, which should map to groff's (already nonbreaking) \0.

> there are bunch of others (hair space, thin space, ideographic space,
> ...) but I don't know what their breaking semantics are in Unicode.


Irrational, IMO.  Unicode considers U+2009 THIN SPACE and
U+200A HAIR SPACE breakable, for no good reason that I can see.  Groff (quite sensibly, since the concept is sort of absurd) does not offer breaking versions of these spaces, and the only reason to add them would be strict compliance with a Unicode property that probably no one who uses those code points actually wants: I can't think of a single real-world use case for a breaking thin space (though perhaps this is merely a failure of my imagination).

This is all another can of worms I intentionally didn't address in what I intended to be a simple change.

> 4. A non-breaking hyphen would then be something that looks
> like \[hy] but doesn't actually break?


Yes.

> You can just use the character as-is in input.


Ah, I guess you used -Tutf8 output, where that does work.  (Somehow your groff command got stripped from your comment.)  All other output formats (notably -Tps and -Tpdf) produce "warning: can't find special character 'u2011'".

Dave <barx>
Project Member
Fri 14 Aug 2020 10:00:02 AM UTC, comment #1: 

It's a little demoralizing that even these baby steps seem fraught with complication.

1. "U+00A0 NO-BREAK SPACE

This character is in the Latin-1 character set, which groff recognizes, and when groff's input is in Latin-1 encoding, it correctly handles this character (though I'm not certain whether it interprets it as "\~" or "\ ")."

None of the above, it seems:

$ cat EXPERIMENTS/spaces.groff
.pl 1v
.if '\ '\ ' \eSP = \eSP
.if '\ '\~' \eSP = \e\[ti]
.if '\ '\[u00A0]' \eSP = \e[u00A0]
.br
.if '\~'\ ' \e\[ti] = \eSP
.if '\~'\~' \e\[ti] = \e\[ti]
.if '\~'\[u00A0]' \e\[ti] = \e[u00A0]
.br
.if '\[u00A0]'\ ' \e[u00A0] = \eSP
.if '\[u00A0]'\~' \e[u00A0] = \e\[ti]
.if '\[u00A0]'\[u00A0]' \e[u00A0] = \e[u00A0]
$ ./build/test-groff -Tutf8
\SP = \SP
\~ = \~
\[u00A0] = \[u00A0]

None of these are equivalent to the others. :-/

2. The behavior of \: when used as the RHS of a .char request does indeed seem a bit strange.  It looks like the transform is just not happening:

.pl 1v
.char \[u200B] \:
.ds a \[u200B]
.length i \*a
\ni
8

.pl 1v
.ds a \[u200B]
.length i \*a
\ni
8

.pl 1v
.char a b
.ds a a
\*a
b

That unchanged length of 8, the exact character count of "\[u2000B]" is highly suspicious to me.

3. Narrow no-break space.  Have you named all of the non-breaking spaces in Unicode in this ticket?  I know there are bunch of others (hair space, thin space, ideographic space, ...) but I don't know what their breaking semantics are in Unicode.

4. A non-breaking hyphen would then be something that looks like \[hy] but doesn't actually break?  I don't know that this is actually the hardest of the tasks on this list.  You can just use the character as-is in input.  groff doesn't know it's a hyphen, and no hyphenation patterns include it, so it never gets a break after it.

$ cat EXPERIMENTS/non-breaking-hyphen.groff
.pl 1v
.ds a a\[u2011]
.nr b 50 -1
.while \n+b \*a\c

troff: warning [p 1, 0.0i]: can't break line
a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑a‑

Let me know what you think of these findings.

G. Branden Robinson <gbranden>
Project AdministratorIn charge of this item.
Mon 10 Aug 2020 02:56:06 PM UTC, original submission:  

One small change that would improve groff's Unicode support would be to recognize Unicode versions of things groff already knows how to do.

Four examples:

U+00A0 NO-BREAK SPACE

This character is in the Latin-1 character set, which groff recognizes, and when groff's input is in Latin-1 encoding, it correctly handles this character (though I'm not certain whether it interprets it as "\~" or "\ ").

But if the input is some other encoding, preconv converts the character into the string "\[u00A0]", which groff does not recognize.  In macro space, a simple

.char \[u00A0] \~

is enough to take care of this; presumably the equivalent mechanism to make the code handle it internally is just as simple.

U+200B ZERO WIDTH SPACE

This is another character implemented in an existing groff escape (\:) but unrecognized as "\[u200B]".

In this case, the simple, obvious, elegant solution that worked above:

.char \[u200B] \:

stupidly, irritatingly, and undocumentedly doesn't work.  (.char being unable to map something to an escape, or at least to this particular escape, is another bug--either in the implementation, or the lack of documentation of the restriction--for another day.)

U+202F NARROW NO-BREAK SPACE

Groff has two nonbreaking thin spaces, \| and \^.  It is perhaps unclear which of these groff should map "\[u202F]" to, but either one would be an improvement over its current mapping to the warning "can't find special character `u202F'".

U+2011 NON-BREAKING HYPHEN

I deem this change "extra credit" as it's the least likely to be easily implementable, groff syntax having no direct correlate.  Groff can only (via \%) make an entire "word" (sequence of non-whitespace, including hyphens) unbreakable, but has no easy way to support a mix of breaking and nonbreaking hyphens in the same word, such as making the first hyphen of "jack-in-the-box" nonbreaking but the other two breakable.  (This can be done with a mix of \% and \: escapes, as "\%jack-in-\:the-\:box" -- or even, taking advantage of the bug/quirk Branden discovered, as "\%jack-in-\:the-box" -- but this is not obvious.)  So it's possible, but convoluted, to represent "jack\[u2011]in-the-box" in groff syntax; whether this means it's equally convoluted in the underlying code, or whether the code actually does have the concept of a nonbreaking hyphen but just doesn't expose a direct representation of it to user space, I cannot guess.

Dave <barx>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-08-14 gbranden StatusNone => Need Info
        Assigned toNone => gbranden

    Back to the top


    Powered by Savane 3.7