bugGNU roff - Bugs: bug #42233, wcwidth(3) used on UCS4/UTF-32...

 
 

bug #42233: wcwidth(3) used on UCS4/UTF-32 codepoints

Submitter:  None
Submitted:  Tue 29 Apr 2014 11:20:30 AM UTC
   
 
Category:  Core Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  Invalid
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 06 Nov 2020 08:12:41 AM UTC, comment #4: 

 |>> Steffen has withdrawn most/all of his other patches and even after
 |>> reading
 |>
 |> I do not know what this has to do with this bug.
 |
 |As I recall (dimly), you said something at some point that I perhaps
 |misinterpreted--that you were going to work on a project called s-roff
 |and were not going to participate in any further "pull requests", if you
 |will, involving groff.
 |
 |If I misunderstood you, I apologize.

Hm, no, it was just frustrating half a decade ago, and yes,
i still have s-roff in the queue and i really, really want to that
roff package (it will not happen before 2022, that is for sure,
i lost almost the entire year, and will not leave the mailer
i maintain before i reached a state where i can).
I tried to switch to it several times, but one thing or the other
interfered, and then i said i w... ah, whatever.
Anyhow, i synchronized it several times, but stopped with the
current groff release, because the effort was too large.

I keep a list of things i want to cherry pick (or re-create around
the topic due to license issues), that is all.  The rest of the
way i will have to go alone.

 |>> this report a few times I'm not clear on what exactly the problem
 |>> is supposed to be.
 |>>
 |>> The "solution", "drop gnulib", is not likely, especially not during
 |>> an RC cycle.
 |>
 |> Sorry, what??
 |
 |I refer to this statement in the original bug report:
 |
 |"The neat side effect of that is that the entire GNULib can be
 |unhooked and removed from groff(1)."
 |
 |However, you're right that this side effect was not proposed as
 |_necessary_, only possible.

Yes, i think by then groff had nothing to do with GNUlib, and
i posted a binary search table of a few kilobytes which could have
been used as a correct implementation (other possibilities would
exist, i think Xorg and mksh, for example, and Plan9Port and such,
they all implement such binary lookup arrays to satisfy the very
problem in question).
Now that ship has sailed, and GNUlib functions could be used,
i presume.  (I only looked into GNUlib once, at that time, and
there were functions which could have been used ... if i recall
correctly.)

 |>> This could be reopened if we had a simple, reproducible case of
 |>> groff actually misbehaving.
 |>>
 |>>> I think currently groff makes false use of wcwidth(3): if it finds
 |>>> the `unicode' property in a `DESC' file it uses wcwidth(3) to
 |>>> determine the visual width, not taking into account the current
 |>>> locale, but which wcwidth(3) depends upon.
 |>>
 |>> I don't understand[.]
 |>
 |> I am too old for this shit, really.  I therefore agree.
 |
 |I am struggling with the non-idiomatic expression "makes false use".  I
 |can interpret it, but only vaguely.  Also, I may lack domain knowledge
 |here.
 |
 |It's my understanding that Unicode defines a property called "East Asian
 |width"--at least that's what my local unicode(1) command calls it.

No, no, Unicode defines a character width (0, 1 and 2, at the time
i last looked into it, which has been a couple of years indeed).
The problem is anyhow that with preconv Unicode is fed into the
machine, the DESC supports a Unicode property, but then the code
does not care at all and uses the wcwidth(3) function, which works
in correspondance with the currently active locale, which is
false.

I have never ("not yet") looked deep enough to know whether it
matters -- but: having the correct function in place there
(preconv==Unicode, DESC==Unicode, .. character handling==Unicode?)
seems to be the right approach.

 |>> [.] why the width of a Unicode character would be locale-dependent.
 |>> As I understand it, the width property (half-width, full-width,

And zero-width.

 |>> undefined) is determined on a per-codepoint basis and while it might
 |>> vary, there's no reason to expect it to vary based on the locale.
 |>> More likely, I think, it would vary due to choices taken by a font
 |>> vendor, and people using the font would be forced to adapt.
 |
 |Thinking about this some more, the possibility of an "ambiguous" or
 |"undefined" character width at the UCS level could mean that the locale
 |is permitted to determine this parameter.

Unicode does not do that.  Again, long time, but if i recall
correctly anything which is not defined otherwise is normal width.

 |>> Closing.
 |>
 |> I think there was a ML thread by the time i opened the bug report,
 |> where the according GNUlib function that could simply be used to
 |> correctly implement the given was named.
 |
 |Hmm. It would be good to find this.  I wonder if Dave Kemper can help;
 |to my eyes, he seems to have a fluttering cape that advertises a deep
 |knowledge of our mailing list history.
 |
 |> Then that piece of cake would be correct, despite possibly non-capable
 |> surroundings.
 |
 |If this would fix the infinite loop Osamu Ayama found, and that I
 |crudely hacked around in bcdf2f4c7c28328c711c6a7ac2ea17f2ecd5cdd4 (also

I do not know.  Zero width can happen.

 |see https://savannah.gnu.org/bugs/?44018 ), that would be terrific.

 |I think we just need someone with a little more gnulib and/or
 |wide-character savvy than I possess right now to articulate the issue so
 |that I can understand it.  Eventually, I'll learn, but Bertrand's trying
 |to get an RC1 out.  :)

Anonymous
Tue 03 Nov 2020 07:29:51 AM UTC, comment #3: 

Discussion of this bug has mostly taken place off-bug:

http://lists.gnu.org/archive/html/groff/2020-10/msg00041.html
http://lists.gnu.org/archive/html/groff/2020-10/msg00132.html
http://lists.gnu.org/archive/html/groff/2020-11/msg00000.html

I am not the Posting Police, but it is helpful if comments about a specific bug report are made within that bug report so they're all in one place and easy(er) to follow.

> > I think there was a ML thread by the time i opened the bug report,
> > where the according GNUlib function that could simply be used to
> > correctly implement the given was named.
>
> Hmm. It would be good to find this.  I wonder if Dave Kemper
> can help; to my eyes, he seems to have a fluttering cape that
> advertises a deep knowledge of our mailing list history.


Don't tell anyone, but the cape is actually just a search box.

But I'm not sure what even to look for here, as I'm also not clear on exactly what problem this bug report is addressing.  My best guess so far is http://lists.gnu.org/archive/html/groff/2015-04/msg00008.html, which includes a patch "to switch from `wcwidth' to `uc_width'."  (The patch appears to have never been applied, as the build failure it was addressing was solved in a different manner.)

uc_width is part of GNU libunistring:

http://www.gnu.org/software/libunistring/manual/libunistring.html#index-uc_005fwidth-646

which appears to be separate from Gnulib itself.

Dave <barx>
Group Member
Wed 21 Oct 2020 06:54:43 AM UTC, comment #2: 

Steffen has withdrawn most/all of his other patches and even after reading this report a few times I'm not clear on what exactly the problem is supposed to be.

The "solution", "drop gnulib", is not likely, especially not during an RC cycle.

This could be reopened if we had a simple, reproducible case of groff actually misbehaving.

> I think currently groff makes false use of wcwidth(3): if it finds the `unicode' property in a `DESC' file it uses wcwidth(3) to determine the visual width, not taking into account the current locale, but which wcwidth(3) depends upon.


I don't understand why the width of a Unicode character would be locale-dependent.  As I understand it, the width property (half-width, full-width, undefined) is determined on a per-codepoint basis and while it might vary, there's no reason to expect it to vary based on the locale.  More likely, I think, it would vary due to choices taken by a font vendor, and people using the font would be forced to adapt.

Closing.

G. Branden Robinson <gbranden>
Group administrator
Tue 29 Apr 2014 12:09:54 PM UTC, comment #1: 

Hm.  After updating my ArchLinux the test program `uniwidth.c' shows two errors which i haven't seen a month ago.

For one wcwidth(3) now returns `2' for 64 HEXAGRAM's (U+4DC0-U+4DFF; HEXAGRAM FOR THE CREATIVE HEAVEN - HEXAGRAM FOR BEFORE COMPLETION).  This i claim wrong again, because the Unicode `EastAsianWidth.txt' clearly classifies them with the East_Asian_Width property `N' (<http://www.unicode.org/reports/tr11/>).

Second, when compiled with `-DLIBUNISTRING -lunistring' the program crashes because the GNULib function uc_width() doesn't seem to allow NULL now; when changing the NULL to "utf-8" i know get 107 inequalities.  What i yet have looked at i am right: e.g., U+2EF4 is not currently assigned in Unicode and therefore must be treated as `N'; it is very likely that it will be fullwidth once it is assigned, since it is in the CJK range, however.  Ditto U+3040 (HIRAGANA), U+A48D (YI SYLLABLE) ... and most likely etc.

P.S.: oops, i have no account!  this is sdaoden AT yandex DOT com.

Anonymous
Tue 29 Apr 2014 11:20:30 AM UTC, original submission:  

I think currently groff makes false use of wcwidth(3): if it finds the `unicode' property in a `DESC' file it uses wcwidth(3) to determine the visual width, not taking into account the current locale, but which wcwidth(3) depends upon.
Also groff uses UCS4 / UTF-32 code points, but (unfortunately) neither ISO C nor POSIX specify the character set that wchar_t uses, and in fact at least the Citrus project (*BSD and deriviatives) uses some homebrew bit pattern for several (asian) locales.

Whereas GNULib also offers wcwidth-replacement functions which are defined to use UCS4 / UTF-32 code points, one of my personal projects is a Unicode aware library, and, short, i'm attaching a patch that implements a `unicode_is_fullwidth()' functionality, which is what the current code requires to have.
The neat side effect of that is that the entire GNULib can be
unhooked and removed from groff(1).
The attached program `uniwidth.c' can be used to verify that the implementation is correct.

So, after applying patches `0001-Drop-wcwidth-3-in-favour-of-Unicode-6.3.0-homebrew-v.patch' and `0002-Unhook-GNUlib.patch' all that is needed to bring groff into a much nicer state is

  $ git rm -rf src/libs/gnulib && git commit -m 'Drop GNUlib'

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #31272:  0002-Unhook-GNUlib.patch added by None (7KiB - text/x-diff)
file #31273:  uniwidth.c added by None (3KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by bgarrigues (Updated the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-06-10 gbranden Summary[PATCH] wcwidth(3) used on UCS4/UTF-32 codepoints wcwidth(3) used on UCS4/UTF-32 codepoints
    2020-10-21 gbranden StatusNone Invalid
        Assigned toNone gbranden
        Open/ClosedOpen Closed
    2020-10-13 barx Carbon-CopyRemoved 93119 -
    2020-10-13 barx Summarywcwidth(3) used on UCS4/UTF-32 codepoints [PATCH] wcwidth(3) used on UCS4/UTF-32 codepoints
    2018-11-26 bgarrigues CategoryNone Core
        Item GroupNone Incorrect behaviour
    2014-04-29 None Attached File- Added 0001-Drop-wcwidth-3-in-favour-of-Unicode-6.3.0-homebrew-v.patch, #31271
        Attached File- Added 0002-Unhook-GNUlib.patch, #31272
        Attached File- Added uniwidth.c, #31273

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code