bugGNU Core Utilities - Bugs: bug #29138, expand doesn't support multibyte...


You are not allowed to post comments on this tracker with your current authentication level.

bug #29138: expand doesn't support multibyte characters (e.g. UTF-8) (patch available)

Submitter:  Herbert Thielen <hthielen>
Submitted:  Tue 09 Mar 2010 02:56:28 PM UTC
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Confirmed
Privacy:  Public Assigned to:  pixelbeat
Open/Closed:  Open

Tue 09 Mar 2010 06:18:09 PM UTC, comment #2: 

Thanks for the hint. As I noted, I'm not experienced with wide characters ... so I didn't realize that unicode may contain combined characters. I've added a wcwidth() call now to forward the column, so it should handle combining characters and multi column characters correctly.

The modified patch is attached.

(file #19895)

Herbert Thielen <hthielen>
Tue 09 Mar 2010 04:10:21 PM UTC, comment #1: 

Thanks for the patch!

However, I'm not sure you can process characters independently like that.
Consider combining characters for example (/usr/bin/printf a\u0301).
I've attached the current i18n patch applied to fedora's expand for reference, which does handle combining chars. Note that does have issues in that it mishandles this case:
printf " \ta\tb\n" | expand --tabs=3 -i
We're currently working on i18n in coreutils but don't let that preclude you from submitting ideas or patches.

(file #19894)

Pádraig Brady <pixelbeat>
Group administrator
Tue 09 Mar 2010 02:56:28 PM UTC, original submission:  

'expand' doesn't replace tab characters by the correct number of spaces in case the input data contains multibyte characters, e.g. german umlauts when using the locale UTF-8.

A small example test file is attached.

I've attached a patch to coreutils 8.4 which works for me, but I'm not experienced in wide character programming, so there might still be some problem - more tests and review needed.

Thanks for the work, best regards


Herbert Thielen <hthielen>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #19895:  expand.c.patch added by hthielen (2KiB - text/x-patch)
file #19894:  expand-i18n.diff added by pixelbeat (5KiB - text/x-patch)
file #19893:  expand.c.patch added by hthielen (2KiB - text/x-patch)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by pixelbeat (Updated the item)
  • -email is unavailable- added by hthielen (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.


    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-03-09 hthielen Attached File- Added expand.c.patch, #19895
    2010-03-09 pixelbeat Attached File- Added expand-i18n.diff, #19894
    2010-03-09 pixelbeat StatusNone Confirmed
        Assigned toNone pixelbeat
    2010-03-09 hthielen Attached File- Added expand-wrong-with-utf8-test.input, #19891
        Attached File- Added expand-wrong-with-utf8-test.expected-output, #19892
        Attached File- Added expand.c.patch, #19893

    Back to the top

    Powered by Savane 3.12