bugGNU Core Utilities - Bugs: bug #29138, expand doesn't support multibyte...


You are not allowed to post comments on this tracker with your current authentication level.

bug #29138: expand doesn't support multibyte characters (e.g. UTF-8) (patch available)

Submitted by:  Herbert Thielen <hthielen>
Submitted on:  Tue 09 Mar 2010 02:56:28 PM UTC  
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Confirmed
Privacy: PublicAssigned to: Pádraig Brady <pixelbeat>
Open/Closed: Open

Tue 09 Mar 2010 06:18:09 PM UTC, comment #2:

Thanks for the hint. As I noted, I'm not experienced with wide characters ... so I didn't realize that unicode may contain combined characters. I've added a wcwidth() call now to forward the column, so it should handle combining characters and multi column characters correctly.

The modified patch is attached.

(file #19895)

Herbert Thielen <hthielen>
Tue 09 Mar 2010 04:10:21 PM UTC, comment #1:

Thanks for the patch!

However, I'm not sure you can process characters independently like that.
Consider combining characters for example (/usr/bin/printf a\u0301).
I've attached the current i18n patch applied to fedora's expand for reference, which does handle combining chars. Note that does have issues in that it mishandles this case:
printf " \ta\tb\n" | expand --tabs=3 -i
We're currently working on i18n in coreutils but don't let that preclude you from submitting ideas or patches.

(file #19894)

Pádraig Brady <pixelbeat>
Project AdministratorIn charge of this item.
Tue 09 Mar 2010 02:56:28 PM UTC, original submission:

'expand' doesn't replace tab characters by the correct number of spaces in case the input data contains multibyte characters, e.g. german umlauts when using the locale UTF-8.

A small example test file is attached.

I've attached a patch to coreutils 8.4 which works for me, but I'm not experienced in wide character programming, so there might still be some problem - more tests and review needed.

Thanks for the work, best regards


Herbert Thielen <hthielen>


Attached Files
file #19895:  expand.c.patch added by hthielen (2KiB - text/x-patch)
file #19894:  expand-i18n.diff added by pixelbeat (5KiB - text/x-patch)
file #19893:  expand.c.patch added by hthielen (2KiB - text/x-patch)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by pixelbeat (Updated the item)
  • -unavailable- added by hthielen (Submitted the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 09 Mar 2010 06:18:09 PM UTChthielenAttached File-=>Added expand.c.patch, #19895
    Tue 09 Mar 2010 04:10:21 PM UTCpixelbeatAttached File-=>Added expand-i18n.diff, #19894
    Tue 09 Mar 2010 04:00:18 PM UTCpixelbeatStatusNone=>Confirmed
      Assigned toNone=>pixelbeat
    Tue 09 Mar 2010 02:56:28 PM UTChthielenAttached File-=>Added expand-wrong-with-utf8-test.input, #19891
      Attached File-=>Added expand-wrong-with-utf8-test.expected-output, #19892
      Attached File-=>Added expand.c.patch, #19893

    Back to the top

    Powered by Savane 3.1-cleanup1