buggrep - Bugs: bug #28275, grep -P should use PCRE_UTF8


You are not allowed to post comments on this tracker with your current authentication level.

bug #28275: grep -P should use PCRE_UTF8

Submitted by:  D.O.A. <tkzv>
Submitted on:  Sun 13 Dec 2009 02:06:05 PM UTC  
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Confirmed
Privacy:  Public Assigned to:  None
Open/Closed:  Open

( Jump to the original submission)

Mon 07 Jan 2013 02:28:23 AM UTC, comment #9: 

I believe that this problem was fixed by the commit:


pcresearch: set UTF-8 flag correctly for UTF-8 locales

NIIBE Yutaka <gniibe>
Tue 22 Dec 2009 08:46:10 AM UTC, comment #8: 

That would require conversion to UTF-8 before searching (for non-single-byte, non-UTF-8 character sets) but it is a good idea.

Yet another thing to rewrite. :-)

Paolo Bonzini <bonzini>
Mon 21 Dec 2009 02:49:29 PM UTC, comment #7: 

> Regarding Perl mode, that's unfortunately a bug in PCRE, not grep.

pcregrep that comes with PCRE works fine if the -u switch is specified. Philip Hazel suggested calling pcre_compile() with PCRE_UTF8 flag set, when UTF-8 is involved. I tried patching grep, but ran into some Gentoo-specific difficulties.

D.O.A. <tkzv>
Thu 17 Dec 2009 01:30:21 AM UTC, comment #6: 

The testcase gave me the following results.

% dd if=/dev/urandom bs=1024 count=1024 \
| iconv -c -f ucs-2 -t utf-8 \
| LANG=en_US.UTF8 grep -oha '[a-z]' \
| hexdump -C \
| sed -e 's/^[^ ]*//; s/|.*//; s/ 0a/\
/g' \
| sed -e 's/^ //; s/  / /g; /^$/d'

c5 a3
c5 b7
c5 ad
c4 81
c4 89
c5 9b
c2 aa
c5 b5
c4 a7
c5 a9
c3 a6
c4 b8
c3 ae
c4 ab
c3 a4
c3 a3
c5 9b
c3 bd

Though I don't know what character "c5 a3", "c5 b7", "c5 ad",
etc show, this behavior is designed in glibc.

Norihirio Tanaka <noritnk>
Mon 14 Dec 2009 11:38:16 PM UTC, comment #5: 

> Is this behavior in glibc by design?

In POSIX actually, but yes.

Regarding Perl mode, that's unfortunately a bug in PCRE, not grep.  (Personally I consider grep -P a bad idea, but maybe that's just me).

Paolo Bonzini <bonzini>
Mon 14 Dec 2009 11:10:11 PM UTC, comment #4: 

Does [a-z] mean any Latin-based symbol? Does [а-я] mean any extended Cyrillic? Is this behavior in glibc by design? Thanks for the explanation.

Still, in Perl mode bracket expressions work incorrectly with non-ASCII symbols.

grep -Poha '[бв]' random-symbols.txt

treats the expression in brackets as 4 1-byte synbols: '\xd0\xb1\xd0\xb2'. It gives a column of 1-byte symbols \xb1 \xb2 and \xd0.

D.O.A. <tkzv>
Mon 14 Dec 2009 09:01:07 PM UTC, comment #3: 

As long as it doesn't match cyrillic or arabic or East Asian scripts, it's the intended behavior.  To filter letters make sure to set the environment variable LC_ALL=C when running grep.

Paolo Bonzini <bonzini>
Mon 14 Dec 2009 08:57:33 PM UTC, comment #2: 

No. It matches various non-ASCII symbols. Like

For example type (on UTF-8 system):

dd if=/dev/urandom bs=1024000 count=1 |iconv -c -f ucs-2 -t utf-8 > random-symbols.txt

grep -oha '[a-z]' random-symbols.txt > 'random [a-z].txt'

and you'll see what I mean.

(file #19265, file #19266)

D.O.A. <tkzv>
Mon 14 Dec 2009 09:38:54 AM UTC, comment #1: 

Do you mean that [a-z] also matches for example uppercase?  If so, that is expected and is a result of how collating sequences are defined by glibc.  For example, try "ls -1" and "LANG=C ls -1" on a directory with both uppecase and lowercase file names.

Paolo Bonzini <bonzini>
Sun 13 Dec 2009 02:06:05 PM UTC, original submission:  

In UTF-8 locale if basic or extended regular expressions are selected, ranges like [a-z] or [а-я] seem to match much more symbols, than they should. Simply enumerating all the symbols, e.g. [abcdefghijklmnopqrstuvwxyz] or [абвгдеёжзийклмнопрстуфхцчшщъыьэюя] works fine.

If perl regular expressions are selected (-P switch), ranges with ASCII-only symbols like [a-z] work correctly, but multibyte (both ranges and enumeration) symbols are interpreted as several 1-byte symbols.

D.O.A. <tkzv>


Attached Files
file #19266:  random [a-z].txt added by tkzv (835B - text/plain - Random text in UTF and what grep matches to '[a-z]')
file #19265:  random-symbols.txt added by tkzv (169KiB - text/plain - Random text in UTF and what grep matches to '[a-z]')


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by gniibe (Posted a comment)
  • -email is unavailable- added by noritnk (Posted a comment)
  • -email is unavailable- added by bonzini (Posted a comment)
  • -email is unavailable- added by tkzv (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-12-22 bonzini StatusInvalid => Confirmed
        Open/ClosedClosed => Open
        SummaryRanges like [a-z] incorrectly match in UTF systems => grep -P should use PCRE_UTF8
    2009-12-14 bonzini Open/ClosedOpen => Closed
        StatusNone => Invalid
    2009-12-14 tkzv Attached File- => Added random [a-z].txt, #19266
        Attached File- => Added random-symbols.txt, #19265

    Back to the top

    Powered by Savane 3.7