buggrep - Bugs: bug #28275, grep -P should use PCRE_UTF8


You are not allowed to post comments on this tracker with your current authentication level.

bug #28275: grep -P should use PCRE_UTF8

Submitted by:  D.O.A. <tkzv>
Submitted on:  Sun Dec 13 14:06:05 2009  
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Confirmed
Privacy: PublicAssigned to: None
Open/Closed: Open

(Jump to the original submission Jump to the original submission)

Mon Jan 7 02:28:23 2013, comment #9:

I believe that this problem was fixed by the commit:


pcresearch: set UTF-8 flag correctly for UTF-8 locales

NIIBE Yutaka <gniibe>
Tue Dec 22 08:46:10 2009, comment #8:

That would require conversion to UTF-8 before searching (for non-single-byte, non-UTF-8 character sets) but it is a good idea.

Yet another thing to rewrite. :-)

Paolo Bonzini <bonzini>
Mon Dec 21 14:49:29 2009, comment #7:

> Regarding Perl mode, that's unfortunately a bug in PCRE, not grep.

pcregrep that comes with PCRE works fine if the -u switch is specified. Philip Hazel suggested calling pcre_compile() with PCRE_UTF8 flag set, when UTF-8 is involved. I tried patching grep, but ran into some Gentoo-specific difficulties.

D.O.A. <tkzv>
Thu Dec 17 01:30:21 2009, comment #6:

The testcase gave me the following results.

% dd if=/dev/urandom bs=1024 count=1024 \
| iconv -c -f ucs-2 -t utf-8 \
| LANG=en_US.UTF8 grep -oha '[a-z]' \
| hexdump -C \
| sed -e 's/^[^ ]//; s/|.//; s/ 0a/\
/g' \
| sed -e 's/^ //; s/ / /g; /^$/d'

c5 a3
c5 b7
c5 ad
c4 81
c4 89
c5 9b
c2 aa
c5 b5
c4 a7
c5 a9
c3 a6
c4 b8
c3 ae
c4 ab
c3 a4
c3 a3
c5 9b
c3 bd

Though I don't know what character "c5 a3", "c5 b7", "c5 ad",
etc show, this behavior is designed in glibc.

Norihirio Tanaka <noritnk>
Mon Dec 14 23:38:16 2009, comment #5:

> Is this behavior in glibc by design?

In POSIX actually, but yes.

Regarding Perl mode, that's unfortunately a bug in PCRE, not grep. (Personally I consider grep -P a bad idea, but maybe that's just me).

Paolo Bonzini <bonzini>
Mon Dec 14 23:10:11 2009, comment #4:

Does [a-z] mean any Latin-based symbol? Does [а-я] mean any extended Cyrillic? Is this behavior in glibc by design? Thanks for the explanation.

Still, in Perl mode bracket expressions work incorrectly with non-ASCII symbols.

grep -Poha '[бв]' random-symbols.txt

treats the expression in brackets as 4 1-byte synbols: '\xd0\xb1\xd0\xb2'. It gives a column of 1-byte symbols \xb1 \xb2 and \xd0.

D.O.A. <tkzv>
Mon Dec 14 21:01:07 2009, comment #3:

As long as it doesn't match cyrillic or arabic or East Asian scripts, it's the intended behavior. To filter letters make sure to set the environment variable LC_ALL=C when running grep.

Paolo Bonzini <bonzini>
Mon Dec 14 20:57:33 2009, comment #2:

No. It matches various non-ASCII symbols. Like

For example type (on UTF-8 system):

dd if=/dev/urandom bs=1024000 count=1 |iconv -c -f ucs-2 -t utf-8 > random-symbols.txt

grep -oha '[a-z]' random-symbols.txt > 'random [a-z].txt'

and you'll see what I mean.

(file #19265, file #19266)

D.O.A. <tkzv>
Mon Dec 14 09:38:54 2009, comment #1:

Do you mean that [a-z] also matches for example uppercase? If so, that is expected and is a result of how collating sequences are defined by glibc. For example, try "ls -1" and "LANG=C ls -1" on a directory with both uppecase and lowercase file names.

Paolo Bonzini <bonzini>
Sun Dec 13 14:06:05 2009, original submission:

In UTF-8 locale if basic or extended regular expressions are selected, ranges like [a-z] or [а-я] seem to match much more symbols, than they should. Simply enumerating all the symbols, e.g. [abcdefghijklmnopqrstuvwxyz] or [абвгдеёжзийклмнопрстуфхцчшщъыьэюя] works fine.

If perl regular expressions are selected (-P switch), ranges with ASCII-only symbols like [a-z] work correctly, but multibyte (both ranges and enumeration) symbols are interpreted as several 1-byte symbols.

D.O.A. <tkzv>


Attached Files
file #19266:  random [a-z].txt added by tkzv (835B - text/plain - Random text in UTF and what grep matches to '[a-z]')
file #19265:  random-symbols.txt added by tkzv (169kB - text/plain - Random text in UTF and what grep matches to '[a-z]')


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by gniibe (Posted a comment)
  • -unavailable- added by noritnk (Posted a comment)
  • -unavailable- added by bonzini (Posted a comment)
  • -unavailable- added by tkzv (Submitted the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue Dec 22 08:46:10 2009bonziniStatusInvalid=>Confirmed
      SummaryRanges like [a-z] incorrectly match in UTF systems=>grep -P should use PCRE_UTF8
    Mon Dec 14 21:01:07 2009bonziniOpen/ClosedOpen=>Closed
    Mon Dec 14 20:57:33 2009tkzvAttached File-=>Added random [a-z].txt, #19266
      Attached File-=>Added random-symbols.txt, #19265

    Back to the top

    Powered by Savane 3.1-cleanup1