buggrep - Bugs: bug #28275, grep -P should use PCRE_UTF8

 
 

You are not allowed to post comments on this tracker with your current authentification level.

bug #28275: grep -P should use PCRE_UTF8

Submitted by:  D.O.A. <tkzv>
Submitted on:  Sun 13 Dec 2009 02:06:05 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Confirmed
Privacy: PublicAssigned to: None
Open/Closed: Open

(Jump to the original submission Jump to the original submission)

Mon 07 Jan 2013 02:28:23 AM UTC, comment #9:

I believe that this problem was fixed by the commit:

67436786c110bbb565f0c1b96f1ca1d4586e1c8d

pcresearch: set UTF-8 flag correctly for UTF-8 locales

NIIBE Yutaka <gniibe>
Tue 22 Dec 2009 08:46:10 AM UTC, comment #8:

That would require conversion to UTF-8 before searching (for non-single-byte, non-UTF-8 character sets) but it is a good idea.

Yet another thing to rewrite. :-)

Paolo Bonzini <bonzini>
Mon 21 Dec 2009 02:49:29 PM UTC, comment #7:

> Regarding Perl mode, that's unfortunately a bug in PCRE, not grep.


pcregrep that comes with PCRE works fine if the -u switch is specified. Philip Hazel suggested calling pcre_compile() with PCRE_UTF8 flag set, when UTF-8 is involved. I tried patching grep, but ran into some Gentoo-specific difficulties.

D.O.A. <tkzv>
Thu 17 Dec 2009 01:30:21 AM UTC, comment #6:

The testcase gave me the following results.

% dd if=/dev/urandom bs=1024 count=1024 \
| iconv -c -f ucs-2 -t utf-8 \
| LANG=en_US.UTF8 grep -oha '[a-z]' \
| hexdump -C \
| sed -e 's/^[^ ]//; s/|.//; s/ 0a/\
/g' \
| sed -e 's/^ //; s/ / /g; /^$/d'

c5 a3
c5 b7
c5 ad
77
c4 81
c4 89
c5 9b
68
c2 aa
c5 b5
c4 a7
c3
a8
c5 a9
c3 a6
c4 b8
c3 ae
78
c4 ab
c3 a4
c3 a3
c5 9b
c3 bd

Though I don't know what character "c5 a3", "c5 b7", "c5 ad",
etc show, this behavior is designed in glibc.

Norihirio Tanaka <noritnk>
Mon 14 Dec 2009 11:38:16 PM UTC, comment #5:

> Is this behavior in glibc by design?


In POSIX actually, but yes.

Regarding Perl mode, that's unfortunately a bug in PCRE, not grep. (Personally I consider grep -P a bad idea, but maybe that's just me).

Paolo Bonzini <bonzini>
Mon 14 Dec 2009 11:10:11 PM UTC, comment #4:

Does [a-z] mean any Latin-based symbol? Does [а-я] mean any extended Cyrillic? Is this behavior in glibc by design? Thanks for the explanation.

Still, in Perl mode bracket expressions work incorrectly with non-ASCII symbols.

grep -Poha '[бв]' random-symbols.txt

treats the expression in brackets as 4 1-byte synbols: '\xd0\xb1\xd0\xb2'. It gives a column of 1-byte symbols \xb1 \xb2 and \xd0.

D.O.A. <tkzv>
Mon 14 Dec 2009 09:01:07 PM UTC, comment #3:

As long as it doesn't match cyrillic or arabic or East Asian scripts, it's the intended behavior. To filter letters make sure to set the environment variable LC_ALL=C when running grep.

Paolo Bonzini <bonzini>
Mon 14 Dec 2009 08:57:33 PM UTC, comment #2:

No. It matches various non-ASCII symbols. Like
ǹûṣőṏŭṋṽẚęčẉįļẹèĕểöâǩǝŏä

For example type (on UTF-8 system):

dd if=/dev/urandom bs=1024000 count=1 |iconv -c -f ucs-2 -t utf-8 > random-symbols.txt

grep -oha '[a-z]' random-symbols.txt > 'random [a-z].txt'

and you'll see what I mean.

(file #19265, file #19266)

D.O.A. <tkzv>
Mon 14 Dec 2009 09:38:54 AM UTC, comment #1:

Do you mean that [a-z] also matches for example uppercase? If so, that is expected and is a result of how collating sequences are defined by glibc. For example, try "ls -1" and "LANG=C ls -1" on a directory with both uppecase and lowercase file names.

Paolo Bonzini <bonzini>
Sun 13 Dec 2009 02:06:05 PM UTC, original submission:

In UTF-8 locale if basic or extended regular expressions are selected, ranges like [a-z] or [а-я] seem to match much more symbols, than they should. Simply enumerating all the symbols, e.g. [abcdefghijklmnopqrstuvwxyz] or [абвгдеёжзийклмнопрстуфхцчшщъыьэюя] works fine.

If perl regular expressions are selected (-P switch), ranges with ASCII-only symbols like [a-z] work correctly, but multibyte (both ranges and enumeration) symbols are interpreted as several 1-byte symbols.

D.O.A. <tkzv>

 

Attached Files
file #19266:  random [a-z].txt added by tkzv (835B - text/plain - Random text in UTF and what grep matches to '[a-z]')
file #19265:  random-symbols.txt added by tkzv (169kB - text/plain - Random text in UTF and what grep matches to '[a-z]')

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gniibe (Posted a comment)
  • -unavailable- added by noritnk (Posted a comment)
  • -unavailable- added by bonzini (Posted a comment)
  • -unavailable- added by tkzv (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 22 Dec 2009 08:46:10 AM UTCbonziniStatusInvalid=>Confirmed
      Open/ClosedClosed=>Open
      SummaryRanges like [a-z] incorrectly match in UTF systems=>grep -P should use PCRE_UTF8
    Mon 14 Dec 2009 09:01:07 PM UTCbonziniOpen/ClosedOpen=>Closed
      StatusNone=>Invalid
    Mon 14 Dec 2009 08:57:33 PM UTCtkzvAttached File-=>Added random [a-z].txt, #19266
      Attached File-=>Added random-symbols.txt, #19265

    Back to the top


    Powered by Savane 3.1-cleanup