Mon 07 Jan 2013 02:28:23 AM UTC, comment #9:
I believe that this problem was fixed by the commit:
67436786c110bbb565f0c1b96f1ca1d4586e1c8d
pcresearch: set UTF-8 flag correctly for UTF-8 locales
|
Tue 22 Dec 2009 08:46:10 AM UTC, comment #8:
That would require conversion to UTF-8 before searching (for non-single-byte, non-UTF-8 character sets) but it is a good idea.
Yet another thing to rewrite. :-)
|
Mon 21 Dec 2009 02:49:29 PM UTC, comment #7:
> Regarding Perl mode, that's unfortunately a bug in PCRE, not grep.
pcregrep that comes with PCRE works fine if the -u switch is specified. Philip Hazel suggested calling pcre_compile() with PCRE_UTF8 flag set, when UTF-8 is involved. I tried patching grep, but ran into some Gentoo-specific difficulties.
|
Thu 17 Dec 2009 01:30:21 AM UTC, comment #6:
The testcase gave me the following results.
% dd if=/dev/urandom bs=1024 count=1024 \
| iconv -c -f ucs-2 -t utf-8 \
| LANG=en_US.UTF8 grep -oha '[a-z]' \
| hexdump -C \
| sed -e 's/^[^ ]//; s/|.//; s/ 0a/\
/g' \
| sed -e 's/^ //; s/ / /g; /^$/d'
c5 a3
c5 b7
c5 ad
77
c4 81
c4 89
c5 9b
68
c2 aa
c5 b5
c4 a7
c3
a8
c5 a9
c3 a6
c4 b8
c3 ae
78
c4 ab
c3 a4
c3 a3
c5 9b
c3 bd
Though I don't know what character "c5 a3", "c5 b7", "c5 ad",
etc show, this behavior is designed in glibc.
|
Mon 14 Dec 2009 11:38:16 PM UTC, comment #5:
> Is this behavior in glibc by design?
In POSIX actually, but yes.
Regarding Perl mode, that's unfortunately a bug in PCRE, not grep. (Personally I consider grep -P a bad idea, but maybe that's just me).
|
Mon 14 Dec 2009 11:10:11 PM UTC, comment #4:
Does [a-z] mean any Latin-based symbol? Does [а-я] mean any extended Cyrillic? Is this behavior in glibc by design? Thanks for the explanation.
Still, in Perl mode bracket expressions work incorrectly with non-ASCII symbols.
grep -Poha '[бв]' random-symbols.txt
treats the expression in brackets as 4 1-byte synbols: '\xd0\xb1\xd0\xb2'. It gives a column of 1-byte symbols \xb1 \xb2 and \xd0.
|
Mon 14 Dec 2009 09:01:07 PM UTC, comment #3:
As long as it doesn't match cyrillic or arabic or East Asian scripts, it's the intended behavior. To filter letters make sure to set the environment variable LC_ALL=C when running grep.
|
Mon 14 Dec 2009 08:57:33 PM UTC, comment #2:
No. It matches various non-ASCII symbols. Like
ǹûṣőṏŭṋṽẚęčẉįļẹèĕểöâǩǝŏä
For example type (on UTF-8 system):
dd if=/dev/urandom bs=1024000 count=1 |iconv -c -f ucs-2 -t utf-8 > random-symbols.txt
grep -oha '[a-z]' random-symbols.txt > 'random [a-z].txt'
and you'll see what I mean.
(file #19265, file #19266)
|
Mon 14 Dec 2009 09:38:54 AM UTC, comment #1:
Do you mean that [a-z] also matches for example uppercase? If so, that is expected and is a result of how collating sequences are defined by glibc. For example, try "ls -1" and "LANG=C ls -1" on a directory with both uppecase and lowercase file names.
|
Sun 13 Dec 2009 02:06:05 PM UTC, original submission:
In UTF-8 locale if basic or extended regular expressions are selected, ranges like [a-z] or [а-я] seem to match much more symbols, than they should. Simply enumerating all the symbols, e.g. [abcdefghijklmnopqrstuvwxyz] or [абвгдеёжзийклмнопрстуфхцчшщъыьэюя] works fine.
If perl regular expressions are selected (-P switch), ranges with ASCII-only symbols like [a-z] work correctly, but multibyte (both ranges and enumeration) symbols are interpreted as several 1-byte symbols.
|