Sun 31 Dec 2006 11:47:06 AM UTC, comment #5:
> Part of my confusion was due to other tools (such as sed, awk
> and vi), not using the collation order specified in the locale
For the first two that would be a bug. I don't know about 'awk', but here at least 'sed' uses the specified collation order.
> Anyway, thanks for helping me come to a better understanding
> of grep and locale, and apologies for assuming it was a bug.
No problem.
|
Sat 30 Dec 2006 09:38:38 PM UTC, comment #4:
Yes, when I first posted the 'bug' I hadn't realised that it was due to the collation order of en_GB.UTF-8. It is logical for grep to follow the collation order specified in the locale, so I appreciate it is not a bug.
Part of my confusion was due to other tools (such as sed, awk and vi), not using the collation order specified in the locale (although there is no reason that they should not do things differently). Also, whilst the significance of this is documented in http://www.gnu.org/software/grep/doc/grep_8.html#IDX178 there seems to be a lack of wider awareness of the issue (it seems to be generally assumed that [a-d] is equivalent to [abcd]). This is not a criticism of grep, but I don't think it can hurt to have this discussion as there must be other users who are unaware of their locale collation order and the implications of this on grep (tho perhaps this is the wrong place to have such a discussion!).
Anyway, thanks for helping me come to a better understanding of grep and locale, and apologies for assuming it was a bug.
|
Sat 30 Dec 2006 08:28:18 PM UTC, comment #3:
> I've also tested LC_ALL-en_US.UTF-8 and this suffers from the
> same problem.
Well, it's not really a problem, it's how it's supposed to work.
> http://www.gnu.org/software/grep/doc/grep_8.html#IDX178
> "It matches any single character that sorts between the two
> characters, inclusive, using the locale's collating sequence
> and character set"
Read on a bit: "Many locales sort characters in dictionary order, and in these locales `[a-d]' is typically not equivalent to `[abcd]'; it might be equivalent to `[aBbCcDd]', for example."
If you do not like that ordering, set somewhere in your startup or login scripts 'export LC_COLLATE=POSIX'.
|
Sat 30 Dec 2006 03:25:42 PM UTC, comment #2:
Yes, it does seem to be a locale problem. If I type grep '^[a-E]' filename then aAbBcCdDeE are included. When using LC_ALL=C grep '^[a-e]' grep then only abcde are included. My friend who has a different locale has no problems unless he types LC_ALL=en_GB.UTF-8 grep '^[a-e]' filename
My locale is:
rob@fred:~/UnixProgramming$ locale
LANG=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=
output of type -a grep:
rob@fred:~/UnixProgramming$ type -a grep
grep is /usr/local/bin/grep
grep is /bin/grep
I've also tested LC_ALL-en_US.UTF-8 and this suffers from the same problem. My friend has done a bit of investigation which might be of interest:
"Link to the page of the manual that i'm kindof talking about.
http://www.gnu.org/software/grep/doc/grep_8.html#IDX178
"It matches any single character that sorts between the two
characters, inclusive, using the locale's collating sequence and
character set"
So, I think this is implying that the character set itself has no
ordering, as it probably shouldn't, although you could infer one based upon the actually encoding underneath (however this is fairly nasty, as it's indirectly exposing the underlying storage structures, which shouldn't have to be sensibly ordered).
Which if you've made it through that sentence - we're onto the
locale's collating sequence. This would be the bit that says how to sort the letters. I'm guessing grep calls a function called get range, taking 2 arguments, start and end.. so, for a-c, it would call getRange(a,c), which would return abc, (or aAbBc) and then use that value in the [ any one from here brackets ]."
|
Sat 30 Dec 2006 02:58:04 PM UTC, comment #1:
> When I type: grep '^[a-e]' filename
> it returns all lines beginning with an a,b,c,d,e,A,B,C, or D
> (but not an E).
Sounds like a locale problem. If you type:
grep '^[a-E]' filename
do all ten letters get included?
And what happens when using:
LC_ALL=C grep '^[A-Ea-e]' filename
What is the output of 'locale' and of 'type -a grep'?
(In CVS there's another problem, "grep -i --color '^[A-E]'" does
not colour the matches, whereas "grep -i --color '^[a-e]'" does.)
|
Fri 29 Dec 2006 08:08:23 PM UTC, original submission:
Hi,
I've run into some problems with grep's handling of regular expressions. When I specify a range it returns characters outside the range.
When I type: grep '^[a-e]' filename
it returns all lines beginning with an a,b,c,d,e,A,B,C, or D (but not an E). Similarly, if I type grep '^[A-C]' filename is returns all lines beginning with A,B,C, b or c (but not a).
rob@fred:~/UnixProgramming$ grep '^[a-e]' addresses
DangerMouse, Trusty Red Pillar Box, Baker Street, London
Count Duckula, Castle Duckula, Translyvania
Donatello, Sewers, Manhattan
Babar, The Castle, Celestville
I found this behaviour very surprising and have been doing some investigating to try and get to the bottom of the problem. No-one else seems to have reported the same issue, however, I've discussed the issue with another developer and their version on Windows (2.4.2) suffers from a similar problem. The version I am running is grep (GNU grep) 2.5.1 for Linux.
Are you aware of this problem?
Thanks,
Rob
|