buggrep - Bugs: bug #18633, regular expressions...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #18633: regular expressions case-insensitivity

Submitted by:  Rob Hinks <robhinks>
Submitted on:  Fri 29 Dec 2006 08:08:23 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Works For Me
Privacy: PublicAssigned to: None
Open/Closed: Closed

Sun 31 Dec 2006 11:47:06 AM UTC, comment #5:

> Part of my confusion was due to other tools (such as sed, awk
> and vi), not using the collation order specified in the locale

For the first two that would be a bug. I don't know about 'awk', but here at least 'sed' uses the specified collation order.

> Anyway, thanks for helping me come to a better understanding
> of grep and locale, and apologies for assuming it was a bug.

No problem.

Benno Schulenberg <bens>
Sat 30 Dec 2006 09:38:38 PM UTC, comment #4:

Yes, when I first posted the 'bug' I hadn't realised that it was due to the collation order of en_GB.UTF-8. It is logical for grep to follow the collation order specified in the locale, so I appreciate it is not a bug.

Part of my confusion was due to other tools (such as sed, awk and vi), not using the collation order specified in the locale (although there is no reason that they should not do things differently). Also, whilst the significance of this is documented in http://www.gnu.org/software/grep/doc/grep_8.html#IDX178 there seems to be a lack of wider awareness of the issue (it seems to be generally assumed that [a-d] is equivalent to [abcd]). This is not a criticism of grep, but I don't think it can hurt to have this discussion as there must be other users who are unaware of their locale collation order and the implications of this on grep (tho perhaps this is the wrong place to have such a discussion!).

Anyway, thanks for helping me come to a better understanding of grep and locale, and apologies for assuming it was a bug.

Rob Hinks <robhinks>
Sat 30 Dec 2006 08:28:18 PM UTC, comment #3:

> I've also tested LC_ALL-en_US.UTF-8 and this suffers from the
> same problem.


Well, it's not really a problem, it's how it's supposed to work.

> http://www.gnu.org/software/grep/doc/grep_8.html#IDX178
> "It matches any single character that sorts between the two
> characters, inclusive, using the locale's collating sequence
> and character set"


Read on a bit: "Many locales sort characters in dictionary order, and in these locales `[a-d]' is typically not equivalent to `[abcd]'; it might be equivalent to `[aBbCcDd]', for example."

If you do not like that ordering, set somewhere in your startup or login scripts 'export LC_COLLATE=POSIX'.

Benno Schulenberg <bens>
Sat 30 Dec 2006 03:25:42 PM UTC, comment #2:

Yes, it does seem to be a locale problem. If I type grep '^[a-E]' filename then aAbBcCdDeE are included. When using LC_ALL=C grep '^[a-e]' grep then only abcde are included. My friend who has a different locale has no problems unless he types LC_ALL=en_GB.UTF-8 grep '^[a-e]' filename

My locale is:
rob@fred:~/UnixProgramming$ locale
LANG=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

output of type -a grep:
rob@fred:~/UnixProgramming$ type -a grep
grep is /usr/local/bin/grep
grep is /bin/grep

I've also tested LC_ALL-en_US.UTF-8 and this suffers from the same problem. My friend has done a bit of investigation which might be of interest:

"Link to the page of the manual that i'm kindof talking about.
http://www.gnu.org/software/grep/doc/grep_8.html#IDX178

"It matches any single character that sorts between the two
characters, inclusive, using the locale's collating sequence and
character set"

So, I think this is implying that the character set itself has no
ordering, as it probably shouldn't, although you could infer one based upon the actually encoding underneath (however this is fairly nasty, as it's indirectly exposing the underlying storage structures, which shouldn't have to be sensibly ordered).

Which if you've made it through that sentence - we're onto the
locale's collating sequence. This would be the bit that says how to sort the letters. I'm guessing grep calls a function called get range, taking 2 arguments, start and end.. so, for a-c, it would call getRange(a,c), which would return abc, (or aAbBc) and then use that value in the [ any one from here brackets ]."

Rob Hinks <robhinks>
Sat 30 Dec 2006 02:58:04 PM UTC, comment #1:

> When I type: grep '^[a-e]' filename
> it returns all lines beginning with an a,b,c,d,e,A,B,C, or D
> (but not an E).


Sounds like a locale problem. If you type:
grep '^[a-E]' filename
do all ten letters get included?
And what happens when using:
LC_ALL=C grep '^[A-Ea-e]' filename

What is the output of 'locale' and of 'type -a grep'?

(In CVS there's another problem, "grep -i --color '^[A-E]'" does
not colour the matches, whereas "grep -i --color '^[a-e]'" does.)

Benno Schulenberg <bens>
Fri 29 Dec 2006 08:08:23 PM UTC, original submission:

Hi,

I've run into some problems with grep's handling of regular expressions. When I specify a range it returns characters outside the range.

When I type: grep '^[a-e]' filename
it returns all lines beginning with an a,b,c,d,e,A,B,C, or D (but not an E). Similarly, if I type grep '^[A-C]' filename is returns all lines beginning with A,B,C, b or c (but not a).

rob@fred:~/UnixProgramming$ grep '^[a-e]' addresses
DangerMouse, Trusty Red Pillar Box, Baker Street, London
Count Duckula, Castle Duckula, Translyvania
Donatello, Sewers, Manhattan
Babar, The Castle, Celestville

I found this behaviour very surprising and have been doing some investigating to try and get to the bottom of the problem. No-one else seems to have reported the same issue, however, I've discussed the issue with another developer and their version on Windows (2.4.2) suffers from a similar problem. The version I am running is grep (GNU grep) 2.5.1 for Linux.

Are you aware of this problem?

Thanks,

Rob

Rob Hinks <robhinks>

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by taa (Updated the item)
  • -unavailable- added by bens (Posted a comment)
  • -unavailable- added by robhinks (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 22 Mar 2008 07:41:42 PM UTCtaaStatusNone=>Works For Me
      Open/ClosedOpen=>Closed
    Fri 29 Dec 2006 08:08:23 PM UTCrobhinksCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup1