buggrep - Bugs: bug #18633, regular expressions...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #18633: regular expressions case-insensitivity

Submitter:  Rob Hinks <robhinks>
Submitted:  Fri 29 Dec 2006 08:08:23 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Works For Me
Privacy:  Public Assigned to:  None
Open/Closed:  Closed

Sun 31 Dec 2006 11:47:06 AM UTC, comment #5: 


> Part of my confusion was due to other tools (such as sed, awk
> and vi), not using the collation order specified in the locale

For the first two that would be a bug.  I don't know about 'awk', but here at least 'sed' uses the specified collation order.

> Anyway, thanks for helping me come to a better understanding
> of grep and locale, and apologies for assuming it was a bug.

No problem.

Benno Schulenberg <bens>
Sat 30 Dec 2006 09:38:38 PM UTC, comment #4: 

Yes, when I first posted the 'bug' I hadn't realised that it was due to the collation order of en_GB.UTF-8. It is logical for grep to follow the collation order specified in the locale, so I appreciate it is not a bug.

Part of my confusion was due to other tools (such as sed, awk and vi), not using the collation order specified in the locale (although there is no reason that they should not do things differently). Also, whilst the significance of this is documented in http://www.gnu.org/software/grep/doc/grep_8.html#IDX178 there seems to be a lack of wider awareness of the issue (it seems to be generally assumed that [a-d] is equivalent to [abcd]). This is not a criticism of grep, but I don't think it can hurt to have this discussion as there must be other users who are unaware of their locale collation order and the implications of this on grep (tho perhaps this is the wrong place to have such a discussion!).

Anyway, thanks for helping me come to a better understanding of grep and locale, and apologies for assuming it was a bug.

Rob Hinks <robhinks>
Sat 30 Dec 2006 08:28:18 PM UTC, comment #3: 


> I've also tested LC_ALL-en_US.UTF-8 and this suffers from the
> same problem.


Well, it's not really a problem, it's how it's supposed to work.

> http://www.gnu.org/software/grep/doc/grep_8.html#IDX178
> "It matches any single character that sorts between the two
> characters, inclusive, using the locale's collating sequence
> and character set"


Read on a bit: "Many locales sort characters in dictionary order, and in these locales `[a-d]' is typically not equivalent to `[abcd]'; it might be equivalent to `[aBbCcDd]', for example."

If you do not like that ordering, set somewhere in your startup or login scripts 'export LC_COLLATE=POSIX'.


Benno Schulenberg <bens>
Sat 30 Dec 2006 03:25:42 PM UTC, comment #2: 

Yes, it does seem to be a locale problem. If I type grep '^[a-E]' filename then aAbBcCdDeE are included. When using LC_ALL=C grep '^[a-e]' grep then only abcde are included. My friend who has a different locale has no problems unless he types LC_ALL=en_GB.UTF-8 grep '^[a-e]' filename

My locale is:
rob@fred:~/UnixProgramming$ locale
LANG=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

output of type -a grep:
rob@fred:~/UnixProgramming$ type -a grep
grep is /usr/local/bin/grep
grep is /bin/grep


I've also tested LC_ALL-en_US.UTF-8 and this suffers from the same problem. My friend has done a bit of investigation which might be of interest:

"Link to the page of the manual that i'm kindof talking about.
http://www.gnu.org/software/grep/doc/grep_8.html#IDX178

"It matches any single character that sorts between the two
characters, inclusive, using the locale's collating sequence and
character set"

So, I think this is implying that the character set itself has no
ordering, as it probably shouldn't, although you could infer one based upon the actually encoding underneath (however this is fairly nasty, as it's indirectly exposing the underlying storage structures, which shouldn't have to be sensibly ordered).

Which if you've made it through that sentence - we're onto the
locale's collating sequence. This would be the bit that says how to sort the letters. I'm guessing grep calls a function called get range, taking 2 arguments, start and end.. so, for a-c, it would call getRange(a,c), which would return abc, (or aAbBc) and then use that value in the [ any one from here brackets ]."

Rob Hinks <robhinks>
Sat 30 Dec 2006 02:58:04 PM UTC, comment #1: 


> When I type: grep '^[a-e]' filename
> it returns all lines beginning with an a,b,c,d,e,A,B,C, or D
> (but not an E).


Sounds like a locale problem.  If you type:
  grep '^[a-E]' filename
do all ten letters get included?
And what happens when using:
  LC_ALL=C grep '^[A-Ea-e]' filename

What is the output of 'locale' and of 'type -a grep'?


(In CVS there's another problem, "grep -i --color '^[A-E]'" does
not colour the matches, whereas "grep -i --color '^[a-e]'" does.)

Benno Schulenberg <bens>
Fri 29 Dec 2006 08:08:23 PM UTC, original submission:  

Hi,

I've run into some problems with grep's handling of regular expressions. When I specify a range it returns characters outside the range.

When I type: grep '^[a-e]' filename
it returns all lines beginning with an a,b,c,d,e,A,B,C, or D (but not an E). Similarly, if I type grep '^[A-C]' filename is returns all lines beginning with A,B,C, b or c (but not a).

rob@fred:~/UnixProgramming$ grep '^[a-e]' addresses
DangerMouse, Trusty Red Pillar Box, Baker Street, London
Count Duckula, Castle Duckula, Translyvania
Donatello, Sewers, Manhattan
Babar, The Castle, Celestville

I found this behaviour very surprising and have been doing some investigating to try and get to the bottom of the problem. No-one else seems to have reported the same issue, however, I've discussed the issue with another developer and their version on Windows (2.4.2) suffers from a similar problem. The version I am running is grep (GNU grep) 2.5.1 for Linux.

Are you aware of this problem?

Thanks,

Rob

Rob Hinks <robhinks>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by taa (Updated the item)
  • -email is unavailable- added by bens (Posted a comment)
  • -email is unavailable- added by robhinks (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-03-22 taa StatusNone Works For Me
        Open/ClosedOpen Closed
    2006-12-29 robhinks Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code