buggrep - Bugs: bug #36682, Ignore case handling of special...

 
 

You are not allowed to post comments on this tracker with your current authentification level.

bug #36682: Ignore case handling of special unicode characters (case folding)

Submitted by:  Johannes Meixner <jsmeix>
Submitted on:  Tue 19 Jun 2012 10:35:53 AM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: In Progress
Privacy: PublicAssigned to: None
Open/Closed: Open

Tue 19 Jun 2012 03:59:20 PM UTC, comment #2:

Thanks for the bug report. Attached is a proposed patch to the documentation. This wouldn't be in the man page, which is
supposed to be terse, but it would be in the full documentation.

(file #26063)

Paul Eggert <eggert>
Project Member
Tue 19 Jun 2012 10:40:48 AM UTC, comment #1:

A typo:

I worte
-----------------------------------------------
but 'HEISS' could be also written as
'HEI[LATIN SMALL LETTER SHARP S]'
-----------------------------------------------
which should be
-----------------------------------------------
but 'HEISS' could be also written as
'HEI[LATIN CAPITAL LETTER SHARP S]'
-----------------------------------------------

Johannes Meixner <jsmeix>
Tue 19 Jun 2012 10:35:53 AM UTC, original submission:

Basically this is related to what is mentioned in the
"POSIX and --ignore-case" and "Unicode and --ignore-case"
sections in grep's TODO file.

The current behaviour is not a bug in grep
but a missing feature.

But I think that the current behaviour is not sufficiently
described in the grep documentation and this is probably
a (minor) bug in current grep.

Currently "grep -i" does not implement "case folding"
according to what is described in
http://www.unicode.org/versions/Unicode6.1.0/ch05.pdf

Here "case folding" means to have a list of mappings
for special characters (like the Greek sigma or
the German sharp s) how to convert them in both the
RE and the buffer-to-search into a sequence of bytes
which is appropriate for caseless matching using
a binary comparison.

Currently for "grep -i" such special characters
do not match.

For example:

Unicode | UTF-8 (oct.) | name
----------------------------------------------------------
U+03A3 | 0316 0243 | GREEK CAPITAL LETTER SIGMA
U+03C2 | 0317 0202 | GREEK SMALL LETTER FINAL SIGMA
U+03C3 | 0317 0203 | GREEK SMALL LETTER SIGMA
U+00DF | 0303 0237 | LATIN SMALL LETTER SHARP S
U+1E9E | 0341 0272 0236 | LATIN CAPITAL LETTER SHARP S

Note that according to the Unicode specification
converting LATIN SMALL LETTER SHARP S
to uppere case results 'SS' (two ASCII 'S')
and not a LATIN CAPITAL LETTER SHARP S
but converting LATIN CAPITAL LETTER SHARP S
to lower case results LATIN SMALL LETTER SHARP S

Converting to lower case results that
GREEK CAPITAL LETTER SIGMA gets converted
to GREEK SMALL LETTER SIGMA which does not match
GREEK SMALL LETTER FINAL SIGMA.

There is the German lowercase word
'hei[LATIN SMALL LETTER SHARP S]' (English 'hot')
and when it is uppercased it becomes 'HEISS'
but 'HEISS' could be also written as
'HEI[LATIN SMALL LETTER SHARP S]'

Current results:
----------------------------------------------------------
$ export LC_ALL=el_GR.utf8 ; export LANG=el_GR.utf8

$ echo -e '\0316\0243\0316\0243' >SS

$ echo -e '\0316\0243\0317\0202' >Sf

$ echo -e '\0317\0203\0317\0202' >sf

$ echo -e '\0317\0203\0317\0203' >ss

$ grep -q -i -f SS ss && echo yes || echo no
yes

$ grep -q -i -f ss SS && echo yes || echo no
yes

$ grep -q -i -f Sf sf && echo yes || echo no
yes

$ grep -q -i -f sf Sf && echo yes || echo no
yes

$ grep -q -i -f SS sf && echo yes || echo no
no

$ grep -q -i -f sf SS && echo yes || echo no
no

$ export LC_ALL=de_DE.utf8 ; export LANG=de_DE.utf8

$ echo -e 'hei\0303\0237' >heif

$ echo -e 'HEI\0341\0272\0236' >HEIF

$ echo 'HEISS' >HEISS

$ grep -q -i -f heif HEIF && echo yes || echo no
yes

$ grep -q -i -f HEIF heif && echo yes || echo no
yes

$ grep -q -i -f heif HEISS && echo yes || echo no
no

$ grep -q -i -f HEISS heif && echo yes || echo no
no

$ grep -q -i -f HEIF HEISS && echo yes || echo no
no

$ grep -q -i -f HEISS HEIF && echo yes || echo no
no

$ grep -V
grep (GNU grep) 2.12
...
----------------------------------------------------------

The "no" results should be "yes" if "case folding"
would be used according to what is described in
http://www.unicode.org/versions/Unicode6.1.0/ch05.pdf

Missing documentation:

Currently "man grep" does not mention the issue.

The README file mentiones only the "tr_TR.UTF-8" locale.
Probably the "Turkish I/i with/without dot" issue
is meant there.

I think that "man grep" should explicitly mention
the current shortcoming of "-i, --ignore-case".

I suggest to enhance "man grep" by something like this:
------------------------------------------------------------
-i, --ignore-case
Ignore case distinctions in both the PATTERN
and the input files. (-i is specified by POSIX.)
Currently grep does not implement "case folding"
accodring to the Unicode Standard. For special
unicode characters caseless matching fails.
For example GREEK SMALL LETTER FINAL SIGMA
does not match GREEK CAPITAL LETTER SIGMA and
LATIN SMALL LETTER SHARP S does not match 'SS'.
------------------------------------------------------------

Johannes Meixner <jsmeix>

 

Attached Files
file #26063:  grep-doc-diff.txt added by eggert (1kB - text/plain - Proposed documentation patch.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by eggert (Updated the item)
  • -unavailable- added by jsmeix (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 19 Jun 2012 03:59:20 PM UTCeggertAttached File-=>Added grep-doc-diff.txt, #26063
      StatusNone=>In Progress

    Back to the top


    Powered by Savane 3.1-cleanup