buggrep - Bugs: bug #36682, Ignore case handling of special...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #36682: Ignore case handling of special unicode characters (case folding)

Submitter:  Johannes Meixner <jsmeix>
Submitted:  Tue 19 Jun 2012 10:35:53 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  In Progress
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Tue 19 Jun 2012 03:59:20 PM UTC, comment #2: 

Thanks for the bug report.  Attached is a proposed patch to the documentation.  This wouldn't be in the man page, which is
supposed to be terse, but it would be in the full documentation.


(file #26063)

Paul Eggert <eggert>
Group Member
Tue 19 Jun 2012 10:40:48 AM UTC, comment #1: 

A typo:

I worte
-----------------------------------------------
but 'HEISS' could be also written as
'HEI[LATIN SMALL LETTER SHARP S]'
-----------------------------------------------
which should be
-----------------------------------------------
but 'HEISS' could be also written as
'HEI[LATIN CAPITAL LETTER SHARP S]'
-----------------------------------------------

Johannes Meixner <jsmeix>
Tue 19 Jun 2012 10:35:53 AM UTC, original submission:  

Basically this is related to what is mentioned in the
"POSIX and --ignore-case" and "Unicode and --ignore-case"
sections in grep's TODO file.

The current behaviour is not a bug in grep
but a missing feature.

But I think that the current behaviour is not sufficiently
described in the grep documentation and this is probably
a (minor) bug in current grep.

Currently "grep -i" does not implement "case folding"
according to what is described in
http://www.unicode.org/versions/Unicode6.1.0/ch05.pdf

Here "case folding" means to have a list of mappings
for special characters (like the Greek sigma or
the German sharp s) how to convert them in both the
RE and the buffer-to-search into a sequence of bytes
which is appropriate for caseless matching using
a binary comparison.

Currently for "grep -i" such special characters
do not match.

For example:

Unicode | UTF-8 (oct.)   | name
----------------------------------------------------------
U+03A3  | 0316 0243      | GREEK CAPITAL LETTER SIGMA
U+03C2  | 0317 0202      | GREEK SMALL LETTER FINAL SIGMA
U+03C3  | 0317 0203      | GREEK SMALL LETTER SIGMA
U+00DF  | 0303 0237      | LATIN SMALL LETTER SHARP S
U+1E9E  | 0341 0272 0236 | LATIN CAPITAL LETTER SHARP S

Note that according to the Unicode specification
converting LATIN SMALL LETTER SHARP S
to uppere case results 'SS' (two ASCII 'S')
and not a LATIN CAPITAL LETTER SHARP S
but converting LATIN CAPITAL LETTER SHARP S
to lower case results LATIN SMALL LETTER SHARP S

Converting to lower case results that
GREEK CAPITAL LETTER SIGMA gets converted
to GREEK SMALL LETTER SIGMA which does not match
GREEK SMALL LETTER FINAL SIGMA.

There is the German lowercase word
'hei[LATIN SMALL LETTER SHARP S]' (English 'hot')
and when it is uppercased it becomes 'HEISS'
but 'HEISS' could be also written as
'HEI[LATIN SMALL LETTER SHARP S]'

Current results:
----------------------------------------------------------
$ export LC_ALL=el_GR.utf8 ; export LANG=el_GR.utf8

$ echo -e '\0316\0243\0316\0243' >SS

$ echo -e '\0316\0243\0317\0202' >Sf

$ echo -e '\0317\0203\0317\0202' >sf

$ echo -e '\0317\0203\0317\0203' >ss

$ grep -q -i -f SS ss && echo yes || echo no
yes

$ grep -q -i -f ss SS && echo yes || echo no
yes

$ grep -q -i -f Sf sf && echo yes || echo no
yes

$ grep -q -i -f sf Sf && echo yes || echo no
yes

$ grep -q -i -f SS sf && echo yes || echo no
no

$ grep -q -i -f sf SS && echo yes || echo no
no

$ export LC_ALL=de_DE.utf8 ; export LANG=de_DE.utf8

$ echo -e 'hei\0303\0237' >heif

$ echo -e 'HEI\0341\0272\0236' >HEIF

$ echo 'HEISS' >HEISS

$ grep -q -i -f heif HEIF && echo yes || echo no
yes

$ grep -q -i -f HEIF heif && echo yes || echo no
yes

$ grep -q -i -f heif HEISS && echo yes || echo no
no

$ grep -q -i -f HEISS heif && echo yes || echo no
no

$ grep -q -i -f HEIF HEISS && echo yes || echo no
no

$ grep -q -i -f HEISS HEIF && echo yes || echo no
no

$ grep -V
grep (GNU grep) 2.12
...
----------------------------------------------------------

The "no" results should be "yes" if "case folding"
would be used according to what is described in
http://www.unicode.org/versions/Unicode6.1.0/ch05.pdf

Missing documentation:

Currently "man grep" does not mention the issue.

The README file mentiones only the "tr_TR.UTF-8" locale.
Probably the "Turkish I/i with/without dot" issue
is meant there.

I think that "man grep" should explicitly mention
the current shortcoming of "-i, --ignore-case".

I suggest to enhance "man grep" by something like this:
------------------------------------------------------------
  -i, --ignore-case
         Ignore case distinctions in both the PATTERN
         and the input files. (-i is specified by POSIX.)
         Currently grep does not implement "case folding"
         accodring to the Unicode Standard. For special
         unicode characters caseless matching fails.
         For example GREEK SMALL LETTER FINAL SIGMA
         does not match GREEK CAPITAL LETTER SIGMA and
         LATIN SMALL LETTER SHARP S does not match 'SS'.
------------------------------------------------------------

Johannes Meixner <jsmeix>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #26063:  grep-doc-diff.txt added by eggert (1KiB - text/plain - Proposed documentation patch.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by eggert (Updated the item)
  • -email is unavailable- added by jsmeix (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-06-19 eggert Attached File- Added grep-doc-diff.txt, #26063
        StatusNone In Progress

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code