bugGNU nano - Bugs: bug #47626, composed and decomposed characters...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #47626: composed and decomposed characters are not seen as equal

Submitter:  Benno Schulenberg <bens>
Submitted:  Tue 05 Apr 2016 12:15:18 PM UTC
   
 
Severity:  2 - Minor Status:  Wont Fix
Assigned to:  None Open/Closed:  Closed

Mon 07 Dec 2020 10:15:17 AM UTC, comment #2: 

Well, normalizing Unicode strings is complicated.  As the FAQ (http://www.macchiato.com/unicode/nfc-faq_ says, it's best not to roll your own, but I don't want nano to have another dependency (in this case: libicu), so... won't fix.

Anyway, Vim and Emacs don't recognize precomposed and composed characters as equivalent either, so it can hardly be said that nano is falling short.

(Strangely, when xterm and urxvt display a letter that is followed by a combining character, they actually display the equivalent precomposed character.  Which means that copy-pasting something from the screen can give you something different from what is actually in the file.  Xfce Terminal displays the file as it is.)

Benno Schulenberg <bens>
Group administrator
Tue 05 Apr 2016 01:23:51 PM UTC, comment #1: 

Proposed solution: when reading in a file (in a Unicode locale), nano should normalize the encoding.  And, when any normalization occurs, it should note this on the statusbar (just like when a DOS or Mac file is converted -- the difference is that now writing out the file does modify it).  This normalization step should probably be behind a configuration and/or runtime flag.

Benno Schulenberg <bens>
Group administrator
Tue 05 Apr 2016 12:15:18 PM UTC, original submission:  

When a Unicode-encoded text contains characters that are canonically equivalent but encoded in differing manners, then nano will not see them as the same when searching for them.

To reproduce, use nano to open the attached file, which contains the words caña and coöp, each encoded in two ways: once with single code point for the ñ and ö, and once with a normal n/o followed by a combining tilde/diaeresis.  Then, when searching for caña or coöp, nano will find only the first occurrence, not the equivalent second one.

(As a small consolation: vim and emacs and geany and pico don't get this right either.  But gedit does.)

(To confuse matters further: what I call "composed" is what Unicode calls "decomposed": two or more code points that together make up a single character.  And what I call "precomposed" is what Unicode calls "composed": a single code point that signifies a single character that consists of two or more visual elements.)

Benno Schulenberg <bens>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #36840:  equivalents added by bens (29B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by bens (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-12-21 bens Open/ClosedOpen Closed
    2020-12-07 bens StatusNone Wont Fix
    2016-04-05 bens Summarycomposed and precomposed characters are not seen as equal composed and decomposed characters are not seen as equal
    2016-04-05 bens Attached File- Added equivalents, #36840

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code