bugGNU nano - Bugs: bug #47626, composed and decomposed characters...

 
 

bug #47626: composed and decomposed characters are not seen as equal

Submitted by:  Benno Schulenberg <bens>
Submitted on:  Tue 05 Apr 2016 12:15:18 PM UTC  
 
Severity: 2 - MinorStatus: None
Assigned to: NoneOpen/Closed: Open

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 05 Apr 2016 01:23:51 PM UTC, comment #1:

Proposed solution: when reading in a file (in a Unicode locale), nano should normalize the encoding. And, when any normalization occurs, it should note this on the statusbar (just like when a DOS or Mac file is converted -- the difference is that now writing out the file does modify it). This normalization step should probably be behind a configuration and/or runtime flag.

Benno Schulenberg <bens>
Project Administrator
Tue 05 Apr 2016 12:15:18 PM UTC, original submission:

When a Unicode-encoded text contains characters that are canonically equivalent but encoded in differing manners, then nano will not see them as the same when searching for them.

To reproduce, use nano to open the attached file, which contains the words caña and coöp, each encoded in two ways: once with single code point for the ñ and ö, and once with a normal n/o followed by a combining tilde/diaeresis. Then, when searching for caña or coöp, nano will find only the first occurrence, not the equivalent second one.

(As a small consolation: vim and emacs and geany and pico don't get this right either. But gedit does.)

(To confuse matters further: what I call "composed" is what Unicode calls "decomposed": two or more code points that together make up a single character. And what I call "precomposed" is what Unicode calls "composed": a single code point that signifies a single character that consists of two or more visual elements.)

Benno Schulenberg <bens>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #36840:  equivalents added by bens (29B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by bens (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 05 Apr 2016 01:23:51 PM UTCbensSummarycomposed and precomposed characters are not seen as equal=>composed and decomposed characters are not seen as equal
    Tue 05 Apr 2016 12:15:18 PM UTCbensAttached File-=>Added equivalents, #36840

    Back to the top


    Powered by Savane 3.1-cleanup1