bugGNU nano - Bugs: bug #50773, [Request] improve the handling of...


You are not allowed to post comments on this tracker with your current authentication level.

bug #50773: [Request] improve the handling of combining characters

Submitted by:  Mike Frysinger <vapier>
Submitted on:  Mon 10 Apr 2017 05:04:38 AM UTC  
Severity:  1 - Wish Status:  None
Assigned to:  None Open/Closed:  Open

( Jump to the original submission)

Thu 04 May 2017 08:43:38 AM UTC, comment #6: 

> GTK+ wipes out the entire grapheme, Qt only removes the last
> combining character. (Both remove the entire grapheme on Delete.)

Thanks for the information, Egmont.  So... what I am suggesting is that nano should behave like Qt.  Maybe later we'll add an option to make Backspace work like in GTK+.  (But even these combining characters themselves are not high on my list of things to do.)

Benno Schulenberg <bens>
Project Administrator
Wed 03 May 2017 09:26:02 PM UTC, comment #5: 

Just FYI:

GTK+ and Qt behave differently on Backspace in their simplest one-line text entry fields (e.g. gnome-terminal or konsole -> profile preferences -> rename the profile).

GTK+ wipes out the entire grapheme, Qt only removes the last combining character. (Both remove the entire grapheme on Delete.)

Egmont Koblinger <egmont>
Tue 11 Apr 2017 08:33:52 PM UTC, comment #4: 

backspace-vs-delete sounds like a reasonable compromise.  hopefully the user's keyboard provides both :).

i don't feel strongly on the char counting.  if you do, that's fine.

wrt gnulib, i'm not familiar with their modules, and a dumb grep shows they have a bunch.  a search of the manual doesn't help narrow things down.  you should just e-mail bug-gnulib@, describe what you're looking for, and if gnulib has any APIs that might help.  they're a friendly bunch :).

Mike Frysinger <vapier>
Project Member
Tue 11 Apr 2017 08:54:37 AM UTC, comment #3: 

I've read a bit on the Unicode website and encountered this: "on a given system the backspace key might delete by code point, while the delete key may delete an entire cluster." [1]  I think that is the perfect compromise: deleting the entire grapheme when <Delete> is pressed avoids the suprprise of combining character(s) jumping to the preceding character, but still allows to remove combining characters individually by using <Backspace>.  So I don't think a config option is needed there.

For moving the cursor I don't want a config option either: it should always move per grapheme.  And the character count should continue to simply count codepoints, so that the user can realize that there is a compound character somewhere when the cursor moves just one position and the character count changes by two or more.  The man page will have to explain this behavior.

So... all we now need is some complicated logic to make <Left> and <Right> move per grapheme instead of per codepoint.  Surely gnulib has a function to help with this, no?  Something like is_combining(char) or get_combining_class(char)?  :)

[1] http://www.unicode.org/reports/tr29/ -- section 3, fourth paragraph

Benno Schulenberg <bens>
Project Administrator
Mon 10 Apr 2017 08:09:21 PM UTC, comment #2: 

i think either behavior wrt deletion is defensible.  i don't feel too strongly either way since i don't use/encounter them normally, so i can't speak to common workflows.  being able to delete combining characters like emacs does seem like a reasonable position, but so does deleting an entire grapheme.  maybe time for a config option ? :)

i agree that the chars field is, generally speaking, not an exact field one would care about (where as the cols field gets way more usage).  personally i use it as a rough equiv to byte count rather than character (codepoints or graphemes) count.  if nano had the ability to "move" over codepoints w/out updating the cursor, counting codepoints would be ideal as it would be clear that the cursor "moved" in the datastream even if visually it didn't move (row/col position is unchanged).

so if we had a config option that allowed users to specify which they cared about more (codepoints or graphemes), i think answers to all of this would fall out naturally ?

Mike Frysinger <vapier>
Project Member
Mon 10 Apr 2017 03:16:55 PM UTC, comment #1: 

When I open a file that contains just "0ax̊bc" (without the quotes and without a newline) with vim, it says "1L, 7C".  Seven characters!  That's even worse than nano, which correctly identifies six codepoints, six individual characters.

Anyway:  1) Yes, the "stutter" when the cursor moves over a combining character is confusing and annoying.  Although joe and Geany behave the same way as nano currently does, vim and gedit and emacs don't, and I agree it would be better if nano behaved like the latter three: treat a character plus its subsequent combining characters as a single entity when moving the cursor around.

(However, emacs is clever here: when you put the cursor on the "b" and type <Backspace>, it will delete just the combining-ring-above.  And if you put the cursor on the "x" and type <Delete>, only the x disappears and the combining-ring-above moves to the "a".  This may seem weird, but if a user knows about combining characters, she most probably is able to type them too, and then it would be strange that one can type such characters separately but not delete them separately.  I think nano should behave like emacs in this case.)

2) Don't know what I would prefer nano to count: codepoints or graphemes...  How often will a user be interested in exactly how many visual characters her file contains?  What I can imagine is that someone wants their lines to be at most a certain number of columns, but who does an M-D and wants their file to be an exact number of characters?  Also: nano counts newlines as characters, and I am certain that normal users don't count an end-of-line as a character.  So... when newlines are counted, then it is most logical to count also combining characters separately.

Benno Schulenberg <bens>
Project Administrator
Mon 10 Apr 2017 05:04:38 AM UTC, original submission:  

consider this content:
$ printf '0ax\U0000030Abc\n'
there's a combining character (U+030A Combining Ring Above) after the x.

nano shows this status:
[ line 1/2 (50%), col 0/6 (16%), char 0/7 (0%) ]

there's a few weird things here:
- when moving the cursor over the combined character, it stops and lets you delete the combining character, except the position renders after the char (over the "b").  if you put in multiple combining chars, it does the same but more so.
- the chars field is arguably correct ... it's showing codepoints.  but it's also arguably incorrect ... users don't count distinct codepoints, they could graphemes.

comparing it to other programs (like vim and Chrome), they only operate on graphemes.  that means the cursor moves over x̊ in one step, deletes it with one keystroke, etc...  i think nano should do the same.

Mike Frysinger <vapier>
Project Member


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by egmont (Posted a comment)
  • -email is unavailable- added by bens (Posted a comment)
  • -email is unavailable- added by vapier (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-05-27 bens Severity3 - Normal => 1 - Wish
        Summaryhandling of combining characters => [Request] improve the handling of combining characters

    Back to the top

    Powered by Savane 3.5