bugGNU nano - Bugs: bug #61487, in a UTF-8 locale, '\xčć' is...


You are not allowed to post comments on this tracker with your current authentication level.

bug #61487: in a UTF-8 locale, '\xčć' is colorized as valid hex in a C file

Submitted by:  Benno Schulenberg <bens>
Submitted on:  Mon 15 Nov 2021 02:44:13 PM UTC  
Severity:  2 - Minor Status:  Fixed
Assigned to:  Benno Schulenberg <bens> Open/Closed:  Open

Mon 22 Nov 2021 10:54:56 AM UTC, comment #5: 

Thanks for digging into this.

On my local systems and three remote systems (all Debian-based), echo 'čđ' | grep '[A-Fa-f]' matches.  On five other remote systems (Alpine, Gentoo, the three BSDs), it doesn't match.  (With LANG either nl_NL.UTF-8 or en_US.UTF-8.)

And indeed, after running a default ./configure (or with --with-included-regex), then in a nano syntax [A-Fa-f] does not match čđ.  Strange.  I think the latter is a bug in gnulib (or in the way nano uses gnulib), but... as long as no one complains about a specific use case they expect to work... I'm not going to bother with that.

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Sun 21 Nov 2021 09:50:39 PM UTC, comment #4: 

Putting `setlocale(LC_ALL, "")` in the test program makes the [0-9A-Fa-f] pattern match "čđ" on Linux systems, but not macOS.

Looks like there's no guarantee that a system changes its regular-expression behavior based on locale. macOS doesn't, and neither does the GNU C library included with nano.

Brad Town <townba>
Sun 21 Nov 2021 07:13:41 PM UTC, comment #3: 

I was able to reproduce this bug on Debian (rodete) by using "./configure --without-included-regex". It goes away with a default configuration (what I'd used before) or "./configure --with-included-regex".

The small test program in my other comment is using the system libraries but doesn't exhibit the problem, which I find interesting.

Brad Town <townba>
Sun 21 Nov 2021 06:42:17 PM UTC, comment #2: 

This is a good change, but it shows that there's something else going on, perhaps with your system.

Those characters should not be considered part of that range. On the systems I tried (macOS and Debian [rodete]), syncing to before your patch, nano doesn't have that bug. The installed versions don't have it, either. (LANG=en_US.UTF-8)

Some tests with other programs:

  • In JavaScript, "čđ".match(/[0-9A-Fa-f]/) returns null.
  • With egrep, echo 'čđ' | egrep '[0-9A-Fa-f]+' does not match.
  • The following code does not match. I tried it on macOS, a Raspberry Pi 3 running Buster, and a Debian system.

#include <regex.h>
#include <stdio.h>

int main(int argc, const char *argv[]) {
  const char *pattern = "[0-9A-Fa-f]+";
  const char *input = "čđ";  // "\xc4\x8d\xc4\x91"
  regex_t re;
  if (regcomp(&re, pattern, REG_EXTENDED) != 0) {
    return 1;
  regmatch_t match;
  if (regexec(&re, input, 1, &match, 0) == 0) {
    fprintf(stderr, "\"%s\" matches pattern /%s/.\n", input, pattern);
  } else {
    fprintf(stderr, "\"%s\" does not match pattern /%s/.\n", input, pattern);

Brad Town <townba>
Mon 15 Nov 2021 03:22:55 PM UTC, comment #1: 

Fixed in git, commit f757c4a5, by changing all ranges [0-9A-Fa-f] to [[:xdigit:]] -- and [0-9ABCDEFabcdef] too, for shortness and symmetry.

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Mon 15 Nov 2021 02:44:13 PM UTC, original submission:  

To reproduce, run in a UTF-8 locale:

src/nano --rc=syntax/c.nanorc +25 src/text.c

Then paste this line into the file: '\xcd' '\xčđ'
See that both things are colorized as being valid hexadecimal.  But the second is clearly not.

Apparently most UTF-8 locales include several accented characters in the A-F and a-f ranges.  So the syntax rules (for C and other files) should not use those ranges but instead use the character class [:xdigit:].  (The full enumeration ABCDEFabcdef would work too, but [:xdigit:] is shorter.)

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by townba (Posted a comment)
  • -email is unavailable- added by bens (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-11-15 bens StatusNone => Fixed
        Summaryin a UTF-8 locale, '\xčđ' is colorized as if it were valid hex in a C file => in a UTF-8 locale, '\xčć' is colorized as valid hex in a C file

    Back to the top

    Powered by Savane 3.7