bugGNU nano - Bugs: bug #55982, the regex for Ruby constants...


You are not allowed to post comments on this tracker with your current authentication level.

bug #55982: the regex for Ruby constants matches too many things

Submitted by:  Rory Thrasher <thrasher>
Submitted on:  Fri 22 Mar 2019 10:39:24 PM UTC  
Severity:  3 - Normal Status:  Need Info
Assigned to:  Benno Schulenberg <bens> Open/Closed:  Closed

( Jump to the original submission)

Sun 09 Feb 2020 10:36:06 AM UTC, comment #17: 

Closing due to a lack of information.  When you have new info, please post it and I will reopen the issue.

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Tue 21 Jan 2020 06:11:40 PM UTC, comment #16: 

Ping...  How is things with this?  Is this still a problem on your machine?  If yes, did you check (as suggested in the previous comment #15) how a self-compiled nano behaves?

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Sun 27 Oct 2019 03:53:27 PM UTC, comment #15: 

On an inherited machine, I've run a Fedora 30 live image inside VirtualBox, have installed the nano package (version 4.2) and have dowloaded the global.rb file.  But I don't see any miscolorings: "require" and other words that start with a lowercase letter are in the default color, strings are green, keywords like "module" and "class" are brown, and only the words that start with a capital are blue.

Only when I change the relevant 'color' command to 'icolor', do most words turn blue.

Could you try compiling nano yourself on the affected machine?  The latest version, 4.5?  Maybe the regex stuff on your machine is weird, and compiling nano yourself should use the regex stuff from gnulib that is included in the nano tarball.  When compilation is complete, you can check that the resultant nano uses gnulib's regex routines by running:

  objdump -t src/nano | grep rpl_re

It should list a dozen or so functions.

Then see if this src/nano binary exhibits the same problem.

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Sat 18 May 2019 06:36:42 AM UTC, comment #14: 

Sorry for the delay on this.  I haven't forgotten, just haven't gotten around to it yet.  I'm hoping to try a live-usb next week.

I'll note that a colleague using fedora didn't have this issue, however a new user on my personal machine still had the issue even without my personalized bashrc/nanorc/etc.

Rory Thrasher <thrasher>
Sun 21 Apr 2019 10:21:12 AM UTC, comment #13: 

Any news on this, Rory?  Does it affect also a fresh Fedora install?

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Sat 30 Mar 2019 09:51:18 AM UTC, comment #12: 

It must be some locale setting on your system.  It's not really treating regexes case-insensitively, because if that were the case, [[:upper:]] wouldn't work either.  (When in nano you search for, for example, "[[:upper:]]om", it finds also "some" and "command".  You have to toggle case-sensitivity on to make it find only "Home" and such.)

If you can check in a virtual machine how a default Fedora install behaves in this respect, that would be great.

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Fri 29 Mar 2019 08:57:42 PM UTC, comment #11: 

Yup. "zeep" is in blue.

I've also swapped out my usual ~/.nanorc with one that only has `include "/usr/share/nano/ruby.nanorc"` just to be sure that it wasn't some other setting messing with it.

What else might be causing the regex to be case insensitive?

I wonder if this is something specific to my system somehow, or if its something a blank fedora VM would see as well...

Rory Thrasher <thrasher>
Fri 29 Mar 2019 08:20:42 PM UTC, comment #10: 

Strange...  For me, the en_US.UTF-8 locale does not cause regexes to become case insensitive.

Can you type the lowercase word "zeep" into a ruby file and see if it gets highlighted in blue (when using [A-Z] instead of [[:upper:]])?

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Tue 26 Mar 2019 07:51:50 PM UTC, comment #9: 

$ locale

Rory Thrasher <thrasher>
Tue 26 Mar 2019 07:50:08 PM UTC, comment #8: 

To see your locale, please paste the full output of the 'locale' command.  I'm guessing that you have a changed setting for LC_COLLATE, which makes that [A-Z] does not cover only uppercase letters but also lowercase ones (at least a-y, probably not z).

Yes, word boundaries are working fine.  The problem is that the nano syntax regexes are not locale-resistant -- they should always use [[:upper:]] instead of [A-Z], and similar character classes.  There is some work to be done.  :|

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Tue 26 Mar 2019 07:32:22 PM UTC, comment #7: 

Sorry, missed the second comment.  It is only finding words that begin with "he".  So I think that means word boundaries are working fine.

Rory Thrasher <thrasher>
Tue 26 Mar 2019 07:24:47 PM UTC, comment #6: 

Interesting.  Changing to [[:upper:]] seems to work pretty well. It doesn't color global/instance variables that are lowercase (@args or @failed or @messages in global.rb) - but I assume that is what the oversight missed?

I'm using the nano that comes with fedora 28.

$ cat /etc/locale.conf
$ nano --version
 GNU nano, version 2.9.8
 (C) 1999-2011, 2013-2018 Free Software Foundation, Inc.
 (C) 2014-2018 the contributors to nano
 Email: -email is unavailable- Web: https://nano-editor.org/
 Compiled options: --enable-utf8

@@ are class variables: https://www.ruby-lang.org/en/documentation/faq/8/

Rory Thrasher <thrasher>
Tue 26 Mar 2019 06:21:57 PM UTC, comment #5: 

To exclude a problem with word boundaries, could you run:

  nano --ignore NEWS

and type: ^W M-R \<he <Enter> M-W M-W M-W

(That is: Ctrl+W  Alt+R  \<he  Ctrl+M  Alt+W Alt+W Alt+W)

Are only words that start with "he" or "He" found?  Or also words like "the" and "other"?

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Tue 26 Mar 2019 06:15:08 PM UTC, comment #4: 

Looking at that global.rb file, for me all the words "require" have the normal color, the strings are green, and only the words that start with an uppercase letter (I18n, ActiveSupport, Inflector, and so on) are bright blue.  It would seems that the regex functions on your system somehow use REG_ICASE by default.

What platform are you on?  What is your locale?  Which version of nano?

If, in the regex for the constants, you change [A-Z]+ to [[:upper:]]+, does it work better then?  If you remove the plus (+) (it is redundant), does it work better then?

Looking at the regex, it indeed seems that, besides constants, it was also meant to color global variables and instance variables, but it failed to do that because of an oversight.  (What the @@ means in Ruby, I have no idea, I can't find it.)

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Mon 25 Mar 2019 09:05:00 PM UTC, comment #3: 

Hmmm.  I guess I was focusing on class/instance/global variables.  I'm not sure if that regex is also supposed to get generic class/instance/variables that are all lowercase (eg, @foo) or if it is only constants (that might also be class/instance/global - eg @BAR).  Highlighting class/instance/global variables that aren't constants might also be nice if those aren't intended as part of this regex.

I'll use https://github.com/Homebrew/brew/blob/master/Library/Homebrew/global.rb as it has examples of constants and instance variables.

Using the default syntax highlighting, almost everything ends up being brightblue.  Starting from the first word on the first word, which is `require` - which should not match.  Later on in the file you can see `def` and `end` colored blue as well, which should be yellow but are being overwritten.

My regex abilities aren't good enough to really understand what the problem or fix is (my change is definitely not right, as you pointed out).  While moving the color rule for constants before the reserved words will let all the reserved words come through as yellow - there is still way too much matching brightblue.  Perhaps its something to do with word boundaries?

Rory Thrasher <thrasher>
Mon 25 Mar 2019 04:35:34 PM UTC, comment #2: 

Hmm...  I don't see how the color rule for Ruby constants could miscolor any of the reserved words (apart from "BEGIN" and "END"), because most of those reserved word are in all-lowercase, and constants must start with an uppercase letter, says the color rule.  So... if you really see this miscoloring, please give me the URL of a Ruby file where you see it (because I have no Ruby files on my machine and zero knowledge of the language).

Your proposed fix to avoid the miscoloring is not right, because it would color only capitalized words that are preceded with "$" or "@" or "@@"; it would not color capitalized words by themselves (that is: actual constants -- see https://ruby-doc.org/docs/ruby-doc-bundle/Manual/man-1.4/syntax.html#variable ).

To avoid the miscoloring of "BEGIN" and "END", wouldn't it be enough to place the color rule for constants before the color rule for reserved words?

Benno Schulenberg <bens>
Project AdministratorIn charge of this item.
Sun 24 Mar 2019 05:21:18 PM UTC, comment #1: 

On second though, `{1}` is redundant, isn't it?  So...

## Constants
color brightblue "(\$|@|@@)\<[A-Z]+[0-9A-Z_a-z]*"

Rory Thrasher <thrasher>
Fri 22 Mar 2019 10:39:24 PM UTC, original submission:  

The regex for ruby constants has an error where it matches almost every word, including reserved words and variables.  It tries to match with things starting with $|@|@@, but it does it 0 or 1 times - which matches basically everything.  This overwrites the reserved words that are defined right before this.

    ## Constants
    color brightblue "(\$|@|@@)?\<[A-Z]+[0-9A-Z_a-z]*"

It should instead be something like...

    ## Constants
    color brightblue "(\$|@|@@){1}\<[A-Z]+[0-9A-Z_a-z]*"

...so that it properly matches things that start with $|@|@@ exactly once.

Rory Thrasher <thrasher>


Attached Files
file #46627:  patch added by thrasher (597B - application/octet-stream - Patchfile with possible fix)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by bens (Posted a comment)
  • -email is unavailable- added by thrasher (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-02-09 bens Open/ClosedOpen => Closed
    2019-03-26 bens Severity2 - Minor => 3 - Normal
    2019-03-25 bens Severity3 - Normal => 2 - Minor
        StatusNone => Need Info
        Assigned toNone => bens
        SummaryRuby syntax highlighting regex for constants matches too many things => the regex for Ruby constants matches too many things
    2019-03-22 thrasher Attached File- => Added patch, #46627

    Back to the top

    Powered by Savane 3.5