bugGNU Octave - Bugs: bug #35911, Backslash escaping in regexprep...

 
 

bug #35911: Backslash escaping in regexprep replacement string incompatibility

Submitted by:  Burkart Lingner <burkart>
Submitted on:  Tue 20 Mar 2012 05:00:24 PM UTC  
 
Category: InterpreterSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Matlab Compatibility
Status: FixedAssigned to: None
Originator Name: Open/Closed: Closed
Release: 3.6.1Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 10 Apr 2012 09:16:22 PM UTC, comment #13:

I checked in the following partial compatibility fix. Note the FIXME in the do_regexp_string_escapes function.

http://hg.savannah.gnu.org/hgweb/octave/rev/e47d929fde8f

John W. Eaton <jwe>
Project Administrator
Fri 30 Mar 2012 05:50:39 PM UTC, comment #12:

That is indeed a much better way to handle the different escaping behavior. Thanks for the heads-up!

Burkart Lingner <burkart>
Fri 30 Mar 2012 05:39:41 PM UTC, comment #11:

Hmm, the text box with the code snippet as it is displayed in my browser is not wide enough by default. If you widen it, you can see that the expression checks that the result of size is equal to 1.

John W. Eaton <jwe>
Project Administrator
Fri 30 Mar 2012 05:37:54 PM UTC, comment #10:

The behavior should be fixed to be compatible for single-quoted strings. The change is simple enough, and I see no reason to be incompatible here. There's no need for new configuration variables, or to enable this feature only for --traditional.

If you need to check the behavior to make your script compatible now, I suggest using a test similar to the following to check whether regexprep does backslash escape processing on it third argument instead of comparing version numbers.

John W. Eaton <jwe>
Project Administrator
Fri 30 Mar 2012 05:29:13 PM UTC, comment #9:

What I meant by "interoperable code" is a script that runs and returns the same output on all of MATLAB, octave --traditional, and Octave without --traditional.

Burkart Lingner <burkart>
Fri 30 Mar 2012 05:24:05 PM UTC, comment #8:

> Regarding an additional configuration variable, it
> would complicate interoperable code to something like this:


No, the idea is that you can run "octave --traditional" and have all of the silly things that Matlab does.

Jordi GutiƩrrez Hermoso <jordigh>
Project Administrator
Fri 30 Mar 2012 04:55:52 PM UTC, comment #7:

Here are the MATLAB 2011b/Octave 3.6.1 comparison results as per request:

yields "XX" in both MATLAB and Octave which means neither interprets backslash escapes in the single-quoted input string.

yields "aXb" in both MATLAB and Octave. However, I wouldn't say that this is due to interpolation of the regex string but rather an inherent feature of regular expressions. See for example

which yields "X.X" in both environments. Here "\w" is recognized by the regex engine just like the "\n" is in the example above.

Nevertheless both MATLAB and Octave can handle

and both return "aXb". Not that I would personally use regular expressions containing un-escaped newlines. The point is that Octave's regexprep function is already compatible with MATLAB's regarding single-quoting both the input string and the regular expression string.


Regarding an additional configuration variable, it would complicate interoperable code to something like this:

If single-quoted strings in regexprep were unconditionally interpolated, this would make the if condition a little simpler (granted, a minor improvement) and would make it possible to omit the if statement altogether once the script supports only Octave versions 3.8 and newer. The latter may not be desirable on the release day of 3.8, but what about a few years from now?

Burkart Lingner <burkart>
Fri 30 Mar 2012 12:23:13 AM UTC, comment #6:

For the original reporter, can you verify that the only string which undergoes interpolation in Matlab is the replacement string? Or do the original string and the pattern string also need this same fix?

For the original string, try the following

This will return either 'X' or 'XX' depending on whether interpolation is done.

For the pattern string, try

If interpolation is active this should come out 'aXb'.

Rik <rik5>
Project Administrator
Fri 30 Mar 2012 12:15:55 AM UTC, comment #5:

John, your patch works. I thought there must be an existing routine for doing string escapes but couldn't recall the name.

I'd like to limit this behavior, because it is counter-intuitive, to a Matlab compatibility mode. Do you object to a new configuration variable? We already have things like allow_noninteger_range_as_index and do_braindead_shortcircuit_evaluation. Some possible names

interpolate_single_quotes
interpolate_matlab_single_quotes
do_braindead_quote_interpolation
matlab_single_quote_interpolation
matlab_single_quote_mode

Rik <rik5>
Project Administrator
Thu 29 Mar 2012 03:31:14 AM UTC, comment #4:

I think what you want to do is something like this:

See the attached diff. Does this change do the right thing? If so, feel free to add a test that checks that backslash escaping is handled specially for single-quoted strings and check it in.

(file #25502)

John W. Eaton <jwe>
Project Administrator
Thu 29 Mar 2012 02:41:57 AM UTC, comment #3:

You ask a very perceptive question. I'll speak for myself, as a frequent Octave contributor, and say that I consider Octave's behavior to be superior. Computers are tools and I think it is useful to be able to say exactly what I want to have happen ('single quotes') or have interpolation happen naturally the way it does in all UNIX shell interfaces ("double quotes"). But, I don't think there would be any hostility to adding a compatibility patch. There is already an Octave mode that attempts to be as compatible with Matlab as possible (--traditional, also accessible as --braindead). If interpolation inside single quoted strings was confined to a few places in Octave, and only when explicitly enabled, that would actually make sense. Octave tries to be a superset of Matlab in that we can always run their code, but they may not be able to run ours.

If you're interested in working on this we can take the discussion to the Octave Maintainers mailing list and work out a strategy. Just to see how this would go I looked at the case for regexprep. The relevant code is line 1049 of regexp.cc in the src/DLD-FUNCTIONS directory.

const std::string replacement = args(2).string_value ();

At this point, a C++ string is created from the single-quoted, non-interpreted string that was typed at the command prompt. It would be easy to drop the "const" declaration and then implement the interpolation manually.

if (matlab_traditional_mode)
{
// Series of replacements such as "\\" => "\"
}

Rik <rik5>
Project Administrator
Wed 28 Mar 2012 05:50:07 PM UTC, comment #2:

Thanks for refreshing my memory regarding the different treatment of single-quoted and double-quoted strings in Octave. So my MATLAB example

equals

in Octave, presumably for all escape sequences mentioned at http://www.gnu.org/software/octave/doc/interpreter/Escape-Sequences-in-String-Constants.html#Escape-Sequences-in-String-Constants.

My question is, does this MATLAB incompatibility exist because nobody has taken it on, yet, (which I would totally understand) or is there a consensus amongst Octave developers that the current Octave behavior is actually superior to MATLAB's, i.e. "it's not a bug, it's a feature"?

Burkart Lingner <burkart>
Sun 25 Mar 2012 09:45:49 PM UTC, comment #1:

This is similar to bug #33611. It is unlikely that this will be fixed. Matlab made a poor decision not to have interpolated strings (double quote) and then they worked around it by occasionally having interpolation depending on the context (examples: printf format strings, regexprep replacement strings). Chasing this kind of bug-for-bug compatibility is not easy and not fun.

For the examples you gave, it would be easiest to switch to interpolated strings in Octave by changing the single quote to a double quote. Then things like "\\" will be transformed to just '\'. This isn't too much of a problem if the number of regexprep calls in your code is small. Otherwise, you could write a Perl or Octave script to convert the last argument of all regexprep calls from single to double quote.

Rik <rik5>
Project Administrator
Tue 20 Mar 2012 05:00:24 PM UTC, original submission:

MATLAB expects backslashes in the replacement string of regexprep(...) to be escaped as '\\' whereas Octave expects a single backslash. Example:

yields "\oo\ar" in MATLAB and "\\oo\\ar" in Octave.

yields "\oo\ar" in both MATLAB and Octave. However, using single backslashes is not a real solution. Let's say you want to replace the "f" and "b" in "foobar" with the character sequence "\n" but not a newline. This would be achieved with the replacement string "\n" in Octave only whereas that would cause a newline to be inserted in MATLAB which requires the replacement string "\\n" for the desired functionality. Another scenario would be that you want to replace something with two consecutive backslashes. This calls for the replacement string "\\" in Octave but for "\\\\" in MATLAB.

Burkart Lingner <burkart>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #25502:  diffs.txt added by jwe (549B - text/plain)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jordigh (Posted a comment)
  • -unavailable- added by jwe (Updated the item)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by burkart (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 8 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 11 Apr 2012 06:12:50 AM UTCrik5StatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Thu 05 Apr 2012 07:03:12 PM UTCrik5StatusIn Progress=>Confirmed
      Assigned torik5=>None
    Wed 04 Apr 2012 12:34:48 AM UTCrik5StatusNone=>In Progress
      Assigned toNone=>rik5
    Thu 29 Mar 2012 03:31:14 AM UTCjweAttached File-=>Added diffs.txt, #25502
    Sun 25 Mar 2012 09:45:49 PM UTCrik5Dependencies-=>Depends on bugs #33611

    Back to the top


    Powered by Savane 3.1-cleanup