bugGNU Octave - Bugs: bug #35911, Backslash escaping in regexprep...

 
 

bug #35911: Backslash escaping in regexprep replacement string incompatibility

Submitter:  Burkart Lingner <burkart>
Submitted:  Tue 20 Mar 2012 05:00:24 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 3.6.1 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 10 Apr 2012 09:16:22 PM UTC, comment #13: 

I checked in the following partial compatibility fix.  Note the FIXME in the do_regexp_string_escapes function.

http://hg.savannah.gnu.org/hgweb/octave/rev/e47d929fde8f

John W. Eaton <jwe>
Group administrator
Fri 30 Mar 2012 05:50:39 PM UTC, comment #12: 

That is indeed a much better way to handle the different escaping behavior. Thanks for the heads-up!

Burkart Lingner <burkart>
Fri 30 Mar 2012 05:39:41 PM UTC, comment #11: 

Hmm, the text box with the code snippet as it is displayed in my browser is not wide enough by default.  If you widen it, you can see that the expression checks that the result of size is equal to 1.

John W. Eaton <jwe>
Group administrator
Fri 30 Mar 2012 05:37:54 PM UTC, comment #10: 

The behavior should be fixed to be compatible for single-quoted strings.  The change is simple enough, and I see no reason to be incompatible here.  There's no need for new configuration variables, or to enable this feature only for --traditional.

If you need to check the behavior to make your script compatible now, I suggest using a test similar to the following to check whether regexprep does backslash escape processing on it third argument instead of comparing version numbers.


exist ('OCTAVE_VERSION') && size (regexprep ('a', 'a', '\n')) == 1


John W. Eaton <jwe>
Group administrator
Fri 30 Mar 2012 05:29:13 PM UTC, comment #9: 

What I meant by "interoperable code" is a script that runs and returns the same output on all of MATLAB, octave --traditional, and Octave without --traditional.

Burkart Lingner <burkart>
Fri 30 Mar 2012 05:24:05 PM UTC, comment #8: 


> Regarding an additional configuration variable, it
> would complicate interoperable code to something like this:


No, the idea is that you can run "octave --traditional" and have all of the silly things that Matlab does.

Jordi GutiƩrrez Hermoso <jordigh>
Group Member
Fri 30 Mar 2012 04:55:52 PM UTC, comment #7: 

Here are the MATLAB 2011b/Octave 3.6.1 comparison results as per request:


regexprep('\n', '.', 'X')


yields "XX" in both MATLAB and Octave which means neither interprets backslash escapes in the single-quoted input string.


str = sprintf('a\nb');
regexprep(str, '\n', 'X')


yields "aXb" in both MATLAB and Octave. However, I wouldn't say that this is due to interpolation of the regex string but rather an inherent feature of regular expressions. See for example


regexprep('a.b', '\w', 'X')


which yields "X.X" in both environments. Here "\w" is recognized by the regex engine just like the "\n" is in the example above.

Nevertheless both MATLAB and Octave can handle


str = sprintf('a\nb');
newline = sprintf('\n');
regexprep(str, newline, 'X')


and both return "aXb". Not that I would personally use regular expressions containing un-escaped newlines. The point is that Octave's regexprep function is already compatible with MATLAB's regarding single-quoting both the input string and the regular expression string.




Regarding an additional configuration variable, it would complicate interoperable code to something like this:


if strcmp(environment, 'Octave') ...
   && ((version < 3.8) || (~interpolate_single_quotes))
  % Octave default: don't interpolate single-quoted strings
  repl = '\n';
else
  % MATLAB-style: do interpolate regexprep replacement strings
  repl = '\\n';
end
out = regexprep(in, 'n', repl);


If single-quoted strings in regexprep were unconditionally interpolated, this would make the if condition a little simpler (granted, a minor improvement) and would make it possible to omit the if statement altogether once the script supports only Octave versions 3.8 and newer. The latter may not be desirable on the release day of 3.8, but what about a few years from now?

Burkart Lingner <burkart>
Fri 30 Mar 2012 12:23:13 AM UTC, comment #6: 

For the original reporter, can you verify that the only string which undergoes interpolation in Matlab is the replacement string?  Or do the original string and the pattern string also need this same fix?

For the original string, try the following


regexprep ('\n', '.', 'X')


This will return either 'X' or 'XX' depending on whether interpolation is done.

For the pattern string, try


str = sprintf ('a\nb');
regexprep (str, '\n', 'X')


If interpolation is active this should come out 'aXb'.

Rik <rik5>
Group administrator
Fri 30 Mar 2012 12:15:55 AM UTC, comment #5: 

John, your patch works.  I thought there must be an existing routine for doing string escapes but couldn't recall the name. 

I'd like to limit this behavior, because it is counter-intuitive, to a Matlab compatibility mode.  Do you object to a new configuration variable?  We already have things like allow_noninteger_range_as_index and do_braindead_shortcircuit_evaluation.  Some possible names

interpolate_single_quotes
interpolate_matlab_single_quotes
do_braindead_quote_interpolation
matlab_single_quote_interpolation
matlab_single_quote_mode

Rik <rik5>
Group administrator
Thu 29 Mar 2012 03:31:14 AM UTC, comment #4: 

I think what you want to do is something like this:


  std::string replacement = args(2).string_value ();
  if (error_state)
    return retval;
  if (args(2).is_sq_string ())
    replacement = do_string_escapes (replacement);


See the attached diff.  Does this change do the right thing?  If so, feel free to add a test that checks that backslash escaping is handled specially for single-quoted strings and check it in.

(file #25502)

John W. Eaton <jwe>
Group administrator
Thu 29 Mar 2012 02:41:57 AM UTC, comment #3: 

You ask a very perceptive question.  I'll speak for myself, as a frequent Octave contributor, and say that I consider Octave's behavior to be superior.  Computers are tools and I think it is useful to be able to say exactly what I want to have happen ('single quotes') or have interpolation happen naturally the way it does in all UNIX shell interfaces ("double quotes").  But, I don't think there would be any hostility to adding a compatibility patch.  There is already an Octave mode that attempts to be as compatible with Matlab as possible (--traditional, also accessible as --braindead).  If interpolation inside single quoted strings was confined to a few places in Octave, and only when explicitly enabled, that would actually make sense.  Octave tries to be a superset of Matlab in that we can always run their code, but they may not be able to run ours.

If you're interested in working on this we can take the discussion to the Octave Maintainers mailing list and work out a strategy.  Just to see how this would go I looked at the case for regexprep.  The relevant code is line 1049 of regexp.cc in the src/DLD-FUNCTIONS directory.

const std::string replacement = args(2).string_value ();

At this point, a C++ string is created from the single-quoted, non-interpreted string that was typed at the command prompt.  It would be easy to drop the "const" declaration and then implement the interpolation manually.

if (matlab_traditional_mode)
  {
    // Series of replacements such as "\\" => "\"
   }

Rik <rik5>
Group administrator
Wed 28 Mar 2012 05:50:07 PM UTC, comment #2: 

Thanks for refreshing my memory regarding the different treatment of single-quoted and double-quoted strings in Octave. So my MATLAB example


regexprep('foobar', '[fb]', '\\') % single quotes replacement


equals


regexprep('foobar', '[fb]', "\\") % double quotes replacement


in Octave, presumably for all escape sequences mentioned at http://www.gnu.org/software/octave/doc/interpreter/Escape-Sequences-in-String-Constants.html#Escape-Sequences-in-String-Constants.

My question is, does this MATLAB incompatibility exist because nobody has taken it on, yet, (which I would totally understand) or is there a consensus amongst Octave developers that the current Octave behavior is actually superior to MATLAB's, i.e. "it's not a bug, it's a feature"?

Burkart Lingner <burkart>
Sun 25 Mar 2012 09:45:49 PM UTC, comment #1: 

This is similar to bug #33611.  It is unlikely that this will be fixed.  Matlab made a poor decision not to have interpolated strings (double quote) and then they worked around it by occasionally having interpolation depending on the context (examples: printf format strings, regexprep replacement strings).  Chasing this kind of bug-for-bug compatibility is not easy and not fun.

For the examples you gave, it would be easiest to switch to interpolated strings in Octave by changing the single quote to a double quote.  Then things like "\\" will be transformed to just '\'.  This isn't too much of a problem if the number of regexprep calls in your code is small.  Otherwise, you could write a Perl or Octave script to convert the last argument of all regexprep calls from single to double quote.

Rik <rik5>
Group administrator
Tue 20 Mar 2012 05:00:24 PM UTC, original submission:  

MATLAB expects backslashes in the replacement string of regexprep(...) to be escaped as '\\' whereas Octave expects a single backslash. Example:


regexprep('foobar', '[fb]', '\\')


yields "\oo\ar" in MATLAB and "\\oo\\ar" in Octave.


regexprep('foobar', '[fb]', '\')


yields "\oo\ar" in both MATLAB and Octave. However, using single backslashes is not a real solution. Let's say you want to replace the "f" and "b" in "foobar" with the character sequence "\n" but not a newline. This would be achieved with the replacement string "\n" in Octave only whereas that would cause a newline to be inserted in MATLAB which requires the replacement string "\\n" for the desired functionality. Another scenario would be that you want to replace something with two consecutive backslashes. This calls for the replacement string "\\" in Octave but for "\\\\" in MATLAB.

Burkart Lingner <burkart>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #25502:  diffs.txt added by jwe (549B - text/plain)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jordigh (Posted a comment)
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by burkart (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-04-11 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2012-04-05 rik5 StatusIn Progress Confirmed
        Assigned torik5 None
    2012-04-04 rik5 StatusNone In Progress
        Assigned toNone rik5
    2012-03-29 jwe Attached File- Added diffs.txt, #25502
    2012-03-25 rik5 Dependencies- Depends on bugs #33611

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code