bugGNU Octave - Bugs: bug #35910, Incorrect regex matching of...


bug #35910: Incorrect regex matching of multi-byte UTF-8 characters

Submitted by:  Burkart Lingner <burkart>
Submitted on:  Tue 20 Mar 2012 04:29:25 PM UTC  
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Patch Submitted Assigned to:  Rik <rik5>
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Tue 05 Mar 2019 04:59:51 PM UTC, comment #6: 

The attached patch prepends "(*UTF8)" to the pattern string for regular expressions.
I hope this is the correct place to smuggle that string in.

(file #46432)

Markus Mützel <mmuetzel>
Project Member
Sun 03 Mar 2019 08:10:13 PM UTC, comment #5: 

Hmm, if we can't detect any perform difference (probably because when the UTF8 directive, but neither the string or pattern use UTF8, PCRE falls back to regular matching), then we should just add it.  I don't think Octave can be made to compile well on systems 10 years or older.

So, I guess you could prepare a changeset and we could do benchmarking with/without the cset and see if there is any difference before committing

Rik <rik5>
Project AdministratorIn charge of this item.
Sun 03 Mar 2019 05:03:08 PM UTC, comment #4: 

This is still the case with Octave 5.1.
As a work-around, it is possible to start the pattern with "(*UTF8)" with PCRE 7.9 and newer (Apr 2009):

>> string = regexprep('§x', '(*UTF8)^(.)', '$1;')

string = §;x
>> fprintf('%x\n', string)


I didn't see a performance impact with a primitive test:

>> tic, for i=1:1e5, string = regexprep('§x', '(*UTF8)^(.)', '$1;'); end, toc
Elapsed time is 1.20157 seconds.
>> tic, for i=1:1e5, string = regexprep('§x', '^(.)', '$1;'); end, toc
Elapsed time is 1.23969 seconds.

Should we add this automatically? Or at least document it in the docstring for "regexp"?

I can't find a relevant thread in the mailing list archives.

Markus Mützel <mmuetzel>
Project Member
Sat 19 Nov 2016 09:08:45 PM UTC, comment #3: 

This issue is still present in Octave 4.2.0.

Hartmut <hardy>
Thu 04 Jun 2015 01:35:26 AM UTC, comment #2: 

Updating status to the development release, still present.

Mike Miller <mtmiller>
Project Administrator
Sun 25 Mar 2012 08:30:49 PM UTC, comment #1: 

Confirmed on a recent tip.

Currently Octave does not support UTF8 in regular expressions, although it is not hard to add.  I've made an experimental patch that does so but it does impact performance.  UTF8 is 23% slower than non-UTF8 pattern matching. 

Adding support for UTF8 is a big enough change that I am going to take the issue to the Octave Maintainers mailing list so that we can make a decision about how best to implement this.  It is also big enough that this will come out not as a bug fix, but in the next major release of Octave (3.8.0).

Rik <rik5>
Project AdministratorIn charge of this item.
Tue 20 Mar 2012 04:29:25 PM UTC, original submission:  

When matching a single character at a position where there's a multi-byte UTF-8 character, only the first byte is matched. Depending on how this match is then processed, it can result in invalid UTF-8. Example:

string = regexprep('§x', '^(.)', '$1;')
fprintf('%x\n', string)


string = ?;?x

where "?" is the replacement character and the UTF-8 codes for "§", ";", and "x" are "0xC2 0xA7", "0x3B", and "0x78", respectively.

The expected output would have been

string = §;x

Burkart Lingner <burkart>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #46432:  bug35910_regexp_utf8.patch added by mmuetzel (894B - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by apjanke (Adding self to Cc list)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by hardy (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by burkart (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-03-05 mmuetzel Attached File- => Added bug35910_regexp_utf8.patch, #46432
        StatusIn Progress => Patch Submitted
    2019-03-05 apjanke Carbon-Copy- => Added floss@apjanke.net
    2019-03-03 mtmiller Carbon-CopyRemoved 80942 => -
    2015-06-04 mtmiller Release3.6.1 => dev
        Operating SystemGNU/Linux => Any
    2012-03-25 rik5 CategoryInterpreter => Libraries
        StatusNone => In Progress
        Assigned toNone => rik5

    Back to the top

    Powered by Savane 3.5