bugGNU Octave - Bugs: bug #38149, regexp: doesn't return empty...

 
 

bug #38149: regexp: doesn't return empty tokens -- they just disappear

Submitter:  None
Submitted:  Wed 23 Jan 2013 10:55:06 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  3 - Low Item Group:  Matlab Compatibility
Status:  Postponed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 22 Jul 2022 08:57:51 PM UTC, comment #17: 

this appears to still be an issue in curred dev branch for v8.0.0

in the comment #0 code, the 'this works' example matches between octave v8.0.0 and matlab 2022a. The second two work in matlab but in Octave produce:

error: t(2): out of bound 1 (dimensions are 1x1)


for Rik's test script in comment #13, matlab 2022a currently produces:


>> te = regexp ('abc', 'abc(z?)', 'tokenExtents');
celldisp (te)
te{1} =
     4     3
>> te = regexp ('abce', '((a)(z)?b(c)(z*))(z)?(e)?', 'tokenExtents');
celldisp (te)
te{1} =
     1     3
     4     3
     4     4
>> [T, TE, NM] = regexp (sprintf('%s','John Davis\nRogers, James'), '(?<first>\w+)\s+(?<last>\w+)|(?<last>\w+),\s+(?<first>\w+)', 'tokens', 'tokenExtents', 'names');
>> celldisp (T)
T{1}{1} =
John
T{1}{2} =
Davis
T{2}{1} =
nRogers
T{2}{2} =
James
>> celldisp (TE)
TE{1} =
     1     4
     6    10
TE{2} =
    12    18
    21    25
>> NM(1).first
ans =
    'John'
>> NM(1).last
ans =
    'Davis'
>> NM(2).first
ans =
    'James'
>> NM(2).last
ans =
    'nRogers'


while octave produces:


>> te = regexp ('abc', 'abc(z?)', 'tokenExtents');
>> celldisp (te)
te{1} =

   4   3

>> te = regexp ('abce', '((a)(z)?b(c)(z*))(z)?(e)?', 'tokenExtents')
;
>> celldisp (te)
te{1} =

   1   3
   1   1
   3   3
   4   3
   4   4

>> [T, TE, NM] = regexp (sprintf('%s','John Davis\nRogers, James'),
'(?<first>\w+)\s+(?<last>\w+)|(?<last>\w+),\s+(?<first>\w+)', 'token
s', 'tokenExtents', 'names');
>> celldisp (T)
T{1}{1} =

John

T{1}{2} =

Davis

T{2}{1} =

nRogers

T{2}{2} =

James

>> celldisp (TE)
TE{1} =

    1    4
    6   10

TE{2} =

   12   18
   21   25

>> NM(1).first
ans = John
>> NM(1).last
ans = Davis
>> NM(2).first
ans = James
>> NM(2).last
ans = nRogers


where the only difference appears in the 2nd "celldisp (te)" line.

 Nicholas Jankowski <nrjank>
Group Member
Thu 04 Jun 2015 02:10:50 AM UTC, comment #16: 

Updating bug status based on comment #15, still confirmed with the current development version.

 Mike Miller <mtmiller>
Group Member
Sun 07 Jul 2013 09:55:50 PM UTC, comment #15: 

I've had a look at this and it is going to be much harder than I anticipated.  I think a full re-write of Octave's interface with PCRE is required.

 Rik <rik5>
Group administrator
Fri 01 Feb 2013 06:04:40 PM UTC, comment #14: 

Better, I'd say:

>> te = regexp ('abc', 'abc(z?)', 'tokenExtents');
>> celldisp (te)

 
te{1} =
 
     4     3

 

>>
>> te = regexp ('abce', '((a)(z)?b(c)(z*))(z)?(e)?', 'tokenExtents');
>> celldisp (te)

 
te{1} =
 
     1     3
     4     3
     4     4

 

>>
>> [T, TE, NM] = regexp (sprintf('%s','John Davis\nRogers, James'), '(?<first>\w+)\s+(?<last>\w+)|(?<last>\w+),\s+(?<first>\w+)', 'tokens', 'tokenExtents', 'names');
>>
>> celldisp (T)

 
T{1}{1} =
 
John
 
 
T{1}{2} =
 
Davis
 
 
T{2}{1} =
 
nRogers
 
 
T{2}{2} =
 
James
 

>> celldisp (TE)

 
TE{1} =
 
     1     4
     6    10

 
 
TE{2} =
 
    12    18
    21    25

 

>> NM(1).first


ans =

John

>> NM(1).last


ans =

Davis

>> NM(2).first


ans =

James

>> NM(2).last


ans =

nRogers

>>


Michael Godfrey <godfrey>
Group Member
Fri 01 Feb 2013 05:48:02 PM UTC, comment #13: 

Okay, at least I got 1 out of 5 answers.

Let's try the following:


te = regexp ('abc', 'abc(z?)', 'tokenExtents');
celldisp (te)

te = regexp ('abce', '((a)(z)?b(c)(z*))(z)?(e)?', 'tokenExtents');
celldisp (te)

[T, TE, NM] = regexp (sprintf('%s','John Davis\nRogers, James'), '(?<first>\w+)\s+(?<last>\w+)|(?<last>\w+),\s+(?<first>\w+)', 'tokens', 'tokenExtents', 'names');

celldisp (T)
celldisp (TE)
NM(1).first
NM(1).last
NM(2).first
NM(2).last



 Rik <rik5>
Group administrator
Fri 01 Feb 2013 05:25:25 PM UTC, comment #12: 

Matlab is being a bit opaque, as usual:

>> version


ans =

7.9.0.529 (R2009b)

>> te = regexp ('abc', 'abc(z)?', 'tokenExtents');
>> typeinfo (te)

??? Undefined function or method 'typeinfo' for input arguments of type 'cell'.
 

>> celldisp (te)

 
te{1} =
 
     4     3

 

>> te{:}


ans =

     4     3

>> te[:]

??? te[:]
      |
Error: Unbalanced or unexpected parenthesis or bracket.
 
Here is the result of "help disp" in Matlab:

>> help disp

 DISP Display array.
    DISP(X) displays the array, without printing the array name.  In
    all other ways it's the same as leaving the semicolon off an
    expression except that empty arrays don't display.
 
    If X is a string, the text is displayed.
 
=====================
This not quite the way disp() works in Octave, or actually
the way it seems to work in Matlab.

 Michael Godfrey <godfrey>
Group Member
Fri 01 Feb 2013 05:18:02 PM UTC, comment #11: 

This is infuriating.  How does one see the results of a calculation in Matlab?  I don't understand why they are making it so difficult to print a data structure.

Let's try just one regexp until we can get it working.


te = regexp ('abc', 'abc(z)?', 'tokenExtents');
typeinfo (te)
celldisp (te)
te{:}
te[:]



 Rik <rik5>
Group administrator
Fri 01 Feb 2013 04:58:34 PM UTC, comment #10: 

Oops.  I cut and pasted from email.
Here is try from savannah
(with TE{:} at the end -- did not help)

>> [T, TE, NM] = regexp (sprintf('%s','John Davis\nRogers, James'), '(?<first>\w+)\s+(?<last>\w+)|(?<last>\w+),\s+(?<first>\w+)', 'tokens', 'tokenExtents', 'names');
>> disp (T)

    {1x2 cell}    {1x2 cell}

>> disp (TE)

    [2x2 double]    [2x2 double]

>> disp (NM)

1x2 struct array with fields:
    last
    first

>>
>> te = regexp ('abce', '((a)(z)?b(c)(z*))(z)?(e)?', 'tokenExtents');
>> disp (te)

    [3x2 double]

>>
>> [S, E, TE, M, T, NM, SP] = regexp ('abc', '(?<all>(a)(?<miss1>z?)b(?<hit1>c)(z*))')


S =

     []


E =

     []


TE =

     {}


M =

     {}


T =

     {}


NM =

0x0 struct array with fields:
    all


SP =

    'abc'

>>
>> disp (TE)
>> disp (T)
>> disp (NM)
>>
>> TE{:}
>>


Michael Godfrey <godfrey>
Group Member
Fri 01 Feb 2013 04:46:49 PM UTC, comment #9: 

Michael,

Are you getting the test code from an e-mail, or cutting and pasting from the verbatim block on Savannah?  The verbatim block in the bug report has lines that are longer than 80 characters.  In the samples you ran it appears that Matlab is seeing only 80 characters at a time followed by a newline which is leading to bad input messages.

Maybe TE{:} will display the actual contents of the tokenExtents?

Rik <rik5>
Group administrator
Fri 01 Feb 2013 04:39:22 PM UTC, comment #8: 

Maybe this more explicit code will work.


well, does not look like it:

>> te = regexp ('abc', 'abc(z?)', 'tokenExtents');
>> disp (te)

    [1x2 double]

>>
>> te = regexp ('abc', 'abc(z)?', 'tokenExtents');
>> disp (te)

    [1x2 double]

>>
>> [T, TE, NM] = regexp (sprintf('%s','John Davis\nRogers, James'),

??? [T, TE, NM] = regexp (sprintf('%s','John Davis\nRogers, James'),
                                                                    |
Error: Expression or statement is incorrect--possibly unbalanced (, {, or [.
 

>> '(?<first>\w+)\s+(?<last>\w+)|(?<last>\w+),\s+(?<first>\w+)', 'tokens',


ans =

(?<first>\w+)\s+(?<last>\w+)|(?<last>\w+),\s+(?<first>\w+)


ans =

tokens

>> 'tokenExtents', 'names');

??? 'tokenExtents', 'names');
                           |
Error: Unbalanced or unexpected parenthesis or bracket.
 

>> disp (T)

??? Undefined function or variable 'T'.
 

>> disp (TE)

??? Undefined function or variable 'TE'.
 

>> disp (NM)

??? Undefined function or variable 'NM'.
 

>>
>> te = regexp ('abce', '((a)(z)?b(c)(z*))(z)?(e)?', 'tokenExtents');
>> disp (te)

    [3x2 double]

>>
>> [S, E, TE, M, T, NM, SP] = regexp ('abc',

??? [S, E, TE, M, T, NM, SP] = regexp ('abc',
                                             |
Error: Expression or statement is incorrect--possibly unbalanced (, {, or [.
 

>> '(?<all>(a)(?<miss1>z?)b(?<hit1>c)(z*))')

??? '(?<all>(a)(?<miss1>z?)b(?<hit1>c)(z*))')
                                            |
Error: Unbalanced or unexpected parenthesis or bracket.
 

>>
>> disp (TE)

??? Undefined function or variable 'TE'.
 

>> disp (T)

??? Undefined function or variable 'T'.
 

>> disp (NM)

??? Undefined function or variable 'NM'.
 

>>
>>


Michael Godfrey <godfrey>
Group Member
Fri 01 Feb 2013 04:11:52 PM UTC, comment #7: 

Thanks Michael.  I guess I don't know how to write Matlab code very well because I was expecting that it would actually print the matrix of tokenExtents rather than just declaring it to be a matrix.

Maybe this more explicit code will work.


te = regexp ('abc', 'abc(z?)', 'tokenExtents');
disp (te)

te = regexp ('abc', 'abc(z)?', 'tokenExtents');
disp (te)

[T, TE, NM] = regexp (sprintf('%s','John Davis\nRogers, James'), '(?<first>\w+)\s+(?<last>\w+)|(?<last>\w+),\s+(?<first>\w+)', 'tokens', 'tokenExtents', 'names');
disp (T)
disp (TE)
disp (NM)

te = regexp ('abce', '((a)(z)?b(c)(z*))(z)?(e)?', 'tokenExtents');
disp (te)

[S, E, TE, M, T, NM, SP] = regexp ('abc', '(?<all>(a)(?<miss1>z?)b(?<hit1>c)(z*))')

disp (TE)
disp (T)
disp (NM)


 Rik <rik5>
Group administrator
Fri 01 Feb 2013 02:42:33 PM UTC, comment #6: 


>> regexp ('abc', 'abc(z?)', 'tokenExtents')


ans =

    [1x2 double]

>>
>> regexp ('abc', 'abc(z)?', 'tokenExtents')


ans =

    [1x2 double]

>>
>> version


ans =

7.9.0.529 (R2009b)

 

>> t = regexp (sprintf('%s','John Davis\nRogers, James'), '(?<f.

??? t = regexp (sprintf('%s','John Davis\nRogers, James'), '(?<f.
                                                           |
Error: A MATLAB string constant is not terminated properly.
 

Michael Godfrey <godfrey>
Group Member
Thu 31 Jan 2013 11:29:33 PM UTC, comment #5: 

A fifth test I need run to see how Matlab is handling patterns which don't match in an alternation operator.


t = regexp (sprintf('%s','John Davis\nRogers, James'), '(?<first>\w+)\s+(?<last>\w+)|(?<last>\w+),\s+(?<first>\w+)', 'tokens')


 Rik <rik5>
Group administrator
Thu 31 Jan 2013 05:30:35 PM UTC, comment #4: 

I have one more set of regexp tests to run if you have the time.


regexp ('abc', 'abc(z?)', 'tokenExtents')

regexp ('abc', 'abc(z)?', 'tokenExtents')


 Rik <rik5>
Group administrator
Mon 28 Jan 2013 06:59:10 PM UTC, comment #3: 

I have an implementation that works for tokens.  Getting the tokenExtents to work correctly is more difficult.  Could you run the following corner cases and tell me what Matlab returns?


regexp ('abce', '((a)(z)?b(c)(z*))(z)?(e)?', 'tokenExtents')

[S, E, TE, M, T, NM, SP] = regexp ('abc', '(?<all>(a)(?<miss1>z?)b(?<hit1>c)(z*))')


 Rik <rik5>
Group administrator
Fri 25 Jan 2013 07:18:54 AM UTC, comment #2: 

MATLAB (2011b):

>> [S, E, TE, M, T, NM, SP] = regexp ('bar2','^(foo)?bar(\d*)$')

S =
     1
E =
     4
TE =
    [2x2 double]
M =
    'bar2'
T =
    {1x2 cell}
NM =
1x1 struct array with no fields.
SP =
    ''    ''

>> TE{:}

ans =
     1     0
     4     4

>> T{:}

ans =
    ''    '2'

>> [S, E, TE, M, T, NM, SP] = regexp ('bar2','^bar(z)?(\d*)$')

S =
     1
E =
     4
TE =
    [2x2 double]
M =
    'bar2'
T =
    {1x2 cell}
NM =
1x1 struct array with no fields.
SP =
    ''    ''

>> TE{:}

ans =
     4     3
     4     4

>> T{:}

ans =
    ''    '2'

 Anonymous
Thu 24 Jan 2013 10:38:22 PM UTC, comment #1: 

This is confirmed on the development code branch as well.

For reference, could you try the following code bits on Matlab and post what it returns?  I'm curious about how they handle token extents for matched but empty substrings which is where the issue lies.


[S, E, TE, M, T, NM, SP] = regexp ('bar2','^(foo)?bar(\d*)$')
[S, E, TE, M, T, NM, SP] = regexp ('bar2','^bar(z)?(\d*)$')


 Rik <rik5>
Group administrator
Wed 23 Jan 2013 10:55:06 AM UTC, original submission:  

Parenthesized subexpressions create tokens. If the subexpression matches an empty string, an empty string should be returned. Now they just disappear: If I have an expression with two parenthesized subexpressions and the first match empty, only the second token is returned. This is inconsistent with other regex implementations like perl and MATLAB.

This works:
t=regexp('1bar2','^(\d*)bar(\d*)$','tokens');
isequal(t{1}{2},'2')

This doesn't:
t=regexp('bar2','^(\d*)bar(\d*)$','tokens');
isequal(t{1}{2},'2')

t=regexp('bar2','^(foo)?bar(\d*)$','tokens');
isequal(t{1}{2},'2')

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Carbon-Copy List
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by godfrey (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-07-22 nrjank Priority5 - Normal 3 - Low
    2019-09-16 mtmiller Dependencies- bugs #56899 is dependent
    2016-08-24 mtmiller Dependencies- bugs #48883 is dependent
    2015-06-04 mtmiller StatusIn Progress Postponed
        Release3.6.2 dev
        Operating SystemMicrosoft Windows Any
    2014-01-12 rik5 Dependencies- bugs #41211 is dependent
    2013-10-28 mtmiller CategoryNone Octave Function
        Item GroupIncorrect Result Matlab Compatibility
    2013-07-07 rik5 Summaryregexp doesn\'t return empty tokens -- they just disappear regexp: doesn't return empty tokens -- they just disappear
    2013-01-28 rik5 StatusConfirmed In Progress
    2013-01-24 rik5 StatusNone Confirmed

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-f13d.
    Corresponding source code