bugGNU Octave - Bugs: bug #58008, textscan: literals are managed...

 
 

bug #58008: textscan: literals are managed differently from Matlab depending on delimiters

Submitter:  None
Submitted:  Wed 18 Mar 2020 04:58:16 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  2 Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  None
Originator Name:  rampin76 Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 28 Mar 2020 11:04:26 PM UTC, comment #10: 

It was difficult to find the exact spot where this bug manifested in the code, but once found was easy to fix.  I checked in a fix on the development branch here https://hg.savannah.gnu.org/hgweb/octave/rev/56349d9ca566.  This will be part of Octave 7.1.

Closing report.

Rik <rik5>
Group administrator
Sat 28 Mar 2020 07:08:15 PM UTC, comment #9: 

Re-categorizing as Matlab Compatibility.

The following code works in Matlab R2020a.


txt = sprintf ('literal_other_1_1;literal_other_1_2\nliteral_other_2_1;literal_other_2_2\nliteral_other_3_1;literal_other_3_2')
nm = textscan (txt, 'literal%s literal%s', 'Delimiter', ';')


However, the following does not


nm = textscan (txt, 'literal%s;literal%s', 'Delimiter', ';')


Clearly, there are subtle issues with both Octave's and Matlab's implementation.  If I have a data format that is even remotely complicated I would choose an external solution, such as Perl, to process the data in to a more ordinary form that would be easier to parse by Octave.  Or figure out a convenient workaround as there is in this case by including ';' as a literal and not just as a delimiter.

Rik <rik5>
Group administrator
Fri 20 Mar 2020 08:49:48 PM UTC, comment #8: 

To the OP:
For tabs you can also try with double rather than single quotes. AFAIK it shouldn't make a difference, but who knows.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 20 Mar 2020 08:47:16 PM UTC, comment #7: 

FWIW, my experience with textscan (in Octave and in Matlab) is that even for just-above-simple text file formats, considerable experimentation may be required to get it right. Sometimes so much that I write dedicated scripts or programs rather than trying to invoke textscan.
The clever trick that Mike came up with fits perfectly in this picture, even if it just works around a bug. I'd just advise "if it works for you, it works".

Years ago we tried to find out exactly how delimiters, whitespace, endofline and literals + their default values would influence each other and in what priority, but we never managed to come up with a definitive "script" or "flowchart". To that end rigorous reverse-engineering is required but this volunteer project simply lacks time.
So IMO textscan is a still a bit of a black box, or a very dark grey one at that ;-)
That said Matlab has greatly improved their version, but also added a lot of bells and whistles that IMO mainly add to the confusion.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 20 Mar 2020 10:46:55 AM UTC, comment #6: 


comment #5:

> Looks a bit weird to me that a delimiter is also a literal, and/or vice versa. AFAIK delimiters and literals are supposed to be separate entities, although in a way one could say that they work similarly.
>
> But maybe this points to where the issue lies.
>


I agree. The Octave manual (v 5.1.0), under the textscan description reports "As in fscanf, any (non-whitespace) text in the format that is not one of these specifiers is considered a literal." So the semicolon should be considered a literal, not a delimiter.

Anyway, thank you very much to Mike Miller for the hint. This solves my issue, as I have to deal with many external files with semicolon delimiter.

As far as I can see, however, the general problem is not solved. Indeed, I was not able to manage the tab-delimiter case.

I tried with

mm=textscan(fid,'literal%s\t literal%s','Delimiter','\t')

and also tried to change from single to double quote to check how the string is interpreted, but without success.

Do you have a workaround also for the tab-delimiter case?

Anonymous
Thu 19 Mar 2020 08:41:12 PM UTC, comment #5: 

Looks a bit weird to me that a delimiter is also a literal, and/or vice versa. AFAIK delimiters and literals are supposed to be separate entities, although in a way one could say that they work similarly.

But maybe this points to where the issue lies.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 19 Mar 2020 04:26:59 PM UTC, comment #4: 

The original example works fine if you use


mm = textscan (fid, 'literal%s;literal%s', 'Delimiter', ';')


This seems to me like the right call. I don't see why it should be expected to work with a space in the literal pattern string.

Mike Miller <mtmiller>
Group Member
Wed 18 Mar 2020 08:28:09 PM UTC, comment #3: 

Forgot to add that with ';' as delimiter Matlab also shows the "incorrect" / "unexpected" answer.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 18 Mar 2020 08:27:12 PM UTC, comment #2: 

BTW, In Matlab r2014a your example doesn't work either:

>> nm = textscan (txt, 'literal%s literal%s', 'Delimiter', ' ')
nm =
    {2x1 cell}    {2x1 cell}
>> nm{1}
ans =
    '_other_1_1'
    '_other_2_2\nliteral_other_3_1'
>> nm{2}
ans =
    '_other_1_2\nliteral_other_2_1'
    '_other_3_2'
>>


(empty lines removed)

Adding "\n" to delimiters or to 'endofline' makes no difference. '\r\n' as EOL won't work either.

Matlab has known issues with "cuddling" literals (literals immediately bordering fields). At the time (long ago) I was working on strread.m I've put a lot of effort in properly processing of cuddling literals (because I needed it myself). Unfortunately Octave's textscan, when turned into a (much faster) compiled function, couldn't follow.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 18 Mar 2020 08:07:47 PM UTC, comment #1: 

Confirmed with Octave-5.2.0 & 7.0.0.

FYI, there is a slow workaround using textread (or strread when reading from string):

>> [a, b] = strread (txt, 'literal%s literal%s', 'Delimiter', ';')
a =
{
  [1,1] = _other_1_1
  [2,1] = _other_2_2 literal_other_3_1
}

b =
{
  [1,1] = _other_1_2 literal_other_2_1
  [2,1] = _other_3_2
}

>> [a, b] = strread (txt, 'literal%s literal%s', 'Delimiter', ';\n')
a =
{
  [1,1] = _other_1_1
  [2,1] = _other_2_1
  [3,1] = _other_3_1
}

b =
{
  [1,1] = _other_1_2
  [2,1] = _other_2_2
  [3,1] = _other_3_2
}


so you'll note that you do need to add the EOL ("\n") to the delimiter collection for textread (and strread, textread's workhorse behind the scenes) to work.

Release -> dev

Philip Nienhuis <philipnienhuis>
Group Member
Wed 18 Mar 2020 04:58:16 PM UTC, original submission:  

Octave version: 5.1.0
Operating system: Windows 10 Enterprise, 64-bit

The textscan function exhibits a strange behaviour when using “literals”. The behaviour depends on the used delimiter. With a “space” delimiter the behaviour seems correct, with other types of delimiters (e.g., “;”) not.

Case 1, with space delimiter. I use the following input file

literal_other_1_1 literal_other_1_2
literal_other_2_1 literal_other_2_2
literal_other_3_1 literal_other_3_2

and I scan it with the commands

clear;
fid=fopen('test.txt');
mm=textscan(fid,'literal%s literal%s','Delimiter',' ')
fclose(fid);

getting the expected output

mm =
{
  [1,1] =
  {
    [1,1] = _other_1_1
    [2,1] = _other_2_1
    [3,1] = _other_3_1
  }

  [1,2] =
  {
    [1,1] = _other_1_2
    [2,1] = _other_2_2
    [3,1] = _other_3_2
  }

}

Case 2, with semicolon delimiter. I use the following input file

literal_other_1_1;literal_other_1_2
literal_other_2_1;literal_other_2_2
literal_other_3_1;literal_other_3_2

and I scan it with the commands

clear;
fid=fopen('test.txt');
mm=textscan(fid,'literal%s literal%s','Delimiter',';')
fclose(fid);

getting the unexpected output

mm =
{
  [1,1] =
  {
    [1,1] = _other_1_1
    [2,1] = _other_2_1
    [3,1] = _other_3_1
  }

  [1,2] =
  {
    [1,1] = literal_other_1_2
    [2,1] = literal_other_2_2
    [3,1] = literal_other_3_2
  }

}

I think this is a bug and I didn’t find it in the list, though in the past other bugs concerning textscan were reported and solved.
The unexpected behaviour occurs also with tab ('\t') and comma (',') delimiters.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-03-28 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
        Operating SystemMicrosoft Windows Any
    2020-03-28 rik5 Priority5 - Normal 2
        Item GroupIncorrect Result Matlab Compatibility
        StatusWorks For Me Confirmed
        Summarytextscan: literals are managed differently depending on delimiters textscan: literals are managed differently from Matlab depending on delimiters
    2020-03-19 mtmiller StatusConfirmed Works For Me
    2020-03-18 philipnienhuis CategoryInterpreter Octave Function
        Item GroupNone Incorrect Result
        StatusNone Confirmed
        Release5.1.0 dev

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code