bugGNU Octave - Bugs: bug #50485, textscan() auto-detection of...


bug #50485: textscan() auto-detection of format is not Matlab compatible

Submitter:  sergey plotnikov <nul0m>
Submitted:  Tue 07 Mar 2017 05:30:00 PM UTC
Category:  Octave Function Severity:  3 - Normal
Priority:  3 - Low Item Group:  Matlab Compatibility
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Sun 29 Mar 2020 07:08:58 PM UTC, comment #7: 

Update: 3/29/2020.

The motivating example from comment #0

s = textscan(sprintf('2,1,\n,,\n2,,3'),'',-1,'delimiter',','); s{:}

works if the explicit format '%f %f %f' is given.

Re-titling report to indicate that the current problem is that the format is not automatically detected in the same way as Matlab.

Lowering the priority as there is a simple workaround which is to give Octave a hint about the the data by supplying a format.

Rik <rik5>
Group administrator
Thu 23 Nov 2017 11:24:21 PM UTC, comment #6: 

If this is not a regression, and it involves deep changes to the textscan code, then it is not appropriate for the stable branch and 4.2.2 release.  The more experimental work to add new features, including better compatibility, belongs on the development branch.

Rik <rik5>
Group administrator
Wed 22 Nov 2017 01:14:28 PM UTC, comment #5: 

Please disregard comment #3, I've opened a new item for it as suggested by comment #4, available here: https://savannah.gnu.org/bugs/index.php?52479

Motius <motius>
Tue 21 Nov 2017 09:38:47 PM UTC, comment #4: 

Actually, I think this (#3) may be a different bug, but in the same area of code.  I believe that bug is quite simply relative to the initial bug.  It looks like the scanning routines all do skipping of white space in some form:

  textscan::scan_one (delimited_stream& is, const textscan_format_elt& fmt,
                      octave_value& ov, Array<octave_idx_type> row)
    skip_whitespace (is);

However, at the instantiation, the default white space includes the \t character:

  textscan::textscan (const std::string& who_arg)
    : who (who_arg), buf (), whitespace_table (), delim_table (),
      delims (), comment_style (), comment_len (0), comment_char (-2),
      buffer_size (0), date_locale (), inf_nan (init_inf_nan ()),
      empty_value (numeric_limits<double>::NaN ()), exp_chars ("edED"),
      header_lines (0), treat_as_empty (), treat_as_empty_len (0),
      whitespace (" \b\t"), eol1 ('\r'), eol2 ('\n'),
      return_on_error (1), collect_output (false),
      multiple_delims_as_one (false), default_exp (true), lines (0)
  { }

What needs to be done is remove any delimiters from the whitespace string.  That is, if the user selects

 'delimiter', sprintf(','))

as you've done, then that character should be subtracted from the string.

We should probably create another bug report for this if there isn't one already.

Dan Sebald <sebald>
Tue 21 Nov 2017 12:14:05 PM UTC, comment #3: 

I think a found a related bug:

a = "\t\ta\tb\tc\n"
textscan(a, '%s', 'delimiter', sprintf('\t'))

does not have the same output on Octave 4.0.3 and 4.2.1.
Version 4.0.3 matches MatLab's (v2017a) interpretation.

With v4.0.3:

  [1,1] =
    [1,1] =
    [2,1] =
    [3,1] = a
    [4,1] = b
    [5,1] = c

With 4.2.1:

  [1,1] =
    [1,1] = a
    [2,1] = b
    [3,1] = c

Motius <motius>
Sun 26 Mar 2017 01:54:56 AM UTC, comment #2: 

I can describe what is wrong (fixing is another matter).

There is a function called textscan::read_format_once() which reads one line of of delimited fields, and this is in a loop until either N-times or indefinite (-1).  The code breaks out of that loop if there was an error.  There is an error, which is "nothing worked", but it is a bogus error because in fact there were two successful reads on the first line.

And that is what is missing from the code.  There should be either some return value, or some means of deducing that textscan::read_format_once() did actually do something.  Instead, the decision is solely based upon is.fail().  So, yes there was a fail (i.e., the third element of '2,1,'), but it's also true there were some successes (and one might even want to consider the NaN a success).

Simply oversight, but this goes pretty deep into multiple function calls, so I think this one would take time to fix and is a summer project sort of thing.  NaN needs to be handled properly and some means of avoiding the bogus error needs to be implemented.

Dan Sebald <sebald>
Tue 07 Mar 2017 09:13:32 PM UTC, comment #1: 

Confirmed in 4.3.0+ & checked with Matlab r2017a prerelease.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 07 Mar 2017 05:30:00 PM UTC, original submission:  

textscan seems to ignore the content of the scanned text after an empty line


>> s = textscan(sprintf('2,1,\n,,\n2,,3'),'',-1,'delimiter',','); s{:}
ans =
ans =
ans =


 s = textscan(sprintf('2,1,\n,,\n2,,3'),'',-1,'delimiter',','); s{:}
ans =  2
ans =  1

sergey plotnikov <nul0m>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by motius (Posted a comment)
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by nul0m (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.


    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-03-29 rik5 Priority5 - Normal 3 - Low
        Summarytextscan() behavior incompatible with matlab in case of empty lines textscan() auto-detection of format is not Matlab compatible
    2017-08-14 mtmiller CategoryNone Octave Function
    2017-03-07 philipnienhuis StatusNone Confirmed
    2017-03-07 philipnienhuis Release4.2.1 dev

    Back to the top

    Powered by Savane 3.13-b921.
    Corresponding source code