bugGNU Octave - Bugs: bug #53685, textscan() with Delimiter...

 
 

bug #53685: textscan() with Delimiter specified always treats multiple delimiters as one

Submitter:  None
Submitted:  Wed 18 Apr 2018 02:46:48 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Works For Me Assigned to:  None
Originator Name:  frankd Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 4.2.2
Operating System:  * Microsoft Windows Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 19 Apr 2018 03:35:07 PM UTC, comment #3: 

Glad it is working for you now.  Closing report.

Rik <rik5>
Group administrator
Thu 19 Apr 2018 11:28:57 AM UTC, comment #2: 

Thanks for your reply. I tried your examples on both Octave 4.2.2 and Matlab 2017b. In some cases, Matlab is different in how it displays "ans" cell array returns, so I used celldisp() for both.

Your first example:

  • textscan ("A,B C,D", "%s", "delimiter", ',')

is same on both Octave and Matlab

Your second example:

  • textscan ("A,,,B C,D", "%s", "delimiter", ',')

is same on both Octave and Matlab.

Your number and string example is like the problem I'm trying to solve. Both Malab and Octave give the same results as your example.

Now I see my problem is that I was using commas as separators in the format strings, as I would for sscanf(). In some cases, Matlab and Octave give different results, for example:

  • celldisp(textscan ("A,,,B C,D", "%s,%s,%s,%s,%s", "delimiter", ','))

but now this seems like incorrect usage.

Both Matlab and Octave documentation examples show spaces for separators in the format string of textscan(). The octave documentation is a little more complete in the description.

I changed the commas in my format string to spaces, now it works as expected.


Frank Demarest <frankd65>
Wed 18 Apr 2018 05:25:14 PM UTC, comment #1: 

Can you run your test with Matlab and see if it fails there?  textscan() is a very complicated function, and unfortunately Octave has to try and reproduce Matlab behavior exactly because many users have gotten used to its idiosyncracies.

As a start, see the documentation at http://www.mathworks.com/help/matlab/ref/textscan.html.

For issue #1, "spaces in the string fields act as delimiters, creating extra fields", Matlab says


Within each row of data, the default field delimiter is white-space. White-space can be any combination of space (' '), backspace ('\b'), or tab ('\t') characters. If you do not specify a delimiter, then:

    the delimiter characters are the same as the white-space characters. The default white-space characters are ' ', '\b', and '\t'. Use the 'Whitespace' name-value pair argument to specify alternate white-space characters.

    textscan interprets repeated white-space characters as a single delimiter.


It is essentially as if they set the Delimiter option to whitespace and MultipleDelimsAsOne to true.

So, if your delimiter is actualy a comma then you will need to say so.  For example,


textscan ("A,B C,D", "%s", "delimiter", ',')
ans =
{
  [1,1] =
  {
    [1,1] = A
    [2,1] = B C
    [3,1] = D
  }

}


which works to preserve the space in the string.

For the last issue, I find that MultipleDelimsAsOne works.  For example


textscan ("A,,,B C,D", "%s", "delimiter", ',')
ans =
{
  [1,1] =
  {
    [1,1] = A
    [2,1] =
    [3,1] =
    [4,1] = B C
    [5,1] = D
  }

}


As expected, there were two empty strings created where there were extra commas.  Now switching on the MultipleDelimsAsOne option


textscan ("A,,,B C,D", "%s", "delimiter", ',', "multipledelimsasone", 1)
ans =
{
  [1,1] =
  {
    [1,1] = A
    [2,1] = B C
    [3,1] = D
  }

}


I don't exactly know your file, but lets say your trying to read a number, string, number, string.


textscan ("1.1,A,2.2,B C", "%f %s %f %s", "delimiter", ',')
{
  [1,1] =  1.1000
  [1,2] =
  {
    [1,1] = A
  }

  [1,3] =  2.2000
  [1,4] =
  {
    [1,1] = B C
  }
}


That seems right.


textscan ("1.1,A,,B C", "%f %s %f %s", "delimiter", ',')


If one of the numbers is missing, that seems to work too.


textscan ("1.1,A,,B C", "%f %s %f %s", "delimiter", ',')
ans =
{
  [1,1] =  1.1000
  [1,2] =
  {
    [1,1] = A
  }

  [1,3] =  NaN
  [1,4] =
  {
    [1,1] = B C
  }

}


Is there a one-line example that shows how Octave textscan is not behaving identically to the Matlab textscan function?

Rik <rik5>
Group administrator
Wed 18 Apr 2018 02:46:48 AM UTC, original submission:  

I have a data file that contains comma separated fields of known types. The file is from an external device, so I can't change the format.

  • Some numeric fields may sometimes be null, nothing between two adjacent commas
  • The text fields may include spaces, and are not quoted


Using textscan() to read the file with %f and %s:

  • Without using the Delimiter option, the null numeric fields correctly appear as NaN, but the spaces in the string fields act as delimiters, creating extra fields.
  • Using the Delimiter option, the null numeric fields disappear, shifting all subsequent fields
  • The MultipleDelimsAsOne option seems to have no effect when used with the Delimiter option.


A .m file with examples (using strings) is attached.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #43964:  textscan_bug.m added by None (3KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by frankd65 (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-04-19 rik5 StatusNeed Info Works For Me
        Open/ClosedOpen Closed
    2018-04-18 rik5 StatusNone Need Info
    2018-04-18 None Attached File- Added textscan_bug.m, #43964

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code