bugGNU Octave - Bugs: bug #57612, textscan omits final column value...

 
 

bug #57612: textscan omits final column value for CSV is value is empty

Submitter:  Will Carhart <wcarhart>
Submitted:  Thu 16 Jan 2020 06:50:04 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Will Carhart Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 09 Jan 2021 02:30:40 PM UTC, comment #5: 

Ok, sorry for the noise, the same problem has been already reported in #58079

Fabio <efferre79>
Fri 08 Jan 2021 04:07:33 PM UTC, comment #4: 

I have applied the patch on 6.1.0 but I can still see a problem when the number of separators on a line is different than the fields in the format string.


str = sprintf (['101,' '\n' '201,']);
C = textscan (str, '%s%s%s', 'Delimiter', ',')

C =
{
  [1,1] =
  {
    [1,1] = 101
  }

  [1,2] =
  {
    [1,1] =
  }

  [1,3] =
  {
    [1,1] = 201
  }

}


Matlab fills the output cell as expected (besides the difference put in evidence in comment #3)

Fabio <efferre79>
Sun 29 Mar 2020 06:34:51 PM UTC, comment #3: 

I checked in a fix for this on the development branch here: https://hg.savannah.gnu.org/hgweb/octave/rev/abcff237241f.  This will be a part of the 7.1 release of Octave.

Octave now behaves in a way I find sensible, but which is Matlab incompatible.  Here is simplified test code and the result 


str = sprintf (['101,' '\n' '201,']);
C = textscan (str, '%s%q', 'Delimiter', ',')

C =
{
  [1,1] =
  {
    [1,1] = 101
    [2,1] = 201
  }

  [1,2] =
  {
    [1,1] =
    [2,1] =
  }

}


As you can see, the second output in the cell array contains two empty entries.  This makes sense to me because there is no value after the ',' delimiter to read for both row 1 and row 2.  Matlab, however, returns just a single empty value.  Apparently, it treats end-of-file (EOF) differently from end-of-line (EOL) which I find unnecessary.

Marking as fixed and closing report.

Rik <rik5>
Group administrator
Fri 17 Jan 2020 02:34:38 AM UTC, comment #2: 

Just to acknowledge the title of this bug report.  Octave 5.1.0, 5.1.90, and dev and Matlab R2019a behave the same for the example with final CSV value:


newline = @() sprintf('\n');
str = ['102,19700101,,0.485,42', newline(), '111,19700101,,0.48,42']
C   = textscan (str, '%s%f%q%f%q', 'Delimiter', ',')


Kai Torben Ohlhus <siko1056>
Group Member
Fri 17 Jan 2020 02:27:05 AM UTC, comment #1: 

Thank you for your bug report.  I created an easier equivalent code to reproduce in Octave and Matlab:


newline = @() sprintf('\n');
str = ['102,19700101,,0.485,', newline(), '111,19700101,,0.48,']
C   = textscan (str, '%s%f%q%f%q', 'Delimiter', ',')


Your expected output is exactly that of Matlab R2019a where C{1}{1} and C{1}{2} are "char-arrays".

In Octave 5.1.0, 5.1.90, and dev the output looks gibberish as in comment #0.  Thus changing to dev.

Kai Torben Ohlhus <siko1056>
Group Member
Thu 16 Jan 2020 06:50:04 PM UTC, original submission:  

Running Octave 4.2.2 on Ubuntu 16.04.

textscan will skip the last column value in a CSV file if the column value is empty. For example, consider this code snippet:

lineFormat = [repmat('%s',1,1), ...
         repmat('%f',1,1), ...
         repmat('%q',1,1), ...
         repmat('%f',1,1), ...
         repmat('%q',1,1)];
C = textscan(fid, lineFormat, 'Delimiter', ',')


With the following CSV data:

102,19700101,,0.485,
111,19700101,,0.48,


I would expect the result of textscan to be:

C =
{
  [1,1] =
  {
    [1,1] = 102
    [2,1] = 111
  }
  [1,2] =
     19700101
     19700101
  [1,3] =
  {
    [1,1] =
    [2,1] =
  }
  [1,4] =
     0.48500
     0.48000
  [1,5] =
  {
    [1,1] =
    [2,1] =
  }
}


However, in reality Octave's textscan produces the following:

C =
{
  [1,1] =
  {
    [1,1] = 102
    [2,1] = 19700101
  }
  [1,2] =

     1.9700e+07
            NaN

  [1,3] =
  {
    [1,1] =
    [2,1] = 0.48
  }
  [1,4] =

       0.48500
     110.00000

  [1,5] =
  {
    [1,1] = 111
    [2,1] = 19700101
  }
}


Note that all of the data is shifted by one cell because the last column value for each line in the CSV is empty. This is not the behavior in MATLAB. In MATLAB, textscan can parse the above CSV data as expected.

I have tried the options 'MultipleDelimsAsOne'=1 and 'EndOfLine'='\n' and ='\r', but neither changed the behavior.

Is this expected? If so, is there a way to configure Octave's textscan to perform like MATLAB's?

Will Carhart <wcarhart>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by efferre79 (Posted a comment)
  • -email is unavailable- added by efferre79
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by wcarhart (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-01-08 efferre79 Carbon-Copy- Added efferre79
    2020-03-29 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2020-01-17 siko1056 StatusNone Confirmed
        Release4.2.2 dev

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code