bugGNU Octave - Bugs: bug #50813, importdata returns cell-of-strings...

 
 

bug #50813: importdata returns cell-of-strings instead of struct

Submitter:  Marc Dirix <dirixmjm>
Submitted:  Sun 16 Apr 2017 12:57:02 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 01 Apr 2020 09:02:43 AM UTC, comment #13: 

Another example where Octave's importdata output differs from Matlab's:


>> urlwrite('https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv','time_series_covid19_confirmed_global.csv');
>> M = importdata('time_series_covid19_confirmed_global.csv')
M =
  struct with fields:
        data: [256×72 double]
    textdata: {257×74 cell}


Octave returns a 257x1 cell array with each row stored as a string. With a importdata('...',',',1) call, Matlab's output is unchanged while Octave then returns a struct with fields of size:


    data: [256x73 double]
textdata: {514x1 cell}


Guillaume <gyom>
Wed 01 Apr 2020 08:35:19 AM UTC, comment #12: 

Running the example from comment #1, Octave returns:


A =
  scalar structure containing the fields:
    data =
       1   2
       3   4
    textdata =
    {
      [1,1] = A  B
    }
    colheaders =
    {
      [1,1] = A
      [1,2] = B
    }


while Matlab would return {'A','B'} for the textdata field. A solution might be to use the row_entries variable instead of row in:
http://hg.savannah.gnu.org/hgweb/octave/file/c26838a3afa6/scripts/io/importdata.m#l194

Guillaume <gyom>
Sun 23 Apr 2017 11:31:40 AM UTC, comment #11: 

If no one beats me to it, I'll have a look in the coming weeks.

I think the delimiter assessment code, currently acting on the header lines if these are present, ultimately should be based on the purely numeric section.
This implies some fseek-ing hence and forth, but I suppose the required execution time for that is peanuts compared to reading (large amounts of) numeric data further on.
I also have a basic idea on how to deal with column headers comprising units enclosed in parens or brackets separated by whitespace, once the delimiter assessment works reliably.  But I won't try AI here :-), just basic guesswork and if that fails just return cellstr.

(Just wondering what to do if row headers are also present...)

Philip Nienhuis <philipnienhuis>
Group Member
Tue 18 Apr 2017 08:04:26 PM UTC, comment #10: 

@JWE:
AFAICT importdata() invokes dlmread once it knows the delimiter so it shouldn't be that slow.

IMO user-convenience functions like importdata.m quickly tend to turn into dinosaurs with a lot of built-in guesswork (or if you really want, AI :-) ).
I think attempts to mimick this guesswork/AI is a nice job for a real hobbyist; maybe we shouldn't pursue this too much.
Looking at Octave's code I see a lot of opportunities for misinterpretation of the header line. E.g., when one of more of the column headers comprises just digits.

The line of hyphens is skipped because it is fed to str2double that returns a NaN (IOW, not numeric data) and then the line is added to the headerlines collection.


Philip Nienhuis <philipnienhuis>
Group Member
Tue 18 Apr 2017 07:48:33 PM UTC, comment #9: 

... and it also fails on Linux =>

OS => Any

Philip Nienhuis <philipnienhuis>
Group Member
Tue 18 Apr 2017 07:47:40 PM UTC, comment #8: 

Hmmm, the "bug" I mentioned refers of course to the failing regexp in L.174.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 18 Apr 2017 07:46:07 PM UTC, comment #7: 

@OP:
You can at least recover the numeric data using:

AData = importdata('Plot5G.txt', ' ')


i.e., specify the delimiter explicitly.

A quick analysis shows that the regexp in L.174 (in subfunc importdata_asci) turns up "g" rather than a space as delimiter.
But even when specifying the delimiter explicitly the header isn't parsed correctly, so there's a bug I think.
That Matlab seems a bit smarter than Octave as regards parsing the header is a mere compatibility issue.

Once you got the data read fine, the header entries can be parsed using some magic provided you know what the header is expected to look like. In this specific case:

>> strtrim (cell2mat (regexpi (A.textdata{1}, '([a-z() \.\[]+?\])', "tokens")))
        ans = {
          [1,1] = Theta [deg.]
          [1,2] = Phi   [deg.]
          [1,3] = Abs(Grlz)[dB    ]
          [1,4] = Abs(Cross)[dB    ]
          [1,5] = Phase(Cross)[deg.]
          [1,6] = Abs(Copol)[dB    ]
          [1,7] = Phase(Copol)[deg.]
          [1,8] = Ax.Ratio[dB    ]
        }
>>


I also leave this open as I have other priorities the next weeks.

Status => confirmed
Release => dev

Philip Nienhuis <philipnienhuis>
Group Member
Tue 18 Apr 2017 07:45:08 PM UTC, comment #6: 

Yes, please leave this open.

How does Matlab recognize column headers when they contain spaces and the field separator is also spaces?  Does it do some paren matching?  There must be some kind of fuzzy matching/guessing that it is doing.

Also, how does it know to skip the line of "-" characters?  Again, some kind of guesswork?  A human can easily tell what the header is.  Maybe Matlab looks for the first line that has something that looks like a row of numbers?  I don't know.

I downloaded the file and it appears to be 9+MB.  I truncated it to 10 lines of data and uploaded it.  It should be attached to this comment.  The abbreviated file still shows the same problem, so we don't really need all the extra lines for this report.

OTOH, I also noticed that importdata is quite slow for large files of only numbers (I removed the header lines to see what it would do with this file).  But that should probably be a separate report.


(file #40439)

John W. Eaton <jwe>
Group administrator
Tue 18 Apr 2017 05:59:44 PM UTC, comment #5: 

Ok, maybe, I don't know for certain if the file size is the issue or not, I usually only deal with attachments that are kB rather than MB.

Regardless, I will leave this open for someone who knows more about importdata to ponder. I don't use the function or know how it is intended to work, but this seems like a Matlab compatibility issue that someone may want to address.

Mike Miller <mtmiller>
Group Member
Tue 18 Apr 2017 05:54:37 PM UTC, comment #4: 

Ok, I thought the file-size was ok, because windows reports 9.673 KB which is under the limit which is shown.

As the file is an export from CST Microwave Studio (a tool we use often), changing the header everytime is not an option for.

Although the behavior differs from Matlab, as mentioned before, I have a fix for our use in place. So, I'm ok with closing this bug.

Marc Dirix <dirixmjm>
Tue 18 Apr 2017 05:08:44 PM UTC, comment #3: 

You should be able to attach the file if you compress it, there is a file size limitation.

Anyway, I can confirm, and changing the file header from this


Theta [deg.]  Phi   [deg.]  Abs(Grlz)[dB    ]   Abs(Cross)[dB    ]  Phase(Cross)[deg.]  Abs(Copol)[dB    ]  Phase(Copol)[deg.]  Ax.Ratio[dB    ]
------------------------------------------------------------------------------------------------------------------------------------------------------


to this


Theta   Phi     Abs(Grlz)   Abs(Cross)  Phase(Cross)  Abs(Copol)  Phase(Copol)  Ax.Ratio
------------------------------------------------------------------------------------------------------------------------------------------------------


fixes it for me. In other words eliminating the extra labels in square brackets in the first row.

Mike Miller <mtmiller>
Group Member
Tue 18 Apr 2017 03:11:42 PM UTC, comment #2: 

I won't upload, I get the following error:
No group chosen: nogroup.

Please find the file here: http://www.dirix.nu/Plot5G.txt

I have further found that in this case "textscan("Plot5G.txt',' ',2) (E.g. space-separated, and 2 headerlines) at least parses the file.

Marc Dirix <dirixmjm>
Tue 18 Apr 2017 03:01:52 PM UTC, comment #1: 

I cannot reproduce your problem with a simple .txt file as follows:
A  B
1  2
3  4
saved as test.txt

Matlab r2016b:

>> format compact
>> A = importdata ('test.txt')
A =
  struct with fields:

          data: [2×2 double]
      textdata: {'A'  'B'}
    colheaders: {'A'  'B'}
>>


Octave 4.2.0 and 4.3.0+ (the latter dev. version):

>> format compact
>> A = importdata ('test.txt')
A =
  scalar structure containing the fields:
    data =
       1   2
       3   4
    textdata =
    {
      [1,1] = A,B
    }
    colheaders =
    {
      [1,1] = A
      [1,2] = B
    }

>>


If you want us to look further into it please upload a test file that works in Matlab and provokes the bug in Octave.

Status => Need info

Philip Nienhuis <philipnienhuis>
Group Member
Sun 16 Apr 2017 12:57:02 PM UTC, original submission:  

In Octave, using importdata directly on a file return a cell containing strings for each line of the file.
However in Matlab the same command returns a struct containing separate textdata cell and data maxtrix.

Matlab output:
AData = importdata('Plot5G.txt')
AData =

        data: [65160x8 double]
    textdata: {2x1 cell}


Octave:

AData = importdata('Plot5G.txt')
AData =
{
  [1,1] = Theta [deg.]  Phi   [deg.]  Abs(Grlz)[dB    ]   Abs(Cross)[dB    ]  Phase(Cros
s)[deg.]  Abs(Copol)[dB    ]  Phase(Copol)[deg.]  Ax.Ratio[dB    ]
  [2,1] = ------------------------------------------------------------------------------
------------------------------------------------------------------------
  [3,1] =    0.000           0.000            1.322e+01          -1.560e-01
194.336           1.302e+01              62.815           1.588e+01
  [4,1] =    1.000           0.000            1.330e+01           2.045e-01
186.123           1.309e+01              54.157           1.566e+01
  [5,1] =    2.000           0.000            1.338e+01           5.023e-01
178.192           1.315e+01              45.538           1.554e+01
.....

Marc Dirix <dirixmjm>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #40439:  Plot5G.txt added by jwe (2KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gyom (Posted a comment)
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by dirixmjm (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-04-18 philipnienhuis Operating SystemMicrosoft Windows Any
    2017-04-18 philipnienhuis StatusNeed Info Confirmed
        Release4.2.1 dev
    2017-04-18 jwe Attached File- Added Plot5G.txt, #40439
    2017-04-18 philipnienhuis StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code