bugGNU Octave - Bugs: bug #50813, importdata returns cell-of-strings...

 
 

bug #50813: importdata returns cell-of-strings instead of struct

Submitted by:  Marc Dirix <dirixmjm>
Submitted on:  Sun 16 Apr 2017 12:57:02 PM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Matlab Compatibility
Status: ConfirmedAssigned to: None
Originator Name: Open/Closed: Open
Release: devOperating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 23 Apr 2017 11:31:40 AM UTC, comment #11:

If no one beats me to it, I'll have a look in the coming weeks.

I think the delimiter assessment code, currently acting on the header lines if these are present, ultimately should be based on the purely numeric section.
This implies some fseek-ing hence and forth, but I suppose the required execution time for that is peanuts compared to reading (large amounts of) numeric data further on.
I also have a basic idea on how to deal with column headers comprising units enclosed in parens or brackets separated by whitespace, once the delimiter assessment works reliably. But I won't try AI here :-), just basic guesswork and if that fails just return cellstr.

(Just wondering what to do if row headers are also present...)

Philip Nienhuis <philipnienhuis>
Project Member
Tue 18 Apr 2017 08:04:26 PM UTC, comment #10:

@JWE:
AFAICT importdata() invokes dlmread once it knows the delimiter so it shouldn't be that slow.

IMO user-convenience functions like importdata.m quickly tend to turn into dinosaurs with a lot of built-in guesswork (or if you really want, AI :-) ).
I think attempts to mimick this guesswork/AI is a nice job for a real hobbyist; maybe we shouldn't pursue this too much.
Looking at Octave's code I see a lot of opportunities for misinterpretation of the header line. E.g., when one of more of the column headers comprises just digits.

The line of hyphens is skipped because it is fed to str2double that returns a NaN (IOW, not numeric data) and then the line is added to the headerlines collection.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 18 Apr 2017 07:48:33 PM UTC, comment #9:

... and it also fails on Linux =>

OS => Any

Philip Nienhuis <philipnienhuis>
Project Member
Tue 18 Apr 2017 07:47:40 PM UTC, comment #8:

Hmmm, the "bug" I mentioned refers of course to the failing regexp in L.174.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 18 Apr 2017 07:46:07 PM UTC, comment #7:

@OP:
You can at least recover the numeric data using:

i.e., specify the delimiter explicitly.

A quick analysis shows that the regexp in L.174 (in subfunc importdata_asci) turns up "g" rather than a space as delimiter.
But even when specifying the delimiter explicitly the header isn't parsed correctly, so there's a bug I think.
That Matlab seems a bit smarter than Octave as regards parsing the header is a mere compatibility issue.

Once you got the data read fine, the header entries can be parsed using some magic provided you know what the header is expected to look like. In this specific case:

I also leave this open as I have other priorities the next weeks.

Status => confirmed
Release => dev

Philip Nienhuis <philipnienhuis>
Project Member
Tue 18 Apr 2017 07:45:08 PM UTC, comment #6:

Yes, please leave this open.

How does Matlab recognize column headers when they contain spaces and the field separator is also spaces? Does it do some paren matching? There must be some kind of fuzzy matching/guessing that it is doing.

Also, how does it know to skip the line of "-" characters? Again, some kind of guesswork? A human can easily tell what the header is. Maybe Matlab looks for the first line that has something that looks like a row of numbers? I don't know.

I downloaded the file and it appears to be 9+MB. I truncated it to 10 lines of data and uploaded it. It should be attached to this comment. The abbreviated file still shows the same problem, so we don't really need all the extra lines for this report.

OTOH, I also noticed that importdata is quite slow for large files of only numbers (I removed the header lines to see what it would do with this file). But that should probably be a separate report.

(file #40439)

John W. Eaton <jwe>
Project Administrator
Tue 18 Apr 2017 05:59:44 PM UTC, comment #5:

Ok, maybe, I don't know for certain if the file size is the issue or not, I usually only deal with attachments that are kB rather than MB.

Regardless, I will leave this open for someone who knows more about importdata to ponder. I don't use the function or know how it is intended to work, but this seems like a Matlab compatibility issue that someone may want to address.

Mike Miller <mtmiller>
Project Administrator
Tue 18 Apr 2017 05:54:37 PM UTC, comment #4:

Ok, I thought the file-size was ok, because windows reports 9.673 KB which is under the limit which is shown.

As the file is an export from CST Microwave Studio (a tool we use often), changing the header everytime is not an option for.

Although the behavior differs from Matlab, as mentioned before, I have a fix for our use in place. So, I'm ok with closing this bug.

Marc Dirix <dirixmjm>
Tue 18 Apr 2017 05:08:44 PM UTC, comment #3:

You should be able to attach the file if you compress it, there is a file size limitation.

Anyway, I can confirm, and changing the file header from this

to this

fixes it for me. In other words eliminating the extra labels in square brackets in the first row.

Mike Miller <mtmiller>
Project Administrator
Tue 18 Apr 2017 03:11:42 PM UTC, comment #2:

I won't upload, I get the following error:
No group chosen: nogroup.

Please find the file here: http://www.dirix.nu/Plot5G.txt

I have further found that in this case "textscan("Plot5G.txt',' ',2) (E.g. space-separated, and 2 headerlines) at least parses the file.

Marc Dirix <dirixmjm>
Tue 18 Apr 2017 03:01:52 PM UTC, comment #1:

I cannot reproduce your problem with a simple .txt file as follows:
A B
1 2
3 4
saved as test.txt

Matlab r2016b:

Octave 4.2.0 and 4.3.0+ (the latter dev. version):

If you want us to look further into it please upload a test file that works in Matlab and provokes the bug in Octave.

Status => Need info

Philip Nienhuis <philipnienhuis>
Project Member
Sun 16 Apr 2017 12:57:02 PM UTC, original submission:

In Octave, using importdata directly on a file return a cell containing strings for each line of the file.
However in Matlab the same command returns a struct containing separate textdata cell and data maxtrix.

Matlab output:
AData = importdata('Plot5G.txt')
AData =

data: [65160x8 double]
textdata: {2x1 cell}

Octave:

AData = importdata('Plot5G.txt')
AData =
{
[1,1] = Theta [deg.] Phi [deg.] Abs(Grlz)[dB ] Abs(Cross)[dB ] Phase(Cros
s)[deg.] Abs(Copol)[dB ] Phase(Copol)[deg.] Ax.Ratio[dB ]
[2,1] = ------------------------------------------------------------------------------
------------------------------------------------------------------------
[3,1] = 0.000 0.000 1.322e+01 -1.560e-01
194.336 1.302e+01 62.815 1.588e+01
[4,1] = 1.000 0.000 1.330e+01 2.045e-01
186.123 1.309e+01 54.157 1.566e+01
[5,1] = 2.000 0.000 1.338e+01 5.023e-01
178.192 1.315e+01 45.538 1.554e+01
.....

Marc Dirix <dirixmjm>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #40439:  Plot5G.txt added by jwe (2KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jwe (Updated the item)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by dirixmjm (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 18 Apr 2017 07:48:33 PM UTCphilipnienhuisOperating SystemMicrosoft Windows=>Any
    Tue 18 Apr 2017 07:46:07 PM UTCphilipnienhuisStatusNeed Info=>Confirmed
      Release4.2.1=>dev
    Tue 18 Apr 2017 07:45:08 PM UTCjweAttached File-=>Added Plot5G.txt, #40439
    Tue 18 Apr 2017 03:01:52 PM UTCphilipnienhuisStatusNone=>Need Info

    Back to the top


    Powered by Savane 3.1-cleanup1