bugGNU Octave - Bugs: bug #58293, importdata doesn't like empty...

 
 

bug #58293: importdata doesn't like empty header lines

Submitter:  Fabio <efferre79>
Submitted:  Sat 02 May 2020 05:26:06 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 24 May 2020 02:02:08 AM UTC, comment #8: 

I changed the detection algorithm quite a bit in this changeset: 28277:01358faf49c9.  Along with a number of other changesets, I think the issues in this report have all been fixed.  All of the changes were on the development branch so to acces the fix you will need to get importdata.m from the Octave Mercurial repository.


Rik <rik5>
Group administrator
Thu 07 May 2020 06:13:24 PM UTC, comment #7: 

ok, opened #58328 for the column headers topic

Fabio <efferre79>
Thu 07 May 2020 05:59:20 PM UTC, comment #6: 

Why don't you open up a new bug report that is on the correct topic.

For reference, Matlab does not recognize column headers unless they use the correct delimiter

So,


HeaderB
1,2,3


results in textdata being equal to "HeaderB", no colheaders output, and data field of [1 2 3].

Only if the delimiter and the number of fields match does Matlab assign colheaders, rather than textdata.  For example,


HeaderA HeaderB HeaderC
1,2,3


doesn't produce colheaders because the delimiter isn't a ','.

Also,


HeaderA,HeaderB
1,2,3


doesn't work because the delimiter is correct, but there aren't three text elements.

Rik <rik5>
Group administrator
Thu 07 May 2020 05:44:45 PM UTC, comment #5: 

no the problem is that colheaders is blindly looked into last line of headers.


      if (! isempty (output.textdata))
        if (delimiter == " ")
          output.colheaders = regexp (strtrim (output.textdata{end}),
                                      ' +', 'split');
        else
          output.colheaders = ostrsplit (output.textdata{end}, delimiter);
        endif
      endif


I think this problem is not related to this bug report, I could have a look and open a new bug. What do you think?

We should decide if the number of column headers must be equal to the number of columns in output.data, or if we can have empty col headers like in my example file attached initially. For instance we can decide that there are column headers only if there is a separator in the last line of headers

Fabio <efferre79>
Thu 07 May 2020 05:34:20 PM UTC, comment #4: 

Exactly.  I think Octave is having problem detecting header lines.  My guess, I haven't checked yet, is that the fact that "header2" contains a number is throwing the detection algorithm off.  It seems to correctly find all text header lines such as "headerB".

Rik <rik5>
Group administrator
Thu 07 May 2020 05:14:16 PM UTC, comment #3: 

Ok, I see what you mean, header2 should not appear in colheaders, right?

Fabio <efferre79>
Thu 07 May 2020 05:09:49 PM UTC, comment #2: 

I get the following result which seems correct:


octave:1> importdata('hdr.dos.txt',',')
ans =

  scalar structure containing the fields:

    data =

       1   2   3
       4   5   6

    textdata =
    {
      [1,1] = headerA
      [2,1] =
      [3,1] =
      [4,1] = headerB
    }

    colheaders =
    {
      [1,1] = headerB
    }



Fabio <efferre79>
Thu 07 May 2020 03:11:50 PM UTC, comment #1: 

I think this issue is more complicated than it appears.  Could you try


[a, b, c] = importdata ("hdr.dos.txt")


with the attached file and report the results?


(file #49031)

Rik <rik5>
Group administrator
Sat 02 May 2020 05:26:06 PM UTC, original submission:  

The commit http://hg.savannah.gnu.org/hgweb/octave/rev/75f7e8728e17 introduced a regression (octave 4.4.1 was working fine), moreover empty lines are now discarded from the header section of the datafile.

I am adding here a patch to solve the problem, you can see a line of code where I consider the possibility to encounter old Mac files (the \r is used as line separator), feel free to remove that part if you don't consider that support important in octave.

I have updated also the BITS to test the new functionality, here I have also attached two sample log file to analyze. All the BITS have been successfully tested on Linux and Windows.


Fabio <efferre79>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49031:  hdr.dos.txt added by rik5 (36B - text/plain)
file #48986:  importdata.m.patch added by efferre79 (1KiB - text/x-patch)
file #48987:  test_CRLF.log added by efferre79 (93B - text/x-log)
file #48988:  test_LF.log added by efferre79 (85B - text/x-log)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by efferre79 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-24 rik5 StatusNeed Info Fixed
        Open/ClosedOpen Closed
        Release5.2.0 dev
    2020-05-07 rik5 Attached File- Added hdr.dos.txt, #49031
        StatusNone Need Info
    2020-05-02 efferre79 Attached File- Added importdata.m.patch, #48986
        Attached File- Added test_CRLF.log, #48987
        Attached File- Added test_LF.log, #48988

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code