bugGNU Octave - Bugs: bug #64074, importdata returns column vector...

 
 

bug #64074: importdata returns column vector of textdata when there is an empty field in the imported data

Submitter:  None
Submitted:  Thu 20 Apr 2023 09:00:12 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Confirmed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * 8.1.0
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 21 Apr 2023 10:46:57 PM UTC, comment #5: 

I re-titled the bug report to reflect the issue about returning a column vector, instead of an array, of text data.

Interestingly, if every field (that is an identified text field) has some text in it then Octave correctly returns an array.

For this data:


title;casename;0
a;b;
load case;c;
height;h;1.3


Octave returns


d2 = importdata ('import2csv.txt', ';')
d2 =

  scalar structure containing the fields:

    data =

            0
           NA
           NA
       1.3000

    textdata =
    {
      [1,1] = title
      [2,1] = a
      [3,1] = load case
      [4,1] = height
      [1,2] = casename
      [2,2] = b
      [3,2] = c
      [4,2] = h
    }


Clearly, it is the empty fields which are causing the issue.

Rik <rik5>
Group administrator
Fri 21 Apr 2023 10:43:05 PM UTC, comment #4: 

For starters, if you just want the numerical data then using dlmread() will be faster.

This works:


dlmread ('myfile.csv', ';', 'emptyvalue', NaN)
ans =

      NaN      NaN        0
      NaN      NaN      NaN
      NaN      NaN      NaN
      NaN      NaN   1.3000


And you can retrieve the data in the final column with ans(:,3).

As for importdata, it is guessing wrong about the delimiter.  You can specify the delimiter yourself and that will get you the correct numerical data.


data = importdata ('myfile.csv', ';')
data =

  scalar structure containing the fields:

    data =

            0
           NA
           NA
       1.3000

    textdata =
    {
      [1,1] = title
      [2,1] = load case
      [3,1] = height
      [4,1] = casename
      [5,1] =
      [6,1] = h
    }


There are still two issues.  First, the auto-detection of the delimiter is unsuccessful and that is an issue with the regular expression.

Second, even with the correct delimiter Octave is omitting some blank results in 'textdata' and is returning a column vector instead of an array.

Rik <rik5>
Group administrator
Fri 21 Apr 2023 01:18:47 PM UTC, comment #3: 

yep this two are in fact two separate issues. The second one is a problem with the regexp. For the first one I habe a patch ready (still testing).

Anonymous
Fri 21 Apr 2023 12:55:25 PM UTC, comment #2: 

guess this happens in importdata_ascii subfunction in import_data.m


      ## Look for number, DELIMITER, DELIMITER*, number
      delim = regexpi (row, '[-+]?\d*[.]?\d+(?:[ed][-+]?\d+)?[ij]?([^-+\d.deij])\1*[-+]?\d*[.]?\d+(?:[ed][-+]?\d+)?[ij]?',
                       'tokens', 'once');


the trailing "1" in casename1 is recognized as a number, and "1;0" matches, "e;0" doesn't

A.R. Burgers <arb>
Thu 20 Apr 2023 09:19:50 AM UTC, comment #1: 

funnily change the csv file to (note the "1" in casename)


title;casename1;0
;;
load case;;
height;h;1.3


produces


A =

  scalar structure containing the fields:

    data =

            0
           NA
           NA
       1.3000

    textdata =
    {
      [1,1] = title
      [2,1] = load case
      [3,1] = height
      [4,1] = casename1
      [5,1] =
      [6,1] = h
    }


which is quite unexpected

Anonymous
Thu 20 Apr 2023 09:00:12 AM UTC, original submission:  

When calling importdata on a csv file with empty fields, octave groups them all in a single column:

this csv "myfile.csv"


title;casename;0
;;
load case;;
height;h;1.3


with this call


A=importdata("myfile.csv")


produces this output


A =
{
  [1,1] = title;casename;test;0
  [2,1] = ;;;
  [3,1] = load case;;;
  [4,1] = height;h;m;1.3
}


while matlab produces this output


A =

  scalar structure containing the fields:

    data =

            0
           NA
           NA
       1.3000

    textdata =
    {
      [1,1] = title
      [2,1] =
      [3,1] = load case
      [4,1] = height
      [1,2] = casename
      [2,2] =
      [3,2] =
      [4,2] = h
    }


which I would also expect from octave

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by arb (Posted a comment)
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-04-21 rik5 Item GroupIncorrect Result Matlab Compatibility
        StatusNone Confirmed
        Summaryimportdata does not like empty fields importdata returns column vector of textdata when there is an empty field in the imported data
    2023-04-20 jwe Item GroupMatlab Compatibility Incorrect Result

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code