bugGNU Octave - Bugs: bug #60533, some Octave functions unable to...

 
 

bug #60533: some Octave functions unable to read ascii data file if D used as exponential separator

Submitter:  Nicholas Jankowski <nrjank>
Submitted:  Wed 05 May 2021 02:50:01 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  3 - Low Item Group:  Matlab Compatibility
Status:  Confirmed Assigned to:  None
Originator Name:  Nicholas Jankowski Open/Closed:  * Open
Release:  * 6.2.0 Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 06 May 2021 04:33:42 PM UTC, comment #4: 

textscan() is a class in oct-stream.cc.  It implements its own read_double() function rather than relying on the read_value() function in lo-utils.cc.  It's just what I suspected.  The read_double() function has to parse each character, effectively replicating the work of the C++ library.

Rik <rik5>
Group administrator
Thu 06 May 2021 04:23:25 PM UTC, comment #3: 

interesting.  what does textscan do differently?

Nicholas Jankowski <nrjank>
Group Member
Wed 05 May 2021 10:03:26 PM UTC, comment #2: 

Localizing the problem.  dlmread calls read_value<double> which is located in liboctave/util/lo-utils.cc.  In turn, this eventually calls read_fp_value in the same file.  The eventual code is


        is >> val;


We are relying on the C++ stream libraries for reading "val" and those libraries understand 'e' for exponent but not 'd' for exponent.  There might be a way to hack the libraries in to recognizing 'd' for 'e' by making one's own locale.  This sounds difficult though.  Otherwise, we would need to write our own stream input routine that would read character-by-character and assemble a number.  That is precisely what the C++ library does and it sounds like a lot work to replicate that.

Rik <rik5>
Group administrator
Wed 05 May 2021 09:53:49 PM UTC, comment #1: 

Annoying, but there is the simple workaround of search and replace on the original data file.  That's probably the fastest answer.

The 'd' or 'D' indicator for exponent is valid within the Octave interpreter.


octave:1> 1.5D3
ans = 1500


It would be nice, but not critical to support this.

Rik <rik5>
Group administrator
Wed 05 May 2021 02:50:01 PM UTC, original submission:  

attempting to load a text file containing data using D as the exponential separator (apparently used in some systems to distinguish double and single precision data), load produces an error and dlmread reads the data ignoring the D and numbers the D.

e.g., a text file 'testdata.dat' containing:


1.00000 .509844D-02 .334798D-02 .100602D-03
2.00000 .511026D-02 .334856D-02 .101089D-03


(as well as a separate testdata2.dat where the D has been replaced with E)

in matlab:

>> format long

>> a = load('testdata.dat')
a =

   1.000000000000000   0.005098440000000   0.003347980000000   0.000100602000000
   2.000000000000000   0.005110260000000   0.003348560000000   0.000101089000000

>> b = dlmread('testdata.dat')

b =

   1.000000000000000   0.005098440000000   0.003347980000000   0.000100602000000
   2.000000000000000   0.005110260000000   0.003348560000000   0.000101089000000

>> c = textread('testdata.dat')

c =

   1.000000000000000   0.005098440000000   0.003347980000000   0.000100602000000
   2.000000000000000   0.005110260000000   0.003348560000000   0.000101089000000

>> fid = fopen('testdata.dat');d = textscan(fid, '%f %f %f %f')

d =

  1×4 cell array

    {2×1 double}    {2×1 double}    {2×1 double}    {2×1 double}

>> d{4}

ans =

   1.0e-03 *

   0.100602000000000
   0.101089000000000

>> e = importdata('testdata.dat', ' ')

e =

  2×1 cell array

    {'1.00000 .509844D-02 .334798D-02 .100602D-03'}
    {'2.00000 .511026D-02 .334856D-02 .101089D-03'}

>> e = importdata('testdata2.dat', ' ')

e =

   1.000000000000000   0.005098440000000   0.003347980000000   0.000100602000000
   2.000000000000000   0.005110260000000   0.003348560000000   0.000101089000000

>> f = readtable('testdata.dat')

f =

  2×4 table

    Var1       Var2          Var3          Var4
    ____    __________    __________    ___________

     1      0.00509844    0.00334798    0.000100602
     2      0.00511026    0.00334856    0.000101089


Not sure if I've missed any that should be tested.  So other than importdata, all matlab data reading functions appear able to properly interpret the D.

in Octave 6.2.1 (hg id: 6fc423987872):

>> format long

>> a = load('testdata.dat')
error: load: failed to read matrix from file 'testdata.dat'

>> b = dlmread('testdata.dat')
b =

   1.000000000000000   0.509844000000000   0.334798000000000   0.100602000000000
   2.000000000000000   0.511026000000000   0.334856000000000   0.101089000000000

>> c = textread('testdata.dat')
c =

     1
   NaN
   NaN
   NaN
     2
   NaN
   NaN
   NaN

>> fid = fopen('testdata.dat');d = textscan(fid, '%f %f %f %f')
d =
{
  [1,1] =

     1
     2

  [1,2] =

     5.098440000000001e-03
     5.110260000000000e-03

  [1,3] =

     3.347980000000000e-03
     3.348560000000001e-03

  [1,4] =

     1.006020000000000e-04
     1.010890000000000e-04

}

>>
>> d{4}
ans =

   1.006020000000000e-04
   1.010890000000000e-04

>>  e = importdata('testdata.dat', ' ')
e =

   1.000000000000000   0.509844000000000   0.334798000000000   0.100602000000000
   2.000000000000000   0.511026000000000   0.334856000000000   0.101089000000000

>> f = readtable('testdata.dat')
error: 'readtable' undefined near line 1, column 1

The 'readtable' function is not yet implemented in Octave.


So, it looks like in octave only textscan is able to interpret a D exponential separator.

Nicholas Jankowski <nrjank>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by nrjank (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-05-05 rik5 StatusNone Confirmed
    2021-05-05 rik5 Priority5 - Normal 3 - Low

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code