bugGNU Octave - Bugs: bug #57086, textscan "TreatAsEmpty"...


bug #57086: textscan "TreatAsEmpty" option produces a garbled cell array

Submitted by:  None
Submitted on:  Sun 20 Oct 2019 08:40:19 AM UTC  
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  Nicky McLean Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  5.1.0
Operating System:  Any

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Mon 21 Oct 2019 02:14:52 AM UTC, comment #3: 

There is something wrong with the "TreatAsEmpty" option.  I don't know if this is the only problem, but it certainly exists.

I've attached a simpler example for this.  When there is a pattern that should be treated as empty and it is in the last position in the file, it is returned as 0 instead of as NaN.

Incidentally, since the expression in Hic.m is using "TreatAsEmpty", "?" it should be replacing "?" with NaN everywhere in the file.  If I open up the data file using a text editor and search and replace to change "?" to NaN and then use Hic.m to read the file, then there are 7440 data lines as expected.

(file #47724, file #47725)

Rik <rik5>
Project Administrator
Sun 20 Oct 2019 02:44:42 PM UTC, comment #2: 

FWIW, there are some open textscan bug reports. It may well be that this a duplicate, esp. of bug #52867.

Philip Nienhuis <philipnienhuis>
Project Member
Sun 20 Oct 2019 02:06:14 PM UTC, comment #1: 

Thanks for providing a full reproduction case with your bug report! This'll make it easy to work on.

Indeed looks like a textscan bug.

I can reproduce on Octave 4.4.1 on macOS 10.14.6. Doesn't sound like anything OS-specific is going on.

One thing I can see is that the Dave.csv file looks like it's in ISO 8859-1 encoding, where I think recent Octaves expect the input to be in UTF-8 encoding. I don't see how that could cause textscan to screw up and leave out lines; none of the over-128 characters appear in data rows, as far as I can tell.

Andrew Janke <apjanke>
Sun 20 Oct 2019 08:40:19 AM UTC, original submission:  

   Exactly the same misbehaviour was found with Octave 4.4.0 on a wunduhs XP system (32-bit) and on rebooting to Linux Mint 64-bit, Octave 4.2.2. On noticing that the "release" options in the bug submission form now go up to 5.1.0, after an extensive exercise with "flatpak" (there being no apparent upgrade arrangement for 4.2.2), the same test gives the same results.
   The test run is via file Hic.m, which reads data from file Dave.csv.
   The first problem is that lines of data are lost. The data file has three heading lines followed by data in records 4 to 7443. But only 6778, or instead, 6777 sets of values appear in the cell array. The first column of data contains a date, and the conversion of the date to a daynumber via datenum failed (this was the initial encounter, involving a much more general 700-line script)  Grist{1}(6777) = 25/03/2013 but Grist{1}(6778) = .49, which is not valid input for datenum.
   The second problem is that zero values are supplied when the input data file does not have them. A command scatter(Grist{8},Grist{10}); (temperatures at the Baring Head site vs. temperatures at Dave's home) shows two zero values (outside expectations for these data) at elements 1159 and 6777 of Grist{10}. Inspection of the source data file Dave.csv shows no such zero values.

   The original script file Gnash.m was developed for Matlab and used the name GnashData instead of Grist as in Hic.m. To the best of my recollection, the above problems did not occur via MatLab. The data file Dave.csv was written by a prog. Gnash.exe that has the ability to read in its output format; it found all data records correct and produced no extraneous zero values.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #47724:  tst_empty.m added by rik5 (139B - text/x-matlab)
file #47725:  tst_empty.csv added by rik5 (229B - text/csv)
file #47719:  Hic.m added by None (611B - text/x-objcsrc - Hic.m should read Dave.csv and show some of the odd results.)
file #47720:  Dave.csv added by None (566KiB - text/csv - Hic.m should read Dave.csv and show some of the odd results.)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by siko1056 (Updated the item)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-10-21 rik5 Attached File- => Added tst_empty.m, #47724
        Attached File- => Added tst_empty.csv, #47725
        Summarytextscan produces a garbled cell array: some data are missing, some values are wrong. => textscan "TreatAsEmpty" option produces a garbled cell array
    2019-10-20 siko1056 StatusNone => Confirmed
        Operating SystemGNU/Linux => Any
    2019-10-20 None Attached File- => Added Hic.m, #47719
        Attached File- => Added Dave.csv, #47720

    Back to the top

    Powered by Savane 3.5