bugGNU Octave - Bugs: bug #52867, textscan produces an incorrect...

 
 

bug #52867: textscan produces an incorrect cell array

Submitter:  Pedro Pena <pedrolpena>
Submitted:  Fri 12 Jan 2018 02:04:55 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Pedro Pena Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 29 Mar 2020 08:28:36 PM UTC, comment #8: 

This changeset https://hg.savannah.gnu.org/hgweb/octave/rev/abcff237241f from today (3/29/2020) fixes the issue in this report.  This will be a part of the 7.1 release of Octave.

Closing report.

Rik <rik5>
Group administrator
Mon 15 Jan 2018 02:27:13 PM UTC, comment #7: 

@Pedro:
A workaround is the following code that works for me to read your file:

## Open & read file
fid = fopen ("cruise_params.cfg.tmp");
txt = fread (fid, Inf, "char=>char")';
fclose (fid);
## Split file on line endings and remove comment lines
txt = strsplit (txt, "\r\n")';
txt = txt(find (cellfun (@isempty, strfind (txt, "#"))));
## Concatenate strings + apply textscan
txta = strjoin (txt, "\n");
C = textscan (txta, "%s%s", "delimiter", {"=", "\n"});


Philip Nienhuis <philipnienhuis>
Group Member
Mon 15 Jan 2018 02:20:15 AM UTC, comment #6: 

Thank you for pointing me in the right direction.
I've replaced the textscan portion of my code with
a textread conversion.I was not able to produce consistent results with what I was doing and it was really confusing me. After some massaging, I managed to get it to work realizing that there might be a bug in textread or one of its backend functions. I've searched for the bug but I can't seem to find one like it.
I'm going to submit a bug report for textread.

Pedro Pena <pedrolpena>
Sun 14 Jan 2018 11:43:23 PM UTC, comment #5: 

O.k., good tip.
I'll implement it that way in the meantime.
Thanks

Pedro Pena <pedrolpena>
Sat 13 Jan 2018 05:00:20 PM UTC, comment #4: 

BTW you can try to read the file with textread.m
That still invokes strread.m (that IIRC was textscan's backend in Octave-4.0.0) and takes the same paraneters and format strings.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 12 Jan 2018 07:32:10 PM UTC, comment #3: 


The cell array structure is good, so I agree it's
likely being read incorrectly.

In the submitted example a text file is prcoessed by textscan.
This text files contains text representing variable names and the values assigned to the variable names.
textscan should produce a cell matrix containing variable names along with the values for those variables.

cruiseVars{1} should contain the all of the variable names(left of "=").
cruiseVars{2} should contain the values assigned to the variables(right of "=").

This makes it easy to search for values using the variable name.

For example, say I want to retrieve the value assigned to variable "cruise_id" after cruiseVars is created.


>> X = strfind(cruiseVars{1},'cruise_id');
>> i=find(~cellfun(@isempty,X),1);
>> cruiseVars{2}{i}
ans = AB1705


I have a function that does this.
https://github.com/pedrolpena/visbeck_ladcp_processing/blob/master/m/misc/get_cruise_variable_value.m

 
In the above example "AB1705" is indeed the value assigned to variable "cruise_id" this was taken from the following line in the text file.

cruise_id=AB1705
cruise_id_prefix=
cruise_id_suffix=


However when I attempt to extract the value for "cruise_id_prefix", the value returned is "cruise_id_suffix", which
means that the data that should be contained in cruiseVars{1} has been placed in cruiseVars{2}. It's as if textscan  "sees" the next line
as being after the "=" on the same line.

This following is what it looks like to textscan.

cruise_id_prefix=cruise_id_suffix


It looks like the CRLF is being seen as another delimiter (I imagine it does use it as a new line delimiter).
Maybe there is some logic that applies the newline delimiter to the delimiter used in the text tokenization process?
A newline detection next to a delimiter detection issue?


With this in mind I tried one of your test's and I replaced the blank with a character and this made it work as expected even though it failed for you.


cruise_id=AB1705
cruise_id_prefix=a
cruise_id_suffix=b


the values on the left of the "=" are placed in cruiseVars{1} and the values on the right of "=" are placed in cruiseVars{2} as it should.


Pedro Pena <pedrolpena>
Fri 12 Jan 2018 08:29:13 AM UTC, comment #2: 

Furthermore, it makes no difference whether textscan reads from file or from text:

>> fid =  fopen ("cruise_params.cfg.txt")
fid =  3
>> txt = fread (fid, Inf, "char=>char")';
>> C = textscan (txt, '%s%s', 'Delimiter', '=', 'CommentStyle', '#');
>> C
C =
{
  [1,1] =
  {
    [1,1] = working_directory
    [2,1] = cruise_id
    [3,1] = cruise_id_prefix
    [4,1] = correct_year
    [5,1] = use_mat_for_nav
:


(note I changed file name extension but that shouldn't matter)

and

  • commentstyle (I tried with removing the comment lines),
  • whitespace setting,
  • endofline (your file contains CRLF, I also tried with just LF) and
  • delimiter (changed it into ";")
  • empty fields (I added "a" after the "=" in the "cruise_***fix=" lines)

make no difference either.

I cannot confirm that the cell array is "garbled" (the cell array structure itself looks good), it's just that the file has been read incorrectly, a.o. it consistently skips the line "cruise_id_suffix=".

-> Title adapted

Philip Nienhuis <philipnienhuis>
Group Member
Fri 12 Jan 2018 08:10:06 AM UTC, comment #1: 

Confirmed, it looks like it is still the same issue in 4.3.0+ (4.3.0+ tip from Jan. 10, Windows 7).

FTR, Matlab r2018a prerelease confirms that Octave-4.0.2 did it right.

Release -> dev
OS -> Any

Philip Nienhuis <philipnienhuis>
Group Member
Fri 12 Jan 2018 02:04:55 AM UTC, original submission:  

textscan under octave 4.2.1 produces a garbled cell array from a text file input but under Octave 4.02 the output is as one
would it expect it to be.

A commented text file(configuration file) with variables and assigned values is read. The delimiter is "=" with "#" read as a comment.


the file 'cruise_params.cfg.tmp'
is first opened and then read with textscan.

The contents are then placed in a cell array.

All works fine under Octave 4.02 but under octave 4.2.1 the cell array is garbled.

I've included the input file "cruise_params.cfg.tmp"  for testing and I've included a text file(octaveComparison.txt) with the results from the textscan for each octave version.


Below is a snippet of the output.

Octave 4.02 (correct)
============

fid=fopen('cruise_params.cfg.tmp','r');
cruiseVars=textscan(fid,'%s%s','Delimiter','=','CommentStyle','#');
cruiseVars
cruiseVars =
{
  [1,1] =
  {
    [1,1] = working_directory
    [2,1] = cruise_id
    [3,1] = cruise_id_prefix
    [4,1] = cruise_id_suffix



Octave 4.21 (incorrect)
===========

fid=fopen('cruise_params.cfg.tmp','r');
cruiseVars=textscan(fid,'%s%s','Delimiter','=','CommentStyle','#');
cruiseVars
cruiseVars =
{
  [1,1] =
  {
    [1,1] = working_directory
    [2,1] = cruise_id
    [3,1] = cruise_id_prefix
    [4,1] = correct_year


I'm not sure if it's a problem with textscan or one of its underlying functions.

This is part of a processing script that runs fine under octave 4.02
https://github.com/pedrolpena/visbeck_ladcp_processing

Pedro Pena <pedrolpena>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #42889:  cruise_params.cfg.tmp added by pedrolpena (44KiB - application/octet-stream - files)
file #42890:  octaveComparison.txt added by pedrolpena (11KiB - text/plain - files)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by pedrolpena (Submitted the item)
  • -email is unavailable- added by pedrolpena (octave 4.2.1 textscan bug)
  • -email is unavailable- added by pedrolpena (octave 4.2.1 textscan bug)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-03-29 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2018-01-12 philipnienhuis Summarytextscan reads text file using file handle and produces a garbled cell array textscan produces an incorrect cell array
    2018-01-12 philipnienhuis StatusNone Confirmed
        Release4.2.1 dev
        Operating SystemGNU/Linux Any
    2018-01-12 pedrolpena Attached File- Added cruise_params.cfg.tmp, #42889
        Attached File- Added octaveComparison.txt, #42890
        Carbon-Copy- Added -email is unavailable-
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code