bugGNU Octave - Bugs: bug #52892, textread incorrectly reads a text...

 
 

bug #52892: textread incorrectly reads a text file when empty lines are present

Submitter:  Pedro Pena <pedrolpena>
Submitted:  Mon 15 Jan 2018 02:53:13 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  None Assigned to:  None
Originator Name:  Pedro Pena Open/Closed:  * Open
Release:  * 4.2.1 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 17 Jan 2018 01:13:28 PM UTC, comment #9: 

Great info, thanks for pointing me in the right direction.



Pedro Pena <pedrolpena>
Wed 17 Jan 2018 06:43:32 AM UTC, comment #8: 

I'm using the latest development code, so there's a good chance behavior is different.  These file-reading routines have seen a lot of bug-fixes over the years.

There is no IDE that is recommended or supported by development.  (The GUI layout can work with qtcreator, I think, but that's about it.)

There are some suggestions for developers and creating patches/changesets here:

https://octave.sourceforge.io/developers.php

Basically, useful tools are grep and some type of Mercurial source control viewer such as "thg".  That just makes it convenient to look at changesets.  The online Mercurial interface is OK, but not fast.  To become familiar with code and where it resides in th source tree I suggest using thg and then type "textread", or "textscan" is a another related file that I know has had some changesets recently.  thg will then give a list of changeset that you may look at to see just the changes, and also the names of the files and full/relative directory path so that you may quickly locate where some of these routines are.  For example, here's a recent change I made regarding the EOL behavior for textscan:

http://hg.savannah.gnu.org/hgweb/octave/rev/f77da8da0f3f

When you think you've got something useful, commit your changes either from the command line or from thg (which is convenient).  Use the changeset message format that is commonly used in all the other changesets.  Once committed, do something like

hg export tip > my_spiffy_changeset-abc2018jan17.patch

where abc is initials.  That's just the naming convention I use for my organizational purpose because often I will create multiple revs over the course of a few days, but any file naming convention is suitable.  The extensions "patch" and "diff" are good because the file extension often is a signal for people's editors (e.g., gvim) about the format and automatic highlighting will be used.

Include a bug report number in the first line of the changeset message, if appropriate.  Try to make the changesets atomic in the sense of only one conceptual change per changeset.  E.g., don't address file reading and plotting images with the same changeset.

Dan Sebald <sebald>
Wed 17 Jan 2018 03:50:36 AM UTC, comment #7: 

@Dan

I've tried your examples using the same files but our results differ.
That said, specifying eol as "\r" works nicely and also clears things up a bit.
I'll have to test it out some more tomorrow.

@Anyone
BTW, I have some experience with C and C++ and I'm interested in looking at the textscan code.
Is there a recommended development environment for octave or is it pretty much just use your basic text editor and gdb?
I have the source code and I've seen the documentation produced with doxygen but no mention of a development environment.

 


>> ver
----------------------------------------------------------------------
GNU Octave Version: 4.2.1
GNU Octave License: GNU General Public License
Operating System: Linux 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64
----------------------------------------------------------------------
Package Name        | Version | Installation directory
--------------------+---------+-----------------------
             image  |   2.6.1 | /home/pedro/octave/image-2.6.1
instrument-control  |   0.2.2 | /home/pedro/octave/instrument-control-0.2.2
                io  |   2.4.2 | /home/pedro/octave/io-2.4.2
               nan  |   3.1.4 | /home/pedro/octave/nan-3.1.4
        statistics  |   1.2.4 | /home/pedro/octave/statistics-1.2.4
>> [a,b]=textread('cruise_params_with_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#');
>> a{1}
ans =
>> b{1}
ans = working_directory
>> a{end}
ans = nan
>> b{end}
ans =
>> [a,b]=textread('cruise_params_no_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#',"endofline","\n");
>> a{1}
ans = working_directory
>> b{1}
ans = .
>> a{end}
ans = beam2earth_bad_down_beam
>> b{end}
ans = nan
>> [a,b]=textread('cruise_params_no_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#',"endofline","\r");
>> a{1}
ans = working_directory
>> b{1}
ans = .
>> a{end}
ans =
>> b{end}
ans =
>> [length(a) length(b)]
ans =

   123   123


Pedro Pena <pedrolpena>
Tue 16 Jan 2018 06:49:58 PM UTC, comment #6: 

@Dan:
you can also specify .., "endofline, "\r\n", ... (same with textscan). endofile can be "" (empty), "\n", "\r", or "\r\n", see help.

Behind the scenes, textread.m's backend strread.m strips away the leading CR before any further processing of EOLs.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 16 Jan 2018 04:38:37 PM UTC, comment #5: 

I have several versions of the data files and it is possible I uploaded a version that I did not intend to, I'll have to check it out.
 
I like your theory that the bug might just be that textread is not auto detecting CR+LF, almost sounds like an easy fix.....

I hope to have some time later in the day to test some texread scenarios.

With all of your suggestions, I've been able to get my program mostly working and at the moment I can use what I have.

Unfortunately there are differences on how this program runs under 4.2.1 and 4.0.2.
I hope these are due to poor programming on my part and that they are not actual bugs....
 



Pedro Pena <pedrolpena>
Tue 16 Jan 2018 09:41:58 AM UTC, comment #4: 

There is another difference between these two attached data files.  The EOL character for cruise_params_with_empty_lines.cfg is <0A> LF while the EOL characters for cruise_params_no_empty_lines.cfg is <0D><0A> CR+LF.  For me, the following work:


octave:33> [a,b]=textread('cruise_params_with_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#');
octave:34> a{1}
ans = working_directory
octave:35> b{1}
ans = .
octave:36> a{end}
ans = beam2earth_bad_down_beam
octave:37> b{end}
ans = nan
octave:38>
octave:38>
octave:38> [a,b]=textread('cruise_params_no_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#',"endofline","\n");
octave:39> a{1}
ans = working_directory
octave:40> b{1}
ans = .
octave:41> a{end}
ans = beam2earth_bad_down_beam
octave:42> b{end}
ans = nan


That is, if I tell textread that the "endofline" is "\n" (the LF or CR+LF) then the file reads fine.  I suppose the CR before the LF gets treated as whitespace and is ignored.

The following doesn't quite work:


octave:43> [a,b]=textread('cruise_params_no_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#',"endofline","\r");
octave:44> a{1}
ans = working_directory
octave:45> b{1}
ans = .
octave:46> a{end}
ans =
octave:47> b{end}
ans =
octave:48> [length(a) length(b)]
ans =

   123   123


There's an extra entry at the end because after the last CR is still a LF and that gets treated as another line.

Anyway, could it be that the bug here is simply that textread() is not auto-detecting CR+LF when the documentation suggests that it should?

Dan Sebald <sebald>
Mon 15 Jan 2018 02:25:35 PM UTC, comment #3: 

BTW @Pedro:
Please be aware that textread (and strread) also has an "endofline" parameter. EOLs are implicitly added to the delimiter collection rather than to the whitespace collection. So empty lines will "break" the flow and make the process of reading the file getting out of sync with the format string.

It would be better if textscan got fixed, your other bug report (bug #52867) looks like a real bug to me, still in Octave dev version. But fixing textscan is a bit beyond me (I'm only a little familiar with C++)

Philip Nienhuis <philipnienhuis>
Group Member
Mon 15 Jan 2018 02:02:17 PM UTC, comment #2: 

Well, I'm the one who suggested Pedro to use textread.m, but to my own diappointment I soon found out that textread.m (in fact its backend strread.m) chokes on the specific form of the comment lines in the file. It looks like consecutive comment chars (rather than one comment char + text + EOL) screw up somewhere; that happens with any single comment char, also with % rather than #. I dived into it a bit but ran out of time.

FYI some years ago I have done a lot of work on strread.m (and textread.m) but the comment processing code is Rik's.

I'm not sure if at this stage a separate bug report is required for that, maybe later. Note that strread.m and textread.m are not so much deprecated but rather disadvised by the Mathworks and I suppose Octave follows suite there.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 15 Jan 2018 08:21:46 AM UTC, comment #1: 

There has been quite a bit of activity concerning textscan, textread, etc.  As you mentioned, there is an internal C++ routine that is used by script files.  I forget exactly which is which, but some bug reports are here:

https://savannah.gnu.org/bugs/index.php?52550
https://savannah.gnu.org/bugs/index.php?52479

However, this testread/scan is one voluminous piece of code to account for so many different scenarios, and there is a good chance that more adjustments need to be done.

I have the latest development code and can test your demos here.  You mention snippet, so I'm assuming you manually removed a big chunk of the output because I see many more lines...so I too will snip the results to the first ten:  I see


octave:1> [a,b]=textread('cruise_params_with_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#');
octave:2> a
a =
{
  [1,1] =
  [2,1] = .
  [3,1] = AB1705
  [4,1] =
  [5,1] =
  [6,1] = 2017
  [7,1] = 0
  [8,1] = 1
  [9,1] = 0
  [10,1] = psc
***SNIP***
  [123,1] = nan
}

octave:3> b
b =
{
  [1,1] = working_directory
  [2,1] = cruise_id
  [3,1] = cruise_id_prefix
  [4,1] = cruise_id_suffix
  [5,1] = correct_year
  [6,1] = use_mat_for_nav
  [7,1] = make_nav
  [8,1] = use_sadcp
  [9,1] = print_formats
  [10,1] = remove_zctd_downcast
***SNIP***
  [22,1] = position_fixed
 ESCOD
{
  [1,1] = working_directory
  [2,1] = cruise_id
  [3,1] = cruise_id_prefix
  [4,1] = cruise_id_suffix
  [5,1] = correct_year
  [6,1] = use_mat_for_nav
  [7,1] = make_nav
  [8,1] = use_sadcp
  [9,1] = print_formats
***SNIP***
  [122,1] = beam2earth_bad_down_beam
  [123,1] =
}



octave:4> [a,b]=textread('cruise_params_no_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#');
error: str(0): subscripts must be either integers 1 to (2^63)-1 or logicals
error: called from
    strread at line 446 column 5
    textread at line 249 column 31


You mentioned that the only difference between these files is the blank lines.  However, when I do a diff comparison ignoring white space, I see the following differences:


linux@ ~/octave/bug/52892 $ diff cruise_params_no_empty_lines.cfg cruise_params_with_empty_lines.cfg -wu
--- cruise_params_no_empty_lines.cfg        2018-01-15 01:11:56.643370310 -0600
+++ cruise_params_with_empty_lines.cfg        2018-01-15 01:11:36.559370114 -0600
@@ -4,12 +4,13 @@
 # Any line that starts with a "#" will be ignored. Don't add comments #
 # after a variable because this can mess up the parsing of this file  #
 # in some versions of Matlab and Octave.                              #
-# Using the "percent" percent symbol in comments on a line before a line    #
+# Using the "%" percent symbol in comments on a line before a line    #
 # with a variable to read can cause that variable to be ignored.      #
-# Avoid  using "percent"                                                    #
+# Avoid  using "%"                                                    #
 #                                                                     #
 #                                                                     #
 #######################################################################
+
 #######################################################################
 # to process the cast in the current directory, set this variable to  #
 # "." without quotes. All the paths in the script  should be          #
@@ -724,3 +725,4 @@
 beam2earth_bad_down_beam=nan
 #                                                                     #
 #######################################################################
+


Is there an ancillary bug here that you ran across?

The reason that this is being shifted is that there is one empty line added that is being read as though it was an entry.  Take a look at a[1,1]; it's blank.  So the first non-blank, non-comment item ends up in b[1,1].  Perhaps that is wrong behavior; don't know.  I've tried adding the option ...,"whitespace","\n") and that seems to have no effect.  The documentation indicates:


octave:24> help strread
'strread' is a function from the file /home/sebald/octave/octave/octave/scripts/io/strread.m

 -- [A, ...] = strread (STR)
 -- [A, ...] = strread (STR, FORMAT)
 -- [A, ...] = strread (STR, FORMAT, FORMAT_REPEAT)
 -- [A, ...] = strread (STR, FORMAT, PROP1, VALUE1, ...)
 -- [A, ...] = strread (STR, FORMAT, FORMAT_REPEAT, PROP1, VALUE1,
          ...)
     Read data from a string.
***SNIP***
     "delimiter"
          Any character in VALUE will be used to split STR into words
          (default value = any whitespace).  Note that whitespace is
          implicitly added to the set of delimiter characters unless a
          "%s" format conversion specifier is supplied; see "whitespace"
          parameter below.  The set of delimiter characters cannot be
          empty; if needed Octave substitutes a space as delimiter.
***SNIP***
     "whitespace"
          Any character in VALUE will be interpreted as whitespace and
          trimmed; the string defining whitespace must be enclosed in
          double quotes for proper processing of special characters like
          "\t".  In each data field, multiple consecutive whitespace
          characters are collapsed into one space and leading and
          trailing whitespace is removed.  The default value for
          whitespace is " \b\r\n\t" (note the space).  Whitespace is
          always added to the set of delimiter characters unless at
          least one "%s" format conversion specifier is supplied; in
          that case only whitespace explicitly specified in "delimiter"
          is retained as delimiter and removed from the set of
          whitespace characters.  If whitespace characters are to be
          kept as-is (in e.g., strings), specify an empty value (i.e.,
          "") for "whitespace"; obviously, whitespace cannot be a
          delimiter then.


I think the reason that the blank line is being treated as an item (i.e., a[1,1]) is the fact that your example uses at least one %s.  The %s means a string, and it is conceivable that that string is empty.  Hence any blank lines are considered empty strings, I guess.

I'm not sure there is a bug here.  What do you think?

Dan Sebald <sebald>
Mon 15 Jan 2018 02:53:13 AM UTC, original submission:  


textread or one if its backend functions incorrectly reads a text file when empty lines are present.
In this example I used two text files that differ only in that one has empty lines and the other has  none.
"cruise_params_no_empty_lines.cfg" and "cruise_params_with_empty_lines.cfg"

In this example
the contents of a and b are seemingly swapped and shifted but who knows what's really going on here.

I believe I also saw this behaviour in octave 4.02.


Here is a snippet the result when the file is read containing empty lines


>> [a,b]=textread('cruise_params_with_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#');
>> a
a =
{
  [1,1] =
  [2,1] = .
  [3,1] = AB1705
  [4,1] =
  [5,1] =
  [6,1] = 2017
  [7,1] = 0
  [8,1] = 1
  [9,1] = 0
  [10,1] = psc

>> b
b =
{
  [1,1] = working_directory
  [2,1] = cruise_id
  [3,1] = cruise_id_prefix
  [4,1] = cruise_id_suffix
  [5,1] = correct_year
  [6,1] = use_mat_for_nav
  [7,1] = make_nav
  [8,1] = use_sadcp
  [9,1] = print_formats
  [10,1] = remove_zctd_downcast


Here is a snippet of the result when the file is read containing no empty lines(This is what I was expecting)


>> [a,b]=textread('cruise_params_no_empty_lines.cfg','%s%s','Delimiter','=','CommentStyle','#');
>> a
a =
{
  [1,1] = working_directory
  [2,1] = cruise_id
  [3,1] = cruise_id_prefix
  [4,1] = cruise_id_suffix
  [5,1] = correct_year
  [6,1] = use_mat_for_nav
  [7,1] = make_nav
  [8,1] = use_sadcp
  [9,1] = print_formats
  [10,1] = remove_zctd_downcast

>> b
b =
{
  [1,1] = .
  [2,1] = AB1705
  [3,1] =
  [4,1] =
  [5,1] = 2017
  [6,1] = 0
  [7,1] = 1
  [8,1] = 0
  [9,1] = psc
  [10,1] = 1


Pedro Pena <pedrolpena>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #42912:  cruise_params_no_empty_lines.cfg added by pedrolpena (44KiB - application/x-wine-extension-cfg)
file #42913:  cruise_params_with_empty_lines.cfg added by pedrolpena (44KiB - application/x-wine-extension-cfg)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by pedrolpena (textread bug on octave 4.2.1)
  • -email is unavailable- added by pedrolpena (textread bug on octave 4.2.1)
  • -email is unavailable- added by pedrolpena (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-01-15 pedrolpena Attached File- Added cruise_params_no_empty_lines.cfg, #42912
        Attached File- Added cruise_params_with_empty_lines.cfg, #42913
        Carbon-Copy- Added -email is unavailable-
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code