bugGNU Octave - Bugs: bug #37023, strread with leading white space...

 
 

bug #37023: strread with leading white space in new lines

Submitter:  Xuejin Wang <fclxc10>
Submitted:  Fri 03 Aug 2012 10:58:48 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  philipnienhuis
Originator Name:  Xuejin Wang Open/Closed:  * Closed
Release:  * 3.6.1 Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 30 Aug 2012 07:00:16 PM UTC, comment #4: 
Philip Nienhuis <philipnienhuis>
Group Member
Thu 16 Aug 2012 09:48:58 AM UTC, comment #3: 

Sorry, that should read:

Change lines 378-379 in strread.m from:

      rxp_dlmwsp = sprintf ('( [%s]|[%s] )', dlmstr, dlmstr);
      str = regexprep (str, rxp_dlmwsp, delimiter_str(1));


into these three lines:

      if (! isempty (dlmstr))
        rxp_dlmwsp = sprintf ('( [%s] | [%s]|[%s] )', dlmstr, dlmstr, dlmstr);
        str = regexprep (str, rxp_dlmwsp, delimiter_str(1));
      endif


(watch out for line wrap)

Philip Nienhuis <philipnienhuis>
Group Member
Thu 16 Aug 2012 09:45:52 AM UTC, comment #2: 

Your analysis is spot on, Xuejin.
While strread.m by itself parses similar strings OK, this bug was hidden because of the way strread.m is invoked by textscan.m and textscan.m only has tests specific to specific textscan options.


octave-3.6.2.exe:23> [a, b] = strread (" 1. 1 \n  2 3 ", "%f %f")
a =
   1
   2
b =
   1
   3


... but:


octave-3.6.2.exe:25> [a, b] = strread (" 1. 1 \n  2 3 \n", "%f %f", "endofline", "\n")
a =
     1
   NaN
     3
b =
     1
     2


Cause: I didn't realize in full that regexprep() doesn't "do" overlapping matches. I don't know if that is documented.

Once I'm back at my dev machine I'll push a fix + extra tests (hopefully end of next week).
You'll probably have to wait for Octave-3.8 or 4.0, so that's why I suggest the fix below if you're in a hurry:

FIX (for Octave-3.6.2):
Change lines 378 in strread.m from:

      rxp_dlmwsp = sprintf ('( [%s]|[%s] )', dlmstr, dlmstr);

into these three lines:


      if (! isempty (dlmstr))
        rxp_dlmwsp = sprintf ('( [%s] | [%s]|[%s] )', dlmstr, dlmstr, dlmstr);
        str = regexprep (str, rxp_dlmwsp, delimiter_str(1));
      endif

(watch out for line wrap!)

Philip Nienhuis <philipnienhuis>
Group Member
Tue 14 Aug 2012 09:53:07 AM UTC, comment #1: 

(Sorry for late reply, I'm on vacation abroad w/o much Internet access)

The example file you posted contains an empty line. Is that correct?
When I interpret your datafile as

"   1. 1 \n 2 3\n"

I get:


octave-3.6.2.exe:13> data = textscan("   1. 1 \n 2 3\n", '%f %f')
data =
{
  [1,1] =

       1
     NaN
       3

  [1,2] =

     1
     2

}


...which indeed looks suspicious. I'll have a look at it in the next weeks.

As a workaround, you can also specify the "whitespace" parameter:


octave-3.6.2.exe:14> data = textscan("   1. 1 \n 2 3\n", "%f %f", "whitespace", "")
data =
{
  [1,1] =

     1
     2

  [1,2] =

     1
     3

}


Thanks for the report.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 03 Aug 2012 10:58:48 PM UTC, original submission:  

When using textscan to read some data from a text file, if a new line starts with a space and the previous line ends with a space, the first data in that line becomes "NaN".

For example, if the text file is as follows:

  1. 1

 2 3
It has two lines, each line starts with a space. The following command is used for reading the file:

data = textscan(fid,'%f %f')

The first line is always OK. If the first line has no space at its end, the second line is also OK. However if the first line has a space at the end (after the number "1" at the end of line 1), the first column becomes [0 NaN 3]', and the second column is [1 2]'.

I traced the problem to the function strread, line 373-380:

    ## Check for single delimiter followed/preceded by whitespace
    ## FIXME: Double strrep on str is enormously expensive of CPU time.
    ## Can this be eliminated
    if (! isempty (delimiter_str))
      dlmstr = setdiff (delimiter_str, " ");
      rxp_dlmwsp = sprintf ("( [%s]|[%s] )", dlmstr, dlmstr);
      str = regexprep (str, rxp_dlmwsp, delimiter_str(1));
    endif

My workaround is to run line 381-387 before the above lines. Maybe you'll have a better solution for it.

Thanks,
Xuejin

Xuejin Wang <fclxc10>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by philipnienhuis (Posted a comment)
  • -email is unavailable- added by fclxc10 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-08-30 philipnienhuis StatusIn Progress Fixed
        Originator Name Xuejin Wang
        Open/ClosedOpen Closed
    2012-08-16 philipnienhuis StatusConfirmed In Progress
    2012-08-14 philipnienhuis StatusNone Confirmed
        Assigned toNone philipnienhuis
        Operating SystemMicrosoft Windows Any

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code