bugGNU Octave - Bugs: bug #62152, Textscan fails in rare occurences

 
 

bug #62152: Textscan fails in rare occurences

Submitter:  None
Submitted:  Tue 08 Mar 2022 02:27:28 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  maiky76 Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 19 Apr 2022 10:33:39 PM UTC, comment #4: 

No responses.  Assuming patch worked as expected.  Marking as Fixed and closing report.

Rik <rik5>
Group administrator
Sun 13 Mar 2022 12:36:49 PM UTC, comment #3: 

I pushed a change to the default branch that changes to `peek_undelim` or `get_undelim` where I think the end of the buffer can be reached. In many places, that character might be put back to the stream if it doesn't meet some conditions. This is realized by decrementing the pointer to the current position in the buffer. That wouldn't work with the current design of the buffer after a refresh.
To prevent that issue from causing trouble, I changed the buffer refresh to retain an overlap with the previous buffer content. I chose a rather large number of overlapping characters (25). Most probably an overlap of 3 or 4 would already be fine. But I'm not sure about that.
See here:
https://hg.savannah.gnu.org/hgweb/octave/rev/cfb708de1fc9

IIUC, there are still a few places where `tellg` and `seekg` are used when the buffer might have been refreshed in the meantime. Afaict, that will most likely cause that parts of the (buffered) stream are skipped.
That's not a regression from this change but was already an issue before. I'm not sure how that can be resolved with the current design.
I'm wondering if there aren't some standard functions we could use instead of this manual buffering...

This bug still exists on the stable branch. But all of this buffering seems to be quite delicate, and I'd prefer to not make any intrusive changes in that area just before a planned release.
Maybe, we could consider cherry-picking this for a dot release when we are more certain this is fine.

Anyway, marking as ready for test for the issue reported here.

Markus Mützel <mmuetzel>
Group administrator
Tue 08 Mar 2022 04:56:39 PM UTC, comment #2: 

A someone stripped down reproducer for "Spectrum_ABEC_3.txt":

Path2filename = 'Spectrum_ABEC_3.txt';
fileID     = fopen (Path2filename, 'r');
formatSpec = repmat ('%f', 1, 75);
delimiter  = ' ';
startRow   = 105;
endRow     = 144;
dataArray  = textscan (fileID, formatSpec, endRow-startRow+1, 'Delimiter', delimiter, 'MultipleDelimsAsOne', true, 'HeaderLines', startRow-1, 'ReturnOnError', true, 'EndOfLine', '\r\n');
fclose (fileID);
assert (dataArray{24}(27), 7.45644633363295e-5)


I didn't write that here before: When I remove one of the digits of that number in the source text file, the entire file reads correctly. That's what makes me suspicious about the buffering that seems to be happening in chunks...

Markus Mützel <mmuetzel>
Group administrator
Tue 08 Mar 2022 10:37:15 AM UTC, comment #1: 

Thanks for your report.

No solution yet. But I believe the issue is that `delimited_stream::peek ()` doesn't work as expected if peeking beyond what is currently in the buffer `delimited_stream::m_buf`:
https://hg.savannah.gnu.org/hgweb/octave/file/0826c503f294/libinterp/corefcn/oct-stream.cc#l1267

    int peek (void)
    { return eof () ? std::istream::traits_type::eof () : *m_idx; }


The part of the code where I think this might be an issue is around here:
https://hg.savannah.gnu.org/hgweb/octave/file/0826c503f294/libinterp/corefcn/oct-stream.cc#l2868

    // look for exponent part in, e.g.,  6.023E+23
    bool used_exp = false;
    if (valid && width_left > 1 && m_exp_chars.find (ch) != std::string::npos)
      {
        int ch1 = is.peek ();


Maybe we could use `peek_undelim` here? Or somehow fix `delimited_stream::peek ()` when at the end of the buffer?

Markus Mützel <mmuetzel>
Group administrator
Tue 08 Mar 2022 02:27:28 AM UTC, original submission:  

Bug filed from:
https://octave.discourse.group/t/help-on-textscan-rare-occurrences-of-errors/2332/7

I have been having some very rare occurrences (about 0.15% based on 15k iterations) where the following code does not work.
The files are all generated by a single program and are supposed to strictly follow the same format.

In some instances the code fails to properly read the content of the file and stop somewhere for no apparent reason.
The equivalent function in Matlab never fails.

4 files examples are attached to test,
Spectrum_ABEC_OK.txt load with no issue
Spectrum_ABEC_KO.txt only load with the revised code
Spectrum_ABEC_3.txt stops line 27
Spectrum_ABEC_4.txt stops line 5


clear all
close all
clc
Pref       = 2e-5;

% EmptyData = 25 about 0.1%
% Total_iter = 24122 37h
% size(SPL) should return 37x40

 Path2filename   = 'Spectrum_ABEC_OK.txt'; % Vast majority of the files
% Path2filename   = 'Spectrum_ABEC_KO.txt'; %
% Path2filename   = 'Spectrum_ABEC_3.txt'; % stops line 27
 Path2filename   = 'Spectrum_ABEC_4.txt'; % stops line 5

% ------------------------------------------------------------------------------
% Read first part of the file never fails
delimiter  = ' ';
startRow   = 39;
endRow     = 78;
formatSpec = '%f%f%f%[^\n]';

fileID     = fopen(Path2filename,'r');
dataArrayZ = textscan(fileID, formatSpec, endRow-startRow+1, 'Delimiter', delimiter, 'MultipleDelimsAsOne', true, 'HeaderLines', startRow-1, 'ReturnOnError', false, 'EndOfLine', '\n');
fclose(fileID);
FreqZ      = cell2mat(dataArrayZ(1));
ZRe        = cell2mat(dataArrayZ(2));
ZIm        = cell2mat(dataArrayZ(3));
RadZ       = [ZRe ZIm];
% Clear temporary variables
clearvars delimiter startRow endRow formatSpec dataArrayZ ans;


if exist('OCTAVE_VERSION', 'builtin') == 0

  delimiter  = ' ';
  startRow   = 105;
  endRow     = 144;
  formatSpec = '%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%[^\n\r]';
  fileID     = fopen(Path2filename,'r');
  dataArray  = textscan(fileID, formatSpec, endRow-startRow+1, 'Delimiter', delimiter, 'MultipleDelimsAsOne', true, 'TextType', 'string', 'HeaderLines', startRow-1, 'ReturnOnError', false, 'EndOfLine', '\r\n');
  fclose(fileID);

  Freq           = cell2mat(dataArray(1)); % should be indentical to FreqZ
  SpectrumABEC   = cell2mat(dataArray(2:end-1));
  [~, col]       = size(SpectrumABEC);
  NbSpectra      = col/2

else

  delimiter  = ' ';
  startRow   = 105;
  endRow     = 144;

  % ------------------------------------------------------------------------------
  % original
   formatSpec = '%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%[^\n]';
   fileID     = fopen(Path2filename,'r');
   dataArray  = textscan(fileID, formatSpec, endRow-startRow+1, 'Delimiter', delimiter, 'MultipleDelimsAsOne', true, 'HeaderLines', startRow-1, 'ReturnOnError', false, 'EndOfLine', '\n');
   fclose(fileID);
   Freq           = cell2mat(dataArray(1));
   SpectrumABEC   = cell2mat(dataArray(2:end-1));
   [~, col]       = size(SpectrumABEC);
   NbSpectra      = col/2;

  % ------------------------------------------------------------------------------
  % Octave help
  %  formatSpec = '%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f';
  %  fileID     = fopen(Path2filename,'r');
  %  dataArray  = textscan(fileID, formatSpec, endRow-startRow+1, 'Delimiter', delimiter, 'MultipleDelimsAsOne', true, 'HeaderLines', startRow-1, 'ReturnOnError', true, 'EndOfLine', '\r\n');
  %  fclose(fileID);
  %
  %  Freq           = cell2mat(dataArray(1));
  %  SpectrumABEC   = cell2mat(dataArray(2:end));
  %  [~, col]       = size(SpectrumABEC);
  %  NbSpectra      = col/2;
  % ------------------------------------------------------------------------------

  [~, col]       = size(SpectrumABEC);
  NbSpectra      = col/2 % should return 37x40

end

for k = 1:NbSpectra
  SPL(:,k)       = 20*log10( abs( ( ( SpectrumABEC(:,2*k-1) + 1i*SpectrumABEC(:,2*k) ) /sqrt(2) ) / Pref ) ); % RMS value
  deg(k,1)       = (k-1)*5;
end

SizeSPL = size(SPL)

figure()
  subplot (211)
       semilogx(FreqZ, RadZ(:,1), 'k')
       hold on
       semilogx(FreqZ, RadZ(:,2), 'k--')
       grid on
       ylabel('Z')
       xlabel('Frequency [Hz]')
       legend('ReZ', 'ImZ')
       title('Waveguide Z');
  subplot (212)
       semilogx(Freq, SPL)
       grid on
       ylabel('SPL [dB]')
       xlabel('Frequency [Hz]')
       legend('SPL 5 deg step')
       title('Waveguide SPL');


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #52968:  Spectrum_ABEC_OK.txt added by None (66KiB - text/plain)
file #52969:  Spectrum_ABEC_3.txt added by None (66KiB - text/plain)
file #52970:  Spectrum_ABEC_KO.txt added by None (66KiB - text/plain)
file #52971:  Spectrum_ABEC_4.txt added by None (66KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by jwe (Updated the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-04-19 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2022-03-13 mmuetzel StatusConfirmed Ready For Test
        Release6.4.0 dev
    2022-03-08 mmuetzel StatusNone Confirmed
        Operating SystemMicrosoft Windows Any
    2022-03-08 jwe Item GroupMatlab Compatibility Incorrect Result
    2022-03-08 None Attached File- Added Spectrum_ABEC_OK.txt, #52968
        Attached File- Added Spectrum_ABEC_3.txt, #52969
        Attached File- Added Spectrum_ABEC_KO.txt, #52970
        Attached File- Added Spectrum_ABEC_4.txt, #52971

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code