bugGNU Octave - Bugs: bug #66161, Strange behaviour of fscanf with...

 
 

bug #66161: Strange behaviour of fscanf with CRLF in text mode

Submitter:  Johannes <johannes>
Submitted:  Mon 02 Sep 2024 08:14:10 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * 9.1.0 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 24 Oct 2024 07:11:44 AM UTC, comment #6: 

Thank you for looking into this.
I don't know where exactly the bug is either. If you find an issue with a particular function, the output of "which" is usually a good place to start looking. In this case:

>> which fscanf
'fscanf' is a built-in function from the file libinterp/corefcn/file-io.cc
>>


So, that source file might be a good idea to start investigating.

Markus Mützel <mmuetzel>
Group administrator
Wed 23 Oct 2024 06:44:43 PM UTC, comment #5: 

@mmuetzel, I am happy to work on it, but I am not familiar with the overall code structure of octave. I will need some pointers where to start.

can you let me know what unit I should be looking at that is most relevant to this fscanf behavior?

When I was testing a toolbox I wrote on windows yesterday, I realized that this issue goes back to even older versions on Windows, including 7.3 and 6.4.

Specifically, fscanf(fid, '%f', [4, inf]) fails to parse any line ending with -1 1 in the below testing file on Windows


fid=fopen('testdata.txt', 'rt');
data=fscanf(fid, '%f', [4, inf])';
fclose(fid);



20  3  0  0
   0    1  1  1
   1    1  1  -1
   2    1  -1  1
   3    1  -1  -1
   4    -1  1  1
   5    -1  1  -1
   6    -1  -1  1
   7    -1  -1  -1
   8    1  0  1
   9    0  1  1
  10    0  -1  -1
  11    1  -1  0
  12    -1  0  -1
  13    -1  -1  0
  14    0  1  -1
  15    1  0  -1
  16    1  1  0
  17    -1  1  0
  18    -1  0  1
  19    0  -1  1


Qianqian Fang <fangq>
Sat 19 Oct 2024 06:35:49 AM UTC, comment #4: 

@fangq: If you like to work on this issue with high priority, I'll be happy to review your patches. 👍

Markus Mützel <mmuetzel>
Group administrator
Fri 18 Oct 2024 05:28:46 AM UTC, comment #3: 

this should be a high-priority bug and deserves immediate attention and fix

fscanf is such a frequently used function and it is broken on windows.


I just noticed this bug when my github testing script failed on windows

https://github.com/fangq/iso2mesh/actions/runs/11396953367/job/31711637777

after debugging this problem, I found the culprit was fscanf failing to read the data file. Here is a very simple test

create testdata.txt with the following content


3 5088 5089 4737
3 4336 8 7
3 2922 6 5


then run

fid=fopen('testdata.txt','rt');
elem=fscanf(fid, '%f', [4 3])'
fclose(fid)


the output looks like


elem =

      3   5088   5089   4737
      3   4336      8      3
   2922      6      0      0


basically, everytime when there are two single-digit number at the end of the line, the second number will be dropped. I saw this pattern in a larger file as well.

Tested on windows 10 with Octave 9.2. I also have octave 8.x/7.x/5.x installed on this system, and none of them had this problem. Must be introduced in the 9.x version.

Qianqian Fang <fangq>
Wed 04 Sep 2024 08:23:40 AM UTC, comment #2: 

Hello Markus, thanks for your quick answer.

I do not know exactly with which version it worked, but it worked a year ago. My students almost all use Windows, and they had no problems in the period of October 2023 to January 2024. Most of them used the version that was the latest in October/November 2023.

I will try the workaround you proposed.

Johannes <johannes>
Wed 04 Sep 2024 07:17:50 AM UTC, comment #1: 

This seems to depend on CRLF line endings. If I convert the files to LF line endings, all tests pass...
Alternatively,  if the files are opened in binary mode, all tests also pass with CRLF line endings.

Windows does some weird things with the file position when the line endings are converted on-the-fly in text mode. Maybe, the issue is somewhere around that.

For the time being opening the files in binary mode seems to be a work-around. I.e., `fid = fopen(fname, "rb");` or simply `fid = fopen(fname, "r");`.


You marked this as a regression. Do you know in which version of Octave this was still working?

Markus Mützel <mmuetzel>
Group administrator
Mon 02 Sep 2024 08:14:10 AM UTC, original submission:  

Under Windows, fscanf sometimes has problems to read the last number in the line, see attached file bugdemo.m and the comments in the script. All tests of this script work correctly with Linux.

System information:


Gerätename      DESKTOP-VKR568P
Prozessor       Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz   2.90 GHz
Installierter RAM       16,0 GB (15,9 GB verwendbar)

Systemtyp       64-Bit-Betriebssystem, x64-basierter Prozessor
Stift- und Toucheingabe Für diese Anzeige ist keine Stift- oder Toucheingabe verfügbar.

Edition Windows 10 Enterprise
Version 22H2
Installiert am  <200e>23.<200e>05.<200e>2021
Betriebssystembuild     19045.4780
Leistung        Windows Feature Experience Pack 1000.19060.1000.0


Johannes <johannes>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #56403:  bugdemo.m added by johannes (2KiB - text/x-objcsrc)
file #56404:  file2.txt added by johannes (150B - text/plain)
file #56405:  file1.txt added by johannes (149B - text/plain)
file #56406:  file3.txt added by johannes (150B - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by fangq (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by johannes (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-09-04 mmuetzel StatusNone Confirmed
        SummaryStrange behaviour of fscanf Strange behaviour of fscanf with CRLF in text mode
    2024-09-02 johannes Attached File- Added bugdemo.m, #56403
        Attached File- Added file2.txt, #56404
        Attached File- Added file1.txt, #56405
        Attached File- Added file3.txt, #56406

    Back to the top

    Powered by Savane 3.14-3b9d.
    Corresponding source code