bugGNU Octave - Bugs: bug #57644, fseek fread fail for files >4GB...

 
 

bug #57644: fseek fread fail for files >4GB 5.1.0 on Windows10 64-bit x64 proc

Submitter:  Emmanuel Brandt <emmanuelbrandt>
Submitted:  Tue 21 Jan 2020 11:15:40 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  lostbard
Originator Name:  Emmanuel Brandt Open/Closed:  * Closed
Release:  * 5.1.0 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 12 Feb 2020 08:33:41 AM UTC, comment #41: 

Thank you for reporting back.  Closing report.

Kai Torben Ohlhus <siko1056>
Group Member
Wed 12 Feb 2020 08:30:01 AM UTC, comment #40: 

Tried yesterday it works fine
With large files: 4 to 40gb
This for solving this issue ;)

Emmanuel Brandt <emmanuelbrandt>
Wed 12 Feb 2020 07:56:14 AM UTC, comment #39: 

Did anyone try out if this now works with Octave 5.2.0 in Windows?  Can this report be closed?

Kai Torben Ohlhus <siko1056>
Group Member
Sun 02 Feb 2020 10:13:12 PM UTC, comment #38: 

This should appear in the 5.2 (mxe-octave gnulib patch) as well as upcoming 6.X (updated gnulib) builds

John Donoghue <lostbard>
Group Member
Sun 26 Jan 2020 01:06:55 PM UTC, comment #37: 

The commit to gitlib repo:
http://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=a51a5f5d24ce8e9be82c6a564576500d0f4d4428

Patched in mxe-octave

https://hg.octave.org/mxe-octave/rev/db215107402f


The patch will have to be undone in mxe-octave when octave gets another update for gnulib.

John Donoghue <lostbard>
Group Member
Sat 25 Jan 2020 10:03:22 PM UTC, comment #36: 

gotcha... attempt #2:

Hint: use +nomarkup+ <literal text> -nomarkup- to get rid of unintended italics when typing underscores.


Philip Nienhuis <philipnienhuis>
Group Member
Sat 25 Jan 2020 10:01:16 PM UTC, comment #35: 

Hint: use <literal text> to get rid of unintended italics when typing underscores.
I often forget that myself :-)

Philip Nienhuis <philipnienhuis>
Group Member
Sat 25 Jan 2020 09:06:42 PM UTC, comment #34: 

Actually woulkd be :

diff --git a/lib/lseek.c b/lib/lseek.c
index 013f665..b249839 100644
--- a/lib/lseek.c
+++ b/lib/lseek.c
@@ -63,7 +63,7 @@ rpl_lseek (int fd, off_t offset, int whence)
       return -1;
     }
 #endif
-#if _GL_WINDOWS_64_BIT_OFF_T
+#if GL_WINDOWS_64_BIT_OFF_T || (defined _MINGW32_ && defined
_FILE_OFFSET_BITS && (_FILE_OFFSET_BITS == 64))
   return _lseeki64 (fd, offset, whence);
 #else
   return lseek (fd, offset, whence);



(A typo correction for _MINGW32_)

John Donoghue <lostbard>
Group Member
Sat 25 Jan 2020 08:52:11 PM UTC, comment #33: 

Posted to gnulib at https://lists.gnu.org/archive/html/bug-gnulib/2020-01/msg00146.html

its an issue with gnulib, but effects mingw/windows only.

They have posted as suggest fix already, which is somewhat similar to what I posted in comment 19.


diff --git a/lib/lseek.c b/lib/lseek.c
index 013f665..b249839 100644
--- a/lib/lseek.c
+++ b/lib/lseek.c
@@ -63,7 +63,7 @@ rpl_lseek (int fd, off_t offset, int whence)
       return -1;
     }
 #endif
-#if _GL_WINDOWS_64_BIT_OFF_T
+#if GL_WINDOWS_64_BIT_OFF_T || (defined __MINGW32_ && defined
_FILE_OFFSET_BITS && (_FILE_OFFSET_BITS == 64))
   return _lseeki64 (fd, offset, whence);
 #else
   return lseek (fd, offset, whence);


I have tried this exact fix yet to verify, but it should work.

John Donoghue <lostbard>
Group Member
Fri 24 Jan 2020 09:54:02 AM UTC, comment #32: 

@JohnD going through the discussion thread, I'm still not sure to understand the root cause at this point:

  • Is this with gnulib/win?
  • Did you managed to make it work on large files?
  • Any temp work around I may use?
  • Use me for testing if you need at some point


PS. As a side note and historical background: I faced this issue as I routinely work on very large video astronomical files [4-90GB]. Never had any issue on Linux (Ubuntu) and then running same script on Octave/Windows it failed...
Thanks to all for the effort, really appreciated ;)

Emmanuel Brandt <emmanuelbrandt>
Fri 24 Jan 2020 02:08:04 AM UTC, comment #31: 

Looking more, of why its suddently now working ok - the dependancies for gnulib largefile seems far larger when generating the gnulib files in linux to those when generated in msys2.

I guess I'll have to add a quiestion taht as well.

John Donoghue <lostbard>
Group Member
Thu 23 Jan 2020 10:05:14 PM UTC, comment #30: 

And of course suddenly it is working in msys2 without any changes that I can think of.

Only thing I see different is that for some reason it doesnt do the check for fseeki64 and doesnt attempt to use gnulibs fseek , fseeko modules - descibe not actual changes except to regenerate the m4/configure files.

John Donoghue <lostbard>
Group Member
Thu 23 Jan 2020 04:03:06 PM UTC, comment #29: 

Yeah am slowing working on something to send up to them for discussion.



I also tried compiling in msys2 using the i686-w64-mingw32 compiler - it does the same fail.

John Donoghue <lostbard>
Group Member
Thu 23 Jan 2020 03:52:17 PM UTC, comment #28: 

With all that data, I'm pretty sure this is an upstream bug in gnulib.
@John: You have done more debugging here. Could you please ask upstream? Is -email is unavailable- the correct mailing list?

Markus Mützel <mmuetzel>
Group administrator
Thu 23 Jan 2020 03:38:18 PM UTC, comment #27: 

I created a simple barebones app using gnulib and that does a fseek to end and a ftell and compiled in a actual msys2 x86_64 install, and it fails at the seek to end just like we are seeing.


John Donoghue <lostbard>
Group Member
Thu 23 Jan 2020 08:18:29 AM UTC, comment #26: 

Maybe files >4 GiB are rare and no one noticed yet. But I also think that this is somewhat unlikely.

MS discourages calling 'lseek' at all. [1]

As I understand it, I'd guess the following change might be correct:

diff -r c34f677e6117 lib/lseek.c
--- a/lib/lseek.c        Sun Mar 17 19:27:20 2019 +0100
+++ b/lib/lseek.c        Thu Jan 23 09:12:26 2020 +0100
@@ -52,6 +52,7 @@
       errno = ESPIPE;
       return -1;
     }
+  return _lseeki64 (fd, offset, whence);
 #else
   /* BeOS lseek mistakenly succeeds on pipes...  */
   struct stat statbuf;
@@ -62,10 +63,6 @@
       errno = ESPIPE;
       return -1;
     }
-#endif
-#if _GL_WINDOWS_64_BIT_OFF_T
-  return _lseeki64 (fd, offset, whence);
-#else
   return lseek (fd, offset, whence);
 #endif
 }


Gnulib makes sure with its #define macros that off_t is always 64 bit wide. So it should be save to always call "_lseeki64".
But I might be missing something.

[1]: https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/lseek?redirectedfrom=MSDN&view=vs-2019

Markus Mützel <mmuetzel>
Group administrator
Wed 22 Jan 2020 10:11:38 PM UTC, comment #25: 

Yeah, like I said, the Gnulib code logic and code looks right to me, my guess is it's something wrong in the way Octave is configuring or compiling itself for MinGW.

Mike Miller <mtmiller>
Group Member
Wed 22 Jan 2020 10:09:37 PM UTC, comment #24: 

It seems odd that we would be the first to find this problem.  Are we doing something wrong with our use of gnulib here?  If it looks like a gnulib bug, then can we please report it and fix the problem there?

John W. Eaton <jwe>
Group administrator
Wed 22 Jan 2020 10:05:26 PM UTC, comment #23: 

I'm bowing out of this report again. Just trying to explain why I think the Gnulib logic and lseek.c source code looks correct to me, and what I think the intended behavior is.

Mike Miller <mtmiller>
Group Member
Wed 22 Jan 2020 10:01:02 PM UTC, comment #22: 

_GL_WINDOWS_64_BIT_OFF_T gets set in libgnu/sys_types.in.h




/* Override off_t if Large File Support is requested on native Windows.  */
#if @WINDOWS_64_BIT_OFF_T@
/* Same as int64_t in <stdint.h>.  */
# if defined _MSC_VER
#  define off_t __int64
# else
#  define off_t long long int
# endif
/* Indicator, for gnulib internal purposes.  */
# define _GL_WINDOWS_64_BIT_OFF_T 1
#endif


See largefile.m4 in comment #12 for WINDOWS_64_BIT_OFF_T




John Donoghue <lostbard>
Group Member
Wed 22 Jan 2020 09:42:05 PM UTC, comment #21: 

And for comment #17, since it is is resolving to  use lseek.c which in turn calls the windows _lseek, it is bypassing the use of off_t ect

John Donoghue <lostbard>
Group Member
Wed 22 Jan 2020 09:38:39 PM UTC, comment #20: 

btw the patch in comment #19 is really this makes it work rather than gnulib should be changing to do it that way :)

John Donoghue <lostbard>
Group Member
Wed 22 Jan 2020 09:37:33 PM UTC, comment #19: 

This patch appears to work:


--- a/libgnu/lseek.c    2019-12-09 08:37:43.205846694 -0500
+++ a/libgnu/lseek.c    2020-01-22 13:42:47.180859301 -0500
@@ -63,7 +63,7 @@
       return -1;
     }
 #endif
-#if _GL_WINDOWS_64_BIT_OFF_T
+#if defined _GL_WINDOWS_64_BIT_OFF_T || defined __MINGW64__
   return _lseeki64 (fd, offset, whence);
 #else
   return lseek (fd, offset, whence);



I think the issue really is that gnulib detects that mingw64 natively supports largefile support - which looks like it means it then doesn't set _GL_WINDOWS_64_BIT_OFF_T anywhere.

the fseeko in gnulib is the same - using _GL_WINDOWS_64_BIT_OFF_T to determine wther is should call skeeo. skeeko64 or _fseeki64


John Donoghue <lostbard>
Group Member
Wed 22 Jan 2020 09:27:17 PM UTC, comment #18: 

'lseek' is undefined in lseek.c. So it resolves to msvcrt!_lseek (see John's comment #9). IIUC, "offset" is implicitly cast from 'off_t' to 'long' in that case.

Markus Mützel <mmuetzel>
Group administrator
Wed 22 Jan 2020 09:12:42 PM UTC, comment #17: 

But if _FILE_OFFSET_BITS is set to 64, 'lseek' is a #define alias to 'lseek64', and 'off_t' is a #define alias to 'off64_t', which is a 'long long int' (see the include files io.h and _mingw_off_t.h).

Mike Miller <mtmiller>
Group Member
Wed 22 Jan 2020 09:08:18 PM UTC, comment #16: 

The offset argument for _lseek is of type long (see the link John provided). Contrary to what one might expect, long has 32 bit on 64 bit Windows systems, see e.g. [1].
So I think _lseeki64 must be used for files > 4 GiB no matter if Windows is 32 bit or 64 bit.

[1]: https://docs.microsoft.com/en-us/cpp/build/common-visual-cpp-64-bit-migration-issues?redirectedfrom=MSDN&view=vs-2019

Markus Mützel <mmuetzel>
Group administrator
Wed 22 Jan 2020 08:50:02 PM UTC, comment #15: 

My expectation is that 'lseek' does the right thing on a 64 bit Windows system, so the condition to call '_lseeki64' is only needed on 32 bit systems. That makes sense to me, but I don't know anything about Windows system libraries or the actual behavior.

Mike Miller <mtmiller>
Group Member
Wed 22 Jan 2020 08:36:00 PM UTC, comment #14: 

Thanks, Mike for helping me understand.
In that case, I would expect that _lseeki64 is called unconditionally in lseek.c because off_t is 64-bit in any case.

I'm still not sure I correctly understand how that should be working.

Markus Mützel <mmuetzel>
Group administrator
Wed 22 Jan 2020 08:22:42 PM UTC, comment #13: 

I haven't been following this, but the way I read the Gnulib code, it's saying that if the test program shows that 'off_t' is already 64 bits, then don't define WINDOWS_64_BIT_OFF_T. So if that value is zero, that means that 'off_t' and file I/O calls are expected to support 64-bit offsets implicitly.

Mike Miller <mtmiller>
Group Member
Wed 22 Jan 2020 08:13:28 PM UTC, comment #12: 

This check in largefile.m4 (@ line 124) seems to be inverted:

      AC_CACHE_CHECK([for 64-bit off_t], [gl_cv_type_off_t_64],
        [AC_COMPILE_IFELSE(
           [AC_LANG_PROGRAM(
              [[#include <sys/types.h>
                int verify_off_t_size[sizeof (off_t) >= 8 ? 1 : -1];
              ]],
              [[]])],
           [gl_cv_type_off_t_64=yes], [gl_cv_type_off_t_64=no])
        ])
      if test $gl_cv_type_off_t_64 = no; then
        WINDOWS_64_BIT_OFF_T=1
      else
        WINDOWS_64_BIT_OFF_T=0
      fi


I'm confused...

Markus Mützel <mmuetzel>
Group administrator
Wed 22 Jan 2020 07:59:07 PM UTC, comment #11: 

That might have been wrong. I see the following in the log for default-octave in MXE Octave:

rm -f sys/types.h-t sys/types.h && \
{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
  sed -e 's|@''GUARD_PREFIX''@|GL|g' \
      -e 's|@''INCLUDE_NEXT''@|include_next|g' \
      -e 's|@''PRAGMA_SYSTEM_HEADER''@|#pragma GCC system_header|g' \
      -e 's|@''PRAGMA_COLUMNS''@||g' \
      -e 's|@''NEXT_SYS_TYPES_H''@|<sys/types.h>|g' \
      -e 's|@''WINDOWS_64_BIT_OFF_T''@|0|g' \
      -e 's|@''WINDOWS_STAT_INODES''@|0|g' \
      < /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libgnu/sys_types.in.h; \
} > sys/types.h-t && \
mv sys/types.h-t sys/types.h


Does that mean that WINDOWS_64_BIT_OFF_T is not defined? Why is that so?

Markus Mützel <mmuetzel>
Group administrator
Wed 22 Jan 2020 07:51:20 PM UTC, comment #10: 

I haven't run anything in the debugger. But if I follow the code correctly, _GL_WINDOWS_64_BIT_OFF_T should be defined (in sys/types.h included from unistd.h) and _lseeki64 should be called in lseek.c @ line 67.

Do you see something different?

Markus Mützel <mmuetzel>
Group administrator
Wed 22 Jan 2020 05:43:04 PM UTC, comment #9: 

lseek calls seems to resolve to msvcrt!_lseek (), which I wouldnt think should be correct - I would have thought  _lseeki64 instead with the larger offset value [1].

[1] https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/lseek-lseeki64?view=vs-2019


John Donoghue <lostbard>
Group Member
Wed 22 Jan 2020 04:51:07 PM UTC, comment #8: 

octave_fseeko_wrapper gets called, which in turn calls fseeko in libgnu (fseek.c) , which in turn fails on a call to lseek (libgnu) at line 117 of that file.

lseek.c in also from libgnu rpl_lseek then calls lseek at line 69


John Donoghue <lostbard>
Group Member
Tue 21 Jan 2020 10:07:05 PM UTC, comment #7: 

I can replicate it in dev octave.


On an initial look into the code in oct-stream.cc, stream::seek(off_t offset, int origin), we try to move to the end of the file:



// Move to end of file.  If successful, find the offset of the end.
status = m_rep->seek (0, SEEK_END);


The status return at that point is -1.

I haven't dug further into it yet.



John Donoghue <lostbard>
Group Member
Tue 21 Jan 2020 05:08:21 PM UTC, comment #6: 

@Markus: yes I do have Full control access to these files.
Installed from octave-5.1.0-w64-64-installer.exe
Same issue when executed from latest archive from gnu ftp

Octave banner clearly state
Octave was configured for "x86_64-w64-mingw32"

I can fopen, fclose

>> f=fopen("M57.ser")
f =  3
>> fseek(f,12)
ans = -1
>> fclose(f)
ans = 0

Let mew know if there is anything else I an do to help.

Emmanuel Brandt <emmanuelbrandt>
Tue 21 Jan 2020 04:17:38 PM UTC, comment #5: 

Thanks, John for that info.
Please, disregard my previous comments on cygwin. That was a different bug I happened to read shortly before.

If I correctly understand the gnulib sources, fseeko should be deferring to _fseeki64 on mingw. That looks good to me.

@emmanuelbrandt: Just to make sure: You do have read access on the large file, do you?


Markus Mützel <mmuetzel>
Group administrator
Tue 21 Jan 2020 03:31:47 PM UTC, comment #4: 

Looking at octave configure output from my mxe install of 5.1.0, I see:

checking for _LARGEFILE_SOURCE value needed for large files ... no

checking for _FILE_OFFSET_BITS value needed for large files ... 64

John Donoghue <lostbard>
Group Member
Tue 21 Jan 2020 02:56:55 PM UTC, comment #3: 

File system is NTFS

Tried all latest versions including the "large data" W64_64 but they are all configured for "x86_64_64-mingw32". As they are installers I don't recall any option for 64 just an option for OpenBLAS.

Version downloaded from ftp, while launching octave-cli.5.1.0 in the mingw64 directory, curiously Octave was configured for "x86_64-mingw32"

I suspect this is the root cause issue in the adressing large files.

Emmanuel Brandt <emmanuelbrandt>
Tue 21 Jan 2020 02:45:47 PM UTC, comment #2: 

"fseek" uses the gnulib wrapper for fseeko [1]. That page references "AC_SYS_LARGEFILE":

> On platforms where off_t is a 32-bit type, fseeko does not work correctly with files larger than 2 GB. (Cf. AC_SYS_LARGEFILE.)


I don't find any reference to AC_SYS_LARGEFILE [2] in the Octave sources.
I don't know if off_t is a 32-bit type on cygwin. But maybe we could use that macro to enable fseeking in large files on affected systems.

[1]: https://www.gnu.org/software/gnulib/manual/html_node/fseeko.html
[2]: https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/System-Services.html

Markus Mützel <mmuetzel>
Group administrator
Tue 21 Jan 2020 12:11:29 PM UTC, comment #1: 

Could also be due to the file system you're reading from. Is that NTFS or FAT32?

Did you use a 32-bit Octave installer or the w64 one?

Philip Nienhuis <philipnienhuis>
Group Member
Tue 21 Jan 2020 11:15:40 AM UTC, original submission:  

Running Octave 5.1.0 on Windows 10 64-bit OS on x65-based processor

Cannot work on file larger than 4GB


>> version
ans = 5.1.0
f=fopen("small.ser")
f =  4
ᰕ >> fseek(f,0,'eof')
ans = 0
>> ftell(f)
ans =  221451138
>> fclose(f)
ans = 0
>> f=fopen('large.ser')
f =  4
>> fseek(f,178)
ans = -1
>> fclose(f)
ans = 0


Emmanuel Brandt <emmanuelbrandt>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by emmanuelbrandt (Submitted the item)
  • -email is unavailable- added by emmanuelbrandt
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-02-12 siko1056 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2020-01-26 lostbard StatusNeed Info Ready For Test
        Assigned toNone lostbard
    2020-01-21 philipnienhuis StatusNone Need Info
    2020-01-21 emmanuelbrandt Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code