bugGNU Octave - Bugs: bug #57644, fseek fread fail for files >4GB...

 
 

bug #57644: fseek fread fail for files >4GB 5.1.0 on Windows10 64-bit x64 proc

Submitted by:  Emmanuel Brandt <emmanuelbrandt>
Submitted on:  Tue 21 Jan 2020 11:15:40 AM UTC  
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  John Donoghue <lostbard>
Originator Name:  Emmanuel Brandt Open/Closed:  Closed
Release:  5.1.0 Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Wed 12 Feb 2020 08:33:41 AM UTC, comment #41: 

Thank you for reporting back.  Closing report.

Kai Torben Ohlhus <siko1056>
Project Member
Wed 12 Feb 2020 08:30:01 AM UTC, comment #40: 

Tried yesterday it works fine
With large files: 4 to 40gb
This for solving this issue ;)

Emmanuel Brandt <emmanuelbrandt>
Wed 12 Feb 2020 07:56:14 AM UTC, comment #39: 

Did anyone try out if this now works with Octave 5.2.0 in Windows?  Can this report be closed?

Kai Torben Ohlhus <siko1056>
Project Member
Sun 02 Feb 2020 10:13:12 PM UTC, comment #38: 

This should appear in the 5.2 (mxe-octave gnulib patch) as well as upcoming 6.X (updated gnulib) builds

John Donoghue <lostbard>
Project MemberIn charge of this item.
Sun 26 Jan 2020 01:06:55 PM UTC, comment #37: 

The commit to gitlib repo:
http://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=a51a5f5d24ce8e9be82c6a564576500d0f4d4428

Patched in mxe-octave

https://hg.octave.org/mxe-octave/rev/db215107402f

The patch will have to be undone in mxe-octave when octave gets another update for gnulib.

John Donoghue <lostbard>
Project MemberIn charge of this item.
Sat 25 Jan 2020 10:03:22 PM UTC, comment #36: 

gotcha... attempt #2:

Hint: use +nomarkup+ <literal text> -nomarkup- to get rid of unintended italics when typing underscores.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 25 Jan 2020 10:01:16 PM UTC, comment #35: 

Hint: use <literal text> to get rid of unintended italics when typing underscores.
I often forget that myself :-)

Philip Nienhuis <philipnienhuis>
Project Member
Sat 25 Jan 2020 09:06:42 PM UTC, comment #34: 

Actually woulkd be :

diff --git a/lib/lseek.c b/lib/lseek.c
index 013f665..b249839 100644
--- a/lib/lseek.c
+++ b/lib/lseek.c
@@ -63,7 +63,7 @@ rpl_lseek (int fd, off_t offset, int whence)
       return -1;
     }
 #endif
-#if _GL_WINDOWS_64_BIT_OFF_T
+#if GL_WINDOWS_64_BIT_OFF_T || (defined _MINGW32_ && defined
_FILE_OFFSET_BITS && (_FILE_OFFSET_BITS == 64))
   return _lseeki64 (fd, offset, whence);
 #else
   return lseek (fd, offset, whence);

(A typo correction for _MINGW32_)

John Donoghue <lostbard>
Project MemberIn charge of this item.
Sat 25 Jan 2020 08:52:11 PM UTC, comment #33: 

Posted to gnulib at https://lists.gnu.org/archive/html/bug-gnulib/2020-01/msg00146.html

its an issue with gnulib, but effects mingw/windows only.

They have posted as suggest fix already, which is somewhat similar to what I posted in comment 19.

diff --git a/lib/lseek.c b/lib/lseek.c
index 013f665..b249839 100644
--- a/lib/lseek.c
+++ b/lib/lseek.c
@@ -63,7 +63,7 @@ rpl_lseek (int fd, off_t offset, int whence)
       return -1;
     }
 #endif
-#if _GL_WINDOWS_64_BIT_OFF_T
+#if GL_WINDOWS_64_BIT_OFF_T || (defined __MINGW32_ && defined
_FILE_OFFSET_BITS && (_FILE_OFFSET_BITS == 64))
   return _lseeki64 (fd, offset, whence);
 #else
   return lseek (fd, offset, whence);

I have tried this exact fix yet to verify, but it should work.

John Donoghue <lostbard>
Project MemberIn charge of this item.
Fri 24 Jan 2020 09:54:02 AM UTC, comment #32: 

@JohnD going through the discussion thread, I'm still not sure to understand the root cause at this point:

  • Is this with gnulib/win?
  • Did you managed to make it work on large files?
  • Any temp work around I may use?
  • Use me for testing if you need at some point

PS. As a side note and historical background: I faced this issue as I routinely work on very large video astronomical files [4-90GB]. Never had any issue on Linux (Ubuntu) and then running same script on Octave/Windows it failed...
Thanks to all for the effort, really appreciated ;)

Emmanuel Brandt <emmanuelbrandt>
Fri 24 Jan 2020 02:08:04 AM UTC, comment #31: 

Looking more, of why its suddently now working ok - the dependancies for gnulib largefile seems far larger when generating the gnulib files in linux to those when generated in msys2.

I guess I'll have to add a quiestion taht as well.

John Donoghue <lostbard>
Project MemberIn charge of this item.
Thu 23 Jan 2020 10:05:14 PM UTC, comment #30: 

And of course suddenly it is working in msys2 without any changes that I can think of.

Only thing I see different is that for some reason it doesnt do the check for fseeki64 and doesnt attempt to use gnulibs fseek , fseeko modules - descibe not actual changes except to regenerate the m4/configure files.

John Donoghue <lostbard>
Project MemberIn charge of this item.
Thu 23 Jan 2020 04:03:06 PM UTC, comment #29: 

Yeah am slowing working on something to send up to them for discussion.

I also tried compiling in msys2 using the i686-w64-mingw32 compiler - it does the same fail.

John Donoghue <lostbard>
Project MemberIn charge of this item.
Thu 23 Jan 2020 03:52:17 PM UTC, comment #28: 

With all that data, I'm pretty sure this is an upstream bug in gnulib.
@John: You have done more debugging here. Could you please ask upstream? Is -email is unavailable- the correct mailing list?

Markus Mützel <mmuetzel>
Project Member
Thu 23 Jan 2020 03:38:18 PM UTC, comment #27: 

I created a simple barebones app using gnulib and that does a fseek to end and a ftell and compiled in a actual msys2 x86_64 install, and it fails at the seek to end just like we are seeing.

John Donoghue <lostbard>
Project MemberIn charge of this item.
Thu 23 Jan 2020 08:18:29 AM UTC, comment #26: 

Maybe files >4 GiB are rare and no one noticed yet. But I also think that this is somewhat unlikely.

MS discourages calling 'lseek' at all. [1]

As I understand it, I'd guess the following change might be correct:

diff -r c34f677e6117 lib/lseek.c
--- a/lib/lseek.c        Sun Mar 17 19:27:20 2019 +0100
+++ b/lib/lseek.c        Thu Jan 23 09:12:26 2020 +0100
@@ -52,6 +52,7 @@
       errno = ESPIPE;
       return -1;
     }
+  return _lseeki64 (fd, offset, whence);
#else
   /* BeOS lseek mistakenly succeeds on pipes...  */
   struct stat statbuf;
@@ -62,10 +63,6 @@
       errno = ESPIPE;
       return -1;
     }
-#endif
-#if _GL_WINDOWS_64_BIT_OFF_T
-  return _lseeki64 (fd, offset, whence);
-#else
   return lseek (fd, offset, whence);
#endif
}

Gnulib makes sure with its #define macros that off_t is always 64 bit wide. So it should be save to always call "_lseeki64".
But I might be missing something.

[1]: https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/lseek?redirectedfrom=MSDN&view=vs-2019

Markus Mützel <mmuetzel>
Project Member
Wed 22 Jan 2020 10:11:38 PM UTC, comment #25: 

Yeah, like I said, the Gnulib code logic and code looks right to me, my guess is it's something wrong in the way Octave is configuring or compiling itself for MinGW.

Mike Miller <mtmiller>
Project Administrator
Wed 22 Jan 2020 10:09:37 PM UTC, comment #24: 

It seems odd that we would be the first to find this problem.  Are we doing something wrong with our use of gnulib here?  If it looks like a gnulib bug, then can we please report it and fix the problem there?

John W. Eaton <jwe>
Project Administrator
Wed 22 Jan 2020 10:05:26 PM UTC, comment #23: 

I'm bowing out of this report again. Just trying to explain why I think the Gnulib logic and lseek.c source code looks correct to me, and what I think the intended behavior is.

Mike Miller <mtmiller>
Project Administrator
Wed 22 Jan 2020 10:01:02 PM UTC, comment #22: 

_GL_WINDOWS_64_BIT_OFF_T gets set in libgnu/sys_types.in.h

/* Override off_t if Large File Support is requested on native Windows.  */
#if @WINDOWS_64_BIT_OFF_T@
/* Same as int64_t in <stdint.h>.  */
# if defined _MSC_VER
#  define off_t __int64
# else
#  define off_t long long int
# endif
/* Indicator, for gnulib internal purposes.  */
# define _GL_WINDOWS_64_BIT_OFF_T 1
#endif

See largefile.m4 in comment #12 for WINDOWS_64_BIT_OFF_T

John Donoghue <lostbard>
Project MemberIn charge of this item.
Wed 22 Jan 2020 09:42:05 PM UTC, comment #21: 

And for comment #17, since it is is resolving to  use lseek.c which in turn calls the windows _lseek, it is bypassing the use of off_t ect

John Donoghue <lostbard>
Project MemberIn charge of this item.
Wed 22 Jan 2020 09:38:39 PM UTC, comment #20: 

btw the patch in comment #19 is really this makes it work rather than gnulib should be changing to do it that way :)

John Donoghue <lostbard>
Project MemberIn charge of this item.
Wed 22 Jan 2020 09:37:33 PM UTC, comment #19: 

This patch appears to work:

--- a/libgnu/lseek.c    2019-12-09 08:37:43.205846694 -0500
+++ a/libgnu/lseek.c    2020-01-22 13:42:47.180859301 -0500
@@ -63,7 +63,7 @@
       return -1;
     }
#endif
-#if _GL_WINDOWS_64_BIT_OFF_T
+#if defined _GL_WINDOWS_64_BIT_OFF_T || defined __MINGW64__
   return _lseeki64 (fd, offset, whence);
#else
   return lseek (fd, offset, whence);

I think the issue really is that gnulib detects that mingw64 natively supports largefile support - which looks like it means it then doesn't set _GL_WINDOWS_64_BIT_OFF_T anywhere.

the fseeko in gnulib is the same - using _GL_WINDOWS_64_BIT_OFF_T to determine wther is should call skeeo. skeeko64 or _fseeki64

John Donoghue <lostbard>
Project MemberIn charge of this item.
Wed 22 Jan 2020 09:27:17 PM UTC, comment #18: 

'lseek' is undefined in lseek.c. So it resolves to msvcrt!_lseek (see John's comment #9). IIUC, "offset" is implicitly cast from 'off_t' to 'long' in that case.

Markus Mützel <mmuetzel>
Project Member
Wed 22 Jan 2020 09:12:42 PM UTC, comment #17: 

But if _FILE_OFFSET_BITS is set to 64, 'lseek' is a #define alias to 'lseek64', and 'off_t' is a #define alias to 'off64_t', which is a 'long long int' (see the include files io.h and _mingw_off_t.h).

Mike Miller <mtmiller>
Project Administrator
Wed 22 Jan 2020 09:08:18 PM UTC, comment #16: 

The offset argument for _lseek is of type long (see the link John provided). Contrary to what one might expect, long has 32 bit on 64 bit Windows systems, see e.g. [1].
So I think _lseeki64 must be used for files > 4 GiB no matter if Windows is 32 bit or 64 bit.

[1]: https://docs.microsoft.com/en-us/cpp/build/common-visual-cpp-64-bit-migration-issues?redirectedfrom=MSDN&view=vs-2019

Markus Mützel <mmuetzel>
Project Member
Wed 22 Jan 2020 08:50:02 PM UTC, comment #15: 

My expectation is that 'lseek' does the right thing on a 64 bit Windows system, so the condition to call '_lseeki64' is only needed on 32 bit systems. That makes sense to me, but I don't know anything about Windows system libraries or the actual behavior.

Mike Miller <mtmiller>
Project Administrator
Wed 22 Jan 2020 08:36:00 PM UTC, comment #14: 

Thanks, Mike for helping me understand.
In that case, I would expect that _lseeki64 is called unconditionally in lseek.c because off_t is 64-bit in any case.

I'm still not sure I correctly understand how that should be working.

Markus Mützel <mmuetzel>
Project Member
Wed 22 Jan 2020 08:22:42 PM UTC, comment #13: 

I haven't been following this, but the way I read the Gnulib code, it's saying that if the test program shows that 'off_t' is already 64 bits, then don't define WINDOWS_64_BIT_OFF_T. So if that value is zero, that means that 'off_t' and file I/O calls are expected to support 64-bit offsets implicitly.

Mike Miller <mtmiller>
Project Administrator
Wed 22 Jan 2020 08:13:28 PM UTC, comment #12: 

This check in largefile.m4 (@ line 124) seems to be inverted:

      AC_CACHE_CHECK([for 64-bit off_t], [gl_cv_type_off_t_64],
        [AC_COMPILE_IFELSE(
           [AC_LANG_PROGRAM(
              [[#include <sys/types.h>
                int verify_off_t_size[sizeof (off_t) >= 8 ? 1 : -1];
              ]],
              [[]])],
           [gl_cv_type_off_t_64=yes], [gl_cv_type_off_t_64=no])
        ])
      if test $gl_cv_type_off_t_64 = no; then
        WINDOWS_64_BIT_OFF_T=1
      else
        WINDOWS_64_BIT_OFF_T=0
      fi

I'm confused...

Markus Mützel <mmuetzel>
Project Member
Wed 22 Jan 2020 07:59:07 PM UTC, comment #11: 

That might have been wrong. I see the following in the log for default-octave in MXE Octave:

rm -f sys/types.h-t sys/types.h && \
{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
  sed -e 's|@''GUARD_PREFIX''@|GL|g' \
      -e 's|@''INCLUDE_NEXT''@|include_next|g' \
      -e 's|@''PRAGMA_SYSTEM_HEADER''@|#pragma GCC system_header|g' \
      -e 's|@''PRAGMA_COLUMNS''@||g' \
      -e 's|@''NEXT_SYS_TYPES_H''@|<sys/types.h>|g' \
      -e 's|@''WINDOWS_64_BIT_OFF_T''@|0|g' \
      -e 's|@''WINDOWS_STAT_INODES''@|0|g' \
      < /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libgnu/sys_types.in.h; \
} > sys/types.h-t && \
mv sys/types.h-t sys/types.h

Does that mean that WINDOWS_64_BIT_OFF_T is not defined? Why is that so?

Markus Mützel <mmuetzel>
Project Member
Wed 22 Jan 2020 07:51:20 PM UTC, comment #10: 

I haven't run anything in the debugger. But if I follow the code correctly, _GL_WINDOWS_64_BIT_OFF_T should be defined (in sys/types.h included from unistd.h) and _lseeki64 should be called in lseek.c @ line 67.

Do you see something different?

Markus Mützel <mmuetzel>
Project Member
Wed 22 Jan 2020 05:43:04 PM UTC, comment #9: 

lseek calls seems to resolve to msvcrt!_lseek (), which I wouldnt think should be correct - I would have thought  _lseeki64 instead with the larger offset value [1].

[1] https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/lseek-lseeki64?view=vs-2019

John Donoghue <lostbard>
Project MemberIn charge of this item.
Wed 22 Jan 2020 04:51:07 PM UTC, comment #8: 

octave_fseeko_wrapper gets called, which in turn calls fseeko in libgnu (fseek.c) , which in turn fails on a call to lseek (libgnu) at line 117 of that file.

lseek.c in also from libgnu rpl_lseek then calls lseek at line 69

John Donoghue <lostbard>
Project MemberIn charge of this item.
Tue 21 Jan 2020 10:07:05 PM UTC, comment #7: 

I can replicate it in dev octave.

On an initial look into the code in oct-stream.cc, stream::seek(off_t offset, int origin), we try to move to the end of the file:

// Move to end of file.  If successful, find the offset of the end.
status = m_rep->seek (0, SEEK_END);

The status return at that point is -1.

I haven't dug further into it yet.

John Donoghue <lostbard>
Project MemberIn charge of this item.
Tue 21 Jan 2020 05:08:21 PM UTC, comment #6: 

@Markus: yes I do have Full control access to these files.
Installed from octave-5.1.0-w64-64-installer.exe
Same issue when executed from latest archive from gnu ftp

Octave banner clearly state
Octave was configured for "x86_64-w64-mingw32"

I can fopen, fclose

>> f=fopen("M57.ser")
f =  3
>> fseek(f,12)
ans = -1
>> fclose(f)
ans = 0

Let mew know if there is anything else I an do to help.

Emmanuel Brandt <emmanuelbrandt>
Tue 21 Jan 2020 04:17:38 PM UTC, comment #5: 

Thanks, John for that info.
Please, disregard my previous comments on cygwin. That was a different bug I happened to read shortly before.

If I correctly understand the gnulib sources, fseeko should be deferring to _fseeki64 on mingw. That looks good to me.

@emmanuelbrandt: Just to make sure: You do have read access on the large file, do you?

Markus Mützel <mmuetzel>
Project Member
Tue 21 Jan 2020 03:31:47 PM UTC, comment #4: 

Looking at octave configure output from my mxe install of 5.1.0, I see:

checking for _LARGEFILE_SOURCE value needed for large files ... no

checking for _FILE_OFFSET_BITS value needed for large files ... 64

John Donoghue <lostbard>
Project MemberIn charge of this item.
Tue 21 Jan 2020 02:56:55 PM UTC, comment #3: 

File system is NTFS

Tried all latest versions including the "large data" W64_64 but they are all configured for "x86_64_64-mingw32". As they are installers I don't recall any option for 64 just an option for OpenBLAS.

Version downloaded from ftp, while launching octave-cli.5.1.0 in the mingw64 directory, curiously Octave was configured for "x86_64-mingw32"

I suspect this is the root cause issue in the adressing large files.

Emmanuel Brandt <emmanuelbrandt>
Tue 21 Jan 2020 02:45:47 PM UTC, comment #2: 

"fseek" uses the gnulib wrapper for fseeko [1]. That page references "AC_SYS_LARGEFILE":

> On platforms where off_t is a 32-bit type, fseeko does not work correctly with files larger than 2 GB. (Cf. AC_SYS_LARGEFILE.)


I don't find any reference to AC_SYS_LARGEFILE [2] in the Octave sources.
I don't know if off_t is a 32-bit type on cygwin. But maybe we could use that macro to enable fseeking in large files on affected systems.

[1]: https://www.gnu.org/software/gnulib/manual/html_node/fseeko.html
[2]: https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/System-Services.html

Markus Mützel <mmuetzel>
Project Member
Tue 21 Jan 2020 12:11:29 PM UTC, comment #1: 

Could also be due to the file system you're reading from. Is that NTFS or FAT32?

Did you use a 32-bit Octave installer or the w64 one?

Philip Nienhuis <philipnienhuis>
Project Member
Tue 21 Jan 2020 11:15:40 AM UTC, original submission:  

Running Octave 5.1.0 on Windows 10 64-bit OS on x65-based processor

Cannot work on file larger than 4GB

>> version
ans = 5.1.0
f=fopen("small.ser")
f =  4
ᰕ >> fseek(f,0,'eof')
ans = 0
>> ftell(f)
ans =  221451138
>> fclose(f)
ans = 0
>> f=fopen('large.ser')
f =  4
>> fseek(f,178)
ans = -1
>> fclose(f)
ans = 0

Emmanuel Brandt <emmanuelbrandt>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by emmanuelbrandt (Submitted the item)
  • -email is unavailable- added by emmanuelbrandt
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-02-12 siko1056 StatusReady For Test => Fixed
        Open/ClosedOpen => Closed
    2020-01-26 lostbard StatusNeed Info => Ready For Test
        Assigned toNone => lostbard
    2020-01-21 philipnienhuis StatusNone => Need Info
    2020-01-21 emmanuelbrandt Carbon-Copy- => Added emmanuel.brandt@orange.fr

    Back to the top


    Powered by Savane 3.5