GNU Octave - Bugs: bug #64933, Stat sometimes produces wrong...

Group

Main
- Main
- View members
- Search
Homepage
Download
Mailing lists
Source code
Bugs
- Submit new
- Browse
- Reset to open
- Digest
- Dependencies
- ~~Export~~
- Get statistics
- Search
Tasks
- ~~Submit new~~
- Browse
- Reset to open
- Digest
- Dependencies
- ~~Export~~
- Get statistics
- Search
Patches
- Submit new
- Browse
- Reset to open
- Digest
- Dependencies
- ~~Export~~
- Get statistics
- Search

bug #64933: Stat sometimes produces wrong timestamp

Submitter:	mspo
Submitted:	Mon 27 Nov 2023 09:08:49 AM UTC


Category:	Octave Function	Severity:	3 - Normal
Priority:	5 - Normal	Item Group:	None
Status:	Need Info	Assigned to:	None
Originator Name:		Open/Closed: *	Open
Release: *	8.4.0	Operating System: *	Microsoft Windows
Fixed Release:	None	Planned Release:	None

* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Tue 28 Nov 2023 05:57:52 PM UTC, comment #17: Another possibility is the following change diff --git a/liboctave/wrappers/stat-wrappers.c b/liboctave/wrappers/stat-wrapp> --- a/liboctave/wrappers/stat-wrappers.c +++ b/liboctave/wrappers/stat-wrappers.c @@ -164,7 +164,12 @@ octave_fstat_wrapper (int fid, mode_t *m { struct stat buf; +#if defined (OCTAVE_USE_WINDOWS_API) + // For consistency with stat and lstat behavior. See bug #64933. + int status = _fstati64 (fid, &buf); +#else int status = fstat (fid, &buf); +#endif assign_stat_fields (&buf, mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime, rdev, blksize, blocks); I expect this change would at least make the behavior of stat ("filename") stat (fopen ("filename")) consistent.	John W. Eaton <jwe>
Tue 28 Nov 2023 05:02:32 PM UTC, comment #16: I wasn't aware of that. But apparently Microsoft deliberately chose that behavior to have an invertible conversion from local time to "UTC" time. See, e.g.: https://www.codeproject.com/Articles/1144/Beating-the-Daylight-Savings-Time-Bug-and-Getting	Markus Mützel <mmuetzel>
Tue 28 Nov 2023 04:56:23 PM UTC, comment #15: The important thing here is that for a single file (however/whenever it was created), the {a,c,m}time elements of the struct returned from the two calls stat ("filename") stat (fopen ("filename")) are different and they should not be. There should be only one number of seconds since the epoch that this file was created, regardless of time zone or DST, correct? And if I understand correctly, the "stat (filename)" call is the one that gets it wrong and that is the one that ultimately calls _wstati64. What happens if we make the following change? Does the gnulib replacement for stat return the correct result? diff --git a/liboctave/wrappers/stat-wrappers.c b/liboctave/wrappers/stat-wrapp> --- a/liboctave/wrappers/stat-wrappers.c +++ b/liboctave/wrappers/stat-wrappers.c @@ -117,13 +117,7 @@ octave_stat_wrapper (const char fname, { struct stat buf; -#if defined (OCTAVE_USE_WINDOWS_API) - wchar_t wfname = u8_to_wchar (fname); - int status = _wstati64 (wfname, &buf); - free ((void ) wfname); -#else int status = stat (fname, &buf); -#endif assign_stat_fields (&buf, mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime, rdev, blksize, blocks); @@ -140,14 +134,7 @@ octave_lstat_wrapper (const char lname, { struct stat buf; -#if defined (OCTAVE_USE_WINDOWS_API) - // Windows doesn't have an lstat. Use stat instead - wchar_t wlname = u8_to_wchar (lname); - int status = _wstati64 (wlname, &buf); - free ((void ) wlname); -#else int status = lstat (lname, &buf); -#endif assign_stat_fields (&buf, mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime, rdev, blksize, blocks); I understand that the calls to _wstati64 were made in http://hg.savannah.gnu.org/hgweb/octave/rev/7fb40efda31f to support wide characters in filenames on Windows to solve bug #49118. How can we solve both problems? It's surprising to me that _wstati64 has this behavior. Is there something I don't understand about why it works this way? Oh, so now I see this thread that provides some background about why gnulib has a replacement for stat: https://lists.gnu.org/r/bug-gnulib/2017-04/msg00126.html and also has some discussion about the crazy DST behavior. I also found this post about what appears to be the same issue: https://www.nu42.com/2015/03/stat-vs-fstat-msvcrt-windows.html	John W. Eaton <jwe>
Tue 28 Nov 2023 04:53:07 PM UTC, comment #14: comment #13: > Why is the "seconds since 1970" changing but not the time zone? Where is time zone in output? Exactly. If mtime measures seconds since a point of time in UTC, the number should not change whatever the local time zone is set to or whatever daylight savings time shifts are applied or not applied. In order to accommodate for that, we need a function like `localtime()`.	mspo
Tue 28 Nov 2023 03:59:33 PM UTC, comment #13: Why is the "seconds since 1970" changing but not the time zone? Where is time zone in output?	Anonymous
Tue 28 Nov 2023 01:53:26 PM UTC, comment #12: I see. Thank you. Then I would say that's exactly the behavior I see.	mspo
Tue 28 Nov 2023 01:47:37 PM UTC, comment #11: my local is USA,so those times correspond with the dst shift here. see https://en.wikipedia.org/wiki/Daylight_saving_time_by_country in both cases I created the new file in the time before the change then checked the stat results before & after the change. In hindsight, though, as i was choosing times just before the change so that I could quickly do before/after, the november change occurs during the time where you roll back and repeat an hour and that might have obscured the issue. here i changed time back to the summer, created an new file, then returned to present time. You can see that there is a discrepancy with the stat(<filename>) return for after returning to normal time. at time of file creation (July 1 2023): >> time ans = 1688215020.935662 >> stat('testfilesummer.txt') ans = scalar structure containing the fields: dev = 2 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 2 size = 0 atime = 1688214989 mtime = 1688214989 ctime = 1688214989 blksize = NaN blocks = NaN >> stat(fopen('testfilesummer.txt')) ans = scalar structure containing the fields: dev = 0 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 0 size = 0 atime = 1688214989 mtime = 1688214989 ctime = 1688214989 blksize = NaN blocks = NaN after return to correct time (nov 28 2023) >> time ans = 1701178973.715555 >> stat('testfilesummer.txt') ans = scalar structure containing the fields: dev = 2 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 2 size = 0 atime = 1688211389 mtime = 1688211389 ctime = 1688211389 blksize = NaN blocks = NaN >> stat(fopen('testfilesummer.txt')) ans = scalar structure containing the fields: dev = 0 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 0 size = 0 atime = 1688214989 mtime = 1688214989 ctime = 1688214989 blksize = NaN blocks = NaN	Nicholas Jankowski <nrjank>
Tue 28 Nov 2023 08:24:13 AM UTC, comment #10: Thank you for working on this. Concerning comment #5, I checked the output of stat and realized that the offset already happens there. The strftime(..., localtime()) call is just for better readability. Concerning comment #9, thank you for testing this. I cannot change this setting on the PC here by myself. I can test it later on a different one. However, I'm wondering why you chose the dates that you did. Both March 12th and November 5th are in standard time, at least here in Germany. Could you try this with a file "saved" in summer - say August - and then switch back to the current time and read out the mtime value?	mspo
Mon 27 Nov 2023 10:22:24 PM UTC, comment #9: don't know if this was useful, but assuming it will work for any file created during the previous dst period: I set the time locally to march 12 2023 01:59 and verified it jumps forward to 03:00. before the time change, created a file testfile.txt: >> time ans = 1678604367.157886 >> stat('testfile.txt') ans = scalar structure containing the fields: dev = 2 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 2 size = 0 atime = 1678604356 mtime = 1678604356 ctime = 1678604397 blksize = NaN blocks = NaN >> stat(fopen('testfile.txt')) ans = scalar structure containing the fields: dev = 0 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 0 size = 0 atime = 1678604356 mtime = 1678604356 ctime = 1678604397 blksize = NaN blocks = NaN >> fclose all after the time change: >> time ans = 1678604408.970258 >> stat('testfile.txt') ans = scalar structure containing the fields: dev = 2 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 2 size = 0 atime = 1678604356 mtime = 1678604356 ctime = 1678604397 blksize = NaN blocks = NaN >> stat(fopen('testfile.txt')) ans = scalar structure containing the fields: dev = 0 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 0 size = 0 atime = 1678604356 mtime = 1678604356 ctime = 1678604397 blksize = NaN blocks = NaN doing the same for Nov 5 before time change, created a new file testfile2.txt: >> time ans = 1699164008.989853 >> stat('testfile2.txt') ans = scalar structure containing the fields: dev = 2 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 2 size = 0 atime = 1699164003 mtime = 1699164003 ctime = 1699164003 blksize = NaN blocks = NaN >> stat(fopen('testfile2.txt')) ans = scalar structure containing the fields: dev = 0 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 0 size = 0 atime = 1699164003 mtime = 1699164003 ctime = 1699164003 blksize = NaN blocks = NaN after time change >> time ans = 1699164025.525178 >> stat('testfile2.txt') ans = scalar structure containing the fields: dev = 2 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 2 size = 0 atime = 1699164003 mtime = 1699164003 ctime = 1699164003 blksize = NaN blocks = NaN >> stat(fopen('testfile2.txt')) ans = scalar structure containing the fields: dev = 0 ino = 0 mode = 33206 modestr = -rw-rw-rw- nlink = 1 uid = 0 gid = 0 rdev = 0 size = 0 atime = 1699164003 mtime = 1699164003 ctime = 1699164003 blksize = NaN blocks = NaN >> fclose all so, no difference seen here. Octave 8.4.0, windows 10.	Nicholas Jankowski <nrjank>
Mon 27 Nov 2023 09:35:42 PM UTC, comment #8: I assume I could take my win10 computer off of automatic time syncing, set the time to just before a DST change, and ensure that the change still happens despite being off the auto-update. what commands run before and after the change would help answer these questions?	Nicholas Jankowski <nrjank>
Mon 27 Nov 2023 06:52:27 PM UTC, comment #7: I'm not sure which one is right. It could be that the following is happening: - That powershell command sets the modification time with the current time zone. - The local time zone of the user changed by one hour (presumably from UTC+2 to UTC+1) between the date that they set and today. That means 12:00 UTC+1 is 11:00 UTC today, but 12:00 UTC+2 was 10:00 UTC back then. - The mtime in the stat structure is taken since an EPOCH (and doesn't care about time zones). `localtime` converts it to the user's current time zone. So, the mtime appears to be one hour off (early). If that is the case, stat(fid) might be wrong. But it's probably also possible that something else is going wrong...	Markus Mützel <mmuetzel>
Mon 27 Nov 2023 06:28:27 PM UTC, comment #6: I wrote: "We can only start to debug this problem if we know what that difference is." but I decided to look for differences in what happens for stat(name) and stat(fid) on Windows systems anyway. In the definition of the stat function in libinterp/corefcn/syscalls.cc, I see if (args(0).is_scalar_type ()) { stream_list& streams = interp.get_stream_list (); int fid = streams.get_file_number (args(0)); sys::file_fstat fs (fid); retval = mk_stat_result (fs); } else { std::string fname = args(0).xstring_value ("stat: NAME must be a string"); sys::file_stat fs (fname); retval = mk_stat_result (fs); } In liboctave/system/file-stat.cc, I see that fstat calls octave_fstat_wrapper and stat calls either octave_stat_wrapper or octave_lstat_wrapper. Those functions are defined in liboctave/wrappers/stat-wrappers.c. For octave_fstat_wrapper, we call the gnulib replacement for fstat. On Windows systems, that apparently calls the function _gl_fstat_by_handle that is defined in gnulib/lib/stat-w32.c and that function calls one or more Windows functions (check out the sources for that function for the details, it's complicated as it is trying to cope with changes in the Windows API and provide as much info as it can to fill the stat structure). For the octave_stat_wrapper and octave_lstat_wrapper functions, I see #if defined (OCTAVE_USE_WINDOWS_API) wchar_t wfname = u8_to_wchar (fname); int status = _wstati64 (wfname, &buf); free ((void ) wfname); #else int status = stat (fname, &buf); #endif and #if defined (OCTAVE_USE_WINDOWS_API) // Windows doesn't have an lstat. Use stat instead wchar_t wlname = u8_to_wchar (lname); int status = _wstati64 (wlname, &buf); free ((void ) wlname); #else int status = lstat (lname, &buf); #endif so for those functions, we are skipping the gnulib wrapper completely on Windows systems and assuming that the Windows _wstati64 function does the right thing. If I understand this bug report correctly, _wstati64 is not doing the right thing, but the gnulib wrapper for fstat is? Or do I have that backward?	John W. Eaton <jwe>
Mon 27 Nov 2023 05:03:38 PM UTC, comment #5: Looking only at the output of localtime doesn't help here. What are the structures returned from stat in each case (all the fields)? I'm surprised that there would be a difference between the following two calls, but maybe there is a reason? stat ("some-file") stat (fopen ("some-file")) We can only start to debug this problem if we know what that difference is.	John W. Eaton <jwe>
Mon 27 Nov 2023 04:31:58 PM UTC, comment #4: To be honest, I'm not sure how Windows handles timestamps. However, after changing the timestamp with the posted powershell command, the timestamp shown in the Windows File Explorer is the same as the one I handed over as a string. Therefore, the mtime value consistent with that string and the Windows timestamp is the one with the fid. So I only know that the powershell command is coherent with the system's handling of timestamps - as I would expect. However, I'm not sure if there is any time zone conversion happening in the background.	mspo
Mon 27 Nov 2023 04:14:40 PM UTC, comment #3: Agreed. So, in your example, does `(Get-Item \\"%s\\").LastWriteTime` set the time in the current time zone? Or does it set it as a "universal" time? Same for `Get-Date` on the right hand side: Current time zone or "universal" time? Which one is correct `stat` with filename? Or `stat` with file id?	Markus Mützel <mmuetzel>
Mon 27 Nov 2023 02:47:21 PM UTC, comment #2: Sticking to the Octave documentation, one can note two things: 1) The `stat()` output `mtime` should return the time in the same form as the `time()` function. 2) The `time` function should "Return the current time as the number of seconds since the epoch. The epoch is referenced to 00:00:00 UTC (Coordinated Universal Time) 1 Jan 1970." Since the reference is in UTC, the number of seconds returned as mtime should not change when the local time zone changes or when there is a switch DST <-> standard time.	mspo
Mon 27 Nov 2023 02:19:26 PM UTC, comment #1: What is the correct behavior of stat in that case? I can imagine that some (long-time) measurement that uses mtime as a time stamp would not want to get a sudden jump of 1 hour (or something else) just because the time zone changed on the system that modified the file. In that case, the behavior of `stat` like you describe it for filenames might be preferred. But there might also be other cases where the local time is preferred over a more "universal" time... It's definitely odd and unexpected that `stat` returns different timestamps when passing file names or file ids. Afaict, `stat` is an Octave-only function. So, Matlab-compatibility is no concern. But cross-platform behavior might be: What is the behavior on Linux?	Markus Mützel <mmuetzel>
Mon 27 Nov 2023 09:08:49 AM UTC, original submission: I'll simply copy from my post on octave.discourse.group: https://octave.discourse.group/t/stat-sometimes-produces-wrong-timestamp/4965 I’m using Windows in Germany. We just switched back to standard time this weekend and I noticed a timestamp issue: Calling stat() directly on a file path which points to files written during daylight savings time, e.g., last week, produces timestamps which are off by 1 hour. Files saved during standard time produce correct results currently. I assume that the opposite was true last week, but that’s harder to test. The issue is due to stat() itself and not localtime() or strftime() since the stat() result structure is already off by 1 hour. I finally found a workaround: Calling stat() on a file ID doesn’t show this problem. This can be reproduced under Windows using the following: % make an empty text file fpath = 'test.txt'; fid = fopen(fpath, 'w'); fclose(fid); % use powershell command to change mtime to last week system(sprintf('powershell (Get-Item \\"%s\\").LastWriteTime = Get-Date (\\"%s\\")', fpath, '2023-10-27 12:00:00')); % retrieve mtime directly from file path t = stat(fpath); strftime('%Y-%m-%d %T', localtime(t.mtime)) % retrieve mtime from fid fid = fopen(fpath); t = stat(fid); fclose(fid); strftime('%Y-%m-%d %T', localtime(t.mtime)) This gives me: ans = 2023-10-27 11:00:00 ans = 2023-10-27 12:00:00 I hope that this is unintended and can be fixed. By the way, this issue doesn’t seem to appear on Ubuntu. My system: OS: Windows 10 (version 22H2) Octave version: Version 8.4.0	mspo

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:


Comment:

No files currently attached

Depends on the following items: None found

Items that depend on this one: None found

Carbon-Copy List

-email is unavailable- added by nrjank (Posted a comment)

-email is unavailable- added by jwe (Posted a comment)

-email is unavailable- added by mmuetzel (Posted a comment)

-email is unavailable- added by mspo (Submitted the item)

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

Only group members can vote.

Please enter the title of George Orwell's famous dystopian book (it's a date):

Follows 1 latest change.

Date	Changed by	Updated Field	Previous Value	=>	Replaced by
2023-11-27	mmuetzel	Status	None		Need Info

GNU Octave - Bugs: bug #64933, Stat sometimes produces wrong...

bug #64933: Stat sometimes produces wrong timestamp

<a href="#postcomment">Post a Comment</a>

<a href="#discussion">Discussion</a>

<a href="#attached">Attached Files</a>

<a href="#dependencies">Dependencies</a>

<a href="#cc">Mail Notification Carbon-Copy List</a>

<a href="#votes">Votes</a>

<a href="#history">History</a>