bugGNU Octave - Bugs: bug #49203, File visible in File Browser, but...

 
 

bug #49203: File visible in File Browser, but interpreter can't see it right away when on network drive

Submitter:  Nicholas Jankowski <nrjank>
Submitted:  Tue 27 Sep 2016 10:44:36 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Duplicate Assigned to:  None
Originator Name:  Nicholas Jankowski Open/Closed:  * Closed
Release:  * 4.0.3 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 23 Feb 2019 11:43:58 AM UTC, comment #21: 

Closing this bug.
If this should re-appear in the future. Feel free to re-open or file a new bug report.

Markus Mützel <mmuetzel>
Group administrator
Fri 22 Feb 2019 08:54:55 PM UTC, comment #20: 

as noted over in bug 31080, unable to recreate either that issue or the example here given in Comment #1 using Octave 5.0.91

Nicholas Jankowski <nrjank>
Group Member
Wed 16 Nov 2016 05:09:40 PM UTC, comment #19: 

Added a dependency on 31080 and marking as a duplicate.  When it is solved, both bugs can be closed.

Rik <rik5>
Group administrator
Wed 16 Nov 2016 04:57:13 PM UTC, comment #18: 

just noticed bug #31080 reported in 2010, with activity as recent as June that seems to cover this bug pretty well.

https://savannah.gnu.org/bugs/?31080

Nicholas Jankowski <nrjank>
Group Member
Mon 31 Oct 2016 09:50:01 PM UTC, comment #17: 

The rehash() function forces modified functions to be reloaded, but only (if I'm reading the code correctly) by updating the internal time stamp that represents when the interactive prompt was last shown. IOW, rehash is the same as pressing Enter in an interactive session, and only forces a check (in a script for example) for updated functions that have already been seen whose timestamp may now be newer than the last time they were loaded into memory.

As far as workarounds go, one thing that ought to work would be to remove the directory from the load path and re-add it to the load path. When the directory is just "current directory", this could be as simple as "cd \somewhere\else" and "cd \back\to\my\dir". I think that should force the directory to be scanned as if it had never been seen before. Can someone test that this does in fact allow new function files to be seen when they are not seen after creating them with the editor?

Mike Miller <mtmiller>
Group Member
Mon 31 Oct 2016 08:36:20 PM UTC, comment #16: 

my Google-fu seems to inform me that this could be stemming from a number of possibilities.

"Time stamps are updated at various times and for various reasons. The only guarantee about a file time stamp is that the file time is correctly reflected when the handle that makes the change is closed." [1]

how might this relate to files created in the editor? Might it explain why he has to close and reopen Octave for it to look for the new file?

Also:
"Explorer gets is information from NTFS, by using a cmd prompt and “dir” we found that the NTFS metadata for the files is not updated until the handle to a file is closed.

Refreshing the information of a FOLDER is just going to go to the (memory resident) metadata cached by NTFS, but querying the file explicitly will force disk I/O to get the properties – this was a design change introduced in Vista to reduce unnecessary disk I/O to improve performance" [2]

(that last is from a 2010 post getting comments for years...)

Seems that Windows Server, and or Windows OS's have options that could render use of 'Date Modified' as unreliable for function checking (if disk I/O optimization is the reason, I can see why this only really surfaced for network drives).

If this is the case, then this would appear to require significant effort. are there any workarounds or alternatives that jump to the mind of those familiar with Octave's inner workings?  (Workarounds would be nice if this is going to be relegated to "WTF Windows... won't fix")


[1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).aspx

[2] https://blogs.technet.microsoft.com/asiasupp/2010/12/14/file-date-modified-property-are-not-updating-while-modifying-a-file-without-closing-it/

Nicholas Jankowski <nrjank>
Group Member
Mon 31 Oct 2016 08:23:31 PM UTC, comment #15: 

I just tried the rehash command; it did not fix the issue.

Bradley Slagle <bslagle>
Mon 31 Oct 2016 07:59:14 PM UTC, comment #14: 

huh... I was expecting 1:28pm to match the mtime report.

A bit of googling does seem to indicate that this could just be a Windows quirk. There doesn't seem to be a standard for NTFS on what should and shouldn't cause a date modified update to a folder (new file, changed file contents, should it always just match newest file in folder, etc.)

I assume the answer is yes, but can a Linux user chime in on whether there is an equivalent specification that is more clear?

If the issue is that Octave wouldn't be able to rely on date-modified to know whether to check for new functions, I'm not sure what other approach would be practical, especially if it's a some-Windows unique problem and Octave-on-Windows developers are few and far in between.

Other than 'Matlab doesn't have this problem'... so it must use some other method not vulnerable to this issue.

Bradley, have you tried the 'rehash' command?

Nicholas Jankowski <nrjank>
Group Member
Mon 31 Oct 2016 07:40:13 PM UTC, comment #13: 

attached screen shot of test folder; time stamp is older.

(file #38847)

Bradley Slagle <bslagle>
Mon 31 Oct 2016 07:26:34 PM UTC, comment #12: 

ok, so just to summarize since I think one bit of info is still sitting offline:

I attached a screenshot from an earlier email showing the modified dates for the files in the test folder. 

In the file window a new text file shows a date of 10/31/2016 1:28PM.  test43.m is saved showing a new date of 10/31/2016 1:31PM.

Octave reports back a mtime of:
Mon Oct 31 13:28:51 2016

the time of the new file that was not created via Octave.

Bradley, the only info we're still missing is what time/date Windows is reporting for the test folder. (it wasn't captured in any of the screenshots, those just got the files.) It should be 1:31pm, but if Octave is still saying the time you mentioned it might show 1:28pm.  That's the discrepancy we need to verify.


Nicholas Jankowski <nrjank>
Group Member
Mon 31 Oct 2016 06:59:59 PM UTC, comment #11: 

dirinfo does not update, see below.
-Brad

>> dirinfo = dir (pwd);disp(ctime(dirinfo(3).statinfo.mtime))

Mon Oct 31 13:28:51 2016

Bradley Slagle <bslagle>
Mon 31 Oct 2016 05:20:41 PM UTC, comment #10: 

Bradley, could you try to following to narrow down whether the problem is inside or outside of Octave:

Open a Windows Explorer window to that test folder on your W drive.  (W:\octave-files\test)

Open a second Windows Explorer file to the parent folder (W:\octave-files)  Make sure one of the visible Explorer columns is Date Modified  (if it's not there you should be able to add it by right clicking on the headings)

Note the modified date for the test folder.

in the first window, right click and create a new text file or something.  you should immediately see the test folder's Date Modified update.

wait a minute, open the text file, add a little text, and save it.  Did the timestamp update?

Repeat the process you used to generate that earlier screenshot you sent. Does the Date Modified date in Explorer ever differ from what is reported by the disp(ctime(... command?

This will help clarify whether the issue is that your system is not seeing the updated Date Modified from the server, or if it is and Octave for some reason is not seeing it.

Please post the results here to the bug tracker, rather than responding by email to help speed the discussion.

Nicholas Jankowski <nrjank>
Group Member
Mon 31 Oct 2016 05:16:20 PM UTC, comment #9: 

Note: the user indicates that the new test43.m file was created in the already open Octave editor and saved from there prior to the first error described in comment #8.

Nicholas Jankowski <nrjank>
Group Member
Mon 31 Oct 2016 05:07:45 PM UTC, comment #8: 

Summarizing some offline discussion with the discoverer of the bug (Bradley, on email distribution):

The originator of the bug has been able regularly recreate the issue. screenshot.png is attached.

extracted from the screenshot:

User has an existing function test42. Octave sees the function and runs it without trouble. Verifies he can run it and gets the local folder modified date with:


>> test42; dirinfo = dir (pwd); disp(ctime(dirinfo(3).statinfo.mtime))
b = 8882443
Mon Oct 24 14:25:26 2016


user edits the file to verify that Octave picks up the change:


>> edit test42   **resaves the file
>> test42; dirinfo = dir (pwd); disp(ctime(dirinfo(3).statinfo.mtime))
b = 8882443
Mon Oct 31 09:43:57 2016

everything works as expected. 

Now, user creates a new file, and Octave doesn't see it or an updated modified date:


>> test43; dirinfo = dir (pwd); disp(ctime(dirinfo(3).statinfo.mtime))
error: 'test43' undefined near line 1 column 1
>> dirinfo = dir (pwd); disp(ctime(dirinfo(3).statinfo.mtime))
Mon Oct 31 09:43:57 2016

>> edit test43  **resaves test43
>> dirinfo = dir (pwd); disp(ctime(dirinfo(3).statinfo.mtime))
Mon Oct 31 09:43:57 2016

>> test43; dirinfo = dir (pwd); disp(ctime(dirinfo(3).statinfo.mtime))
error: 'test43' undefined near line 1 column 1


Note that in the screenshot and as reported by the user, test43 shows up immediately as expected in the File Browser, and the edit function doesn't fail. So, the file is present, visible, and accessible.  But for some reason either mtime is not being updated, or Octave is not seeing it as updated.

I was able to create this on my network only once (see discussion below) and not since. I suspected it was a latency issue that was just delaying my PC seeing the updated modify time, as it always did eventually see the new file.

The user reports that no matter how long he waits, Octave never 'sees' the new file, which it seems means it never sees an updated mtime, until he closes and reopens Octave.


Nicholas Jankowski <nrjank>
Group Member
Tue 18 Oct 2016 06:28:22 PM UTC, comment #7: 

FYI, haven't been able to recreate the issue. Not certain but I suspect it's a lack of sufficient network delay. Might need others on local networks to test this.

Nicholas Jankowski <nrjank>
Group Member
Fri 30 Sep 2016 08:00:41 PM UTC, comment #6: 

ok, I'll rerun when I get the chance

Nicholas Jankowski <nrjank>
Group Member
Fri 30 Sep 2016 07:56:48 PM UTC, comment #5: 

I did mean dirinfo(1) in my original example, though, the relevant piece of information is the timestamp on the directory after the file is newly created.

Mike Miller <mtmiller>
Group Member
Fri 30 Sep 2016 01:32:18 PM UTC, comment #4: 

oh, items (1) and (2) in dirinfo are . and ..
I thought things were being strange and never changing until I realized that. so running your test with "...dirinfo(3)..."

Apparently network latency is low enough on a Friday morning that I can't recreate the issue right now. Here is are the outputs of the commands anyway for comparison with next time I can do so:



>> dir
.   ..

**created file in text editor**

>> dir
.         ..        test11.m
>> dirinfo = dir (pwd);disp(ctime(dirinfo(3).statinfo.mtime))
Fri Sep 30 09:17:45 2016

>> edit test11.m
>> dirinfo = dir (pwd);disp(ctime(dirinfo(3).statinfo.mtime))
Fri Sep 30 09:18:07 2016

>> edit test11.m
>> test11; dirinfo = dir (pwd);disp( ctime( dirinfo(3).statinfo.mtime))
Fri Sep 30 09:18:53 2016

>> delete test11.m
>> edit **create a new test21 file, save, then quickly:**
>> test21;
b = 12345
Fri Sep 30 09:22:00 2016


will try again later if things seem 'slower'.

Nicholas Jankowski <nrjank>
Group Member
Fri 30 Sep 2016 12:55:33 PM UTC, comment #3: 


> Is there a delay between when the file is saved and when the network-mapped directory modification timestamp is updated?


The issue happens for newly created files, not just edited/resaved files. Would this still be a related concern?

Will run through those test scripts shortly. Also, since the file shows up immediately in the File Browser and before the interpreter acknowledges it's existence, I'll pay attention to the time/date modified value in File Browser to see if it registers an updated timestamp right away or not.

Nicholas Jankowski <nrjank>
Group Member
Thu 29 Sep 2016 11:19:14 PM UTC, comment #2: 

I suspect this:

https://technet.microsoft.com/en-us/library/cc755136(v=ws.11).aspx

could make things more complicated...

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 29 Sep 2016 11:10:28 PM UTC, comment #1: 

Is there a delay between when the file is saved and when the network-mapped directory modification timestamp is updated?

After creating the file, what do the following (feel free to play with these functions) show?


>> cd /tmp
>> dirinfo = dir (pwd); disp (ctime (dirinfo(1).statinfo.mtime))
Thu Sep 29 15:17:01 2016

>> edit testfun.m

>> fileinfo = dir (fullfile (pwd, "testfun.m")); disp (ctime (fileinfo.statinfo.mtime))
Thu Sep 29 16:06:11 2016

>> dirinfo = dir (pwd); disp (ctime (dirinfo(1).statinfo.mtime))
Thu Sep 29 16:06:11 2016

## some time passes, mtime changes?


On my system the mtimes of the file and the containing directory (not network-mounted) are identical. When Octave encounters an unknown name, it traverses the load path and looks for any directories that have modification times later than the last time the directory was scanned. If the mtime has not been updated, then there must not be any new files in that directory.

Mike Miller <mtmiller>
Group Member
Tue 27 Sep 2016 10:44:36 PM UTC, original submission:  

Related to a recent help list thread:
http://octave.1599824.n4.nabble.com/octave-4-0-3-issue-td4679865.html

Verified on Octave 4.0.3, 32bit, Windows 7, windows zip file installer, reported on the help thread as occurring in Windows 10.

New files created in the working folder appear immediately in Octave's File Browser, but the interpreter gives a not found error for some undetermined amount of time. This appears to only occur on a network drive. (Tested on a decent,but non-zero latency LAN. Assume a mapped remote drive, like Dropbox or Google Drive would be worse. Never occurs for me on local hard drive.)

Procedures to create error:

Created a shared drive folder outside of octave:

M:\temp\testdata

Octave interpreter:

>> cd m:\temp\testdata
>> edit


Created the following simple function:

function test21
b = 12345

gui saved the file as test21.m in the folder created. Octave's File Browser window showed the test21.m file. Quickly in the Octave interpreter:


>> test21
error: 'test21' undefined near line 1 column 1

*wait a few seconds*

>> test21
b = 12345


created another similar test11 file in the gui, saved it, it showed up right away in the file browser. Without waiting between hitting enter in the interpreter:


>> test11
error: 'test11' undefined near line 1 column 1

>> test11
error: 'test11' undefined near line 1 column 1

>> test11
error: 'test11' undefined near line 1 column 1

>> test11
b = 123456789


Although I was unable to verify, the initiator if the help thread indicated no delay with matlab's editor/interpreter. I guess I can see where this could impede workflow in a networked environment like a school or office.

Nicholas Jankowski <nrjank>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #38847:  test-folder.pdf added by bslagle (168KiB - application/pdf)
file #38846:  screenshot2.png added by nrjank (25KiB - image/png - screenshot showing files and timestap in working folder.)
file #38845:  screenshot.png added by nrjank (67KiB - image/png - screenshot of Octave desktop, showing file on network drive in file browser, but interpreter can't see it.)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by bslagle (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by nrjank (Submitted the item)
  • -email is unavailable- added by nrjank
  • -email is unavailable- added by nrjank
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-02-23 mtmiller Carbon-CopyRemoved 80942 -
    2019-02-23 mmuetzel Open/ClosedOpen Closed
    2016-11-16 rik5 StatusNeed Info Duplicate
    2016-11-16 rik5 Dependencies- Depends on bugs #31080
    2016-10-31 bslagle Attached File- Added test-folder.pdf, #38847
    2016-10-31 nrjank Attached File- Added screenshot2.png, #38846
    2016-10-31 nrjank Attached File- Added screenshot.png, #38845
    2016-09-29 mtmiller StatusNone Need Info
    2016-09-27 nrjank Carbon-Copy- Added -email is unavailable-
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code