bugGNU Octave - Bugs: bug #31080, User scripts or functions created...

 
 

bug #31080: User scripts or functions created during a session on Windows are not found

Submitted by:  J-D Bonjour <jdb>
Submitted on:  Sun 19 Sep 2010 10:42:38 PM UTC  
 
Category: InterpreterSeverity: 3 - Normal
Priority: 3 - LowItem Group: Incorrect Result
Status: ConfirmedAssigned to: None
Originator Name: Open/Closed: Open
Release: devOperating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 16 Nov 2016 04:55:59 PM UTC, comment #30:

just noticed this bug, after a similar issue came up and was reported as bug #49203.

https://savannah.gnu.org/bugs/?49203

Nicholas Jankowski <nrjank>
Mon 13 Jun 2016 03:21:07 AM UTC, comment #29:

I like Abhinav's suggestion of only forcing a rehash of a directory known to have changed. I wouldn't call that a sledgehammer at all.

One way to make it more fine-grained would be to allow the second argument to be a filename rather than a directory. Then adding a file is very efficient -- presumably much more efficient than doing a rehash of the whole directory based on a timestamp. The only need for an inefficient whole-directory rehash is that a file may have been added by a program external to (and not triggered by) Octave.

Also, I think you would be looking for "FindFirstChangeNotification", the Windows equivalent to Linux's inotify. I don't think MinGW supports inotyify. http://stackoverflow.com/questions/36125844/use-inotify-with-mingw

Lachlan Andrew <lachlan>
Project Member
Tue 03 May 2016 06:47:56 PM UTC, comment #28:

We already have a reshash function, but all it does is traverse the directories in the path and look for modification times that are newer than the last timestamp the directory was scanned. This is still failing on Windows in this situation. Adding a way to do some kind of "force rehash" is a sledgehammer that might work around this, but I still think the root issue is something quirky with the way Windows filesystems are working with Octave.

Mike Miller <mtmiller>
Project Administrator
Tue 03 May 2016 06:41:13 PM UTC, comment #27:

@Mike: Adding a rehash command doesn't sound all that bad to me. Certain system shells, like csh, have a rehash command to update internal caches after you have changed the PATH variable.

The root cause seems to be known which is that Octave relies on timestamps, and timestamps on Windows filesystems are not always updated. I don't think more detective work will help.

If we want to fix the root cause, then I like your suggestion of looking at inotify. I've never done anything with it though so I can't say how hard it would be.

rehash seems like a nice workaround if we don't want to address the root cause.

Rik <rik5>
Project Administrator
Tue 03 May 2016 06:25:30 PM UTC, comment #26:

No, what we really need to determine is why new files not being found. Is it because the directory timestamp is actually not updating? Is it because something in the MinGW system library translation layer is not working right? Is it because the timestamp resolution isn't fine enough? Is the directory timestamp actually being updated but Octave isn't noticing that somehow? Does it have to do with the type of directory or the type of filesystem used?

We need some detailed tests showing the results of stat(), for example, on the directory containing a new file, and what the system timestamp is when the directory's modified timestamp actually does get updated, if it does at all.

Mike Miller <mtmiller>
Project Administrator
Tue 03 May 2016 06:11:49 PM UTC, comment #25:

To solve this issue, i propose to add a new function 'force_rehash(dir_name)' to force a rehash of the dir without checking for timestamp or edit the current rehash so that 'rehash' or 'rehash(false)' remain as is. But 'rehash(true, dir_name)' could be used to force rehash irrespective of timestamp!
.
This won't be a huge change and will allow users to force a rehash on windows and certainly solve this issue.
.
Should I do the changes and submit a cset? Does anyone have a problem with this approach (just asking as the issue is old and it hasn't been implemented yet)...

Abhinav Tripathi <genuinelucifer>
Mon 02 May 2016 01:40:19 AM UTC, comment #24:

This issue has resurfaced in the context of the symbolic package:

https://github.com/cbm755/octsympy/issues/431

A valid function file is written to a directory that is already in the load path, and the function is not found until some as-yet-undetermined amount of time later.

Mike Miller <mtmiller>
Project Administrator
Sun 05 Jan 2014 10:20:03 AM UTC, comment #23:

Normally, in root dirs there shouldn't be so awfully many files.

I don't know Octave's function/script file loading internals, so apologies if the following is a dumb question; but I wonder, why stat all files? all that seems to be needed is to just stat the called function file in question, and then only once after entering each Octave command (as changing some called function file contents while Octave is accessing it in e.g. a loop wouldn't be a good idea.)

So if this needs to be fixed at all, maybe modding the function/script load logic for Windows systems is the way to go?

I agree with Rik that this can get quite messy.
NTFS isn't a monolithic static file system; there are several versions around. M$ also still has the WinFS looming somewhere, possibly with still other behavior.
On NTFS it is possible to map other drives somewhere in the tree a la *nix mount (on Windows it's called junctions IIRC); we would need to know how those behave as well: as "local" subdirs, or do they behave as remote "root" dirs? in the latter case we'd also need to detect them as such.

Even while this bug is quite old is hasn't seen many confirmations; so I'd also agree with lowering priority/severity until we get more requests.
Repeating myself (see comment #13), having script or function files in root dirs is something that can be avoided.

- IMO on local fixed drives (HDs) it's a bad idea anyway.

- Only on thumb drives and other removable drives, and in some cases on mapped network drives, it can be more reasonable to keep m-files in the root dir.

Perhaps there is a way to detect a directory to be the root of a removable/network device and act accordingly along Jordi's suggestion.
Note that thumb drives, and to a lesser extent network drives, are usually slow; statting files may give a noticeable performance hit.
(I don't know if mapped network drives behave the same as "physical local" drives as regards directory time stamps.)

Philip Nienhuis <philipnienhuis>
Project Member
Sun 05 Jan 2014 05:12:45 AM UTC, comment #22:

What if we just statted all of the files in the directory instead of only statting the directory? Is this a terrible slowdown, to stat all files in the path directories on every command invocation?

Jordi GutiƩrrez Hermoso <jordigh>
Project Administrator
Sat 04 Jan 2014 11:22:18 PM UTC, comment #21:

Yes, confirmed here (I now have a winxp VM with 3.8.0 to test with). The root directory of an NTFS drive seems to have a constant timestamp:

So yes this looks unavoidable given the current path caching scheme.

One possibility might be to watch for directory inode changes instead of looking at timestamps. This could use FindFirstChangeNotification on Windows and inotify on Linux-based systems. This would most likely be a major overhaul.

Mike Miller <mtmiller>
Project Administrator
Sat 04 Jan 2014 11:05:34 PM UTC, comment #20:

This is all down to the same root cause which is that the directory timestamp is not updated when the directory itself is changed (such as by the addition of a new file). On FAT filesysems, the directory timestamp is not updated period. You can work around that by manually using touch.exe to change the timestamp of the directory. Things are a bit better on NTFS systems, in that directories which are modified have their timestamp updated. But, this being Microsoft, there are exceptions to that as well. The root of the file tree is apparently special and just isn't updated. I tried creating new files, new directories, using touch.exe and the timestamp of the root node is never modified.

The easiest way to see this is from within Octave. Use

and replace 'e' by whatever drive letter you are using.

So, the situation just sort of sucks on Microsoft file systems.

It's probably possible to disable the caching code in Octave (make sure this is only on Windows machines), but the performance hit might be unacceptable. Basically, any time a function wasn't already compiled and in the symbol table, Octave would need to do a full search of every single directory on the path. This would also happen everytime someone entered a typo as well like 'dirt' for 'dir' since Octave couldn't know that you didn't mean to call a function that it doesn't know about.

Rik <rik5>
Project Administrator
Sat 04 Jan 2014 10:09:34 PM UTC, comment #19:

Re-reading comments #7 and #8, you'll see that this is primarily about the directory timestamp, not necessarily the file timestamp. So Octave is expecting the timestamp of the containing directory to be updated. Do drive top-level directories have timestamps in Windows, and are they updated when new files are created at the top level?

What does

show before and after creating a new m-file?

Mike Miller <mtmiller>
Project Administrator
Sat 04 Jan 2014 10:02:24 PM UTC, comment #18:

Forgot to add screen session pic; attached.

(file #30188)

Philip Nienhuis <philipnienhuis>
Project Member
Sat 04 Jan 2014 10:00:55 PM UTC, comment #17:

Sorry, I need to be a bit more exact:

do:
"start Octave"
(or "start octave-cli")
.... from that same cmd32 window.

FYI, "start octave" will detach an octave session from that command window. While you're fiddling in Octave, from that cmd32 window you can run "touch".

============
Anyway I've tried this myself with Octave-3.9.0+ and I get the same as the OP. Screen shot of session attached, just to show.
Like the OP I tried it on an NTFS drive (X: in my case)
"touch"ing the file (after Octave command #7-2) doesn't help.

BTW I'm intrigued by the error msg:

i.e. Octave seems to forget the file name although it was entered on the command line.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 04 Jan 2014 07:27:28 PM UTC, comment #16:

@comment #15:

Au contraire, touch.exe does (extremely) probably exist but it's in your <OCTAVE_HOME>/bin directory. IOW in bin/ in the directory where you've installed Octave.
AFAIK it's included in all MXE-built MinGW 3.8.0 binaries.

You could (in a Command window, cmd32.exe) do something like:

set PATH=%PATH%;/full/path/to/Octave/directory/bin

and then start Octave (from that same cmd32 window).

Philip Nienhuis <philipnienhuis>
Project Member
Sat 04 Jan 2014 06:26:45 PM UTC, comment #15:

The command touch does not exist on Windows

A substitute to this command is : copy /b file +,,
(found on http://stackoverflow.com/questions/51435/windows-version-of-the-unix-touch-command), but it doesn't solve the problem.

J-D Bonjour <jdb>
Sat 04 Jan 2014 05:47:13 PM UTC, comment #14:

Does it work if you use touch.exe to update the file's modification time?

Jordi GutiƩrrez Hermoso <jordigh>
Project Administrator
Sat 04 Jan 2014 05:02:03 PM UTC, comment #13:

Yes I'm following but a bit short of time lately - maybe tonight or otherwise tomorrow I'll check. & I also have to (re)build 3.8.0 first.

From what I've read so far I inferred that Octave relies on subdirectory timestamps, not individual function-/script-file timestamps.
Very probably -if not undoubtedly- for good reason (not hard to guess); but a drawback is this sort of issues.

A workaround is to avoid root dirs. Even on thumb drives (USB sticks) that would be a good way.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 04 Jan 2014 04:31:41 PM UTC, comment #12:

Sorry, it doesn't work for me on an NTFS drive (see my session below) !
Philip Nienhuis, could you also check, please ?

Jean-Daniel

octave:1> ver

GNU Octave Version 3.8.0
GNU Octave License: GNU General Public License
Operating System: MINGW32_NT-5.1 Windows XP Service Pack 3 i686
% so I'm working on Windows XP with the 3.8.0 Octave bundle
% from http://octave.osuv.de/3.8/windows/octave-3.8.0-1-portable.zip

octave:2> system('chkdsk M: /i')

The type of the file system is NTFS
...
% so the filesystem of my M: drive is NTFS !

octave:3> cd M:\octave_test % going to a sub-directory
octave:4> edit test1.m
% new file, inserting the line: disp('Hello from sub-dir of drive M:')
octave:5> test1

Hello from sub-dir of drive M:
% so on a sub-dir it works allways

octave:6> cd M:\ % going to the root of this NTFS drive
octave:7> edit test2.m
% new file, inserting the line: disp('Hello from root of drive M:')
octave:8> ls

[octave_test] test2.m
% the created script is visible, but...

octave:9> type test2.m

error: type: couldn't open '' for reading
error: called from:
error: ...\type.m at line 85, column 9

octave:10> test2

error: 'test2' undefined near line 1 column 1

% *** And now restarting Octave ***
octave:1> cd M:\
octave:2> test2

Hello from root of drive M:
% so it works after restarting Octave !

octave:3> type test2.m

test2.m is the user-defined function defined from: M:\test2.m

disp('Hello from root of drive M:')

octave:4> edit test2.m
% adding the line: disp('Second line added')
octave:5> test2

Hello from root of drive M:
Second line added

J-D Bonjour <jdb>
Sat 04 Jan 2014 04:14:46 AM UTC, comment #11:

I just reformatted a USB memory stick to use NTFS instead of the default FAT32 and Octave works perfectly with it. So the diagnosis about timestamps is accurate.

John, I was able to check the file system type by double-clicking "My Computer" to get a list of disk drives. If I right-click on a disk drive and then click on Properties it brings up a new tabbed window. In the General tab there is an entry called File System.

Rik <rik5>
Project Administrator
Sat 04 Jan 2014 02:10:48 AM UTC, comment #10:

I'm unable to reproduce the problem with Octave 3.8.0 running on Windows 7 in a virtual machine. I tried the GUI and the CLI version.

I assume that this is with an NTFS filesystem, but I don't know for sure. How do I check?

John W. Eaton <jwe>
Project Administrator
Sat 04 Jan 2014 01:04:21 AM UTC, comment #9:

No, sorry, the problem also occurs on NTFS partitions ! So it's not a good idea to lower the priority

J-D Bonjour <jdb>
Fri 03 Jan 2014 10:53:27 PM UTC, comment #8:

This is definitely bad Microsoft mojo. This microsoft support brief (http://support.microsoft.com/kb/299648) explains that NTFS directory timestamps are updated when directory contents change, but FAT formatted filesystems do not update directory timestamps when contents change.

My guess is that most modern installs of Windows have an NTFS partition for C:\ which explains why it works on the root partition. Other drive letters are typically things like USB memory sticks which are probably formatted with FAT filesystems which explains why it tends not to work on them.

I'm lowering the priority for this bug. Overhauling a huge amount of Octave directory code to deal with Microsoft's bad design is not my first choice of things that need improvement in Octave.

Rik <rik5>
Project Administrator
Fri 03 Jan 2014 10:25:04 PM UTC, comment #7:

Very definitely confirmed. Octave seems to be caching the contents of a directory on start-up and then never updating the cache. This could be down to differences in how the operating system updates the inode timestamp for the directory.

On Linux, creating a new file in a directory updates the timestamp. I think Octave is comparing the current directory timestamp against the cached timestamp to figure out if it needs to go and re-read the directory.

On Windows, creating a new file in a directory does not change the timestamp--I know because I tried. However, I have the UNIX tool touch.exe installed and when I touch the directory to bring the timestamp up to the current time then Octave correctly finds the new m-files.

This sequence worked:

Seems like it might require a lot of O/S specific code to solve this one.

Rik <rik5>
Project Administrator
Fri 03 Jan 2014 08:46:21 PM UTC, comment #6:

Yes, this problem is still present on Windows. Tested now with the octave-3.8.0-1-portable.zip packaging distributed on http://octave.osuv.de/3.8/windows/

JDB

J-D Bonjour <jdb>
Fri 03 Jan 2014 09:09:56 AM UTC, comment #5:

Thanks for reminding me, I'll check it out.

[Little bit OT]
A similar problem I hit with recent dev versions (3.7.7 & 3.9.0 and the two 3.8.0 release candidates) is that editing/debugging functions in private/ dirs leads to similar behavior (on MinGW):
Octave suddenly doesn't seem to be able to find certain private/functions anymore (while other private/ functions weren't affected). Adding the private subdit to the load path (as the GUI editor helpfully suggests) didn't help. Only a restart helps.
But as this behavior was erratic, and because I don't know the exact rules for the search path as regards private subdirs, I thought I was probably fubarring up myself so I didn't file a bug report.
After all, things work OK as long as functions in private subdirs aren't touched.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 02 Jan 2014 09:11:57 PM UTC, comment #4:

Is this problem still present in Octave 3.8.0?

John W. Eaton <jwe>
Project Administrator
Thu 17 Oct 2013 08:11:45 PM UTC, comment #3:

Octave 3.7.7+ MXE GUI built for MinGW, installed on drive X, and Octave-3.6.7+ MXE CLI (ditto):

With either script or function files, it doesn't even work in a subdir of root:

After restarting, Octave behaves even stranger:

Striking is that it works upon restart for functions but not for scripts in a subdir, and the other way round in the root dir.

Could these be MinGW bugs (in the gcc runtime libraries)?

Philip Nienhuis <philipnienhuis>
Project Member
Thu 17 Oct 2013 06:34:28 PM UTC, comment #2:

Is this still present in the development version that is about to become the 3.8 release?

Rik <rik5>
Project Administrator
Fri 03 Feb 2012 11:16:47 AM UTC, comment #1:

Still present in 3.6.0 MinGW

Philip Nienhuis <philipnienhuis>
Project Member
Sun 19 Sep 2010 10:42:38 PM UTC, original submission:

On Octave MinGW32 for Windows versions 3.2.0 to 3.2.4, when the user current directory is the root of a Windows partition other than C:\ (e.g. D:\, E:\ ...), the m-files (scripts or functions) created by the user during the session (with command "edit" or with other editors) are not usable : when invoqued, the following message is returned : error: '<script>' undefined near line xxx column 1.

Its very strange, because the command "dir" sees these m-files created during the session, and if I restart Octave I am able to use them... but not new scripts/functions created during the session.

Example :

> cd T:\test
> edit script.m
> % creating script containing line: disp('hello world !')
> script

hello world !

> % so it works !
>
> cd .. % going to the root of this filesystem T:\
> pwd

ans = T:\

> edit script2.m
> % creating script containing line: disp('hello again')
> script2

error: `script2' undefined near line 7 column 1

> % doesn't work, but the script is here :
> dir

. .. script2.m

> quit
>
> % new Octave session
> cd T:\
> script2

hello again

> % now it works, because it's a new Octave session
>

J-D Bonjour <jdb>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -unavailable- added by nrjank (Posted a comment)
  • -unavailable- added by lachlan (Posted a comment)
  • -unavailable- added by genuinelucifer (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by jordigh (Updated the item)
  • -unavailable- added by philipnienhuis (Posted a comment)
  • -unavailable- added by jdb (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 13 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 16 Nov 2016 05:08:49 PM UTCrik5Dependencies-=>bugs #49203 is dependent
    Mon 13 Jun 2016 03:21:07 AM UTClachlanSummaryUser scripts or functions created during a session are not found=>User scripts or functions created during a session on Windows are not found
    Mon 02 May 2016 01:40:19 AM UTCmtmillerRelease3.8.0=>dev
    Mon 02 May 2016 01:18:45 AM UTCmtmillerDependencies-=>bugs #47818 is dependent
    Tue 24 Nov 2015 12:31:21 AM UTCrik5Dependencies-=>bugs #44440 is dependent
    Mon 20 Jan 2014 03:02:12 AM UTCmtmillerDependencies-=>bugs #34731 is dependent
    Sat 04 Jan 2014 10:02:24 PM UTCphilipnienhuisAttached File-=>Added bug31080_session.png, #30188
    Fri 03 Jan 2014 10:53:27 PM UTCrik5Priority5 - Normal=>3 - Low
    Fri 03 Jan 2014 10:25:04 PM UTCrik5StatusNeed Info=>Confirmed
      Release3.6.0=>3.8.0
    Thu 02 Jan 2014 09:11:57 PM UTCjweStatusNone=>Need Info
    Thu 28 Jun 2012 01:56:18 PM UTCjordighRelease3.2.4=>3.6.0
    Fri 03 Feb 2012 11:16:47 AM UTCphilipnienhuisCategoryNone=>Interpreter

    Back to the top


    Powered by Savane 3.1-cleanup1