bugGNU Octave - Bugs: bug #31080, User scripts or functions created...

 
 

bug #31080: User scripts or functions created during a session on Windows are not found

Submitter:  J-D Bonjour <jdb>
Submitted:  Sun 19 Sep 2010 10:42:38 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  3 - Low Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 03 Oct 2024 04:08:16 PM UTC, comment #59: 

ok.  reopening.  we can tie other reports to it / update them if they're identified.

Nicholas Jankowski <nrjank>
Group Member
Thu 03 Oct 2024 02:37:52 PM UTC, comment #58: 

@nrjank
Yeah you're right, it's not reliable.
Here, with (dev) Octave-10.0.0 on Win10, the first try worked fine, the second didn't.

[OT]
In the bug tracker fields I see 9.1.0 for both "Fixed release"  and "Planned release", these fields should be updated.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 03 Oct 2024 01:29:42 PM UTC, comment #57: 

slight correction:  often would be a better descriptor than regularly.  It's not as repeatable as would be nice for an easily replicable bug report.

Nicholas Jankowski <nrjank>
Group Member
Thu 03 Oct 2024 01:27:39 PM UTC, comment #56: 

yes, i see this regularly with 9.2.0 and fairly recent stable builds.  I've gotten used to simply issuing a path(path)) command which usually corrects it.

I think there are a number of bug reports on this issue or closely related ones, i didn't realize this old one was still getting traffic.  before reopening it's probably worth checking for those.

Nicholas Jankowski <nrjank>
Group Member
Thu 03 Oct 2024 12:30:19 PM UTC, comment #55: 


comment #54:

> [~OT]
> Yes this is an oldie :-)
> I'd celebrate it, this is one from the "ergonomy" category and I think quite a few users have hit this bug over the years, with reports scattered over several bug reports (incl. duplicates) and mailing list(s) threads.
>
> But I'm quite sure that there must be still older open bugs, just not listed on Savannah ... some of those might have resurfaced as younger bug reports. Maybe it applies to this one as well.
> I have no clear recollection of when the Octave project started using Savannah. Could be that we used s/th from SourceForge before that.




Nope, this is still happening in Octave 9.2.0. Make a new .m file in a Windows session, try to run it, you will get the very same error, as if the file did not exist.

> There used to be a mailing list "bug-octave@octave.org", in my archive I have emails with that address back to at least 2009. Even older emails mention (IIRC) JWE's former uni affilations but the project might not have had a dedicated bug mailing list then, let alone bug tracker.
> [/OT]
>
> BTW I'd feel a little more sure if nrjank could also confirm the behavior I see. Although releases 8.1.0 and 9.1.0 (dev) clearly behave differently.

Anonymous
Sun 28 May 2023 09:47:05 PM UTC, comment #54: 

[~OT]
Yes this is an oldie :-)
I'd celebrate it, this is one from the "ergonomy" category and I think quite a few users have hit this bug over the years, with reports scattered over several bug reports (incl. duplicates) and mailing list(s) threads.

But I'm quite sure that there must be still older open bugs, just not listed on Savannah ... some of those might have resurfaced as younger bug reports. Maybe it applies to this one as well.
I have no clear recollection of when the Octave project started using Savannah. Could be that we used s/th from SourceForge before that.
There used to be a mailing list "bug-octave@octave.org", in my archive I have emails with that address back to at least 2009. Even older emails mention (IIRC) JWE's former uni affilations but the project might not have had a dedicated bug mailing list then, let alone bug tracker.
[/OT]

BTW I'd feel a little more sure if nrjank could also confirm the behavior I see. Although releases 8.1.0 and 9.1.0 (dev) clearly behave differently.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 28 May 2023 07:31:04 PM UTC, comment #53: 

Thanks for testing.

I guess this bug can finally be closed now (after almost 13 years).
I'm not sure if this is something to celebrate or to mourn about. 😉
That made it the oldest report that was still open afaict.

Closing as fixed in Octave 9.

Markus Mützel <mmuetzel>
Group administrator
Sun 28 May 2023 02:43:33 PM UTC, comment #52: 

(Sorry, took a while as I lost the notification email and had to track down what bug it was about)

With a dev crossbuild from May 24, using the commands from comment #49 in the root of a drive F: on a multipartitioned SSD under Win10:

>> cd f:
>> edit newtestfile
>> newtestfile
>> ls -l
total 38
<snip>
drwxr-xr-x 1 philip None    0 Feb 14  2022 '$RECYCLE.BIN'
-rw-r--r-- 1 philip None 3929 May 25 19:51  128KEY.TXT
drwxr-xr-x 1 philip None    0 Dec 16  2021  DOS_dev
drwxr-xr-x 1 philip None    0 Feb  7  2022 'System Volume Information'
-rw-r--r-- 1 philip None  208 Jan 27  2022  btrfs_UUID.txt
drwxr-xr-x 1 philip None    0 Jan 18 23:46  dev
drwxr-xr-x 1 philip None    0 Dec 14  2021  home
drwxr-xr-x 1 philip None    0 Mar  5 10:16  mingw-binaries
-rw-r--r-- 1 philip None    0 May 28 16:34  newtestfile.m
drwxr-xr-x 1 philip None    0 Nov  7  2022  tmp


so at least on my system this bug seems to have been fixed.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 23 May 2023 08:51:57 PM UTC, comment #51: 

Yes I saw the related discussion elsewhere. I'll prepare a new crossbuild later this week and try again.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 23 May 2023 06:40:57 AM UTC, comment #50: 

Recent changes on the default branch might have affected this behavior (hopefully for the better).

Do you still see this issue with a recent build from the default branch?

Markus Mützel <mmuetzel>
Group administrator
Fri 09 Dec 2022 12:02:49 AM UTC, comment #49: 

this bug is still present in octave 8.0.1 (hg id: 24bd675bceab)

testing on a ntfs local ssd in windows 10:


>> cd d:
>> edit newtestfile (create a new script file and save it)
>> newtestfile
error: 'newtestfile' undefined near line 1, column 1
>> ls -l
total 30
<snip>
drwxr-xr-x 1 Admin None   0 Jan 10  2022 '$RECYCLE.BIN'
drwxr-xr-x 1 Admin None   0 Nov  9 01:02  MapData
drwxr-xr-x 1 Admin None   0 Mar 22  2022  Programs
drwxr-xr-x 1 Admin None   0 Aug  9  2020  Recovery
drwxr-xr-x 1 Admin None   0 Aug  9  2020 'System Volume Information'
drwxr-xr-x 1 Admin None   0 Mar 21  2022  localdata
-rw-r--r-- 1 Admin None   9 Dec  8 18:21  newtestfile.m
drwxr-xr-x 1 Admin None   0 Jan 10  2022  temp
>> newtestfile
error: 'newtestfile' undefined near line 1, column 1



also, trying Markus's stat test, things behave fine in a subfolder:


>> cd temp
>> dir
.   ..
>> stat('.')
ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 1670543181
    mtime = 1641824642
    ctime = 1590511359
    blksize = NaN
    blocks = NaN

>> fclose(fopen('testfile.m','w'))
ans = 0
>> stat('.')
ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 1670543207
    mtime = 1670543207
    ctime = 1590511359
    blksize = NaN
    blocks = NaN


but in the root of the drive:


>> stat('.')
ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 315550800
    mtime = 315550800
    ctime = 315550800
    blksize = NaN
    blocks = NaN

>> fclose(fopen('testfile.m','w'))
ans = 0
>> stat('.')
ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 315550800
    mtime = 315550800
    ctime = 315550800
    blksize = NaN
    blocks = NaN

>> ls -l
total 30
<snip>
drwxr-xr-x 1 Admin None   0 Jan 10  2022 '$RECYCLE.BIN'
drwxr-xr-x 1 Admin None   0 Nov  9 01:02  MapData
drwxr-xr-x 1 Admin None   0 Mar 22  2022  Programs
drwxr-xr-x 1 Admin None   0 Aug  9  2020  Recovery
drwxr-xr-x 1 Admin None   0 Aug  9  2020 'System Volume Information'
drwxr-xr-x 1 Admin None   0 Mar 21  2022  localdata
-rw-r--r-- 1 Admin None   0 Dec  8 18:31  newtestfile.m
drwxr-xr-x 1 Admin None   0 Dec  8 18:46  temp
-rw-r--r-- 1 Admin None   0 Dec  8 18:52  testfile.m
>> testfile.m
error: 'testfile' undefined near line 1, column 1


are those time values normal in a root folder?

Nicholas Jankowski <nrjank>
Group Member
Fri 29 Oct 2021 05:15:11 PM UTC, comment #48: 

Thanks.
So, it looks like the mtime of the parent directory didn't update when the file was created.
It is for me when I do the same:

>> format long
>> which test1
>> stat('.')
ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 1635525754
    mtime = 1635523631
    ctime = 1635523460
    blksize = NaN
    blocks = NaN

>> fclose(fopen('test1.m', 'w'));
>> stat('.')
ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 1635527519
    mtime = 1635527519
    ctime = 1635523460
    blksize = NaN
    blocks = NaN

>> which test1
'test1' is a script from the file D:\SVN\Octave\test\octave-2021-10-28-21-39-w64 release of-audio\octave-2021-10-28-21-39-w64\test1.m


What file system (NTFS, FAT32, exFAT, ...) do you use on your drive D:?

Markus Mützel <mmuetzel>
Group administrator
Fri 29 Oct 2021 04:12:11 PM UTC, comment #47: 

Sorry bad read, I thought I had to write the file with the instruction, here is the correct test (I think). The data drive is an additional SSD disk.

>> format long
>> which test1
>> stat('.')

ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 1635509742
    mtime = 1635509742
    ctime = 1635509742
    blksize = NaN
    blocks = NaN

>> fclose(fopen('test1.m', 'w'));
>> stat('.')

ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 1635509742
    mtime = 1635509742
    ctime = 1635509742
    blksize = NaN
    blocks = NaN

>> which test1

Silvestre <paredes>
Fri 29 Oct 2021 12:51:10 PM UTC, comment #46: 

Thanks for testing. But please run the test in a directory that does not contain `test1.m` before the test.

Markus Mützel <mmuetzel>
Group administrator
Fri 29 Oct 2021 12:32:16 PM UTC, comment #45: 

Hi again:

It must be a problem with the disk drive (D:), I have repeated the process on home disk (C:) and there are no problems....

Octave is installed in home disk (c:) as usual, but data are in data disk (d:).

I will check the data disk (d:) configuration....


Silvestre <paredes>
Fri 29 Oct 2021 12:27:44 PM UTC, comment #44: 


I have logged in from the beginning......


------------------------------------------------------------------------

GNU Octave, version 6.3.0
Copyright (C) 2021 The Octave Project Developers.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  For details, type 'warranty'.

Octave was configured for "x86_64-w64-mingw32".

Additional information about Octave is available at https://www.octave.org.

Please contribute if you find this software useful.
For more information, visit https://www.octave.org/get-involved.html

Read https://www.octave.org/bugs.html to learn how to submit bug reports.
For information about changes from previous versions, type 'news'.

>> dir

.   ..

>> edit test1.m
>> dir

.        ..       test1.m

>> test1

error: 'test1' undefined near line 1, column 1

>> type test1.m

error: type: couldn't open '' for reading
error: called from
    type at line 92 column 9

>> path(path());
>> type test1.m

test1.m is the user-defined function defined from: D:\test\test1.m

format long
which test1
stat('.')
fclose(fopen('test1.m', 'w'));
stat('.')
which test1

>> test1

'test1' is a script from the file D:\test\test1.m
ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 1635509742
    mtime = 1635509742
    ctime = 1635509742
    blksize = NaN
    blocks = NaN

ans =

  scalar structure containing the fields:

    dev = 3
    ino = 0
    mode = 16895
    modestr = drwxrwxrwx
    nlink = 1
    uid = 0
    gid = 0
    rdev = 3
    size = 0
    atime = 1635509742
    mtime = 1635509742
    ctime = 1635509742
    blksize = NaN
    blocks = NaN

'test1' is a script from the file D:\test\test1.m

>> type test1.m

test1.m is the user-defined function defined from: D:\test\test1.m


Silvestre <paredes>
Fri 29 Oct 2021 11:33:15 AM UTC, comment #43: 

Re-starting Octave isn't the solution. I just wanted to see if it is something intrinsic to that file or folder...

Re-reading the discussion, it looks like rehash uses some logic to only re-scan the folders of which it thinks that they have changed.
Does re-creating the load path cache completely with `path(path())` help? (I hope that doesn't use some clever tricks to avoid rehashing paths it thinks it already knows...)

Additionally, IIUC, Mike's suspicion was that the modification date of the parent directory doesn't change. Could you try something like the following in a directory that doesn't contain a 'test1.m' file and show the results here?

format long
which test1
stat('.')
fclose(fopen('test1.m', 'w'));
stat('.')
which test1


Markus Mützel <mmuetzel>
Group administrator
Fri 29 Oct 2021 07:39:59 AM UTC, comment #42: 

Local drive (both Octave and the script)

Yes, I find the file after restart Octave (but I think it is not very elegant and time is wasted)

No, I have the same problem even after using rehash().

Only if I restart OCTAVE or, without restarting, I use the option 'Edit->Set Path->Save options' from the top menu of the GUI, then I can see and use the script.


Silvestre <paredes>
Fri 29 Oct 2021 06:59:01 AM UTC, comment #41: 

Do these files reside on a local drive or a network share?
ISTR that there were missing signals for file updates from UNC shares (maybe also others).

Is the file found on a restart of Octave? Does issuing `rehash()` to manually trigger a load path cache update make a difference?

Markus Mützel <mmuetzel>
Group administrator
Thu 28 Oct 2021 10:40:25 PM UTC, comment #40: 

I have created a simple script call probe.m to show the bug, I enclosed the answer from the system. I'm using GUI Octave (6.3.0)


------------- I can see the file in the directory, using octave dir command

>> dir

.                FUNCIONALIDADES  get_matrix.m     getmat.m         probe.m
..               SCRIPTS          getm.m           pm.m




------------ I can see the content of the file, using octave dos command with command type

>> dos("type probe.m")

AB=randn(2,3);
ans = 0



------------ I cannot see the content of the file if I use octave type command

>> type probe.m

error: type: couldn't open '' for reading
error: called from
    type at line 92 column 9



----------  I cannot run the script from the OCTAVE prompt

>> probe

error: 'probe' undefined near line 1, column 1

Silvestre <paredes>
Wed 27 Oct 2021 06:40:42 AM UTC, comment #39: 

@paredes: Could you please show the exact error message or output you are seeing when calling your script doesn't work?

Markus Mützel <mmuetzel>
Group administrator
Tue 26 Oct 2021 07:13:57 PM UTC, comment #38: 

I have two local drive, the one with the windows system (c:), and another (d:) for data, Octave is installed in the default directory at unit C:. I'm using windows 10 Pro.

I change the working directory to drive d:, and add the path to the OCTAVE one, using addpath.

After using Reload (Set Path from the Edit menu), I can use the functions.

Regards,
 

Silvestre <paredes>
Tue 26 Oct 2021 06:07:14 PM UTC, comment #37: 

@Silvestre - just for reference all of the things that might be relevant: are you working on a local drive, a network drive, are you running Windows and which version, and if so are you in a Onedrive managed folder?


Nicholas Jankowski <nrjank>
Group Member
Tue 26 Oct 2021 05:52:35 PM UTC, comment #36: 

I have the same problem with version 6.3.0

I wrote a scripts, I save it at my working directory, I can see the file with command "dir", I can edit the file with command "edit", but neither I cannot see the file with command "type" and of course I cannot run the script; Yes, if you are asking for, I can do it after restart OCTAVE.

How can I fix it?


Silvestre <paredes>
Sat 23 Feb 2019 11:42:41 AM UTC, comment #35: 

Closing as fixed as per comment #34.
If this behaviour should re-appear, we can re-open this bug or file a new one.

Markus Mützel <mmuetzel>
Group administrator
Fri 22 Feb 2019 08:46:24 PM UTC, comment #34: 

wanted to comment that I cannot recreate the situation from Comment #32 with Octave 5.0.91

Nicholas Jankowski <nrjank>
Group Member
Wed 30 May 2018 07:01:13 PM UTC, comment #33: 

I cannot recreate it with Octave-5.0.0 on Windows 7 and on Windows 10.
In fact it works fine here even with octave-4.2.2 on Windows 10.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 30 May 2018 04:20:23 PM UTC, comment #32: 

I can still recreate the behavior.


>> cd m:\test
>> toto.m
error: 'toto' undefined near line 1 column 1
>> edit toto.m
>> toto
error: 'toto' undefined near line 1 column 1
>> dir
.                         chntpw_1.0-1+b1_i386.deb  reviews
..                        elmerbackset.bat          toto.m
Command Prompt.lnk        elmerset.bat
>> toto
retval =  1
ans =  1
>>


Nicholas Jankowski <nrjank>
Group Member
Wed 30 May 2018 03:45:09 PM UTC, comment #31: 

Just tried the following on Windows 10:

```

>> cd d:\        % toto.m does not exist yet
>> edit toto.m   % create script containing: disp hello
>> toto

```

As reported below, this fails in Octave 4.0 and 4.2 with error 'toto' undefined.

The good news is, it works as expected with Octave 4.4.

Unless someone is able to reproduce the problem with Octave 4.4 on some other version of Windows, this bug can be closed ;-)

Julien Bect <jbect>
Wed 16 Nov 2016 04:55:59 PM UTC, comment #30: 

just noticed this bug, after a similar issue came up and was reported as bug #49203.

https://savannah.gnu.org/bugs/?49203

Nicholas Jankowski <nrjank>
Group Member
Mon 13 Jun 2016 03:21:07 AM UTC, comment #29: 

I like Abhinav's suggestion of only forcing a rehash of a directory known to have changed.  I wouldn't call that a sledgehammer at all.

One way to make it more fine-grained would be to allow the second argument to be a filename rather than a directory.  Then adding a file is very efficient -- presumably much more efficient than doing a rehash of the whole directory based on a timestamp.  The only need for an inefficient whole-directory rehash is that a file may have been added by a program external to (and not triggered by) Octave.

Also, I think you would be looking for "FindFirstChangeNotification", the Windows equivalent to Linux's inotify.  I don't think MinGW supports inotyify.  http://stackoverflow.com/questions/36125844/use-inotify-with-mingw

Lachlan Andrew <lachlan>
Tue 03 May 2016 06:47:56 PM UTC, comment #28: 

We already have a reshash function, but all it does is traverse the directories in the path and look for modification times that are newer than the last timestamp the directory was scanned. This is still failing on Windows in this situation. Adding a way to do some kind of "force rehash" is a sledgehammer that might work around this, but I still think the root issue is something quirky with the way Windows filesystems are working with Octave.

Mike Miller <mtmiller>
Group Member
Tue 03 May 2016 06:41:13 PM UTC, comment #27: 

@Mike: Adding a rehash command doesn't sound all that bad to me.  Certain system shells, like csh, have a rehash command to update internal caches after you have changed the PATH variable.

The root cause seems to be known which is that Octave relies on timestamps, and timestamps on Windows filesystems are not always updated.  I don't think more detective work will help.

If we want to fix the root cause, then I like your suggestion of looking at inotify.  I've never done anything with it though so I can't say how hard it would be.

rehash seems like a nice workaround if we don't want to address the root cause.

Rik <rik5>
Group administrator
Tue 03 May 2016 06:25:30 PM UTC, comment #26: 

No, what we really need to determine is why new files not being found. Is it because the directory timestamp is actually not updating? Is it because something in the MinGW system library translation layer is not working right? Is it because the timestamp resolution isn't fine enough? Is the directory timestamp actually being updated but Octave isn't noticing that somehow? Does it have to do with the type of directory or the type of filesystem used?

We need some detailed tests showing the results of stat(), for example, on the directory containing a new file, and what the system timestamp is when the directory's modified timestamp actually does get updated, if it does at all.

Mike Miller <mtmiller>
Group Member
Tue 03 May 2016 06:11:49 PM UTC, comment #25: 

To solve this issue, i propose to add a new function 'force_rehash(dir_name)' to force a rehash of the dir without checking for timestamp or edit the current rehash so that 'rehash' or 'rehash(false)'  remain as is. But 'rehash(true, dir_name)' could be used to force rehash irrespective of timestamp!
.
This won't be a huge change and will allow users to force a rehash on windows and certainly solve this issue.
.
Should I do the changes and submit a cset? Does anyone have a problem with this approach (just asking as the issue is old and it hasn't been implemented yet)...

Abhinav Tripathi <genuinelucifer>
Mon 02 May 2016 01:40:19 AM UTC, comment #24: 

This issue has resurfaced in the context of the symbolic package:

https://github.com/cbm755/octsympy/issues/431

A valid function file is written to a directory that is already in the load path, and the function is not found until some as-yet-undetermined amount of time later.

Mike Miller <mtmiller>
Group Member
Sun 05 Jan 2014 10:20:03 AM UTC, comment #23: 

Normally, in root dirs there shouldn't be so awfully many files.

I don't know Octave's function/script file loading internals, so apologies if the following is a dumb question; but I wonder, why stat all files? all that seems to be needed is to just stat the called function file in question, and then only once after entering each Octave command (as changing some called function file contents while Octave is accessing it in e.g. a loop wouldn't be a good idea.)

So if this needs to be fixed at all, maybe modding the function/script load logic for Windows systems is the way to go?

I agree with Rik that this can get quite messy.
NTFS isn't a monolithic static file system; there are several versions around. M$ also still has the WinFS looming somewhere, possibly with still other behavior.
On NTFS it is possible to map other drives somewhere in the tree a la *nix mount (on Windows it's called junctions IIRC); we would need to know how those behave as well: as "local" subdirs, or do they behave as remote "root" dirs? in the latter case we'd also need to detect them as such.


Even while this bug is quite old is hasn't seen many confirmations; so I'd also agree with lowering priority/severity until we get more requests.
Repeating myself (see comment #13), having script or function files in root dirs is something that can be avoided.

- IMO on local fixed drives (HDs) it's a bad idea anyway.

- Only on thumb drives and other removable drives, and in some cases on mapped network drives, it can be more reasonable to keep m-files in the root dir.

Perhaps there is a way to detect a directory to be the root of a removable/network device and act accordingly along Jordi's suggestion.
Note that thumb drives, and to a lesser extent network drives, are usually slow; statting files may give a noticeable performance hit.
(I don't know if mapped network drives behave the same as "physical local" drives as regards directory time stamps.)

Philip Nienhuis <philipnienhuis>
Group Member
Sun 05 Jan 2014 05:12:45 AM UTC, comment #22: 

What if we just statted all of the files in the directory instead of only statting the directory? Is this a terrible slowdown, to stat all files in the path directories on every command invocation?

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Sat 04 Jan 2014 11:22:18 PM UTC, comment #21: 

Yes, confirmed here (I now have a winxp VM with 3.8.0 to test with). The root directory of an NTFS drive seems to have a constant timestamp:


octave:1> format long
octave:2> info = stat ('C:\')
...
    atime =  315550800
    mtime =  315550800
    ctime =  315550800

octave:3> ctime (315550800)
ans = Tue Jan 01 00:00:00 1980
octave:4> edit ('C:\foo.m')
octave:5> info = stat ('C:\')
...
    atime =  315550800
    mtime =  315550800
    ctime =  315550800


So yes this looks unavoidable given the current path caching scheme.

One possibility might be to watch for directory inode changes instead of looking at timestamps. This could use FindFirstChangeNotification on Windows and inotify on Linux-based systems. This would most likely be a major overhaul.

Mike Miller <mtmiller>
Group Member
Sat 04 Jan 2014 11:05:34 PM UTC, comment #20: 

This is all down to the same root cause which is that the directory timestamp is not updated when the directory itself is changed (such as by the addition of a new file).  On FAT filesysems, the directory timestamp is not updated period.  You can work around that by manually using touch.exe to change the timestamp of the directory.  Things are a bit better on NTFS systems, in that directories which are modified have their timestamp updated.  But, this being Microsoft, there are exceptions to that as well.  The root of the file tree is apparently special and just isn't updated.  I tried creating new files, new directories, using touch.exe and the timestamp of the root node is never modified.

The easiest way to see this is from within Octave.  Use


finfo = stat ('e:\')


and replace 'e' by whatever drive letter you are using.

So, the situation just sort of sucks on Microsoft file systems.

It's probably possible to disable the caching code in Octave (make sure this is only on Windows machines), but the performance hit might be unacceptable.  Basically, any time a function wasn't already compiled and in the symbol table, Octave would need to do a full search of every single directory on the path.  This would also happen everytime someone entered a typo as well like 'dirt' for 'dir' since Octave couldn't know that you didn't mean to call a function that it doesn't know about.


Rik <rik5>
Group administrator
Sat 04 Jan 2014 10:09:34 PM UTC, comment #19: 

Re-reading comments #7 and #8, you'll see that this is primarily about the directory timestamp, not necessarily the file timestamp. So Octave is expecting the timestamp of the containing directory to be updated. Do drive top-level directories have timestamps in Windows, and are they updated when new files are created at the top level?

What does


format long
info = stat ('X:\')


show before and after creating a new m-file?

Mike Miller <mtmiller>
Group Member
Sat 04 Jan 2014 10:02:24 PM UTC, comment #18: 

Forgot to add screen session pic; attached.


Philip Nienhuis <philipnienhuis>
Group Member
Sat 04 Jan 2014 10:00:55 PM UTC, comment #17: 

Sorry, I need to be a bit more exact:

do:
"start Octave"
(or "start octave-cli")
.... from that same cmd32 window.

FYI, "start octave" will detach an octave session from that command window. While you're fiddling in Octave, from that cmd32 window you can run "touch".

============
Anyway I've tried this myself with Octave-3.9.0+ and I get the same as the OP. Screen shot of session attached, just to show.
Like the OP I tried it on an NTFS drive (X: in my case)
"touch"ing the file (after Octave command #7-2) doesn't help.

BTW I'm intrigued by the error msg:

octave:7> type test2.m
error: type: couldn't open '' for reading


i.e. Octave seems to forget the file name although it was entered on the command line.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 04 Jan 2014 07:27:28 PM UTC, comment #16: 

@comment #15:

Au contraire, touch.exe does (extremely) probably exist but it's in your <OCTAVE_HOME>/bin directory. IOW in bin/ in the directory where you've installed Octave.
AFAIK it's included in all MXE-built MinGW 3.8.0 binaries.

You could (in a Command window, cmd32.exe) do something like:

set PATH=%PATH%;/full/path/to/Octave/directory/bin

and then start Octave (from that same cmd32 window).

Philip Nienhuis <philipnienhuis>
Group Member
Sat 04 Jan 2014 06:26:45 PM UTC, comment #15: 

The command touch does not exist on Windows

A substitute to this command is : copy /b file +,,
(found on http://stackoverflow.com/questions/51435/windows-version-of-the-unix-touch-command), but it doesn't solve the problem.



J-D Bonjour <jdb>
Sat 04 Jan 2014 05:47:13 PM UTC, comment #14: 

Does it work if you use touch.exe to update the file's modification time?

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Sat 04 Jan 2014 05:02:03 PM UTC, comment #13: 

Yes I'm following but a bit short of time lately - maybe tonight or otherwise tomorrow I'll check. & I also have to (re)build 3.8.0 first.

From what I've read so far I inferred that Octave relies on subdirectory timestamps, not individual function-/script-file timestamps.
Very probably -if not undoubtedly- for good reason (not hard to guess); but a drawback is this sort of issues.

A workaround is to avoid root dirs. Even on thumb drives (USB sticks) that would be a good way.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 04 Jan 2014 04:31:41 PM UTC, comment #12: 

Sorry, it doesn't work for me on an NTFS drive (see my session below) !
Philip Nienhuis, could you also check, please ?

Jean-Daniel


octave:1> ver

GNU Octave Version 3.8.0
GNU Octave License: GNU General Public License
Operating System: MINGW32_NT-5.1 Windows XP Service Pack 3 i686
% so I'm working on Windows XP with the 3.8.0 Octave bundle
% from http://octave.osuv.de/3.8/windows/octave-3.8.0-1-portable.zip

octave:2> system('chkdsk M: /i')

The type of the file system is NTFS
...
% so the filesystem of my M: drive is NTFS !

octave:3> cd M:\octave_test  % going to a sub-directory
octave:4> edit test1.m
% new file, inserting the line: disp('Hello from sub-dir of drive M:')
octave:5> test1

Hello from sub-dir of drive M:
% so on a sub-dir it works allways

octave:6> cd M:\   % going to the root of this NTFS drive
octave:7> edit test2.m
% new file, inserting the line: disp('Hello from root of drive M:')
octave:8> ls

[octave_test]  test2.m
% the created script is visible, but...

octave:9> type test2.m

error: type: couldn't open '' for reading
error: called from:
error:   ...\type.m at line 85, column 9

octave:10> test2

error: 'test2' undefined near line 1 column 1

% * And now restarting Octave *
octave:1> cd M:\
octave:2> test2

Hello from root of drive M:
% so it works after restarting Octave !

octave:3> type test2.m

test2.m is the user-defined function defined from: M:\test2.m
 
disp('Hello from root of drive M:')

octave:4> edit test2.m
% adding the line: disp('Second line added')
octave:5> test2

Hello from root of drive M:
Second line added


J-D Bonjour <jdb>
Sat 04 Jan 2014 04:14:46 AM UTC, comment #11: 

I just reformatted a USB memory stick to use NTFS instead of the default FAT32 and Octave works perfectly with it.  So the diagnosis about timestamps is accurate.

John, I was able to check the file system type by double-clicking "My Computer" to get a list of disk drives.  If I right-click on a disk drive and then click on Properties it brings up a new tabbed window.  In the General tab there is an entry called File System.

Rik <rik5>
Group administrator
Sat 04 Jan 2014 02:10:48 AM UTC, comment #10: 

I'm unable to reproduce the problem with Octave 3.8.0 running on Windows 7 in a virtual machine.  I tried the GUI and the CLI version.

I assume that this is with an NTFS filesystem, but I don't know for sure.  How do I check?

John W. Eaton <jwe>
Group administrator
Sat 04 Jan 2014 01:04:21 AM UTC, comment #9: 

No, sorry, the problem also occurs on NTFS partitions ! So it's not a good idea to lower the priority

J-D Bonjour <jdb>
Fri 03 Jan 2014 10:53:27 PM UTC, comment #8: 

This is definitely bad Microsoft mojo.  This microsoft support brief (http://support.microsoft.com/kb/299648) explains that NTFS directory timestamps are updated when directory contents change, but FAT formatted filesystems do not update directory timestamps when contents change.

My guess is that most modern installs of Windows have an NTFS partition for C:\ which explains why it works on the root partition.  Other drive letters are typically things like USB memory sticks which are probably formatted with FAT filesystems which explains why it tends not to work on them.

I'm lowering the priority for this bug.  Overhauling a huge amount of Octave directory code to deal with Microsoft's bad design is not my first choice of things that need improvement in Octave.

Rik <rik5>
Group administrator
Fri 03 Jan 2014 10:25:04 PM UTC, comment #7: 

Very definitely confirmed.  Octave seems to be caching the contents of a directory on start-up and then never updating the cache.  This could be down to differences in how the operating system updates the inode timestamp for the directory.

On Linux, creating a new file in a directory updates the timestamp.  I think Octave is comparing the current directory timestamp against the cached timestamp to figure out if it needs to go and re-read the directory.

On Windows, creating a new file in a directory does not change the timestamp--I know because I tried.  However, I have the UNIX tool touch.exe installed and when I touch the directory to bring the timestamp up to the current time then Octave correctly finds the new m-files.

This sequence worked:


cd e:\
mkdir tmptst
cd tmptst
edit junk.m
## Add some code to junk.m in the editor and then quit
which junk.m
## Returns nothing
## Now, in another shell window execute touch e:\tmptst
which junk.m
'junk.m' is the file e:\tmptst\junk.m


Seems like it might require a lot of O/S specific code to solve this one.

Rik <rik5>
Group administrator
Fri 03 Jan 2014 08:46:21 PM UTC, comment #6: 

Yes, this problem is still present on Windows. Tested now with the octave-3.8.0-1-portable.zip packaging distributed on http://octave.osuv.de/3.8/windows/

JDB

J-D Bonjour <jdb>
Fri 03 Jan 2014 09:09:56 AM UTC, comment #5: 

Thanks for reminding me, I'll check it out.

[Little bit OT]
A similar problem I hit with recent dev versions (3.7.7 & 3.9.0 and the two 3.8.0 release candidates) is that editing/debugging functions in private/ dirs leads to similar behavior (on MinGW):
Octave suddenly doesn't seem to be able to find certain private/functions anymore (while other private/ functions weren't affected). Adding the private subdit to the load path (as the GUI editor helpfully suggests) didn't help. Only a restart helps.
But as this behavior was erratic, and because I don't know the exact rules for the search path as regards private subdirs, I thought I was probably fubarring up myself so I didn't file a bug report.
After all, things work OK as long as functions in private subdirs aren't touched.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 02 Jan 2014 09:11:57 PM UTC, comment #4: 

Is this problem still present in Octave 3.8.0?

John W. Eaton <jwe>
Group administrator
Thu 17 Oct 2013 08:11:45 PM UTC, comment #3: 

Octave 3.7.7+ MXE GUI built for MinGW, installed on drive X, and Octave-3.6.7+ MXE CLI (ditto):

With either script or function files, it doesn't even work in a subdir of root:

octave-cli.exe:1> pwd
ans = X:\Octave\octave-3.7.6+_MXE
octave-cli.exe:2> cd f:/tmp
octave-cli.exe:3> edit script.m
octave-cli.exe:5> dir script.m
script.m
octave-cli.exe:6> script
error: 'script' undefined near line 1 column 1
octave-cli.exe:6> rehash
octave-cli.exe:7> script
error: 'script' undefined near line 1 column 1
octave-cli.exe:7> clear -f
octave-cli.exe:8> script
error: 'script' undefined near line 1 column 1
octave-cli.exe:8>
:
:
octave-cli.exe:8> cd /
octave-cli.exe:9> pwd
ans = f:\
octave-cli.exe:10> edit script2.m
octave-cli.exe:11> dir *.m
script2.m
octave-cli.exe:12> script2
error: 'script2' undefined near line 1 column 1
octave-cli.exe:12>


After restarting, Octave behaves even stranger:


octave-cli.exe:1> cd f:\tmp
octave-cli.exe:2> dir *.m
funcfil.m                  strread_bitw_regexp.m      textread.m
script.m                   strread_bitwidth.m         textscan.m
strread.m                  strread_bitwidth_regexp.m
octave-cli.exe:3> funcfil    ## Works OK
Hello world!

octave-cli.exe:4> script     ## does nothing
octave-cli.exe:5> type script.m
script.m is the user-defined function defined from: .\script.m

disp ("Hello World!\n")';

octave-cli.exe:6>
octave-cli.exe:7> cd /
octave-cli.exe:8> dir *.m
funcfil2.m  script2.m
octave-cli.exe:9> script2    ## Works OK;
Script 2

octave-cli.exe:10> funcfil2
octave-cli.exe:11> funcfil2
octave-cli.exe:12> type funcfil2
funcfil2 is the user-defined function defined from: f:\funcfil2.m

function [retval] = funcfil2 ()

        disp ("Funcfil2\n")';

endfunction

octave-cli.exe:13> type script2
script2 is the user-defined function defined from: f:\script2.m

disp ("Script 2\n");

octave-cli.exe:15>which funcfil2
'funcfil2' is a function from the file f:\funcfil2.m


Striking is that it works upon restart for functions but not for scripts in a subdir, and the other way round in the root dir.

Could these be MinGW bugs (in the gcc runtime libraries)?

Philip Nienhuis <philipnienhuis>
Group Member
Thu 17 Oct 2013 06:34:28 PM UTC, comment #2: 

Is this still present in the development version that is about to become the 3.8 release?

Rik <rik5>
Group administrator
Fri 03 Feb 2012 11:16:47 AM UTC, comment #1: 

Still present in 3.6.0 MinGW

Philip Nienhuis <philipnienhuis>
Group Member
Sun 19 Sep 2010 10:42:38 PM UTC, original submission:  

On Octave MinGW32 for Windows versions 3.2.0 to 3.2.4, when the user current directory is the root of a Windows partition other than C:\ (e.g. D:\, E:\ ...), the m-files (scripts or functions) created by the user during the session (with command "edit" or with other editors) are not usable : when invoqued, the following message is returned : error: '<script>' undefined near line xxx column 1.

Its very strange, because the command "dir" sees these m-files created during the session, and if I restart Octave I am able to use them... but not new scripts/functions created during the session.
 
Example :

> cd T:\test
> edit script.m
>    % creating script containing line:  disp('hello world !')
> script

hello world !

>    % so it works !
>
> cd ..  % going to the root of this filesystem T:\
> pwd

ans = T:\

> edit script2.m
>    % creating script containing line:  disp('hello again')
> script2

error: `script2' undefined near line 7 column 1

>    % doesn't work, but the script is here :
> dir

 .        ..       script2.m

> quit
>
>    % new Octave session
> cd T:\
> script2

hello again

>    % now it works, because it's a new Octave session


J-D Bonjour <jdb>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Carbon-Copy List
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by paredes (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by jbect (Posted a comment)
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by lachlan (Posted a comment)
  • -email is unavailable- added by genuinelucifer (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by jordigh (Updated the item)
  • -email is unavailable- added by jdb (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 30 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-10-03 nrjank Planned Release9.1.0 None
    2024-10-03 nrjank StatusNeed Info Confirmed
        Open/ClosedClosed Open
        Fixed Release9.1.0 None
    2024-10-03 nrjank StatusFixed Need Info
    2023-05-28 mmuetzel StatusNeed Info Fixed
        Open/ClosedOpen Closed
        Fixed ReleaseNone 9.1.0
        Planned ReleaseNone 9.1.0
    2023-05-15 philipnienhuis Carbon-Copy- Added philipnienhuis
    2021-10-29 philipnienhuis Carbon-CopyRemoved 78941 -
    2021-10-29 mmuetzel StatusFixed Need Info
        Open/ClosedClosed Open
    2021-10-29 rik5 Carbon-CopyRemoved 72865 -
    2019-02-23 mtmiller Carbon-CopyRemoved 80942 -
    2019-02-23 mmuetzel Open/ClosedOpen Closed
        StatusConfirmed Fixed
    2016-11-16 rik5 Dependencies- bugs #49203 is dependent
    2016-06-13 lachlan SummaryUser scripts or functions created during a session are not found User scripts or functions created during a session on Windows are not found
    2016-05-02 mtmiller Release3.8.0 dev
    2016-05-02 mtmiller Dependencies- bugs #47818 is dependent
    2015-11-24 rik5 Dependencies- bugs #44440 is dependent
    2014-01-20 mtmiller Dependencies- bugs #34731 is dependent
    2014-01-04 philipnienhuis Attached File- Added bug31080_session.png, #30188
    2014-01-03 rik5 Priority5 - Normal 3 - Low

    Back to the top

    Powered by Savane 3.14-fc07.
    Corresponding source code