bugGNU Octave - Bugs: bug #42242, Octave does not always execute...

 
 

bug #42242: Octave does not always execute most recently saved file in an NFS directory

Submitted by:  Jan <janw>
Submitted on:  Wed 30 Apr 2014 09:20:04 AM UTC  
 
Category:  GUI Severity:  3 - Normal
Priority:  3 - Low Item Group:  Incorrect Result
Status:  Need Info Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  GNU/Linux

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Fri 01 May 2020 11:56:26 PM UTC, comment #36: 

I have made extensive use of NFS. It has always had
problems with synchronisation. And, of course, if
participating systems do not have synchronized clocks
problems will occur that are not entirely the fault of
NFS. Various defensive measured have been used including
file close and read back after all writes.

Certainly, being sure to flush any buffers is necessary
in any case. And, all machines should be running ntp.
And NFS sync should always be used.

My experience suggests that including a file close and
reopen on NFS files may help but is not always reliable.

For full reliability I always ran everything locally with
a lock set and then transferred the results to the nfs
system.

Michael Godfrey <godfrey>
Project Member
Fri 01 May 2020 07:55:13 PM UTC, comment #35: 

The output in comment #34 still shows that the initial time query on the local system is returning a larger value than the mtime of the edited file by 78 seconds, even after the time required to edit the file:

      result from time: 1588360654.74920
  mtime of edited file: 1588360576

Are you sure ntp is running and has had time to synchronize the clocks?  As I recall, that may not happen immediately after starting the ntp service.

John W. Eaton <jwe>
Project Administrator
Fri 01 May 2020 07:21:55 PM UTC, comment #34: 

One of my systems wasn't running ntp, so I started it.  However,the problem is still with me, but clear -f fixes it.  See below.

>> nfsbug

10

>> time

ans =  1588360654.74920

>> stat (file_in_loadpath ("nfsbug.m"))

ans =

  scalar structure containing the fields:

    dev =  50
    ino =  360685
    mode =  33279
    modestr = -rwxrwxrwx
    nlink =  1
    uid =  1027
    gid =  100
    rdev = 0
    size =  16
    atime =  1588347181
    mtime =  1588360486
    ctime =  1588360486
    blksize =  8192
    blocks =  8

>> edit nfsbug.m
>> stat (file_in_loadpath ("nfsbug.m"))

ans =

  scalar structure containing the fields:

    dev =  50
    ino =  360685
    mode =  33279
    modestr = -rwxrwxrwx
    nlink =  1
    uid =  1027
    gid =  100
    rdev = 0
    size =  16
    atime =  1588347181
    mtime =  1588360576
    ctime =  1588360576
    blksize =  8192
    blocks =  8

>> nfsbug

10

>> clear -f
>> nfsbug

20

>>

Jon <accent22>
Fri 01 May 2020 05:57:09 PM UTC, comment #33: 

Those results do appear to show some skew in the system time and the file timestamps.  It seems to me that the mtime after editing the file should be larger than the value of the system time from before editing.

Why the difference?  I'm guessing the clocks on the two systems are not properly synchronized.  Are both systems running ntp?  Are the clocks tracking the network time correctly?

Maybe we can avoid these kinds of problems by doing what I suggested in comment #31.

But even if we make that change for Octave, I'd recommend ensuring that your systems are all correctly synchronized to network time, or at least synchronized among themselves.  NFS file systems work best when system clock skew is minimized.

John W. Eaton <jwe>
Project Administrator
Fri 01 May 2020 05:35:23 PM UTC, comment #32: 

I did as instructed and the results are below.  However, modifying the function changed its output.  (This is a one line script that's either fprintf('10\n') or fprintf('20\n'); )This is the same function used in my previous post.  The only change that I know that I did was to remove the "sync" in my mount options that I put in for the results in Comment #30.  Perhaps I am making an error?

>> nfsbug

20

>> time

ans =  1588353991.83605

>> stat (file_in_loadpath ("nfsbug.m"))

ans =

  scalar structure containing the fields:

    dev =  49
    ino =  360685
    mode =  33279
    modestr = -rwxrwxrwx
    nlink =  1
    uid =  1027
    gid =  100
    rdev = 0
    size =  16
    atime =  1588347181
    mtime =  1588347250
    ctime =  1588347250
    blksize =  8192
    blocks =  8

>> edit nfsbug.m
>> stat (file_in_loadpath ("nfsbug.m"))

ans =

  scalar structure containing the fields:

    dev =  49
    ino =  360685
    mode =  33279
    modestr = -rwxrwxrwx
    nlink =  1
    uid =  1027
    gid =  100
    rdev = 0
    size =  16
    atime =  1588347181
    mtime =  1588353983
    ctime =  1588353983
    blksize =  8192
    blocks =  8

>> nfsbug

10

>>

Jon <accent22>
Fri 01 May 2020 04:49:24 PM UTC, comment #31: 

Could someone who is experiencing this problem try the following and report the results here?

## Call your function.
my_fcn

## Get current Unix time.
time

## Get timestamp and other info about the file.
stat (file_in_loadpath ("my_fcn.m"))

## Edit the file and save changes.

## Get timestamp and other info about the modified file.
## Did editing the file change the mtime value?  How does
## it compare to the Unix system time above?
stat (file_in_loadpath ("my_fcn.m"))

## Check to see whether the old version is still called
## or the modified version is loaded.
my_fcn

Long ago, when first thinking about how to reload functions when timestamps changed, I thought to store the time when the function file was last parsed and compare that with the mtime of the file.  If the current mtime is newer than the time it was last parsed, then Octave reloads the function.  But it occurs to me now that maybe we should just store the mtime of the file when we read it and then when checking to reload, just compare that value to the current mtime for the file and if the values are different, reload the file.  That way we don't rely on filesystem timestamps matching up with the system time.  We only care whether the file timestamps have changed.  Is there any reason that wouldn't work?  Even if the timestamps go backward in time, then we should probably read the file again as something might have changed (file restored to some previous contents with a preserved timestamp, for example).

John W. Eaton <jwe>
Project Administrator
Fri 01 May 2020 03:45:21 PM UTC, comment #30: 

I added sync to my mount options for the NFS mounted directory.  I did a umount and then a mount.

I created a one line script and ran it.  I immediately modified it and ran it again.  Octave ran the original unmodified script.

In a terminal window, I immediately typed cat scriptname.m and cat displayed the source code of the modified script.

I ran the script about 20 times after typing cat and Octave was still displaying output from original script.

I typed clear -f, then immediately ran the script again.  It produced output from the modified script.  So clear -f likely fixed the problem.

Jon <accent22>
Fri 01 May 2020 02:43:45 PM UTC, comment #29: 

The problem seemed to be less likely, if NFS server and client are synced comment #15, but not solved comment #17.  Can you confirm that?

After saving the modified file in the Octave GUI editor, can you check with another editor, if the Octave GUI editor really flushed the modified content back?

Does it help to force Octave to update its internal symbol table?

clear -f

(https://octave.org/doc/v5.2.0/XREFclear.html) or

__parse_file__ ("/path/to/file.m")

Kai Torben Ohlhus <siko1056>
Project Member
Fri 01 May 2020 01:50:06 PM UTC, comment #28: 

I have this problem, running on Ubuntu right now.  I'm connecting to a NAS over NFS.  Making changes to a script means I just shut down Octave and restart it.  It's frustrating.

Matlab is working fine, but the conditions are not the same.  I'm running Matlab in Windows 10 on another computer and not using NFS.

If you'd like more information, just ask.

Jon <accent22>
Tue 02 May 2017 09:09:25 PM UTC, comment #27: 

This happens regularily on Windows, the only way to work reliably with Octave to me is to place a breakpoint, run, stop, and run again my programs form the GUI in order to have the right code executed. Debugging, if this bug is not known, could be very frustrating. Even if I fix it with run/stop/run it is a waste of time.

Alessio Ballotti <ballot>
Mon 21 Dec 2015 11:56:58 PM UTC, comment #26: 

I am encountering this error as well. I noticed this error only after using the same program for over an hour. Restarting Octave did not fix the issue. Have not tried restarting machine.

Changing variable and clicking save and run does not update the variables value in the Workspace Window, nor does it produce the correct results for the new variables in the command window. File is properly saved though.

Manually clearing the workspace using the command window allows the variables to be set to the proper value, but using the "Clear" command in the file does not. Clearing the workspace must be performed after each time the file is run.

This program performs simple multiplication and division using several variables.

Idc=10;
N=6.5;
lm=7.35*2;
lg=0.187;
ur=3000;

Bdc=0.4*pi*N*Idc*10^(-4)/(lg+lm/ur)

I am saving to my local machine, which is 64 bit Windows 7. Octave 4.0.0 Compiled for "i686-w64-mingw32".

Anonymous
Thu 29 May 2014 11:21:48 AM UTC, comment #25: 

It was helpful that you mentioned that you are
using NFS v4. There are a great many changes between
NFS v3 and v4. And, I have not really worked with v4,
so some of my NFS knowledge is irrelevant.

If the problem never happens with Matlab it is
more likely that it is an Octave problem.
The GUI side appears to be doing the right thing
so more checking of how the interpreter detects
updates could be useful.

Michael Godfrey <godfrey>
Project Member
Thu 29 May 2014 10:04:35 AM UTC, comment #24: 

Thank you for your comments!

> It s quite likely that this is still an NFS
> problem. Definitely more likely if NFS sync is not
> set.


We are using Ubuntu with NFS v4. NFS shares are always mounted with sync enabled.
Your are right, it might still be an NFS related problem, but on the other hand, we have this setup running for quite some time and other applications are fine (eg MATLAB).

Of course I am available for testing any patches or suggestions you provide. Thanks for looking into this!

Jan <janw>
Wed 28 May 2014 06:55:15 PM UTC, comment #23: 

The GUI supports external editors in the way that files are opened in this external browser. The "save and run" action is only available in the internal editor not for external editors.

Torsten Lilge <ttl>
Project Member
Wed 28 May 2014 06:49:50 PM UTC, comment #22: 

From Mike's comment #20:

Or, since the editor knows that the file has been modified, should there be
a way for it to indicate that to the interpreter, to invalidate the
interpreter's state or last-read timestamp of the file?
============================================

This could be an important point. How does the interpreter
find that the file has been changed? Does it ever
leave the file open? Or, does it always fclose and
reopen the file?  It should not be necessary for the
editor to tell the interpreter it has changed the file
(I hate flags). But the interpreter may need a better
check, and if all else fails, a "changed_flag."

And, does the interpreter check mtime while the file
is open? This may not be NFS-safe either.

Michael Godfrey <godfrey>
Project Member
Wed 28 May 2014 06:23:52 PM UTC, comment #21: 

The behavior below looks correct. But,
the fact that the flush (sorry, I looked for fflush)
was "successful" does not necesarily mean thet
NFS actually wrote the data back to the server and flushed
its buffers. This depends on NFS options, and,...

I thought that the GUI allowed external editiors like emacs.
Is this not true?

It s quite likely that this is still an NFS
problem. Definitely more likely if NFS sync is not
set.

I doubt if testing for flush fail will help on this
problem, but it could be worth implementing.
That said, I have not idea what should be done if a fail
is reported. This could happen if the NFS system has gome
offline, but then nothing will work anyhow.

Michael Godfrey <godfrey>
Project Member
Wed 28 May 2014 06:19:07 PM UTC, comment #20: 

Just to summarize, this bug was closed because the NFS server time was out of sync with the client machine's time and files had timestamps in the past when they were saved. Octave relies on file mtime to be updated and correct relative to host time.

Now the issue being raised is that Octave's built-in editor in the GUI saves the file, but the interpreter runs an old copy because it does not see the updated mtime. Since it is an integrated editor, and there is a "Save File and Run" button, this should probably be fixed in some way. Reopening and assigning to GUI.

So can this be fixed simply by trying to flush the file when saving from the editor and before returning contol to the interpreter?

Or, since the editor knows that the file has been modified, should there be a way for it to indicate that to the interpreter, to invalidate the interpreter's state or last-read timestamp of the file?

Mike Miller <mtmiller>
Project Administrator
Wed 28 May 2014 05:58:20 PM UTC, comment #19: 

The function that is used for all save actions contains the following code

// save the contents into the file
QTextStream out (&file);
QApplication::setOverrideCursor (Qt::WaitCursor);
out << _edit_area->text ();
out.flush ();
QApplication::restoreOverrideCursor ();
file.flush ();
file.close ();

From the Qt-docs:

  • QTextStream::flush (): Flushes any buffered data waiting to be written to the device.
  • bool QFile::flush (): Flushes any buffered data to the file. Returns true if successful; otherwise returns false.

The file is flushed but there is no check whether the flush was successful. Can it happen that a flush fails on NTFS directories? I can prepare a patch that adds that check. But what should be done if flush returns false?

Torsten Lilge <ttl>
Project Member
Wed 28 May 2014 01:31:45 PM UTC, comment #18: 

It appears that the Octave GUI relies on
the editor to flush its write buffers. (I am
not certain of this, but I could find no fflush
code in the GUI source.)

So, what editor(s) are you using and do
you think that the current text has really been
written back from the editor?

Also, I am not sure that the version of NFS that you are
using has been established.  I assume that it is V3.
Have you set NFS sync writeback? There is also a "safe
async writeback", but I do not think that it is
safe. Setting sync in the NFS mount options is safer.

Of course, even without the GUI there is the
opportunity for an editor to appear to contain
text different from what Octave would see. So,
it is possible for mismatch to occur outside of Octave's
control.

Michael Godfrey <godfrey>
Project Member
Tue 27 May 2014 03:42:00 PM UTC, comment #17: 

As a follow-up: I running the NFS server and client in sync since some time now.

But the problem is still present (occurs less often, though):
From time to time octave still executes an older version of the script. It might be that the NFS write operation has a delay in some cases, which causes this behaviour.

The problem is not that octave executes an older revision of a script in case of a write delay. Problem is that the file content, that is displayed in the octave editor, is not what is actually executed. This is misleading.

It would be nice if one could be sure, that octave actually executes the script as it is displayed in the editor.

Jan <janw>
Mon 05 May 2014 03:26:09 PM UTC, comment #16: 

Right. Not "this might cause the problem" but
this ALWAYS casues NFS failures, and not just for
Octave.

Michael Godfrey <godfrey>
Project Member
Mon 05 May 2014 02:26:03 PM UTC, comment #15: 

> Good to check local and remote times, of course, too.


Good point, server and client time are not in sync. This might cause the problem:

  • Before editing:

>> mytime = time

mytime =  1399299413.65183

>> info = stat ("skript_clustering.m")

info =

  scalar structure containing the fields:

    dev =  23
    ino =  98435089
    mode =  33204
    modestr = -rw-rw-r--
    nlink =  1
    uid =  1001
    gid =  1001
    rdev = 0
    size =  2775
    atime =  1399299360
    mtime =  1399299359
    ctime =  1399299359
    blksize =  1048576
    blocks =  8

  • editing file, octave still executs the previous version
  • After editing

>> info = stat ("skript_clustering.m")

info =

  scalar structure containing the fields:

    dev =  23
    ino =  98435089
    mode =  33204
    modestr = -rw-rw-r--
    nlink =  1
    uid =  1001
    gid =  1001
    rdev = 0
    size =  2775
    atime =  1399299360
    mtime =  1399299373
    ctime =  1399299373
    blksize =  1048576
    blocks =  8

>> mytime = time

mytime =  1399299431.89589

Now I have synchronised the system times of server and client and the problem seems to be solved.

Thanks a lot for your help!

Jan <janw>
Mon 05 May 2014 01:29:07 PM UTC, comment #14: 

Also, is the "Octave" read in a different thread
from the editing?

Michael Godfrey <godfrey>
Project Member
Mon 05 May 2014 01:27:29 PM UTC, comment #13: 

In omment #12 does ## edit the file imply
that the file has been flushed and closed at the end
of the edit?

Good to check local and remote times, of course, too.

Michael Godfrey <godfrey>
Project Member
Mon 05 May 2014 01:13:08 PM UTC, comment #12: 

And what about your system's local time value versus the file timestamps while editing the file? This is how Octave determines that a file has been modified and needs to be reloaded.

mytime = time
info = stat ("myfile.m")
## edit the file
mytime = time
info = stat ("myfile.m")

Mike Miller <mtmiller>
Project Administrator
Mon 05 May 2014 12:49:40 PM UTC, comment #11: 

Thanks for your comments, I did some tests:

- touching the file doesn't change the situation (ie. even if I touch the file, octave does not execute the most recent version)

- NFS is mounted with sync, not async

- File timestamps:

  • Before editing:

  scalar structure containing the fields:

    dev =  23
    ino =  98435089
    mode =  33204
    modestr = -rw-rw-r--
    nlink =  1
    uid =  1001
    gid =  1001
    rdev = 0
    size =  2775
    atime =  1399293568
    mtime =  1399293567
    ctime =  1399293567
    blksize =  1048576
    blocks =  8

  • editing file
  • after editing:

  scalar structure containing the fields:

    dev =  23
    ino =  98435089
    mode =  33204
    modestr = -rw-rw-r--
    nlink =  1
    uid =  1001
    gid =  1001
    rdev = 0
    size =  2775
    atime =  1399293568
    mtime =  1399293577
    ctime =  1399293577
    blksize =  1048576
    blocks =  8

  • type displays the correct new file content, but octave executes the old version

Please let me know if this is helpful and if I can provide additional information.

Jan <janw>
Thu 01 May 2014 02:39:04 PM UTC, comment #10: 

Jan,

More information will help. It is likely that
the problem is with how Octive does its I/O,
i.e. in a manner that is not "NFS-safe."
This happens a lot.

Michael Godfrey <godfrey>
Project Member
Thu 01 May 2014 10:39:06 AM UTC, comment #9: 

Thank you for your comments and suggestions! I'll report back next week...

I'd like to note that I don't think the problem is exclusively with NFS: I am running this configuration since years and never had similar issues with other applications (including ie. MATLAB).

Jan <janw>
Thu 01 May 2014 12:16:39 AM UTC, comment #8: 

Classifying the problem as as NFS issue will not
help to fix it. NFS behavior is fixed. The only
thing that can be done is to work around it. This
has been true for many years.

Checking clock sync on the system that is failing
is the first thing to do. Next is to ensure that
updates are correctly synced in the Octave code.
It may be necessary to introduce a wait while the
NFS write actually completes. NFS reports completion
before it is actually true (depending on the NFS
mount options -- these also need checking).

Many installations use async options on the mount command
for claimed efficiency reasons. This will guarantee
consistency problems.

But, this is hardly the beginning of the NFS story...

Oh, yes, worth checking if this is happenning under
NFS3 or 4.  4 is quite different from 3.

Michael Godfrey <godfrey>
Project Member
Wed 30 Apr 2014 11:59:11 PM UTC, comment #7: 

Agree with all of the above suggestions, this is likely some kind of NFS filesystem issue and not necessarily something that Octave is doing wrong.

Some more things you could try:

## Compare the atime, ctime, and mtime stamps before and after
## editing the script file with
format long
t = time
info = stat ("thescript.m")
## Confirm that Octave can read the new file content
type thescript

This might help point to whether it's that the updated saved file contents are not being read or if it's simply that the file's timestamp is not being updated or is skewed from the local clock.

Mike Miller <mtmiller>
Project Administrator
Wed 30 Apr 2014 09:49:35 PM UTC, comment #6: 

Something to also check is whether the NFS server clock
is correctly synced with all the clients. If this is not
true, chaos results.

Michael Godfrey <godfrey>
Project Member
Wed 30 Apr 2014 06:54:44 PM UTC, comment #5: 

Michael's advice about flushing NFS updates is good.  The other issue is the resolution of the NFS timestamp.  There was a similar issue on Windows machines using FAT filesystem where saves that occurred to close together were given the same timestamp.  This resulted in Octave believing the first save was up-to-date and so no load was required.

To test this you might try using 'touch filename' from a shell to see if that corrects things.

Rik <rik5>
Project Administrator
Wed 30 Apr 2014 04:07:25 PM UTC, comment #4: 

Everyone has trouble with NFS consistency.
There are various beliefs about how to avoid
the problem described here. It would be good to
check that updates are being flushed and that
there is only one process that does all the
open/read/write/close operations.

And, of course, there are detailed descriptions on the
web. Many are inaccurate, but some are helpful.

Michael Godfrey <godfrey>
Project Member
Wed 30 Apr 2014 02:02:25 PM UTC, comment #3: 

Hi Mike,
Thanks for the response.

> Does this bug depend on how long you wait before attempting to execute the script in Octave?


No, I think it does not depend on the time. For example, I save the script, wait a couple of seconds and then execute. It stills runs the old version.

> In other words, when it runs the old version of the file, if you wait a minute or two and try again without saving again or changing anything else, does it eventually run the new version?


I tried to wait up to five minutes, but it does not run the new version.

Do you have an idea about a quick fix or patch I could try? The current behaviour makes octave quite difficult to use for me.

Jan <janw>
Wed 30 Apr 2014 01:21:06 PM UTC, comment #2: 

And this is probably not GUI-related, the same behavior would probably happen in command-line mode as well.

Mike Miller <mtmiller>
Project Administrator
Wed 30 Apr 2014 01:19:47 PM UTC, comment #1: 

Thanks for your bug report. I don't have an NFS setup to reproduce this, but this behavior certainly seems possible. Does this bug depend on how long you wait before attempting to execute the script in Octave? In other words, when it runs the old version of the file, if you wait a minute or two and try again without saving again or changing anything else, does it eventually run the new version?

Mike Miller <mtmiller>
Project Administrator
Wed 30 Apr 2014 09:20:04 AM UTC, original submission:  

- Using Octave 3.8.1 on Linux 3.11.0-19-generic #33~precise1-Ubuntu

- Working with m-files stored on NFS filesystem

Steps to reproduce:

  • Use Octave GUI editor to create a simple m-file that produces an output to commandline or to a plot
  • The file is stored on NFS
  • Do the following multiple times:
    • Change a parameter in the script to trigger different output
    • Save file
    • execute using <F5> or GUI console
    • Observe ouput

At some point, the output does not change even if you change the parameters in the m-file. Octave does not execute the most recent version of the m-file, but some previous version.

This means that the m-code displayed in the editor is not what is actually executed by Octave. Very confusing!

I couldn't reproduce this with locally stored files, but only with file stored on NFS.

Please let me know if you need additional information.

Jan <janw>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by accent22 (Posted a comment)
  • -email is unavailable- added by ballot (Posted a comment)
  • -email is unavailable- added by ttl (Posted a comment)
  • -email is unavailable- added by godfrey (Posted a comment)
  • -email is unavailable- added by janw (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-01 mtmiller Carbon-CopyRemoved 80942 => -
    2018-04-27 mtmiller Priority5 - Normal => 3 - Low
        Release3.8.1 => dev
    2014-05-28 mtmiller CategoryInterpreter => GUI
        Item GroupOther => Incorrect Result
        StatusInvalid => Need Info
        Open/ClosedClosed => Open
    2014-05-27 rik5 Carbon-CopyRemoved 72865 => -
    2014-05-05 mtmiller StatusNeed Info => Invalid
        Open/ClosedOpen => Closed
    2014-04-30 mtmiller CategoryGUI => Interpreter
        SummaryGUI does not always execute most recently saved file (NFS) => Octave does not always execute most recently saved file in an NFS directory
    2014-04-30 mtmiller StatusNone => Need Info

    Back to the top


    Powered by Savane 3.5