bugGNU Octave - Bugs: bug #60122, Save and Run in editor does not...

 
 

bug #60122: Save and Run in editor does not always save

Submitter:  None
Submitted:  Fri 26 Feb 2021 01:48:14 AM UTC
   
 
Category:  GUI Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  Bill Eaton Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * 8.1.0
Operating System:  * Microsoft Windows Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 25 Mar 2023 01:04:18 PM UTC, comment #38: 

From: https://www.ntfs.com/ntfs_optimization.htm

<<<
Unnecessary Access Updates
NTFS automatically updates the last access time and date stamp on folders and files when NTFS traverses its B-tree folder structure. To enhance NTFS performance you can disable this behavior and reduce NTFS's operational overhead without significantly impairing functionality. In the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem Registry key, change the NtfsDisableLastAccessUpdate value of type REG_DWORD from the default value 0 (enabled) to 1 (disabled). This Registry value doesn't exist by default, so you need to enter it manually.

>>>


That is one thing to check.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 25 Mar 2023 03:02:04 AM UTC, comment #37: 

I just tried "stat" test on a directory on a D: drive (which is a usb stick with fat32 filesystem). The only way I can change the stats on directory is rename this directory, otherwise it all remains the same. This is on Win 11.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 25 Mar 2023 02:24:40 AM UTC, comment #36: 

"i'm pretty sure the underlying problem predates the gui"

I guess you could see the same problem if you used the edit function or if you used fopen/fprintf/fclose to generate a .m file on the fly.  The timestamp of the file is probably updated with sufficient resolution for us to notice the change in most cases.  But if I recall correctly, the load path code won't notice if writing a file doesn't also trigger a change in the timestamp of the containing directory.  That's because it will assume that nothing in the directory has changed and then will skip looking for new or modified files inside those directories.  If that is the problem, then we can avoid it by changing the load-path code so that we can mark files (or directories) individually as "modified" when we know that changes have been made.

If that's not sufficient (maybe an external editor writes a file in one of these directories for which the timestamp is not properly updated) then maybe we could try to detect which directories have this bad property and flag them for special treatment?  But I'm not sure how to do that without having write permission to the directory.

Or is there some other efficient solution that I'm not seeing?

John W. Eaton <jwe>
Group administrator
Sat 25 Mar 2023 01:19:38 AM UTC, comment #35: 

that would be useful, but i'm pretty sure the underlying problem predates the gui. I'll look through but I think people had issues with other file write operations not triggering the modified date.

Nicholas Jankowski <nrjank>
Group Member
Sat 25 Mar 2023 12:24:08 AM UTC, comment #34: 

"the new data was written to disk and accessible by other programs. Octave just didn't think it had to look for the change."

Maybe we need a way for the GUI to signal that a file has been written.  Then the interpreter can respond to that signal by  doing whatever is needed to tell the load path and/or function table that the function corresponding to the file should be considered out of date so that it will be parsed again.  Then Octave will not have to rely on directory timestamps being updated correctly.

John W. Eaton <jwe>
Group administrator
Fri 24 Mar 2023 08:31:26 PM UTC, comment #33: 

Reading "man fsync":
<<<
fdatasync()  is similar to fsync(), but does not flush modified metadata unless that metadata is needed in order to allow a subsequent data re‐
       trieval to be correctly handled.  For example, changes to st_atime or st_mtime (respectively, time of last access and time  of  last  modifica‐
       tion;  see  inode(7))  do not require flushing because they are not necessary for a subsequent data read to be handled correctly.  On the other
       hand, a change to the file size (st_size, as made by say ftruncate(2)), would require a metadata flush.

>>>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 08:17:08 PM UTC, comment #32: 

I believe qt flush does the same as C/C++ flush -- flushes iostream buffer to an abstract device.
As I understand FlushFileBuffers flushes OS buffers to the physical drive / persistent storage  (equivalent to "sync" in linux).
Notepad test confirms that the data is "there" (but it still could be in OS buffers), the question is: are all related  metadata updated as well?

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 07:48:20 PM UTC, comment #31: 

We are already using qt's flush function: https://doc.qt.io/qt-6/qtextstream.html#flush after having saved the file in the editor.

Torsten Lilge <ttl>
Group Member
Fri 24 Mar 2023 06:30:44 PM UTC, comment #30: 

adding - the previous testing using a separate program (like notepad++ or similar) showed that the new data was written to disk and accessible by other programs. Octave just didn't think it had to look for the change.

Nicholas Jankowski <nrjank>
Group Member
Fri 24 Mar 2023 06:28:52 PM UTC, comment #29: 

I'm not sure that would help.

Dug out one of the original reports:

https://savannah.gnu.org/bugs/?31080


it simply has to do with whether or not the OS/file system changes the folder Modified Date.  As long as that is how octave decides whether or not to check for new data, it will miss new data if the date is not updated consistently.

Nicholas Jankowski <nrjank>
Group Member
Fri 24 Mar 2023 05:54:29 PM UTC, comment #28: 

Both C:\ and D:\ are NTFS

comment #26:

> Just to make sure -- what is the filesystem on D: ?
>
> Dmitri.
> --
>

Ahmed Othman <ahmed_othman>
Fri 24 Mar 2023 05:49:05 PM UTC, comment #27: 

May be we should force flushing:

https://learn.microsoft.com/en-us/windows/win32/fileio/flushing-system-buffered-i-o-data-to-disk

<<<
An application can force the operating system to write the contents of these data buffers to the disk by using the FlushFileBuffers function. Alternatively, an application can specify that write operations are to bypass the data buffer and write directly to the disk by setting the FILE_FLAG_NO_BUFFERING flag when the file is created or opened using the CreateFile function.

>>>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 05:37:39 PM UTC, comment #26: 

Just to make sure -- what is the filesystem on D: ?

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 05:35:24 PM UTC, comment #25: 

Yes, the problem happens with both D:\ and its subfolders.

I tried many times again on C:\ and the problem doesn't happen.

comment #24:

> @Ahmed - some things I'd be curious about:
>
> on D:\, are you able to trigger the bug in either/both the subfolders and root folders?
>
> can you confirm the same does/doesn't happen on C:?
>

Ahmed Othman <ahmed_othman>
Fri 24 Mar 2023 03:57:59 PM UTC, comment #24: 

@Ahmed - some things I'd be curious about:

on D:\, are you able to trigger the bug in either/both the subfolders and root folders?

can you confirm the same does/doesn't happen on C:?

Nicholas Jankowski <nrjank>
Group Member
Fri 24 Mar 2023 03:05:15 AM UTC, comment #23: 

ok, thank you for the followup.  I'd have to hunt through the bug reports, but there definitely have been cases where, due primarily to how windows does/doesn't update the modified date on folders, and for some reason it is different on the system drive than on other drives, Octave misses file changes and assumes the local cache is still the correct version.  this is the first I had heard of that specific issue also applying to the Save As feature vs command line work, though.

Reopening report and bumping release.

Nicholas Jankowski <nrjank>
Group Member
Thu 23 Mar 2023 09:27:20 PM UTC, comment #22: 

I am on a local drive.
I am using a laptop with a primary SSD for the windows system (c:\) and an HDD for storage (D:\).
It seems that the problem happens only for D:\. But I am not 100% sure due to its random nature.

I tried the steps in comment #1. The external editor detects the changes and loads the modified file. But octave still gives results based on the old code even if I press run many times after that. I get results for the old code.


comment #21:

> ok thank you. 
>
> are you on a network or local drive?  is it your primary drive or a secondary drive, and are you working in the root folder of the drive? (these are a few factors we've seen come into play in different parts of octave regarding file saving)
>
> can you try the steps @ttl mentioned in comment #1? and describe whether some of the other things discussed in subsequent comments might be relevant to your situation?  (trying to determine if it's the same or a related issue)
>

Ahmed Othman <ahmed_othman>
Thu 23 Mar 2023 08:47:44 PM UTC, comment #21: 

ok thank you. 

are you on a network or local drive?  is it your primary drive or a secondary drive, and are you working in the root folder of the drive? (these are a few factors we've seen come into play in different parts of octave regarding file saving)

can you try the steps @ttl mentioned in comment #1? and describe whether some of the other things discussed in subsequent comments might be relevant to your situation?  (trying to determine if it's the same or a related issue)

Nicholas Jankowski <nrjank>
Group Member
Thu 23 Mar 2023 08:33:49 PM UTC, comment #20: 

I make modifications to some code and press run.
The older version of the code (before modifications) is the one that is actually running (according to the results).
The mark next to the file name indicates that the changes have been saved, but the results are those corresponding to the older code.
Even if I press run again many times, the older code executes until I make some more changes and press run again and then it executes the new code.

For example, I made a file named test.m with just one line of code "x=5". It runs fine. If I change it to "x=2", I get x=5 in the terminal, even if I press run multiple times, I still get x=5, until I modify it again by adding an extra space for example,  it runs fine and I get x=2.

This happens randomly (most of the time it works fine). But it is annoying especially that sometimes I don't notice and get wrong results.

I noticed that it usually happens when I leave the code for a few minutes between changes and doesn't happen if I make the modifications fast.

I am using octave 8.1.0 release on windows 10 (actually it always for me on octave 6 and 7) and the files are saved locally on the hard drive.

comment #19:

> @Ahmed - we haven't seen reoccurring reports of the same issue described in the original post.  can you please add some detail to exactly what you're seeing happen on your system?


Ahmed Othman <ahmed_othman>
Thu 23 Mar 2023 02:43:15 PM UTC, comment #19: 

@Ahmed - we haven't seen reoccurring reports of the same issue described in the original post.  can you please add some detail to exactly what you're seeing happen on your system?

Nicholas Jankowski <nrjank>
Group Member
Thu 23 Mar 2023 12:09:49 PM UTC, comment #18: 

Error still happens in octave 8.1.0 release on windows

Ahmed Othman <ahmed_othman>
Sun 29 May 2022 12:36:51 AM UTC, comment #17: 

It's been a month since the last changeset was pushed.  I'm assuming everything works and marking this as Fixed.

Rik <rik5>
Group administrator
Wed 27 Apr 2022 05:17:42 PM UTC, comment #16: 

Pushed the patch to the default branch with http://hg.savannah.gnu.org/hgweb/octave/rev/ba701853b1bf

Torsten Lilge <ttl>
Group Member
Fri 22 Apr 2022 03:20:16 PM UTC, comment #15: 

curious if this change has been effective.  I assume it didn't make it into 7.1.0 gui?

Nicholas Jankowski <nrjank>
Group Member
Mon 22 Nov 2021 08:39:21 PM UTC, comment #14: 

Actually not, I lost sight of bug report. I will test this in the next couple of days.

Torsten Lilge <ttl>
Group Member
Mon 22 Nov 2021 01:41:19 AM UTC, comment #13: 

@Torsten: Did you try jwe's suggestion from comment #11?  Does it help?

Rik <rik5>
Group administrator
Wed 08 Sep 2021 12:52:40 AM UTC, comment #12: 

This problem also happens in Linux when working from a CIFS share.

Bill Eaton <wpeaton>
Tue 11 May 2021 06:43:02 PM UTC, comment #11: 

Torsten:  You might also try this change:


diff --git a/libgui/src/m-editor/file-editor.cc b/libgui/src/m-editor/file-editor.cc
--- a/libgui/src/m-editor/file-editor.cc
+++ b/libgui/src/m-editor/file-editor.cc
@@ -666,6 +666,11 @@ namespace octave
        {
          // INTERPRETER THREAD

+         // Act as though this action was entered at the command propmt
+         // so that the interpreter will check for updated file time
+         // stamps.
+         Vlast_prompt_time.stamp ();
+
          tree_evaluator& tw = interp.get_evaluator ();

          if (tw.in_debug_repl ())


I'm not sure that is the correct place or if it would be better to put it in the interpreter_event lambda function in main_window::run_file_in_terminal.  Maybe it is needed in several more locations in the GUI?  Maybe it is even something that should be done for all these interpreter callback events that are the result of user interaction.  After all, these are similar to user inputs at the command prompt, but entered using a button in the GUI instead of a command in the terminal widget.

John W. Eaton <jwe>
Group administrator
Tue 11 May 2021 06:20:39 PM UTC, comment #10: 

In comment #5, Rik wrote:


Now, when you go to run a function a second time Octave first looks into it's cache of parsed code to see if the function exists.  If it does, it then looks at the timestamp of the m-file and compares it to the recorded timestamp.  If they are the same, nothing has changed and Octave can go ahead and use the cached code.  Otherwise, it has to go out to the disk, parse the new code, and build the internal representation which it then runs.


The check for an out of date file (timestamp of file, obtained using a stat(2) call) only happens if Octave has returned to the command prompt for input since the last time the file timestamp was checked.  If you are just using the GUI editor and the "run" button there to save and execute a script, then the prompt timestamp won't be changing, so Octave won't actually check the timestamp on the file itself.

We added this check on the timestamp to avoid slowing down the interpreter while it was just running code.  You don't want to be  calling stat every time a function is executed.

We also rejected the option of scanning the function table and loadpath at each prompt, or on demand as needed at other locations in the interpreter.  Though this solution seems simple, it could make the action of issuing the prompt appear quite sluggish if the loadpath contained network drives or large directories.

Is there a better way?

John W. Eaton <jwe>
Group administrator
Sat 13 Mar 2021 11:34:42 PM UTC, comment #9: 

100 ms refresh sounds about right.

Rik <rik5>
Group administrator
Sat 13 Mar 2021 09:04:20 PM UTC, comment #8: 

I plan to test the timestamp immediately after saving and start the repeated testing only if this test is negative. However, I expect that this first test will already be positive in the majority of cases. But you are right, the "sampling time" for the repeated test could also be 100ms. If the timestamp did not change after 2s (too short?), 'clear -f FUNCTION_NAME' could be carried out before running the file.

Torsten Lilge <ttl>
Group Member
Sat 13 Mar 2021 08:18:43 PM UTC, comment #7: 

@Torsten: That would probably work, and would get around the issue of nested functions and subfunctions not being cleared.

I bet you want to make the repeat timeout shorter than 500 ms.  People are going to be impatient when they click "Save and Run" and a half a second seems laggy.

Rik <rik5>
Group administrator
Sat 13 Mar 2021 06:46:51 PM UTC, comment #6: 

Would it be safer to check for the changed file timestamps after saving and before running the file? This could be done by reading the file stat and repeat this every 500 ms or so until the timestamp has changed or a timeout is reached.

Torsten Lilge <ttl>
Group Member
Fri 12 Mar 2021 11:03:35 PM UTC, comment #5: 

This could be a problem specifically with Windows and, possibly, specifically with networked drives.

When you first run an m-file, Octave parses it and stores an internal representation of the code.  It also records the last modification time of the file.

Now, when you go to run a function a second time Octave first looks into it's cache of parsed code to see if the function exists.  If it does, it then looks at the timestamp of the m-file and compares it to the recorded timestamp.  If they are the same, nothing has changed and Octave can go ahead and use the cached code.  Otherwise, it has to go out to the disk, parse the new code, and build the internal representation which it then runs.

There are two issues with your current setup.  First, some Windows file systems (for example, VFAT) store timestamps with only 2-second resolution (see https://unix.stackexchange.com/questions/470856/timestamps-of-files-copied-to-usb-drive).  So if you save a file, make a quick change, and then save again Octave may not even notice that a change has been made.  If you can, you might check if you can use NTFS as a file system since it has 100 nanosecond resolution.

Second, if the drive is networked there is going to be some time between issuing a save command and when the timestamp on the disk is actually modified.  It could be that Octave issues a save command to the operating system, which returns quickly because it has sent the command off on the network.  Octave then goes to run the script which begins by checking the timestamp.  Maybe the operating system returns the old timestamp before the write command is executed.  This seems perfectly plausible because once you go on to an IP network order of packets is not guaranteed.

There is a way that we could address this on our side.  The sequence of commands is

1. Save
2. Run

Since the expectation, intention is that the programmer must have made a change and that is why the Save operation is required, it should be okay to invalidate Octave's local cache and force the file to be re-parsed.  In this case, the command sequence would be

1. 'clear -f FUNCTION_NAME'
2. Save
3. Run

It's not perfect if the m-file has nested function or subfunctions, since those should be cleared as well.  But maybe this is a step in the right direction.

Rik <rik5>
Group administrator
Mon 08 Mar 2021 11:00:26 PM UTC, comment #4: 

Whoa. The behavior is even stranger than I thought. Editor saves the file. The file is actually saved. A second editor will prompt me for reload. Modification icon does disappear. But then Octave runs an old copy of the script.

If instead, I press the Save button and then press the Save and Run button, the corrected version of script runs. That's what I've been doing as a workaround.

The only question left now is whether this behavior is only for network drives.

Bill Eaton <wpeaton>
Mon 08 Mar 2021 09:07:47 PM UTC, comment #3: 

Just to be clear. Having the file in question open in a second editor is just for testing whether this editor detects a file change (what octave then is not doing) or whether the file is actually not saved by the octave editor.

Torsten Lilge <ttl>
Group Member
Mon 08 Mar 2021 09:00:50 PM UTC, comment #2: 

Torsen Lilge,

These are some good questions. Right now I can only answer 1. I'll do some more work to figure out 2 or to edit in another editor.

So far I have been doing all of my work with files on a network drive. I will do some more work from local drive and also to see if modification icon disappears after Save & Run button is pressed.

Bill Eaton <wpeaton>
Sat 06 Mar 2021 05:31:54 PM UTC, comment #1: 

Thanks for the report. Two questions:

  1. Is the file to be saved located on a local or network drive?
  2. You are saying that the file is not saved. But the modification icon on the editor tab is disappearing?


Could you open the file in an external editor which automatically reloads modified files or at least detects a mofication? When you edit and save the file in Octave's editor, does the external editor reloads the new version or detects the file change?

Torsten Lilge <ttl>
Group Member
Fri 26 Feb 2021 01:48:14 AM UTC, original submission:  

The save and run button in the editor sometimes fails to save and runs an older version of the script. This also applies to 6.1.0.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by ahmed_othman (Posted a comment)
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by wpeaton (Posted a comment)
  • -email is unavailable- added by ttl (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-03-24 nrjank Open/ClosedClosed Open
    2023-03-24 nrjank StatusNeed Info Confirmed
        Release6.2.0 8.1.0
    2023-03-23 rik5 Carbon-CopyRemoved 72865 -
    2023-03-23 nrjank StatusFixed Need Info
    2022-05-29 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2022-04-27 ttl StatusNone Ready For Test

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code