bugGNU Octave - Bugs: bug #53539, GUI file browser date stamps /...

 
 

bug #53539: GUI file browser date stamps / editor script file appear modified when parsing date strings

Submitted by:  Lars Kindermann <larskindermann>
Submitted on:  Sun 01 Apr 2018 02:41:59 AM UTC  
 
Category: GUISeverity: 3 - Normal
Priority: 5 - NormalItem Group: Unexpected Error
Status: ConfirmedAssigned to: None
Originator Name: Open/Closed: Open
Release: devOperating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Fri 20 Apr 2018 08:36:12 PM UTC, comment #13:

Currently the QFileSystemWatcher is used for scanning open files for changes outside the editor. A solution might be to keep track of the file modification date in UTC and check this date when the fileChanged signal from the watcher is received.

Torsten <ttl>
Project Member
Fri 20 Apr 2018 07:37:31 PM UTC, comment #12:

@Mike, if you plan to use any getenv(TZ) in the the gui, be aware that using setenv() makes also getenv() thread unsave.

The problem is not non-reentrant code but setenv() changes the pointer to the env-string which can cause getenv() to segfault.

Here is a good explanation, especially the part about TZ:
https://rachelbythebay.com/w/2017/01/30/env/

Lars Kindermann <larskindermann>
Fri 20 Apr 2018 06:47:36 PM UTC, comment #11:

If there is a reentrant bug with calling setenv in a multi-threaded program, then we will work around that as needed.

As it is, setenv is a function that is provided to the interpreter for user functions to call, and I don't see that changing. And the Octave interpreter itself is single-threaded.

I see no problem with setting TZ to UTC0 temporarily while strptime is called and restoring it. If there is any problem, it is that the scope could be reduced a bit so it's only around the strptime function.

The bigger problem is that the GUI is reacting to this change, which is giving a wrong alert that files have been modified. That should be fixed regardless of whether the datevec function is changed.

Mike Miller <mtmiller>
Project Administrator
Fri 20 Apr 2018 06:30:55 PM UTC, comment #10:

I think the culprit that really has to be avoided is

setenv ("TZ", "UTC0")

because "Modifications of environment variables are not allowed in multi-threaded programs". In glibc the setenv() function is MT-unsafe. See

https://www.gnu.org/software/libc/manual/html_node/Environment-Access.html#Environment-Access

But I'm aware why Mike has introduced the temporary switch to UTC timezone. strptime() is otherwise almost unusable. It is even recommended to do so in several articles on using time related functions. But serious problems arise in multi-threaded programs. Imagine what happens if several threads tamper uncoordinated with the TZ variable...

Maybe the only 100% safe and correct way is to write an own replacement of strptime, which does not care about timezones and daylight saving at all. Perhaps the code from glibc can be stripped down to the bare minimum.

Lars Kindermann <larskindermann>
Fri 20 Apr 2018 05:28:50 PM UTC, comment #9:

Yes, it's related. See bug #49119 for the problem about the GUI editor opening files when running the test suite.

The new wrinkle is the time stamp / file reload message, since the tests in scripts/time set and reset the TZ environment variable.

It occurs to me this could be avoided if the time checks being done on whether a file is modified are made time-zone-aware, or done with time_t values. So maybe the code that checks whether a file has been modified needs to be reviewed.

Mike Miller <mtmiller>
Project Administrator
Fri 20 Apr 2018 03:58:52 PM UTC, comment #8:

I don't know whether it is related, but if I start the GUI and execute _run_test_suite_, several files are opened in the editor and I eventually dialog windows pop up with warnings about files changed by other applications and asking whether I want to reload them.

John W. Eaton <jwe>
Project Administrator
Sat 14 Apr 2018 02:00:42 PM UTC, comment #7:

The shown columns in the file browser can be selected in a context menu after right mouse click into the header of the file list.

There is a preference for the editor in the settings dialog that allows to automatically reload changed editor files without prompting. Maybe this might be a workaround.

Torsten <ttl>
Project Member
Tue 10 Apr 2018 07:59:43 AM UTC, comment #6:

This makes the gui almost unusable for dealing with large amounts of timestamped datapoints imported from text files. (What I mainly use Octave for)

As a temporary workaround, starting Octave with the command

avoids the permanently appearing "file in editor changed" popup.

Lars Kindermann <larskindermann>
Tue 10 Apr 2018 06:50:54 AM UTC, comment #5:

commenting out the line

in datevec.m completely gets rid of the messages about changed files in the editor and flipping timestamps.

But I have no idea how to use strptime() without this hack.

Perhaps creating a mutex for a strptime wrapper and the relevant gui code?

Lars Kindermann <larskindermann>
Tue 10 Apr 2018 06:50:30 AM UTC, comment #4:

Thanks, I tried clicking on the gear and I tried the Preferences dialog, I couldn't find a place that configured which columns are shown in the file browser.

I thought it may have to do with the datevec code that changes the time zone (which I wrote). I can confirm this behavior by running only 'test datevec' repeatedly in the command window, with any script file open in the editor.

Confirmed, and I don't have any ideas on solutions, other than improving the GUI in a future version to not poll file modification times as aggressively, or while Octave functions are running.

Mike Miller <mtmiller>
Project Administrator
Tue 10 Apr 2018 06:17:14 AM UTC, comment #3:

I have localized the probable source of the problem.

It is indeed caused by massive use of datenum() in my scripts. Here is the minimum code that triggers the error 100% reproducible when there is a large amount of big datafiles to be parsed:

There seems to be a race condition between the interpreter and the gui caused by code in datevec.m:

When the gui checks the timestamp of files, either of those open in the editor or those displayed in the filebrowser while the TZ environment variable is temporarily modified by that code, it assumes the files have changed and redraws the timestamps, which are exactly off by the local TZ then.

The unwind_protect assures that TZ is definitely reset but does not protect against concurrent code.

@Mike: You can turn on the display of filetimes in the filebrowser widget by right-clicking the header line...

The zero size octave-workspace is not the cause, but for some reason it increased the probability of the ocurrence for me.

Lars Kindermann <larskindermann>
Mon 09 Apr 2018 10:59:30 PM UTC, comment #2:

I think you are seeing a couple different effects here that may or may not be related. Let's try to separate them and investigate them individually.

The pop up message saying that the script has been modified is probably due to the script itself changing directories or somehow modifying the path while it is running. I observed the same when running the '__run_test_suite__' command from the editor:

The second effect with timestamps - where are you seeing timestamps changing? The file browser doesn't show timestamps on files for me. What widget shows this or what message or behavior makes you think timestamps are changing?

Can you give an example of the type of script you are running that shows this behavior? Is your script doing anything with Octave's date/time functions?

Mike Miller <mtmiller>
Project Administrator
Mon 09 Apr 2018 10:38:00 PM UTC, comment #1:

This effect is caused by an empty file (0 bytes) called "octave-workspace" in the current folder, which gets occasionally created during a crash.

To trigger it on linux do a

and run a script in the gui which does some file io for a while.

Lars Kindermann <larskindermann>
Sun 01 Apr 2018 02:41:59 AM UTC, original submission:

While running scripts which create some system load for a while two things happen after last pull for current dev:

Frequently a message pops up, incorrectly stating that the file in the editor has been changed externally on disk.

The timestamps in the filebrowser start to change continuously and show wrong times for some files.

Both effects stop when returning to the command prompt.

Some of the timestamps flip exactly two hours forward and back, which is also my UTC offset. Others do show other wrong times...

System: Xubuntu 16.04, ext4 filesystem.

Lars Kindermann <larskindermann>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by ttl (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by larskindermann (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 11 Apr 2018 12:27:04 AM UTCmtmillerSummaryGUI file browser date stamps / editor script file has been modified while using datenum / datevec=>GUI file browser date stamps / editor script file appear modified when parsing date strings
    Tue 10 Apr 2018 06:50:30 AM UTCmtmillerItem GroupOther=>Unexpected Error
      StatusNeed Info=>Confirmed
      SummaryTimestamps in filebrowser continuously change under heavy load=>GUI file browser date stamps / editor script file has been modified while using datenum / datevec
    Mon 09 Apr 2018 10:59:30 PM UTCmtmillerStatusNone=>Need Info

    Back to the top


    Powered by Savane 3.1-cleanup1