bugGNU Octave - Bugs: bug #59711, Performance of `cd` is bad on...

 
 

bug #59711: Performance of `cd` is bad on Windows

Submitter:  Renato <yamane>
Submitted:  Thu 17 Dec 2020 04:46:58 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 6.1.0 Operating System:  * Microsoft Windows
Fixed Release:  9.1.0 Planned Release:  9.1.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 05 Aug 2023 01:31:26 PM UTC, comment #31: 

Remembered to test this with a recent build from the default branch using MXE Octave to cross-build.
With that version, it takes about 1 second to load ltfat for me.
With Octave 8.2.0, it takes between 3.4 and 3.9 seconds to load the same package on the same computer.

I'd consider this a "good enough" improvement. Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Mon 22 May 2023 03:11:27 PM UTC, comment #30: 

The previous change didn't work for files in the root directory of a drive on Windows. I pushed this additional change that should fix that:
https://hg.savannah.gnu.org/hgweb/octave/rev/53e15e754725

Markus Mützel <mmuetzel>
Group administrator
Sat 13 May 2023 11:52:18 AM UTC, comment #29: 

I pushed a modified version of the patch in comment #28 where I tried to move more platform dependent code from the headers to .cc files:
http://hg.savannah.gnu.org/hgweb/octave/rev/7d663f770c5a

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Thu 11 May 2023 06:18:11 PM UTC, comment #28: 

The attached patch is a imho nicer implementation of the changes in comment #27. It introduces a new file_time class to abstract away most of the platform specific code in load-path.cc.
There are still a few platform specific parts due to the fact that `file_stat` is very fast on Linux but terribly slow on Windows. So, we'd like to continue using in on the former platform but try to avoid it (at all costs) on the latter.


(file #54727)

Markus Mützel <mmuetzel>
Group administrator
Sun 07 May 2023 12:13:45 PM UTC, comment #27: 

With the attached patch, `pkg load ltfat` is notably faster on Windows compared to Octave 8.2.0.
However, it's a very dirty hack.

Have to think how to better abstract that. If someone has an idea how to do that, please let me know.


(file #54709)

Markus Mützel <mmuetzel>
Group administrator
Sun 07 May 2023 10:00:37 AM UTC, comment #26: 

I managed to compare the latest version to Octave 8.2.0.
There seems to be a very slight difference only.

I attached a profiler to both versions:
With Octave 8.2.0, `octave_stat_wrapper` is call 68 times while executing `pkg load ltfat`.
With the latest development version, that number reduced to 58 times.

The majority of the remaining calls seems to be come via `subdirs_modified` in "load-path.cc".
It's not as straight forward to replace that one because it does not only check if a file exists. But it also uses other fields from the stat structure to check if a file (or folder in this case) was modified since the last time we checked.

There are native Windows functions that are probably more efficient than stat on that platform, e.g., `GetFileTime`:
https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfiletime

But the FILETIME structure that function uses is not directly compatible to the fields that we use in octave::sys::time.
Not sure how to abstract that to something platform independent (that can still be used efficiently on any platforms and doesn't pollute the code with a lot of `#if defined`).

Maybe, I'll just try and hack something together to check if this will really make a difference before thinking about a nice implementation...

Markus Mützel <mmuetzel>
Group administrator
Sat 06 May 2023 12:48:34 PM UTC, comment #25: 

I pushed a couple of patches here that use the new functions where I thought that would be appropriate:
https://hg.savannah.gnu.org/hgweb/octave/rev/632f9b828de1 (plus a fix for that: https://hg.savannah.gnu.org/hgweb/octave/rev/b0785d76430a )
https://hg.savannah.gnu.org/hgweb/octave/rev/e9fdfebc6db0

I tried to test if that makes a difference for the load time of ltfat. But I'm having trouble installing that package on a native build from the default branch on Windows. (Maybe a bad configuration on my end.)

Maybe, a version that was cross-built from Linux might work. After all, `ltfat` is one of the bundled packages.

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Sat 06 May 2023 08:25:20 AM UTC, comment #24: 

I finally came around to starting to clean up the changes I had lying around concerning this.
I pushed a first change here:
https://hg.savannah.gnu.org/hgweb/octave/rev/ed3a18fe328a

This change doesn't make much of a difference yet. It just adds functions that should be more efficient to check if files or directories exist cross-platform.
In the next steps, we should identify where we are currently using fstat and where we could possibly replace its use with the new functions.

Markus Mützel <mmuetzel>
Group administrator
Tue 29 Nov 2022 12:17:53 PM UTC, comment #23: 
Dmitri A. Sergatskov <dasergatskov>
Tue 29 Nov 2022 12:11:35 PM UTC, comment #22: 

I can confirm a significant difference in the time for loading the ltfat package between Octave 6.4 and Octave 7.1.

Re-opening report.

I attached a profiler (Very Sleepy). And the bottleneck no longer seems to be `canonicalize_file_name`, but `stat`.

`stat` is not a native Windows API function. It takes a lot of API calls to fill the entire stat structure when we are often times only interested in whether a file exist or whether it is a "simple" file or a directory.

I had a patch lying around somewhere in which I tried to add simple wrappers to check whether a file or directory exists (which hopefully would perform better on any platform). I guess it's time to dig that up again, check if it still applies and maybe try and finish it...

Markus Mützel <mmuetzel>
Group administrator
Tue 29 Nov 2022 11:46:12 AM UTC, comment #21: 

The problem is back again on Octave 7.3

Renato <yamane>
Tue 19 Jan 2021 08:00:49 PM UTC, comment #20: 

Alright, the regression in performance seems to have been fixed and this bug report can be closed.

Rik <rik5>
Group administrator
Tue 19 Jan 2021 08:00:46 PM UTC, comment #19: 

Thanks for those tests.
IIUC, the performance seems to be acceptable for this use case again.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Tue 19 Jan 2021 07:02:19 PM UTC, comment #18: 

rerunning the old ones also to better capture any local variance. looks like recent build gets times down around where they were in 5.2.0:

Octave 6.1.1, Jan 18 build, ltfat 2.3.1:

>> profile clear; profile on; pkg load ltfat; profile off; profshow
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 141                     cd             0.663      18.65           41
  77                addpath    R        0.640      18.00           22
 117           is_same_file             0.560      15.76         2134
 102                    dir             0.295       8.30           35
  79             ltfatstart             0.234       6.59            1
  51 canonicalize_file_name             0.211       5.93          130
 142                 source             0.177       4.99           20
 146             javaMethod             0.163       4.59            3
 115                  lstat             0.152       4.28         1057
  17              regexprep             0.044       1.24         2155
  46     installed_packages             0.028       0.78            1
 107                 strcat             0.027       0.76           35
   1                    pkg             0.026       0.72            1
 135              fileparts             0.024       0.66           20
 134                    run             0.022       0.62           20
 144          blockprocinit             0.020       0.57            1
 173       ltfatsetdefaults             0.019       0.55            1
 145          javaclasspath             0.016       0.44            1
  47                   load             0.015       0.43            2
  88                  exist             0.015       0.42           59

Octave 6.1.1, Jan 07 build, ltfat 2.3.1:

>> profile clear; profile on; pkg load ltfat; profile off; profshow
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 141                     cd             2.171      36.98           41
  77                addpath    R        1.411      24.03           22
 117           is_same_file             0.598      10.19         2134
 102                    dir             0.272       4.63           35
 142                 source             0.231       3.94           20
  79             ltfatstart             0.230       3.91            1
  51 canonicalize_file_name             0.216       3.67          130
 146             javaMethod             0.161       2.74            3
 115                  lstat             0.154       2.63         1057
  17              regexprep             0.045       0.77         2155
  47                   load             0.031       0.52            2
 135              fileparts             0.026       0.45           20
 134                    run             0.025       0.43           20
  45          load_packages             0.023       0.39            1
 107                 strcat             0.019       0.33           35
  46     installed_packages             0.017       0.28            1
   1                    pkg             0.016       0.27            1
  88                  exist             0.015       0.26           59
 106                readdir             0.012       0.21           35
 139                 rindex             0.012       0.20           20

Octave 6.1.0, ltfat 2.3.1:

>> profile clear; profile on; pkg load ltfat; profile off; profshow
   #                       Function Attr     Time (s)   Time (%)        Calls
-----------------------------------------------------------------------------
 139                             cd             8.990      40.88           41
  74                        addpath    R        6.839      31.10           22
 113                          lstat             2.454      11.16         1057
 115                   is_same_file             0.542       2.46         2134
 100                            dir             0.493       2.24           35
 140                         source             0.479       2.18           20
  76                     ltfatstart             0.418       1.90            1
  85                          exist             0.209       0.95           59
  27         canonicalize_file_name             0.206       0.94          130
 144                     javaMethod             0.173       0.78            3
  70                           stat             0.130       0.59           59
 133                      fileparts             0.123       0.56           20
 132                            run             0.115       0.52           20
 104                        readdir             0.115       0.52           35
 103                      __wglob__             0.089       0.40           35
 105                         strcat             0.071       0.32           35
 137                         rindex             0.062       0.28           20
  58                      regexprep             0.045       0.21         2152
  49 load_packages_and_dependencies             0.041       0.19            1
  20                  load_packages             0.037       0.17            1

Octave 5.2.0, ltfat 2.3.1:

>> profile clear; profile on; pkg load ltfat; profile off; profshow
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 134                     cd             0.708      30.96           41
  72                addpath    R        0.484      21.19           21
  97                    dir             0.292      12.78           35
 110                  lstat             0.177       7.73         1057
 139             javaMethod             0.162       7.08            3
  74             ltfatstart             0.046       2.02            1
 135                 source             0.042       1.82           20
  85                 regexp             0.038       1.68         1058
 112 canonicalize_file_name             0.033       1.44           36
 127                    run             0.019       0.84           20
  17              regexprep             0.019       0.82         1099
 128              fileparts             0.019       0.81           20
  83                  exist             0.018       0.79           59
  46                   load             0.017       0.73            2
 102                 strcat             0.017       0.73           35
 126               arrayfun             0.014       0.61           35
 101                readdir             0.012       0.54           35
 100              __wglob__             0.012       0.51           35
 113              localtime             0.010       0.45         1057
  68                   stat             0.010       0.44           59


Nicholas Jankowski <nrjank>
Group Member
Tue 19 Jan 2021 05:28:22 PM UTC, comment #17: 

@nrjank: Since you had some benchmark data from 6.1.1, 6.1.0, and 6.2 for loading package ltfat, could you test again after the change made by Markus?

Another useful test would be to run the profiler as before and see if 'cd' and 'addpath' still amount to 50% of the running time or whether the results are now more balanced.

Rik <rik5>
Group administrator
Sun 17 Jan 2021 12:38:02 PM UTC, comment #16: 

I tried a different approach that should not be breaking API compatibility:
If I get it right, instead of calling "canonicalize_file_name" for every entry (including private, +namespace and @class folders) in the load path, it uses it earlier on for the "main" folder that is in the load path.
To check which impact that has and whether this is a step in the correct direction, I locally added some commands to output to std::cout wherever "canonicalize_file_name" could be removed with this approach, ran `make check` and counted how often that string was printed. I used that as the baseline. On my system, I counted 56212 calls.
After the described change, I added similar output commands wherever "canonicalize_file_name" is now called instead (and where it wasn't called before) and ran `make check` again with the same configuration. This time, I counted 3734 calls. So for the test suite, that means a reduction by a factor of approximately 15. (But keep in mind that there are other locations in the code where "canonicalize_file_name" is used which aren't counted in either test.)
I don't know how representative the test suite is though.

I also cross-compiled for Windows and ran the test suite there. All tests still pass.
And `pkg load ltfat` still "feels" faster than with Octave 6.1.
So I pushed the change to stable here:
https://hg.savannah.gnu.org/hgweb/octave/rev/8b65dc1fd34d

Marking as ready for test.
It would be interesting to hear from people who reported slow performance. What is their experience with a nightly build from tomorrow 2021 Jan 18 or later?

Markus Mützel <mmuetzel>
Group administrator
Tue 05 Jan 2021 12:37:10 AM UTC, comment #15: 

For an overview of inodes, see https://en.wikipedia.org/wiki/Inode.

Quoting from the article, "The POSIX standard mandates file-system behavior that is strongly influenced by traditional UNIX file systems. An inode is denoted by the phrase "file serial number", defined as a per-file system unique identifier for a file.[9] That file serial number, together with the device ID of the device containing the file, uniquely identify the file within the whole system.[10]"

So, yes, it appears that a combination of device and inode is unique.  From a code perspective, it makes intuitive sense to use the canonical file name as the key in a map.  But if we have to abandon that because it is slow the only people slightly inconvenienced will be Octave core programmers.  Octave users will never see this.

Rik <rik5>
Group administrator
Mon 04 Jan 2021 04:47:03 PM UTC, comment #14: 

regarding your network shares concern, I recall bug #49203 and bug #31080 recently addressing issues of Octave file checking over networks.  Seems that had more to do with file modified dates so Octave sees new and updated files, and unique quirks in how Windows server and ntfs implement such things in a non-standard way. not sure if there's any relation between that and this issue, though.

Nicholas Jankowski <nrjank>
Group Member
Mon 04 Jan 2021 10:21:25 AM UTC, comment #13: 

IIUC, the canonical file name is used as a unique key for the private function file map.
There is no real reason this key has to be the canonical file name. Any other key would also do as long as it identifies the files in the map uniquely.

The attached patch adds a new type that might be used to identify files uniquely. For this, it also adds a function that queries the device id and inode (with `stat`) on Linux, or the volume serial number and file index (with `GetFileInformationByHandle`) on Windows.
Both function calls should be fast on their respective platforms.
A "simple" call to `stat` might also be slightly more efficient than calling `canonicalize_file_name` on Linux.

If these ids (instead of the canonical file names) are used as the keys in the private function file map, `pkg load ltfat` "feels" faster on my system. (But it wasn't as slow as reported by others here in the first place here. So, it is hard to tell the difference.)

`__run_test_suite__` also finishes without errors. But I haven't done much more testing yet.

Does anyone know whether it is save to assume that a combination of device id and inode uniquely identifies a file?
Might hard links cause problems?
What about file systems on remote systems (network shares)?

Anyway, this patch changes the API. So, if accepted at all, it should go on the default branch.

(file #50641)

Markus Mützel <mmuetzel>
Group administrator
Wed 23 Dec 2020 06:07:33 PM UTC, comment #12: 

Changing the bug title because this is not limited to loading the "ltfat" package (albeit that being a good benchmark...).

CC'ing jwe because he added `canonicalize_file_name` to some `load_path˙ functions in 262cdfc6faf9. (To give some context: IIRC, that was a huge[!] performance gain on Windows wrt a previous regression.)
Is it strictly necessary to use that function there? Or would it be ok to just use `make_absolute`?

Markus Mützel <mmuetzel>
Group administrator
Wed 23 Dec 2020 05:21:57 PM UTC, comment #11: 

Extra info:

The Command Window on Octave 6.1 (Windows 10) is also very slow to the ready after the GUI is loaded.

On 5.2 version, when the GUI is loaded, the Command Window is instantly ready to start typing.

On 6.1 version I need to wait around 10 seconds until the Command Window is available.

On 6.1.1 I need to wait around 2 seconds.

If all info is already provided, can you please change the Bug Status from "Need Info" to something more relevant, please?

Renato <yamane>
Wed 23 Dec 2020 04:50:15 PM UTC, comment #10: 

running one after another on my system, since I threw some slow data up earlier:

6.1.1  (hg id: 0180eaf55fd0)  - 7.4s
6.1.0 (release) - 29.3s
5.2.0 (release) - 3sec



Octave 6.1.1 on windows 10

>> profile clear; profile on; pkg load ltfat; profile off; profshow
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 141                     cd             2.721      35.92           41
  77                addpath    R        1.761      23.25           22
 117           is_same_file             0.667       8.81         2134
 146             javaMethod             0.628       8.29            3
 102                    dir             0.341       4.51           35
  51 canonicalize_file_name             0.272       3.59          130
  79             ltfatstart             0.258       3.41            1
 142                 source             0.211       2.79           20
 115                  lstat             0.210       2.78         1057
  17              regexprep             0.063       0.83         2155
 135              fileparts             0.031       0.41           20
  47                   load             0.030       0.39            2
 134                    run             0.029       0.38           20
  45          load_packages             0.022       0.29            1
   1                    pkg             0.021       0.28            1
  46     installed_packages             0.021       0.28            1
 107                 strcat             0.021       0.27           35
  88                  exist             0.019       0.26           59
 139                 rindex             0.014       0.18           20
  87                 evalin    R        0.013       0.17           22


Octave 6.1.0 on windows 10
>> profile clear; profile on; pkg load ltfat; profile off; profshow
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 141                     cd            12.424      41.56           41
  77                addpath    R        9.384      31.39           22
 115                  lstat             3.261      10.91         1057
 117           is_same_file             0.685       2.29         2134
 102                    dir             0.682       2.28           35
 142                 source             0.543       1.81           20
  79             ltfatstart             0.515       1.72            1
  88                  exist             0.286       0.96           59
  51 canonicalize_file_name             0.285       0.95          130
 146             javaMethod             0.200       0.67            3
  73                   stat             0.166       0.55           59
 135              fileparts             0.158       0.53           20
 134                    run             0.154       0.51           20
 105              __wglob__             0.112       0.37           35
 107                 strcat             0.095       0.32           35
 106                readdir             0.093       0.31           35
 139                 rindex             0.083       0.28           20
  17              regexprep             0.069       0.23         2155
   1                    pkg             0.068       0.23            1
  87                 evalin    R        0.065       0.22           22


Octave 5.2.0 on windows 10
>> profile clear; profile on; pkg load ltfat; profile off; profshow
   #                       Function Attr     Time (s)   Time (%)        Calls
-----------------------------------------------------------------------------
 134                             cd             0.833      25.57           41
  72                        addpath    R        0.595      18.26           21
  97                            dir             0.354      10.88           35
 110                          lstat             0.255       7.83         1057
 135                         source             0.242       7.42           20
 139                     javaMethod             0.190       5.84            3
  74                     ltfatstart             0.118       3.62            1
  85                         regexp             0.047       1.45         1058
  46                           load             0.043       1.33            2
   1                            pkg             0.039       1.20            1
 128                      fileparts             0.037       1.13           20
 112         canonicalize_file_name             0.036       1.10           36
 101                        readdir             0.035       1.06           35
 127                            run             0.030       0.92           20
  44                  load_packages             0.028       0.87            1
 102                         strcat             0.028       0.85           35
  83                          exist             0.026       0.79           59
  62 load_packages_and_dependencies             0.021       0.65            1
  17                      regexprep             0.021       0.64         1099
 131                         rindex             0.019       0.58           20


Nicholas Jankowski <nrjank>
Group Member
Wed 23 Dec 2020 04:45:42 PM UTC, comment #9: 

Thanks.
So, even if the current stable is much faster than Octave 6.1, it is still about 6 times slower than Octave 5.2.

I attached a profiler (Very Sleepy CS) to Octave and ran `pkg load ltfat` on my machine. The profiler slows down execution notably. But I hope the results are still valid.

For me, it is still `canonicalize_file_name` that consumes most of the time in the running thread.
The "slowest" call stacks point to this code:
https://hg.savannah.gnu.org/hgweb/octave/annotate/262cdfc6faf9/libinterp/corefcn/load-path.cc#l1938

According to the commit message, this was added to solve bug #57439.
I wonder if we could use a combination of `make_absolute` and `tolower` instead of `canonicalize_file_name` in that function on Windows (`canonicalize_file_name` is probably much faster on Linux because it is a system call).
This might also be a potential candidate for a `normalize_file_name` function that was discussed in bug #59706.

The second choking point is in `octave::load_path::dir_info::update` where `file_stat` is used to check whether directories exist. There are much more efficient ways to check for this on Windows.
See: https://savannah.gnu.org/bugs/?57439#comment50

Markus Mützel <mmuetzel>
Group administrator
Wed 23 Dec 2020 03:53:19 PM UTC, comment #8: 

Comment #2 = Octave 6.1-w64
Comment #6 = Octave 6.1.1-64 (hg id: 0180eaf55fd0) Wed, Dec 23, 2020
This comment = Octave-5.2.0-w64


>> profile on
>> pkg load ltfat
>> profile off
>> profshow
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 134                     cd             1.025      36.63           41
 72                 addpath    R        0.702      25.10           21
 110                  lstat             0.243       8.67         1057
 97                     dir             0.232       8.28           35
 139             javaMethod             0.113       4.04            3
 135                 source             0.058       2.06           20
 74              ltfatstart             0.054       1.94            1
 112 canonicalize_file_name             0.050       1.79           36
 127                    run             0.028       1.00           20
 85                  regexp             0.027       0.95         1058
 128              fileparts             0.026       0.94           20
 83                   exist             0.025       0.90           59
 102                 strcat             0.019       0.69           35
 100              __wglob__             0.015       0.53           35
 68                    stat             0.014       0.49           59
 17               regexprep             0.013       0.46         1099
 131                 rindex             0.013       0.45           20
 46                    load             0.012       0.42            2
 101                readdir             0.011       0.39           35
 126               arrayfun             0.010       0.37           35


Renato <yamane>
Wed 23 Dec 2020 02:49:12 PM UTC, comment #7: 

Thanks for testing.
This looks faster compared to comment #2.
Is this comparable to the loading time in Octave 5.2?

Markus Mützel <mmuetzel>
Group administrator
Wed 23 Dec 2020 02:29:54 PM UTC, comment #6: 

In Reply to Markus Mützel:

Octave 6.1.1 (hg id: 0180eaf55fd0) Wed, Dec 23, 2020
octave-2020-12-23-00-26-w64-installer

Result to be compared with Comment #2


>> profile on
>> pkg load ltfat
>> profile off
>> profshow
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 141                     cd             6.140      39.49           41
 77                addpath    R        4.007      25.77           22
 117           is_same_file             1.808      11.63         2134
 79             ltfatstart             0.594       3.82            1
 51 canonicalize_file_name             0.589       3.79          130
 146             javaMethod             0.522       3.36            3
 102                    dir             0.416       2.67           35
 142                 source             0.409       2.63           20
 115                  lstat             0.363       2.33         1057
 135              fileparts             0.061       0.39           20
 134                    run             0.057       0.36           20
 17              regexprep             0.055       0.35         2155
 107                 strcat             0.044       0.28           35
 1                    pkg             0.040       0.25            1
 88                  exist             0.035       0.23           59
 45          load_packages             0.035       0.23            1
 46     installed_packages             0.033       0.21            1
 139                 rindex             0.030       0.19           20
 105              __wglob__             0.023       0.15           35
 144          blockprocinit             0.022       0.14            1


Renato <yamane>
Wed 23 Dec 2020 09:28:58 AM UTC, comment #5: 

Loading the package with `pkg load ltfat` doesn't take by far as long for me as reported by others here. So I can't really tell a difference.
But there have been some performance improvements for Octave's (internal) `stat` function on Windows on the stable branch. I haven't completely understood the load-path logic yet. But it might be that it is completely re-constructed after each `cd` which might lead to repeated calls of Octave's (internal) `stat` function.
Could you please try with a recent build if this is still slow for you?
You could download those from Kai's buildbots:
https://buildbot.octave.space/#/download

This is probably something that would need to be improved in core Octave and not in the package itself. (Although, there might be some mitigating work-arounds...)

Markus Mützel <mmuetzel>
Group administrator
Sat 19 Dec 2020 01:39:46 PM UTC, comment #4: 

Thanks for the profshow reports.

It's clear that it is I/O functions (cd and addpath, both working with the file system) that appear to be responsible for most of the initialization time. Elsewhere in the bug tracker (bug #59702) I read that the low level functions used by Octave (Windows libraries) seem to induce degraded performance.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 18 Dec 2020 06:48:05 PM UTC, comment #3: 

just tried it as well running ltfat 2.3.1 on Octave 6.1.0 on Windows 10 with the gui.

I noticed when loading the package, the File Browser window spent a lot of time moving pack and forth between different folders.  It appears it is explicitly traversing the numerous cd and addpath commands, and I have noticed previously (even on v5.1) that in the GUI octave was really slow when changing directory depending on the number of files in that folder.

my profshow results:


>> profile clear; profile on; pkg load ltfat; profile off; profshow
   #                       Function Attr     Time (s)   Time (%)        Calls
-----------------------------------------------------------------------------
 139                             cd             9.417      40.13           41
  74                        addpath    R        7.235      30.83           22
 113                          lstat             2.682      11.43         1057
 144                     javaMethod             0.651       2.77            3
 115                   is_same_file             0.554       2.36         2134
 100                            dir             0.524       2.23           35
 140                         source             0.451       1.92           20
  76                     ltfatstart             0.422       1.80            1
  85                          exist             0.222       0.95           59
  27         canonicalize_file_name             0.211       0.90          130
  70                           stat             0.135       0.57           59
 133                      fileparts             0.125       0.53           20
 132                            run             0.119       0.51           20
 103                      __wglob__             0.095       0.40           35
 105                         strcat             0.078       0.33           35
 137                         rindex             0.062       0.26           20
  58                      regexprep             0.049       0.21         2152
  20                  load_packages             0.035       0.15            1
  49 load_packages_and_dependencies             0.035       0.15            1
   1                            pkg             0.026       0.11            1


running under Octave CLI it doesn't seem any faster. 


octave:1> profile clear; profile on; pkg load ltfat; profile off; profshow
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 141                     cd             9.328      43.49           41
  77                addpath    R        7.021      32.73           22
 115                  lstat             2.287      10.66         1057
 117           is_same_file             0.528       2.46         2134
 102                    dir             0.434       2.02           35
  79             ltfatstart             0.306       1.43            1
  88                  exist             0.216       1.01           59
  51 canonicalize_file_name             0.198       0.92          130
 146             javaMethod             0.151       0.70            3
 142                 source             0.119       0.55           20
  73                   stat             0.119       0.55           59
 135              fileparts             0.105       0.49           20
 134                    run             0.103       0.48           20
 105              __wglob__             0.074       0.35           35
 107                 strcat             0.059       0.28           35
 139                 rindex             0.050       0.23           20
  17              regexprep             0.045       0.21         2155
   1                    pkg             0.027       0.13            1
  45          load_packages             0.026       0.12            1
  46     installed_packages             0.024       0.11            1


Nicholas Jankowski <nrjank>
Group Member
Fri 18 Dec 2020 02:24:23 PM UTC, comment #2: 

Hi, here is.
The laptop is a Workstation HP Zbook 15uG6 with Intel Core i7-8565U, 32Gb RAM, NVME Samsung MZVLB512HAJQ.

>> profile on
>> pkg load ltfat
>> profile off
>> profshow

   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 141                     cd            19.374      43.03           41
  77                addpath    R       14.785      32.84           22
 115                  lstat             4.956      11.01         1057
 117           is_same_file             1.228       2.73         2134
 102                    dir             0.689       1.53           35
  79             ltfatstart             0.673       1.49            1
  88                  exist             0.459       1.02           59
  51 canonicalize_file_name             0.400       0.89          130
 146             javaMethod             0.393       0.87            3
 142                 source             0.365       0.81           20
  73                   stat             0.265       0.59           59
 134                    run             0.223       0.49           20
 135              fileparts             0.216       0.48           20
 105              _wglob_             0.179       0.40           35
 107                 strcat             0.131       0.29           35
 139                 rindex             0.106       0.24           20
 106                readdir             0.094       0.21           35
   1                    pkg             0.052       0.12            1
  45          load_packages             0.048       0.11            1
  46     installed_packages             0.043       0.10            1

>>

Renato <yamane>
Thu 17 Dec 2020 09:37:23 PM UTC, comment #1: 

Could you please run

profile on
pkg load ltfat
profile off
profshow


and copy the results here? (between +verbatim+ and -verbatim- each on a single line)

(Item group "regression" reset until we know what the issue is)


Philip Nienhuis <philipnienhuis>
Group Member
Thu 17 Dec 2020 04:46:58 PM UTC, original submission:  

"pkg load ltfat" is taking too long to be loaded on Octave 6.1 (almost 30 seconds).

On previous versions, it was loaded almost instantly.

Renato <yamane>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #54727:  bug59711-load_path-less-file_stat-v1.patch added by mmuetzel (9KiB - application/octet-stream)
file #54709:  bug59711-load_path-less-file_stat-v0.patch added by mmuetzel (4KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by siko1056 (Updated the item)
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by yamane (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 25 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-08-05 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2023-05-13 mmuetzel StatusIn Progress Ready For Test
        Fixed ReleaseNone 9.1.0
    2023-05-11 mmuetzel Attached File- Added bug59711-load_path-less-file_stat-v1.patch, #54727
    2023-05-07 mmuetzel Attached File- Added bug59711-load_path-less-file_stat-v0.patch, #54709
    2023-05-07 mmuetzel StatusReady For Test In Progress
    2023-05-06 mmuetzel StatusIn Progress Ready For Test
    2023-05-06 mmuetzel StatusConfirmed In Progress
    2022-11-29 mmuetzel StatusFixed Confirmed
        Open/ClosedClosed Open
        Planned ReleaseNone 9.1.0
    2021-01-19 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2021-01-17 mmuetzel StatusConfirmed Ready For Test
    2021-01-04 mmuetzel Attached File- Added bug59711_load-path_oct_uid.patch, #50641
    2020-12-23 mmuetzel StatusNeed Info Confirmed
        Summary30 seconds to load the &quot;ltfat&quot; package Performance of `cd` is bad on Windows
        Carbon-Copy- Added jwe
    2020-12-23 mmuetzel CategoryOctave Package Interpreter
        Item GroupNone Performance
        Summary[octave forge] (ltfat) - 30 seconds to load the package 30 seconds to load the "ltfat" package
    2020-12-23 siko1056 CategoryPerformance Octave Package
        Summaryltfat - 30 seconds to load the package [octave forge] (ltfat) - 30 seconds to load the package
    2020-12-17 philipnienhuis Item GroupRegression None

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code