bugGNU Octave - Bugs: bug #59702, dlmread performance


bug #59702: dlmread performance

Submitted by:  Charles Praplan <charprap>
Submitted on:  Wed 16 Dec 2020 01:07:59 PM UTC  
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  None Assigned to:  None
Originator Name:  Charles Praplan Open/Closed:  Open
Release:  6.1.0 Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Mon 21 Dec 2020 06:48:06 PM UTC, comment #10: 

double-conversion is present on Arch, and is already required by qt5-base and vtk.

Thu 17 Dec 2020 12:42:22 PM UTC, comment #9: 

It is also available on macos via macports

$ port list double-conversion
double-conversion              @3.1.0          devel/double-conversion
$ port contents double-conversion
Port double-conversion contains:

Carlo de Falco <cdf>
Project Member
Wed 16 Dec 2020 08:58:22 PM UTC, comment #8: 

double-conversion is also packaged for openSUSE.

Atri Bhattacharya <badshah400>
Wed 16 Dec 2020 08:02:55 PM UTC, comment #7: 

It is packaged on RedHat/Fedora and related:

rpm -ql double-conversion-devel


Dmitri A. Sergatskov <dasergatskov>
Wed 16 Dec 2020 07:45:42 PM UTC, comment #6: 

I would generally prefer to use standard C++ interfaces and features but if there are problems with those then I don't object to using something else.  In mxe-octave we already build the double-conversion library for Qt, so it shouldn't cause trouble there.  It's also packaged for Debian.  What about other systems?

John W. Eaton <jwe>
Project Administrator
Wed 16 Dec 2020 06:01:25 PM UTC, comment #5: 

When I run the same code in Octave 5.2 and Octave 6.1, `setlocale` is called 24 times in Octave 5.2 vs. 66 times in Octave 6.1.
That difference is probably the main reason for the performance difference between the two versions.

But instead of trying to reduce the number of times we use the `>>` operator (which I think is the reason for the slow code path), it might be worth considering replacing it by a more performant implementation altogether.

Would it be okay if we added a dependency on Google's double-conversion library?

Markus Mützel <mmuetzel>
Project Member
Wed 16 Dec 2020 05:24:35 PM UTC, comment #4: 

Other projects seem to have had a similar experience:

It looks like they opted to use double-conversion instead of standard library functions because of the quite significant performance benefit.

Markus Mützel <mmuetzel>
Project Member
Wed 16 Dec 2020 04:22:57 PM UTC, comment #3: 

I ran the line with `dlmread` in a loop while I attached a profiler to the Octave process.

Most of the time is spent in the function `setlocale` in the Microsoft Visual C Run Time (msvcrt).
The innermost function in one of Octave's libraries that calls this function is `Z20octave_read_fp_valueIdEdRSi`. I tried to demangle that symbol [1] but was unsuccessful.
It is probably an instance of the templated function here:

The top of the call stack reads the following in Very Sleepy CS:

setlocale        msvcrt        [unknown]        0        0x7ffd9c349c3d
ZSt14__convert_to_vIdEvPKcRT_RSt12_Ios_IostateRKPi        libstdc++-6        [unknown]        0        0xdd4e1c
ZNKSt7num_getIcSt19istreambuf_iteratorIcSt11char_traitsIcEEE6do_getES3_S3_RSt8ios_baseRSt12_Ios_IostateRd        libstdc++-6        [unknown]        0        0xd359f9
ZNSi10_M_extractIdEERSiRT_        libstdc++-6        [unknown]        0        0xd59a7e
Z20octave_read_fp_valueIdEdRSi        liboctave-8        [unknown]        0        0x6d2325aa
Z8FdlmreadRN6octave11interpreterERK17octave_value_listi        liboctinterp-8        [unknown]        0        0x6f93e056

It's not (yet) clear to me why that would need a call of `setlocale`, why it is so slow, and why it looks like it was faster in the past.
Might be caused by a different version of GCC that we used for compiling Octave 5.2 and Octave 6.1.

[1]: https://demangler.com/

Markus Mützel <mmuetzel>
Project Member
Wed 16 Dec 2020 04:02:11 PM UTC, comment #2: 

I understand the difference between linux and windows.
But how can you explain the difference between Octave 5 and 6 on win10?

Charles Praplan <charprap>
Wed 16 Dec 2020 02:07:10 PM UTC, comment #1: 

It is got to be one of the windows-related issues. I got 1.7sec on
win10 and 0.057sec on linux (running off the same USB thumb drive).


Dmitri A. Sergatskov <dasergatskov>
Wed 16 Dec 2020 01:07:59 PM UTC, original submission:  

I have noticed that Octave 6.1.0 seems to be slower than Octave 5.2.0.
This is already visible at startup when loading the prolog, but this is not the issue.

Here is another example using dlmread. Below is a summary of the time needed to read the appended data file (from a SSD drive) with the appended script.

               Small data file    Large data file (100x)
octave 5.2.0 : 0.502303 seconds   49.4749 seconds
octave 6.1.0 : 1.99895 seconds    213.084 seconds
matlab 2019a : 0.104778 seconds   4.539602 seconds

There is factor 4 between version 5 and 6!

Can somebody explain this?

Charles Praplan <charprap>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #50488:  data_2n_part_50ksamp.csv added by charprap (1MiB - application/vnd.ms-excel)
file #50489:  V6_slower_than_V5_dlmread_test_case.m added by charprap (157B - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by cdf (Posted a comment)
  • -email is unavailable- added by badshah400 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by charprap (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-12-17 jwe SummaryOctave 6 slower then Octave 5 ? => dlmread performance
    2020-12-16 charprap Attached File- => Added data_2n_part_50ksamp.csv, #50488
        Attached File- => Added V6_slower_than_V5_dlmread_test_case.m, #50489

    Back to the top

    Powered by Savane 3.5