bugPSPP - Bugs: bug #33260, terrible performance when data...

 
 

bug #33260: terrible performance when data paged to disk

Submitter:  Ben Pfaff <blp>
Submitted:  Sat 07 May 2011 06:10:19 PM UTC
   
 
Category:  Graphical User Interface Severity:  5 - Average
Status:  Fixed Assigned to:  None
Open/Closed:  Closed Release:  Before 0.8.0
Effort:  0.00
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 25 Jun 2013 03:46:11 PM UTC, comment #7: 

It certainly is MUCH faster now.  Thanks.

Closing this item.

John Darrington <jmd>
Group administrator
Tue 25 Jun 2013 06:08:25 AM UTC, comment #6: 

I pushed a commit that made your test case much faster.  I believe that it reduces the time to read a row from O(N**2) in the number of columns to O(N), in the common case.

John, can you try it out too?

Ben Pfaff <blp>
Group administrator
Tue 28 May 2013 03:57:37 PM UTC, comment #5: 


> and produced the following syntax:


Thanks, that helps.

Ben Pfaff <blp>
Group administrator
Tue 28 May 2013 12:50:42 PM UTC, comment #4: 

Your test is not representative of typical GUI usage.

For the following experiment I used the dimensions reported by this user:
http://lists.gnu.org/archive/html/pspp-users/2013-02/msg00005.html

and produced the following syntax:

input program.
vector vec(1174).
loop #c = 1 to 27500.
 loop #v = 1 to 1174.
  compute vec(#v) = #v.
 end loop.
 end case.
end loop.
end file.
end input program.


frequencies /variables  vec1.



The results:

1. Running the entire syntax from the command line:

   real    0m21.848s
   user    0m18.893s
   sys     0m2.000s

Let us say 20 seconds.  Acceptable.



2. Load the syntax into the gui, thus: 
     psppire file.sps
   Then click Run|All :
   I timed that at 31 seconds.  I think this is also acceptable.


3. a) Load the syntax into the gui, thus: 
     psppire file.sps
   b) Then select lines 1 to 10. (from INPUT PROGRAM thru END INPUT PROGRAM).
   c ) Click Run|Selection - this loads the data into the sheet.
       This also takes 31 seconds.  Acceptable.
   d) Now select the line "frequencies /variables  vec1." and
      Click Run|Selection
      I waited more than 7 minutes for this to complete (then I got fed up with waiting).  This is NOT acceptable.

These results would agree with what was reported in the bug report above.

So it would seem that there is a very poor performance reading data from the gui's datasheet.
  


John Darrington <jmd>
Group administrator
Tue 28 May 2013 05:03:30 AM UTC, comment #3: 

I'm not sure that this is really a problem anymore.  My test case below run in .283 seconds in the text UI.  It looks like it takes about the same amount of time in the GUI.

John, any opinion?


INPUT PROGRAM.
        LOOP #I=1 TO 1000 * 300.
                COMPUTE X=RND(UNIFORM(10)).
                END CASE.
        END LOOP.
        END FILE.
END INPUT PROGRAM.
frequencies x.


Ben Pfaff <blp>
Group administrator
Sat 07 May 2011 08:11:35 PM UTC, comment #2: 

I pushed some commits that reduce runtime from 2.5 seconds to 1.5 seconds in the terminal UI for this FREQUENCIES test case (with the default workspace size).

The GUI performance is still worse than the terminal UI performance.  I'll look into it.

Ben Pfaff <blp>
Group administrator
Sat 07 May 2011 07:08:38 PM UTC, comment #1: 

The problem seems to be that performance for data paged to disk is terrible.  If I run "set workspace=100000000." before loading data, then I get much better performance, only about a second or so from either UI for FREQUENCIES runs.  It's still perceptibly faster from the terminal UI, but not bad from the GUI.

I'm updating the summary and category accordingly.

Ben Pfaff <blp>
Group administrator
Sat 07 May 2011 06:10:19 PM UTC, original submission:  

Running FREQUENCIES on a data set with about 300,000 cases from the text UI, it takes a few seconds.  This is slow (under a second would be what I'd consider "normal").  But running the same FREQUENCIES from the GUI takes over a minute!  This needs to get tracked down--it shouldn't be any slower than the terminal UI.

Ben Pfaff <blp>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jmd (Original reporter)
  • -email is unavailable- added by jmd (Posted a comment)
  • -email is unavailable- added by blp (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-06-25 jmd StatusReady for Test/Review Fixed
        Open/ClosedOpen Closed
        Carbon-Copy- Added -email is unavailable-
    2013-06-25 blp StatusNone Ready for Test/Review
    2013-05-28 jmd CategoryOther Graphical User Interface
    2011-05-14 blp ReleaseNone Before 0.8.0
    2011-05-07 blp CategoryGraphical User Interface Other
        Summaryterrible performance in GUI for large data sets terrible performance when data paged to disk

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code