bugPSPP - Bugs: bug #17241, Performance optimizations for...

 
 

bug #17241: Performance optimizations for procedures

Submitted by:  Ben Pfaff <blp>
Submitted on:  Mon 31 Jul 2006 02:21:42 AM UTC  
 
Category: OtherSeverity: 1 - Wishlist
Status: NoneAssigned to: Ben Pfaff <blp>
Open/Closed: OpenRelease: Future
Effort: 0.00

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 12 Mar 2009 06:41:28 AM UTC, comment #1:

One simple optimisation case is as follows:

get file='several-billion-cases.sav'.
n of cases 10.
list.

Here, the complete file doesn't need to be iterated, since only the first 10 can be used.

John Darrington <jmd>
Project Administrator
Mon 31 Jul 2006 02:21:42 AM UTC, original submission:

These are the realistic part of a more ambitious idea for performance optimizations that I posted earlier to the pspp-dev mailing list.

1. Eliminate the need to cache procedure output. Currently,
every procedure reads the entire active file and writes out a
new copy of it. For big data sets that's often a huge waste
of time and space.

This was almost impossible to get right before I refactored
the code for implementing procedures. Now, it should only be
a few days of work.

2. Implement an optimizer for transformations. SELECT IF can
often be moved earlier without changing any semantics.
Creation of variables whose values are never used can be
dropped entirely. I suspect that this is especially valuable
after #1 is implemented because #1 will retain any
non-temporary transformations from one procedure to the next
for re-execution, whereas currently that does not happen.

Doing a basic job of this should be a few days of work. It's
probably not worth putting more into it than that (and maybe
not that much) unless it is demonstrably useful.

2a. Implement caching of data between procedures.

Ben Pfaff <blp>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jmd (Posted a comment)
  • -unavailable- added by blp (Updated the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 14 May 2011 04:39:08 AM UTCblpReleasePost-0.6.0=>Future
    Mon 18 Dec 2006 05:21:53 PM UTCblpReleaseNone=>Post-0.6.0

    Back to the top


    Powered by Savane 3.1-cleanup1