bugPSPP - Bugs: bug #17241, Performance optimizations for...

 
 

bug #17241: Performance optimizations for procedures

Submitter:  Ben Pfaff <blp>
Submitted:  Mon 31 Jul 2006 02:21:42 AM UTC
   
 
Category:  Other Severity:  1 - Wishlist
Status:  None Assigned to:  blp
Open/Closed:  Open Release:  Future
Effort:  0.00
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 12 Mar 2009 06:41:28 AM UTC, comment #1: 

One simple optimisation case is as follows:


 get file='several-billion-cases.sav'.
 n of cases 10.
 list.

Here, the complete file doesn't need to be iterated, since only the first 10 can be used.

John Darrington <jmd>
Group administrator
Mon 31 Jul 2006 02:21:42 AM UTC, original submission:  

These are the realistic part of a more ambitious idea for performance optimizations that I posted earlier to the pspp-dev mailing list.

1. Eliminate the need to cache procedure output.  Currently,
   every procedure reads the entire active file and writes out a
   new copy of it.  For big data sets that's often a huge waste
   of time and space.

   This was almost impossible to get right before I refactored
   the code for implementing procedures.  Now, it should only be
   a few days of work.

2. Implement an optimizer for transformations.  SELECT IF can
   often be moved earlier without changing any semantics.
   Creation of variables whose values are never used can be
   dropped entirely.  I suspect that this is especially valuable
   after #1 is implemented because #1 will retain any
   non-temporary transformations from one procedure to the next
   for re-execution, whereas currently that does not happen.

   Doing a basic job of this should be a few days of work.  It's
   probably not worth putting more into it than that (and maybe
   not that much) unless it is demonstrably useful.

2a. Implement caching of data between procedures.

Ben Pfaff <blp>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jmd (Posted a comment)
  • -email is unavailable- added by blp (Updated the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-05-14 blp ReleasePost-0.6.0 Future
    2006-12-18 blp ReleaseNone Post-0.6.0

    Back to the top

    Powered by Savane 3.12.
    Corresponding source code