bugPSPP - Bugs: bug #20161, Better integrate automatic...

 
 

bug #20161: Better integrate automatic (de)compression for system/portable file I/O

Submitter:  Ben Pfaff <blp>
Submitted:  Wed 13 Jun 2007 04:41:50 AM UTC
   
 
Category:  System/Portable File I/O Severity:  1 - Wishlist
Status:  None Assigned to:  None
Open/Closed:  Open Release:  Future
Effort:  0.00
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 12 Oct 2008 07:06:33 PM UTC, comment #7: 

The following library would make it trivial to support automatic decompression: http://ftp.isr.ist.utl.pt/pub/MIRRORS/ftp.suse.com/people/werner/libzio/

It's glibc-only though.

Ben Pfaff <blp>
Group administrator
Mon 30 Jul 2007 05:23:14 PM UTC, comment #6: 

This has been half-fixed by the check-in for bug #18692.  We can do better integration later if it proves worthwhile.

Ben Pfaff <blp>
Group administrator
Mon 30 Jul 2007 03:51:13 AM UTC, comment #5: 

The "solution" proposed for bug #18692 would also fix this bug, so I'm changing this bug's status also.

Ben Pfaff <blp>
Group administrator
Sat 16 Jun 2007 01:22:30 AM UTC, comment #4: 


>I'm saying that if you pipe through gzip (or any compression
>tool), then it produces nothing until it's input is closed.
>Although "you" (pspp) will be reading cases sequentially, the gzip
>process will have to store them all in memory until after the
>very last one.


That's not at all true.  You can demonstrate this easily: run "gzip < /dev/urandom > /dev/null" in one terminal, and "top" in another.  gzip will use as much CPU as it can get, but its memory consumption is O(1).

>It'll mean for instance that reading/writing will be considerably
>slower,


Possibly.  I bet it wouldn't be any slower on any mid-range to high-end computer you buy off-the-shelf this year, though, because all of those are multi-core.

>and the maximum possible number of cases will be smaller.


I doubt it.

Ben Pfaff <blp>
Group administrator
Sat 16 Jun 2007 01:15:47 AM UTC, comment #3: 

I'm not saying that it wouldn't work, or that it wouldn't be a good idea (so long as gzip compression is disabled by default).

I'm saying that if you pipe through gzip (or any compression tool), then it produces nothing until it's input is closed.  Although "you" (pspp) will be reading cases sequentially, the gzip process will have to store them all in memory until after the very last one.  So some of the advantages of the current implementation will be lost.  It'll mean for instance that reading/writing will be considerably slower, and the maximum possible number of cases will be smaller.

Just so long as users realize this, I don't think there will be a problem.

John Darrington <jmd>
Group administrator
Fri 15 Jun 2007 02:25:21 PM UTC, comment #2: 

Not sure what you mean by "no longer serializable".  We currently read system and portable files only sequentially, and with one small exception the same is true for writing system and portable files.  I was planning to implement this feature just by piping data to/from a gzip or bzip2 process.  Can you think of why this wouldn't work?

Ben Pfaff <blp>
Group administrator
Fri 15 Jun 2007 07:46:40 AM UTC, comment #1: 

It would be "better" in the sense that it would achieve a greater compression ratio, but it would also mean that the files would no longer be serializeable.  So reading/writing large ones would be significantly slower.

John Darrington <jmd>
Group administrator
Wed 13 Jun 2007 04:41:50 AM UTC, original submission:  

We could easily support automatic compression and decompression for system file I/O, based on file name extensions (.gz .bz2 etc.) or on file magic numbers.  This would be a much better alternative to the "compression" that SPSS system files support.

Ben Pfaff <blp>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jmd (Posted a comment)
  • -email is unavailable- added by blp (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-05-11 jmd ReleasePost-0.6.0 Future
    2007-07-30 blp StatusReady for Test/Review None
        Summarysupport automatic (de)compression for system/portable file I/O Better integrate automatic (de)compression for system/portable file I/O
    2007-07-30 blp StatusNone Ready for Test/Review
    2007-07-25 blp ReleaseNone Post-0.6.0

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code