bugPSPP - Bugs: bug #20161, Better integrate automatic...

 
 

bug #20161: Better integrate automatic (de)compression for system/portable file I/O

Submitted by:  Ben Pfaff <blp>
Submitted on:  Wed Jun 13 04:41:50 2007  
 
Category: System/Portable File I/OSeverity: 1 - Wishlist
Status: NoneAssigned to: None
Open/Closed: OpenRelease: Future
Effort: 0.00

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun Oct 12 19:06:33 2008, comment #7:

The following library would make it trivial to support automatic decompression: http://ftp.isr.ist.utl.pt/pub/MIRRORS/ftp.suse.com/people/werner/libzio/

It's glibc-only though.

Ben Pfaff <blp>
Project Administrator
Mon Jul 30 17:23:14 2007, comment #6:

This has been half-fixed by the check-in for bug #18692. We can do better integration later if it proves worthwhile.

Ben Pfaff <blp>
Project Administrator
Mon Jul 30 03:51:13 2007, comment #5:

The "solution" proposed for bug #18692 would also fix this bug, so I'm changing this bug's status also.

Ben Pfaff <blp>
Project Administrator
Sat Jun 16 01:22:30 2007, comment #4:

>I'm saying that if you pipe through gzip (or any compression
>tool), then it produces nothing until it's input is closed.
>Although "you" (pspp) will be reading cases sequentially, the gzip
>process will have to store them all in memory until after the
>very last one.


That's not at all true. You can demonstrate this easily: run "gzip < /dev/urandom > /dev/null" in one terminal, and "top" in another. gzip will use as much CPU as it can get, but its memory consumption is O(1).

>It'll mean for instance that reading/writing will be considerably
>slower,


Possibly. I bet it wouldn't be any slower on any mid-range to high-end computer you buy off-the-shelf this year, though, because all of those are multi-core.

>and the maximum possible number of cases will be smaller.


I doubt it.

Ben Pfaff <blp>
Project Administrator
Sat Jun 16 01:15:47 2007, comment #3:

I'm not saying that it wouldn't work, or that it wouldn't be a good idea (so long as gzip compression is disabled by default).

I'm saying that if you pipe through gzip (or any compression tool), then it produces nothing until it's input is closed. Although "you" (pspp) will be reading cases sequentially, the gzip process will have to store them all in memory until after the very last one. So some of the advantages of the current implementation will be lost. It'll mean for instance that reading/writing will be considerably slower, and the maximum possible number of cases will be smaller.

Just so long as users realize this, I don't think there will be a problem.

John Darrington <jmd>
Project Administrator
Fri Jun 15 14:25:21 2007, comment #2:

Not sure what you mean by "no longer serializable". We currently read system and portable files only sequentially, and with one small exception the same is true for writing system and portable files. I was planning to implement this feature just by piping data to/from a gzip or bzip2 process. Can you think of why this wouldn't work?

Ben Pfaff <blp>
Project Administrator
Fri Jun 15 07:46:40 2007, comment #1:

It would be "better" in the sense that it would achieve a greater compression ratio, but it would also mean that the files would no longer be serializeable. So reading/writing large ones would be significantly slower.

John Darrington <jmd>
Project Administrator
Wed Jun 13 04:41:50 2007, original submission:

We could easily support automatic compression and decompression for system file I/O, based on file name extensions (.gz .bz2 etc.) or on file magic numbers. This would be a much better alternative to the "compression" that SPSS system files support.

Ben Pfaff <blp>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jmd (Posted a comment)
  • -unavailable- added by blp (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat May 11 06:32:54 2013jmdReleasePost-0.6.0=>Future
    Mon Jul 30 17:23:14 2007blpStatusReady for Test/Review=>None
      Summarysupport automatic (de)compression for system/portable file I/O=>Better integrate automatic (de)compression for system/portable file I/O
    Mon Jul 30 03:51:13 2007blpStatusNone=>Ready for Test/Review
    Wed Jul 25 15:21:44 2007blpReleaseNone=>Post-0.6.0

    Back to the top


    Powered by Savane 3.1-cleanup