bugPSPP - Bugs: bug #20161, Better integrate automatic...

 
 

bug #20161: Better integrate automatic (de)compression for system/portable file I/O

Submitted by:  Ben Pfaff <blp>
Submitted on:  Wed 13 Jun 2007 04:41:50 AM UTC  
 
Category: System/Portable File I/OSeverity: 1 - Wishlist
Status: NoneAssigned to: None
Open/Closed: OpenRelease: Future
Effort: 0.00

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 12 Oct 2008 07:06:33 PM UTC, comment #7:

The following library would make it trivial to support automatic decompression: http://ftp.isr.ist.utl.pt/pub/MIRRORS/ftp.suse.com/people/werner/libzio/

It's glibc-only though.

Ben Pfaff <blp>
Project Administrator
Mon 30 Jul 2007 05:23:14 PM UTC, comment #6:

This has been half-fixed by the check-in for bug #18692. We can do better integration later if it proves worthwhile.

Ben Pfaff <blp>
Project Administrator
Mon 30 Jul 2007 03:51:13 AM UTC, comment #5:

The "solution" proposed for bug #18692 would also fix this bug, so I'm changing this bug's status also.

Ben Pfaff <blp>
Project Administrator
Sat 16 Jun 2007 01:22:30 AM UTC, comment #4:

>I'm saying that if you pipe through gzip (or any compression
>tool), then it produces nothing until it's input is closed.
>Although "you" (pspp) will be reading cases sequentially, the gzip
>process will have to store them all in memory until after the
>very last one.


That's not at all true. You can demonstrate this easily: run "gzip < /dev/urandom > /dev/null" in one terminal, and "top" in another. gzip will use as much CPU as it can get, but its memory consumption is O(1).

>It'll mean for instance that reading/writing will be considerably
>slower,


Possibly. I bet it wouldn't be any slower on any mid-range to high-end computer you buy off-the-shelf this year, though, because all of those are multi-core.

>and the maximum possible number of cases will be smaller.


I doubt it.

Ben Pfaff <blp>
Project Administrator
Sat 16 Jun 2007 01:15:47 AM UTC, comment #3:

I'm not saying that it wouldn't work, or that it wouldn't be a good idea (so long as gzip compression is disabled by default).

I'm saying that if you pipe through gzip (or any compression tool), then it produces nothing until it's input is closed. Although "you" (pspp) will be reading cases sequentially, the gzip process will have to store them all in memory until after the very last one. So some of the advantages of the current implementation will be lost. It'll mean for instance that reading/writing will be considerably slower, and the maximum possible number of cases will be smaller.

Just so long as users realize this, I don't think there will be a problem.

John Darrington <jmd>
Project Administrator
Fri 15 Jun 2007 02:25:21 PM UTC, comment #2:

Not sure what you mean by "no longer serializable". We currently read system and portable files only sequentially, and with one small exception the same is true for writing system and portable files. I was planning to implement this feature just by piping data to/from a gzip or bzip2 process. Can you think of why this wouldn't work?

Ben Pfaff <blp>
Project Administrator
Fri 15 Jun 2007 07:46:40 AM UTC, comment #1:

It would be "better" in the sense that it would achieve a greater compression ratio, but it would also mean that the files would no longer be serializeable. So reading/writing large ones would be significantly slower.

John Darrington <jmd>
Project Administrator
Wed 13 Jun 2007 04:41:50 AM UTC, original submission:

We could easily support automatic compression and decompression for system file I/O, based on file name extensions (.gz .bz2 etc.) or on file magic numbers. This would be a much better alternative to the "compression" that SPSS system files support.

Ben Pfaff <blp>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jmd (Posted a comment)
  • -unavailable- added by blp (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 11 May 2013 06:32:54 AM UTCjmdReleasePost-0.6.0=>Future
    Mon 30 Jul 2007 05:23:14 PM UTCblpStatusReady for Test/Review=>None
      Summarysupport automatic (de)compression for system/portable file I/O=>Better integrate automatic (de)compression for system/portable file I/O
    Mon 30 Jul 2007 03:51:13 AM UTCblpStatusNone=>Ready for Test/Review
    Wed 25 Jul 2007 03:21:44 PM UTCblpReleaseNone=>Post-0.6.0

    Back to the top


    Powered by Savane 3.1-cleanup