Wed 22 Mar 2017 07:31:11 AM UTC, comment #3:
Having just tried the -v7 option on an 8G memory machine, I suggest being very careful to have no other applications running such that almost all memory is free. Otherwise, the system will get caught in a swap-memory trap and have a really difficult time getting out of it.
First, let me say that the compression option should in theory work faster, not slower. The reason is that, as I pointed out in a previous post, barely any CPU is used for the -mat option and all the time spent is the disk churning away. Compression should utilize all the bandwidth of the CPU(s) creating less memory that needs to be stored to disk and hence less disc activity.
However, my guess is that the -v7 compression may not be done quite right in Octave, or at least it isn't done in a memory efficient way. And it should be memory efficient because the large data elements are what we'd like compressed.
When I ran with the -v7 option, because the x variable was 4G memory, it seemed like Octave wanted to do the compression with some large data buffer to work with. So it needed far more memory than my machine had available. And I think because the needed memory chunk was so big, it wasn't a little bit of memory that went into the swap drive, it was 50%. This brought the system to a standstill. Using Cntrl-C on Octave was problematic as well, because then Octave attempts to store variables to octave-workspace, which of course is big:
-rw-r--r--. 1 sebald sebald 3115411537 Mar 22 01:46 octave-workspace
Killing the process did clear big swaths of memory, but still everything else was stuck in swap memory and remained slow.
It may be worth looking at the -v7 compression and see how memory intensive it is. If it is using a lot of memory--on the order of the elements being saved--then it would definitely be worth attempting some algorithm that can perhaps work on smaller hunks of data at a time while streaming to disk, thereby keeping the extra memory required for compression small.
|