bugGNU Octave - Bugs: bug #45331, unpack hangs when output file...

 
 

bug #45331: unpack hangs when output file exists

Submitter:  Muhali <muhali>
Submitted:  Tue 16 Jun 2015 10:13:36 AM UTC
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Fixed Assigned to:  philipnienhuis
Originator Name:  Open/Closed:  * Closed
Release:  * 4.0.0 Operating System:  * GNU/Linux
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       

 

( Jump to the original submission )

Sun 05 Jul 2015 09:23:57 AM UTC, comment #15: 

Thanks for the detailed explanation.

I'll shut up and stop wasting your time :-)

Philip Nienhuis <philipnienhuis>
Project Member
Sat 04 Jul 2015 12:38:51 PM UTC, comment #14: 

The second patch won't unpack files to the current directory if called like so

unpack ("../foo.gz", [], "gz")

Also, with gunzip ('../foo.gz') I get the following error:

error: octave_base_value::convert_to_str_internal (): wrong type argument 'struct'
error: called from
    unpack at line 152 column 9
    gunzip at line 45 column 5
error: tilde_expand: expecting argument to be char or cellstr object
error: called from
    unpack at line 152 column 9
    gunzip at line 45 column 5

This is because without the default assignment in the prototype of the function, dir is no longer a variable if gunzip is called with a single argument.  Instead, when you call unpack you are calling it with the output of the function dir() which is a structure.

function filelist = gunzip (gzfile, dir)

  if (nargin < 1 || nargin > 2)
    print_usage ();
  endif

  if (nargout > 0)
    filelist = unpack (gzfile, dir, "gunzip");
  else
    unpack (gzfile, dir, "gunzip");
  endif

endfunction

Rik <rik5>
Project Administrator
Thu 02 Jul 2015 08:29:59 PM UTC, comment #13: 

Sure, but a similar stanza is in unpack.m.

How about the attached changeset?  (passes all tests)

(file #34365)

Philip Nienhuis <philipnienhuis>
Project Member
Thu 02 Jul 2015 02:58:51 PM UTC, comment #12: 

The stanzas in gunzip and bunzip are important because they ensure that the archives are unpacked in the directory where the archive resides, rather than always in to the current directory.

Rik <rik5>
Project Administrator
Thu 02 Jul 2015 06:33:34 AM UTC, comment #11: 

@Rik:
Thanks for looking at it.

I see you didn't touch gunzip.m and bunzip.m, while both have redundant if not dead code. I had two stanzas in my patch for that.

Shall I push another patch fixing those things?

Philip Nienhuis <philipnienhuis>
Project Member
Wed 01 Jul 2015 09:21:50 PM UTC, comment #10: 

I checked in a change on stable that does most of what was asked for (http://hg.savannah.gnu.org/hgweb/octave/rev/2fc43288a6c4). 

Unpack no longer hangs because the '-f' switch is supplied to both gzip and bzip (I figured it was better to be consistent even though bzip doesn't require it).

I couldn't use the '-k' switch to keep-around the archive because that is a new gzip addition and not all Linux distros ship with a new enough version.  So that remains a difference between Octave and Matlab.  We could open up a separate Matlab Compatibility report about that.

The calculation of the output dir changed from the original patch because I found it was putting all unpacked files in the current directory rather than the directory where the archive resides.

Rik <rik5>
Project Administrator
Sat 20 Jun 2015 09:12:53 PM UTC, comment #9: 

Changeset attached.

@Rik:
Would you please review and if satisfied push to, well, stable?

I'm not extremely proud of it but it works OK now and tests pass:
- unpacked archives are kept (added "-k" flags)
- "movefile" errors are gone
- no more hangs when gunzip'ing existing files (due to added "-f" flag)
==> as far as bunzip2 goes, I do not know if a forced overwrite is the way to go. Currently bunzip2 complains if it finds an existing file, as it does in a shell, but it doesn't hang Octave.

I think (and I already thought so 5 months ago when I first patched unpack.m) that unpack.m is in need of an overhaul but I lack time for that. It could use more tests as well - AFAICS only gunzip is tested.

(file #34280)

Philip Nienhuis <philipnienhuis>
Project Member
Sat 20 Jun 2015 05:57:15 PM UTC, comment #8: 

In gunzip.m & bunzip2.m the second input arg "dir" is [] (empty) by default.
Then, both functions have a stanza like this:

:
  if (isempty (dir))
    dir = fileparts (bzfile);
  endif
:

...which doesn't do an awful lot of things if dir = [].

Next, in unpack.m there's a problem with strcmp-ing otherwise identical paths with and without a trailing filesep.

I'll try to make something out of this.

Philip Nienhuis <philipnienhuis>
Project Member
Fri 19 Jun 2015 09:19:57 PM UTC, comment #7: 

Indeed:

>> x = pi
x = 355/113
>> save foo.mat pi
>> gzip foo.mat
>> delete foo.mat
>> gunzip foo.mat.gz
error: unpack: mkdir failed to create : No such file or directory
error: called from
    unpack at line 225 column 7
    gunzip at line 49 column 5
>> unpack foo.mat.gz
mv: `C:/Users/philip/MyDocs/octave/foo.mat' and `C:/Users/philip/MyDocs/octave/foo.mat' are the same file
error: unpack: unable to move files to ".":
error: called from
    unpack at line 251 column 9
>>

At least some clues....

Philip Nienhuis <philipnienhuis>
Project Member
Fri 19 Jun 2015 08:40:17 PM UTC, comment #6: 

Thanks Philip.  So it seems that the first part is correct, and that Octave both creates the gzip archive and also retains the original file.

The 'gunzip' action seems to be rather screwed up.

octave:11> x = pi;
octave:12> save foo x
octave:13> gzip foo
octave:14> gunzip foo.gz
error: unpack: mkdir failed to create : No such file or directory
error: called from
    unpack at line 225 column 7
    gunzip at line 49 column 5

It looks like unpack.m needs some attention.

Rik <rik5>
Project Administrator
Fri 19 Jun 2015 08:19:40 PM UTC, comment #5: 

With r2015b prerelease:

>> x = pi

x =

    3.1416

>> save foo x
>> gzip foo.mat
>>

...and ML's file browser shows files "foo.mat" and "foo.mat.gz".

Now,

>> delete foo.mat

% a little wait just to be sure...
% OK, foo.mat is wiped.

>> gunzip foo.mat.gz
>>

makes "foo.mat" reappear and "foo.mat.gz" is not deleted.

bzip2/bunzip2 isn't supported by Matlab on Windows and I haven't had time to install r2015b prerelease on Linux yet.

Philip Nienhuis <philipnienhuis>
Project Member
Fri 19 Jun 2015 07:08:55 PM UTC, comment #4: 

Can someone test the what Matlab does for

x = pi;
save foo x
gzip foo

Does the original file foo remain in the directory?  The ordinary Linux command gzip removes the original file, but I think we were trying for Matlab compatibility in this regard.

Rik <rik5>
Project Administrator
Wed 17 Jun 2015 06:15:43 AM UTC, comment #3: 

@Philip: gzip has -f flag - force overwrite of output file and compress links.

Adding -f to gzip command doesn't hang Octave, but causes  an error since there is command to move the file to itself.

Avinoam Kalma <avinoam>
Project Member
Tue 16 Jun 2015 06:47:48 PM UTC, comment #2: 

same with bzip2. zip is different as it requires two arguments. The tar.* cases don't seem to hang.

Muhali <muhali>
Tue 16 Jun 2015 05:29:10 PM UTC, comment #1: 

Does that happen with gzipped files only? How about bzip / zip / tar.bz / tar.gz / tgz ?

If gzip / gunzip has a sort of "force" or "overwrite" flag the first issue should be an easy fix.

As for the second, I think when g(un)zipping a file in the shell, the original file is deleted - at least that is how gzip/gunzip seems to work on my Linux systems.
However, in gzip.m's help text it is clearly stated that the original files will be kept.
Such a thing isn't mentioned in the gunzip.m help text, however.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 16 Jun 2015 10:13:36 AM UTC, original submission:  

unpack hangs if the output file exists.

x = pi;
save foo x
gzip foo
unpack foo.gz

Only after pressing another return I get

error: unpack: unarchiving program exited with status: 2
gzip: /tmp/oct-qxMHrG/foo already exists; do you wish to overwrite (y or n)? not overwritten

Similarly for gunzip, ...

Moreover, when gzipping a file, the file itself is not deleted, whereas when gunzipping a file, that file is deleted.

Muhali <muhali>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #34365:  bug_45331_part2.m added by philipnienhuis (2KiB - text/x-objective-c)
file #34280:  bug45331.cs added by philipnienhuis (3KiB - text/x-csharp)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by avinoam (Posted a comment)
  • -email is unavailable- added by pantxo (Updated the item)
  • -email is unavailable- added by muhali (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only project members can vote.

     

     

     

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-07-02 philipnienhuis Attached File- Added bug_45331_part2.m, #34365
    2015-07-01 rik5 CategoryNone Octave Function
        StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
        Releasedev 4.0.0
    2015-06-20 philipnienhuis Attached File- Added bug45331.cs, #34280
        StatusIn Progress Patch Submitted
    2015-06-20 philipnienhuis StatusConfirmed In Progress
        Assigned toNone philipnienhuis
    2015-06-16 pantxo StatusNone Confirmed
        Summaryunpack hangs file exists unpack hangs when output file exists

    Back to the top


    Powered by Savane 3.9