bugGNU Octave - Bugs: bug #43206, gunzip deletes source files

 
 

bug #43206: gunzip deletes source files

Submitter:  None
Submitted:  Fri 12 Sep 2014 07:28:42 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  3 - Low Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  None
Originator Name:  Stefan Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 12 Jan 2024 10:39:02 PM UTC, comment #22: 

This bug is really old, and probably not very relevant anymore.  I tested an unknown option with gzip and it throws an error.  I think that's good enough per jwe's thoughts in comment #21.

Marking as Fixed and closing report.

Rik <rik5>
Group administrator
Tue 28 Sep 2021 05:32:33 PM UTC, comment #21: 

When Octave invokes gzip with the -k option on your system, what happens?  I expect something like


octave:1> unpack foo.gz
error: unpack: unarchiving program exited with status: 1
gzip: -k not supported in this version
Try `gzip --help' for more information.


so that we wouldn't need to add a test for the feature or version.  Users would just get an error.  But if we wanted to, we could test to see whether -k works as expected and issue a better error message.

If old versions of gzip do fail with an error like the one above, then I'm OK with just closing this report.  But if gzip silently accepts the -k option and doesn't preserve the original file, then maybe we could do more?

John W. Eaton <jwe>
Group administrator
Wed 08 Sep 2021 02:17:24 AM UTC, comment #20: 

Re-opened because of comment #18.

According to comment #3, gzip 1.6 is required:
- RHEL/CentOS 7 (2014 - 2024) https://centos.pkgs.org/7/centos-x86_64/gzip-1.5-10.el7.x86_64.rpm.html
- Debian 9 "stretch" (2017 - 2022) has 1.6 https://packages.debian.org/search?searchon=sourcenames&keywords=gzip
- Ubuntu 18.04 (2018 - 2023) has 1.6 https://packages.ubuntu.com/bionic/gzip

Agree to comment #19.  Waiting for the last oldtimer to upgrade his system is very luxurious (unpaid?) support.  A warning might be sufficient.

There are plenty of Octave container solutions available + compiling a newer gzip from source.

In addition, the changes happen on Octave 7, which will be released next (or next next 😇) year.  In 2022 RHEL 7 is even more stale than today.

Kai Torben Ohlhus <siko1056>
Group Member
Tue 07 Sep 2021 09:15:32 PM UTC, comment #19: 

Could version check the gzip that is available and issue a one-time warning per Octave session?  8 years is a long time ago.

Rik <rik5>
Group administrator
Tue 07 Sep 2021 08:06:11 PM UTC, comment #18: 

I'm not sure that we should close this as fixed.  With older versions of gzip, the problem remains that source files are silently deleted.  I was hoping that such old versions of gzip would be rare by now and that after my change, unzipping would fail with -k option and we could just recommend updating to a modern version of gzip.  If that's not possible, then I think we should find a way to avoid silently deleting the source file.

John W. Eaton <jwe>
Group administrator
Sun 05 Sep 2021 09:21:40 PM UTC, comment #17: 

My system (Kubuntu 2018.04) has gzip v1.6 installed.  This is a 5-year Long Term Release so there will be users, including myself, with older versions until 2023.  I changed the regular expression to support old and new versions of gzip.  See cset http://hg.savannah.gnu.org/hgweb/octave/rev/3915633fdfd0.

Marking issue as Fixed and closing report.

Rik <rik5>
Group administrator
Sun 05 Sep 2021 08:06:03 PM UTC, comment #16: 

Before my change, unpack.m was matching the output from gzip with


files = regexprep (output, '^.+ -- replaced with (.*)$', '$1');


I changed that to


files = regexprep (output, '^.+ -- created (.*)$', '$1');


because that's what gzip printed when I used the -k option.  I didn't think it was necessary to use


files = regexprep (output, '^.+ -- (created|replaced with) (.*)$', '$2');


to match either the old or new output.  But I also expected gzip to fail if we tried to use -k and it wasn't supported.

John W. Eaton <jwe>
Group administrator
Sun 05 Sep 2021 01:39:33 PM UTC, comment #15: 

In addition to comment #14, the full error message from comment #12 was (https://lists.gnu.org/archive/html/octave-bug-tracker/2021-09/msg00025.html):


%!  unlink_or_error (test_file);
%!  uz_filelist = z.unzip (z_file);
%!  assert (is_same_file (uz_filelist, {test_file}))

The problem seems to be that uz_filelist contains more than just a filename.
Output from the debugger is


debug> uz_filelist
uz_filelist =
{
  [1,1] = /tmp/oct-DRbD3A/oct-hwAtPs.gz:          5.7% -- replaced with
/tmp/oct-DRbD3A/oct-hwAtPs
}



Kai Torben Ohlhus <siko1056>
Group Member
Sun 05 Sep 2021 12:19:43 PM UTC, comment #14: 

The webversion of comment #12 of bug #43206 doesn't display the failing code, the e-mail versions does.

A.R. Burgers <arb>
Sun 05 Sep 2021 09:54:26 AM UTC, comment #13: 

I did get 3 failures (2 in gzip, 1 in unpack), but in my case test/fntests.log reported my gzip didn't support the -k option. After upgrading to gzip 1.11 the failures were gone.


A.R. Burgers <arb>
Sun 05 Sep 2021 04:09:34 AM UTC, comment #12: 

I'm getting 2 failures in the BIST tests for gzip.cc and 1 in unpack.m.  Is anybody else seeing this from "make check"?

I went to debug the first failed test and I found it was in the BIST function test_large_file().  The failing code is only for gzip and is shown below.


%!  unlink_or_error (test_file);
%!  uz_filelist = z.unzip (z_file);
%!  assert (is_same_file (uz_filelist, {test_file}))

The problem seems to be that uz_filelist contains more than just a filename.  Output from the debugger is

+verbatim+
debug> uz_filelist
uz_filelist =
{
  [1,1] = /tmp/oct-DRbD3A/oct-hwAtPs.gz:          5.7% -- replaced with /tmp/oct-DRbD3A/oct-hwAtPs
}



Rik <rik5>
Group administrator
Thu 02 Sep 2021 10:09:52 PM UTC, comment #11: 

Time passes and things change.  The -k option was added to gzip more than 8 years ago now.  When was it added to bzip2?  In any case, at this point I think its fair to just use those options so I added them unconditionally in the following changeset:

http://hg.savannah.gnu.org/hgweb/octave/rev/167747278b7a

Marking as ready for test.

John W. Eaton <jwe>
Group administrator
Mon 19 Dec 2016 07:34:57 PM UTC, comment #10: 

Thanks for testing.  So the incompatibility still exists and Octave removes the original .gz file while Matlab does not.

Rik <rik5>
Group administrator
Mon 19 Dec 2016 07:30:46 PM UTC, comment #9: 

I have checked with Matlab (R2014b). It doesn't do this.

The result from the final "exist" code line in comment #8 is still 2, and at the end both files still exist.

Hartmut <hardy>
Fri 16 Dec 2016 09:58:37 PM UTC, comment #8: 

It mostly works with archives (.tar.gz).  This is about individual gzipped files.  Start with a file "hello" that contains some basic text like "123".


>> flist = gzip ("hello")
flist =
{
  [1,1] = hello.gz
}
>> exist ("hello.gz")
ans =  2
>> flist2 = gunzip ("hello.gz")
flist2 =
{
  [1,1] = /home/rik/wip/Projects_Mine/octave-dev/unp/hello
}
>> exist ("hello.gz")
ans = 0


As you can see from the call to exist, the file "hello.gz" has been removed.

If you have access to Matlab, you could test whether this behavior has changed.

Otherwise, something else needs to be done.


Rik <rik5>
Group administrator
Fri 16 Dec 2016 07:25:56 PM UTC, comment #7: 

This issue might already be FIXED in Octave 4.2.0.

Here is what I did:

  • create two tar archives inside a folder
  • make this folder the current folder of Octave
  • run gunzip('*.gz') in Octave
  • the result seems fine now:
    • the two gz files are BOTH unpacked properly
    • the two original gz files are NOT deleted


This worked fine with both a (self built on Ubuntu 16.04) linux Octave and the (official 32-bit installer on WinXP) mxe-octave.

Or are there any issues of this bug report still open?


Hartmut <hardy>
Tue 07 Oct 2014 09:37:49 PM UTC, comment #6: 

The troublesome part is going to be recursive directory processing.  Not relying on 'gunzip -r' means having to write a directory traversal tool in the style of find.

Rik <rik5>
Group administrator
Tue 07 Oct 2014 08:21:32 PM UTC, comment #5: 

I think a pipe is cleaner because with the copy, you have to ensure that you are using a good tmp file name, that the copy succeeds.  Although disk space is cheap, someone might be working on a relatively full partition that does not have room for the original file, the copy, and the uncompressed extracted contents of the archive.  I think it would be better to avoid the unnecessary copy.

John W. Eaton <jwe>
Group administrator
Tue 07 Oct 2014 07:30:35 PM UTC, comment #4: 

Circling back to this issue since I just finished overhauling unpack.  What is the Matlab-compatible behavior we are trying to duplicate?  Do they definitely keep the original archive?

I think Mike's suggestion is the clean way to do things.  However, we could cheat and just do:


copyfile (orig_gz_fname, tmp_gz_fname);
..gunzip actions..
movefile (tmp_gz_fname, orig_gz_fname);


This would also work across platforms and is simple to implement.  It, does, however, require a copy of an archive file that might be very large.  On the other hand, disk storage space is so cheap that it is probably meaningless to optimize here.


Rik <rik5>
Group administrator
Mon 15 Sep 2014 03:03:57 PM UTC, comment #3: 

Gzip only supports --keep in version 1.6 (2013-06-09) and newer unfortunately. It might be better to use stdin piping instead for portability.

On the mailing list, Stefan Reimann wrote:

>  Matlab overwrites output file always silently, even if the same file is gunzipped again and there is no option to prevent it from.


Stefan, please comment on the bug tracker, email replies are not sent to the bug report.

Mike Miller <mtmiller>
Group Member
Sat 13 Sep 2014 07:37:46 PM UTC, comment #2: 

I think you can just add "-k" options in the bzip2 and gzip commands that Octave executes.

What is supposed to happen when output files exist?  Does Matlab prevent you from overwriting them, or does it just overwrite silently?

John W. Eaton <jwe>
Group administrator
Fri 12 Sep 2014 01:15:07 PM UTC, comment #1: 

Thanks for your bug report. So it seems bunzip2, gunzip, and unpack need to be updated to preserve the archive file (except for URL downloads to temporary files).

It's best to make distinct bug reports for each issue, please open a separate report for the wildcard problem.

Mike Miller <mtmiller>
Group Member
Fri 12 Sep 2014 07:28:42 AM UTC, original submission:  

Matlab-Example using wildcard (taken from Matlab R2011b):

% gunzip all *.gz files in the current directory
gunzip('*.gz');


Furthermore Matlab does not delete *.gz source files as Octave does!!! ==> very dangerous if this Octave behaviour is not known to the user

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by arb (Posted a comment)
  • -email is unavailable- added by hardy (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-01-12 rik5 StatusNeed Info Fixed
        Open/ClosedOpen Closed
    2021-09-08 siko1056 StatusFixed Need Info
        Open/ClosedClosed Open
    2021-09-05 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2021-09-02 jwe StatusConfirmed Ready For Test
    2018-10-01 siko1056 Dependencies- bugs #48597 is dependent
    2018-05-01 mtmiller Release3.8.2 dev
    2016-12-16 rik5 Priority5 - Normal 3 - Low
    2014-09-12 mtmiller StatusNone Confirmed
        Operating SystemMicrosoft Windows Any
        Summarygunzip deletes source files and doesn\'t accept wildcards gunzip deletes source files

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code