bugGNU Octave - Bugs: bug #47578, zip/gzip/bzip2 fail when run from...

 
 

bug #47578: zip/gzip/bzip2 fail when run from inside a path with spaces in name

Submitted by:  Philip Nienhuis <philipnienhuis>
Submitted on:  Wed 30 Mar 2016 09:43:58 PM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: InvalidAssigned to: None
Originator Name: Philip NienhuisOpen/Closed: Closed
Release: devOperating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 02 Apr 2016 05:40:18 PM UTC, comment #13:

@Philip: That sounds like a good plan. Open up a new bug report for any Windows-only oddities as they are discovered.

Rik <rik5>
Project Administrator
Sat 02 Apr 2016 08:31:27 AM UTC, comment #12:

@Rik:
Not only escaping; it could well be that zip in mingw/msys simply behaves differently than its sibling(s) on *nix.

IIRC tar was another fine example of that some time ago.

If you don't mind -as this bug is already closed with "invalid"- I'll investigate a bit more on the Windows side as this is a vital thing for the io package.
(Yet I'm surprised I didn't hit this earlier at work. We all work on network drives there, with names sprinkled with spaces)
(Oh but I do know. For spreadsheet I/O Octave automatically prefers COM interface = Excel as all staff has that available, so no Octave user ever got to run the OCT interface)

Philip Nienhuis <philipnienhuis>
Project Member
Sat 02 Apr 2016 04:26:24 AM UTC, comment #11:

Fixed the problem with requesting the output file list from zip in this cset (http://hg.savannah.gnu.org/hgweb/octave/rev/fa58fcb7c51d).

I suspect there are probably still issues with zip on Windows because the regular expression used to escape shell characters is using the backslash character. This is appropriate for a Unix shell, but the caret character is the usual escape character for cmd.exe.

Rik <rik5>
Project Administrator
Fri 01 Apr 2016 11:47:18 PM UTC, comment #10:

Here's one:

It actually creates the zip file, just fails to get the listing using zipinfo.

I tested other cases with files containing spaces in the second argument, with or without a glob, and a directory with spaces in the third argument, and they all worked fine.

Mike Miller <mtmiller>
Project Administrator
Fri 01 Apr 2016 10:57:40 PM UTC, comment #9:

Is there a simple example of zip.m failing that you can concoct so I can debug what is going on?

Rik <rik5>
Project Administrator
Fri 01 Apr 2016 10:31:20 PM UTC, comment #8:

The last verbatim block should read

Philip Nienhuis <philipnienhuis>
Project Member
Fri 01 Apr 2016 10:29:46 PM UTC, comment #7:

Experimenting with invoking octave's zip. and gzip.m showed the following:
- gzip.m works, after some messing with file and path names
- zip.m does NOT work
for subdirs with spaces in the path name.

Following the code in zip.m I see:

cmd = sprintf ("zip -r %s %s", zipfile, sprintf (" %s", files{:}))
[status, output] = system (cmd)

...which after translation of:

is essentially the same as

in OF-io's private/__OCT_spsh_close__.m

I found that the only thing I needed to do to make my original code work in subdirs with spaces in the name was to change the statement:

into:
+verbatim-
[stts, op] = system (sprintf ('zip -q -r "%s" . .', filename))
-verbatim-

i.e., essentially put double quotes around the zipfile name and single quotes around the sprintf format string.
Patching similarly for the original gnumeric (gzip) files statement made that work as well.

So my point here is:
I'd think similar holds for octave's zip.m (and maybe gzip.m and bzip2.m) for them to work properly on Windows in subdirs with spaces in the name.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 31 Mar 2016 09:47:03 PM UTC, comment #6:

Ah thanks gentlemen for uncovering my ignorance and enlightening me.

At the time I copied the code more or less blindly from a contributor. One reason may well be that at that time there were issues with Octave's wrapper functions for zip/gzip/unpack that have since been fixed (I myself fixed some IIRC). So those fixes were never ported to the io package functions, probably because the latter have always worked until I hit the flaws.

Sorry for wasting your time :-)

I'll close with Invalid

Philip Nienhuis <philipnienhuis>
Project Member
Thu 31 Mar 2016 09:44:02 PM UTC, comment #5:

Wow, that was quite an accident.

So we're back to this seems to be working, haven't seen any examples yet that fail because the source or destination directory contain spaces.

Mike Miller <mtmiller>
Project Administrator
Thu 31 Mar 2016 09:12:58 PM UTC, comment #4:

I just installed the io package and I come to the same conclusion as Mike.

The files

private/__OCT_spsh_open__.m:
private/__OCT_spsh_close__.m

both contain direct system calls to gzip.

But, Mike did find a very obscure bug in unpack.m. I checked in a change for that here (http://hg.savannah.gnu.org/hgweb/octave/rev/feac06371be1).

Rik <rik5>
Project Administrator
Thu 31 Mar 2016 08:11:32 PM UTC, comment #3:

Well one big problem with your example is that the io package is calling

so this is not a valid test and not applicable here. Of course that's going to fail due to lack of quoting, but not the same as this bug.

The only example I've been able to come up with demonstrating something like this bug is

The uncompress succeeds, but the file is not moved to the other directory.

Any other examples of failures with these functions and paths containing spaces?

Mike Miller <mtmiller>
Project Administrator
Thu 31 Mar 2016 11:49:36 AM UTC, comment #2:

<title adapted after more investigation>

Turns out this also failed in 3.8.2. So this bug is really old.

(The examples were cooked a bit. But I am sure of the bug/regression as I hit it when trying the spreadsheet test scripts on a drive with spaces in the name.
Rather than impose installing io I figured I'd make up a simple example. But with the io package it is easier for me to show what fails)

If you have the OF io package installed you can do:

more off
pkg load io
io_ods_testscript ("oct") ## .ods file, invokes zip
- or -
io_ods_testscript ("oct", "test.gnumeric") ## invokes gzip

Run these commands from inside a subdir with, and a subdir without spaces in the name.

That example gives for a subdir w/o spaces in name (empty lines removed) (NOTE: all with octave-4.0.1 but 4.1.0+ behaves identical):

and for a subdir with spaces:

Running a level higher up (in d:\tmp):

..but...

runs fine.

I just tried with 3.8.2, 4.0.1 and a 4.1.0+ from Mar 29 and all show the same bug.

Is it a concealed manifestation of the "do not install Octave in a path with spaces" issue? That would be a very creepy gotcha.

Or is it that gzip / zip / bzip2 just get handed a path to the zip file that is in a format they don't expect? (or w/o double quotes)

(regression -> incorrect result)

Philip Nienhuis <philipnienhuis>
Project Member
Wed 30 Mar 2016 10:08:54 PM UTC, comment #1:

Are you sure you have the syntax right? It is quite different between gzip and zip.

For gzip, the help string says

When called with

You are asking it to compress the file 'testgzip.gz' (and gzip will add the extension so this will become 'testgzip.gz.gz') and place it in the directory '/tmp/tmfil123'. But that location already exists as a file, so Octave can't make a directory to place the output in and fails with

On the other hand, the help string for zip is

so the invocation

probably succeeds and creates a zip file with the compressed results of tmfil123.

Rik <rik5>
Project Administrator
Wed 30 Mar 2016 09:43:58 PM UTC, original submission:

The current dev version isn't able to correctly create zip/gzip/bzip2 files, neither in Linux nor in Windows.

$ hg -v summary
parent: 21559:6619945e4434 tip
maint: Merge stable to default.

Steps to reproduce in Linux:

(in shell)
mkdir "~/dir with space"
mkdir "~/dir_wo_space"
cat > /tmp/tmpfil123
just type some junk text here
^Z

(fire up Octave-4.1.0+)
cd "~/dir with space"
gzip ('testgzip.gz', '/tmp/tmpfil123')

and see messages along the lines of:

error: gzip: Failed to create output directory DIR
error: called from
_xzip_ at line 39 column 7
gzip at line 47 column 5

Similarly for bzip2.

zip.m fails silently - is this another bug ?

When trying in "dir_wo_space" it all works perfectly.

Noting that Rik has recently patched zip and friends I quickly tried with scripts/miscellaneous/zip.m from the stable branch swapped into place but that didn't make a difference. So it must be something outside zip/... that affects the zip.gzip/bzip2 functions.

Philip Nienhuis <philipnienhuis>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by philipnienhuis (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 31 Mar 2016 09:47:03 PM UTCphilipnienhuisStatusWorks For Me=>Invalid
      Open/ClosedOpen=>Closed
    Thu 31 Mar 2016 09:44:02 PM UTCmtmillerStatusNeed Info=>Works For Me
    Thu 31 Mar 2016 08:11:32 PM UTCmtmillerStatusNone=>Need Info
    Thu 31 Mar 2016 11:49:36 AM UTCphilipnienhuisItem GroupRegression=>Incorrect Result
      Summaryzip/gzip/bzip2 fail when writing to path with spaces in name=>zip/gzip/bzip2 fail when run from inside a path with spaces in name

    Back to the top


    Powered by Savane 3.1-cleanup1