bugGNU Octave - Bugs: bug #47578, zip/gzip/bzip2 fail when run from...

 
 

bug #47578: zip/gzip/bzip2 fail when run from inside a path with spaces in name

Submitter:  Philip Nienhuis <philipnienhuis>
Submitted:  Wed 30 Mar 2016 09:43:58 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Invalid / Not an Octave Bug Assigned to:  None
Originator Name:  Philip Nienhuis Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 02 Apr 2016 05:40:18 PM UTC, comment #13: 

@Philip: That sounds like a good plan.  Open up a new bug report for any Windows-only oddities as they are discovered.

Rik <rik5>
Group administrator
Sat 02 Apr 2016 08:31:27 AM UTC, comment #12: 

@Rik:
Not only escaping; it could well be that zip in mingw/msys simply behaves differently than its sibling(s) on *nix.

IIRC tar was another fine example of that some time ago.


If you don't mind -as this bug is already closed with "invalid"- I'll investigate a bit more on the Windows side as this is a vital thing for the io package.
(Yet I'm surprised I didn't hit this earlier at work. We all work on network drives there, with names sprinkled with spaces)
(Oh but I do know. For spreadsheet I/O Octave automatically prefers COM interface = Excel as all staff has that available, so no Octave user ever got to run the OCT interface)

Philip Nienhuis <philipnienhuis>
Group Member
Sat 02 Apr 2016 04:26:24 AM UTC, comment #11: 

Fixed the problem with requesting the output file list from zip in this cset (http://hg.savannah.gnu.org/hgweb/octave/rev/fa58fcb7c51d).

I suspect there are probably still issues with zip on Windows because the regular expression used to escape shell characters is using the backslash character.  This is appropriate for a Unix shell, but the caret character is the usual escape character for cmd.exe.

Rik <rik5>
Group administrator
Fri 01 Apr 2016 11:47:18 PM UTC, comment #10: 

Here's one:


>> mkdir ("dir with spaces");
>> save file1
>> save file2
>> save file3
>> out = zip ("dir with spaces/foo.zip", "file*")
zipinfo:  cannot find or open /home/mike/dir, /home/mike/dir.zip or /home/mike/dir.ZIP.
error: zip: zipinfo failed with exit status = 9
error: called from
    zip at line 79 column 7
>> cd ("dir with spaces");
>> ls
foo.zip


It actually creates the zip file, just fails to get the listing using zipinfo.

I tested other cases with files containing spaces in the second argument, with or without a glob, and a directory with spaces in the third argument, and they all worked fine.

Mike Miller <mtmiller>
Group Member
Fri 01 Apr 2016 10:57:40 PM UTC, comment #9: 

Is there a simple example of zip.m failing that you can concoct so I can debug what is going on?

Rik <rik5>
Group administrator
Fri 01 Apr 2016 10:31:20 PM UTC, comment #8: 

The last verbatim block should read

[stts, op] = system (sprintf ('zip -q -r "%s" *.* .', filename))


Philip Nienhuis <philipnienhuis>
Group Member
Fri 01 Apr 2016 10:29:46 PM UTC, comment #7: 

Experimenting with invoking octave's zip. and gzip.m showed the following:
- gzip.m works, after some messing with file and path names
- zip.m does NOT work
for subdirs with spaces in the path name.

Following the code in zip.m I see:

cmd = sprintf ("zip -r %s %s", zipfile, sprintf (" %s", files{:}))
[status, output] = system (cmd)

...which after translation of:

zipfile                   -> filename
sprintf (" %s", files{:}) -> *.* .


is essentially the same as

[stts, op] = system (sprintf ("zip -q -r %s *.* .", filename))


in OF-io's private/__OCT_spsh_close__.m

I found that the only thing I needed to do to make my original code work in subdirs with spaces in the name was to change the statement:

[stts, op] = system (sprintf ("zip -q -r %s *.* .", filename))


into:
+verbatim-
[stts, op] = system (sprintf ('zip -q -r "%s" . .', filename))
-verbatim-

i.e., essentially put double quotes around the zipfile name and single quotes around the sprintf format string.
Patching similarly for the original gnumeric (gzip) files statement made that work as well.

So my point here is:
I'd think similar holds for octave's zip.m (and maybe gzip.m and bzip2.m) for them to work properly on Windows in subdirs with spaces in the name.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 31 Mar 2016 09:47:03 PM UTC, comment #6: 

Ah thanks gentlemen for uncovering my ignorance and enlightening me.

At the time I copied the code more or less blindly from a contributor. One reason may well be that at that time there were issues with Octave's wrapper functions for zip/gzip/unpack that have since been fixed (I myself fixed some IIRC). So those fixes were never ported to the io package functions, probably because the latter have always worked until I hit the flaws.

Sorry for wasting your time :-)

I'll close with Invalid

Philip Nienhuis <philipnienhuis>
Group Member
Thu 31 Mar 2016 09:44:02 PM UTC, comment #5: 

Wow, that was quite an accident.

So we're back to this seems to be working, haven't seen any examples yet that fail because the source or destination directory contain spaces.

Mike Miller <mtmiller>
Group Member
Thu 31 Mar 2016 09:12:58 PM UTC, comment #4: 

I just installed the io package and I come to the same conclusion as Mike.

The files

private/__OCT_spsh_open__.m:
private/__OCT_spsh_close__.m

both contain direct system calls to gzip.

But, Mike did find a very obscure bug in unpack.m.  I checked in a change for that here (http://hg.savannah.gnu.org/hgweb/octave/rev/feac06371be1).

Rik <rik5>
Group administrator
Thu 31 Mar 2016 08:11:32 PM UTC, comment #3: 

Well one big problem with your example is that the io package is calling


system (sprintf ("gzip -c -S=gnumeric %s > %s", ...


so this is not a valid test and not applicable here. Of course that's going to fail due to lack of quoting, but not the same as this bug.

The only example I've been able to come up with demonstrating something like this bug is


>> gunzip ("/path/to/dir with spaces/file.gz", "/path/to/otherdir");
error: movefile: no files to move
error: called from
    movefile at line 92 column 5
    unpack at line 261 column 15
    gunzip at line 49 column 5


The uncompress succeeds, but the file is not moved to the other directory.

Any other examples of failures with these functions and paths containing spaces?

Mike Miller <mtmiller>
Group Member
Thu 31 Mar 2016 11:49:36 AM UTC, comment #2: 

<title adapted after more investigation>

Turns out this also failed in 3.8.2. So this bug is really old.

(The examples were cooked a bit. But I am sure of the bug/regression as I hit it when trying the spreadsheet test scripts on a drive with spaces in the name.
Rather than impose installing io I figured I'd make up a simple example. But with the io package it is easier for me to show what fails)

If you have the OF io package installed you can do:

more off
pkg load io
io_ods_testscript ("oct")  ## .ods file, invokes zip
- or -
io_ods_testscript ("oct", "test.gnumeric")  ## invokes gzip

Run these commands from inside a subdir with, and a subdir without spaces in the name.

That example gives for a subdir w/o spaces in name (empty lines removed)   (NOTE: all with octave-4.0.1 but 4.1.0+ behaves identical):

>> pwd
ans = D:\tmp\dir_wo_spaces_in_name
>> io_ods_testscript ("oct", "test.gnumeric")
Testing .ods interface OCT using file test.gnumeric...
 1. Initialize arrays.
 2. Insert first empty sheet.
 3. Add data to test sheet.
 4. Add another sheet with just one number in A1.
 5. Explore sheet info.
 6. Read data back.
 7. Tests part 1 (basic I/O):
    ...Numeric array... matches.
    ...Cellstr array... matches.
    ...Boolean... recovered.
 8. Repeat reading, now return formulas as text
 9. Tests part 2 (read back as formula):
    (Note: just a check for a string rather than a numerical value)
    ...OK, formula recovered ('=c2+sin(d3)').
10. Cleaning up..... OK
## output skipped


and for a subdir with spaces:

>> pwd
ans = D:\tmp\dir with space in name
>> io_ods_testscript ("oct", "test.gnumeric")

Testing .ods interface OCT using file test.gnumeric...

 1. Initialize arrays.

 2. Insert first empty sheet.
gzip: with: No such file or directory
gzip: space: No such file or directory
gzip: in: No such file or directory
gzip: name\test.gnumeric: No such file or directory
odsclose: could not gzip gnumeric contents in C:\Users\Nienhu01\AppData\Local\Temp
\oct-3BqGgI to D:\tmp\dir with space in name\test.gnumeric

 3. Add data to test sheet.
gzip: with: No such file or directory
gzip: space: No such file or directory
gzip: in: No such file or directory
gzip: name\test.gnumeric: No such file or directory
odsclose: could not gzip gnumeric contents in C:\Users\Nienhu01\AppData\Local\Temp
\oct-8CBW6X to D:\tmp\dir with space in name\test.gnumeric
gzip: with: No such file or directory
gzip: space: No such file or directory
gzip: in: No such file or directory
gzip: name\test.gnumeric: No such file or directory
odsclose: could not gzip gnumeric contents in C:\Users\Nienhu01\AppData\Local\Temp
\oct-eQQoWv to D:\tmp\dir with space in name\test.gnumeric

 4. Add another sheet with just one number in A1.
gzip: with: No such file or directory
gzip: space: No such file or directory
gzip: in: No such file or directory
gzip: name\test.gnumeric: No such file or directory
odsclose: could not gzip gnumeric contents in C:\Users\Nienhu01\AppData\Local\Temp
\oct-CoQ7Lu to D:\tmp\dir with space in name\test.gnumeric

 5. Explore sheet info.
error: odsopen.m: file test.gnumeric not found


Running a level higher up (in d:\tmp):

>> pwd
ans = D:\tmp
>> io_ods_testscript ("oct", "dir with space in name/test.gnumeric")

Testing .ods interface OCT using file dir with space in name/test.gnumeric...

 1. Initialize arrays.

 2. Insert first empty sheet.
gzip: with: No such file or directory
gzip: space: No such file or directory
gzip: in: No such file or directory
gzip: name/test.gnumeric: No such file or directory
odsclose: could not gzip gnumeric contents in C:\Users\Nienhu01\AppData\Local\Temp
\oct-WrP2uN to D:\tmp\dir with space in name/test.gnumeric
:


..but...

>> io_ods_testscript ("oct", "dir_wo_spaces_in_name/test.gnumeric")

Testing .ods interface OCT using file dir_wo_spaces_in_name/test.gnumeric...

 1. Initialize arrays.

 2. Insert first empty sheet.
:


runs fine.

I just tried with 3.8.2, 4.0.1 and a 4.1.0+ from Mar 29 and all show the same bug.

Is it a concealed manifestation of the "do not install Octave in a path with spaces" issue? That would be a very creepy gotcha.

Or is it that gzip / zip / bzip2 just get handed a path to the zip file that is in a format they don't expect? (or w/o double quotes)

(regression -> incorrect result)

Philip Nienhuis <philipnienhuis>
Group Member
Wed 30 Mar 2016 10:08:54 PM UTC, comment #1: 

Are you sure you have the syntax right?  It is quite different between gzip and zip.

For gzip, the help string says


FILELIST = gzip (FILES, DIR)


When called with


gzip ('testgzip.gz', '/tmp/tmpfil123')


You are asking it to compress the file 'testgzip.gz' (and gzip will add the extension so this will become 'testgzip.gz.gz') and place it in the directory '/tmp/tmfil123'.  But that location already exists as a file, so Octave can't make a directory to place the output in and fails with


error: gzip: Failed to create output directory DIR


On the other hand, the help string for zip is


FILELIST = zip (ZIPFILE, FILES)


so the invocation


zip ('testzip.zip', '/tmp/tmpfil123')


probably succeeds and creates a zip file with the compressed results of tmfil123.


Rik <rik5>
Group administrator
Wed 30 Mar 2016 09:43:58 PM UTC, original submission:  

The current dev version isn't able to correctly create zip/gzip/bzip2 files, neither in Linux nor in Windows.

$ hg -v summary
parent: 21559:6619945e4434 tip
 maint: Merge stable to default.

Steps to reproduce in Linux:

(in shell)
mkdir "~/dir with space"
mkdir "~/dir_wo_space"
cat > /tmp/tmpfil123
just type some junk text here
^Z

(fire up Octave-4.1.0+)
cd "~/dir with space"
gzip ('testgzip.gz', '/tmp/tmpfil123')

and see messages along the lines of:

error: gzip: Failed to create output directory DIR
error: called from
    _xzip_ at line 39 column 7
    gzip at line 47 column 5

Similarly for bzip2.

zip.m fails silently - is this another bug ?

When trying in "dir_wo_space" it all works perfectly.

Noting that Rik has recently patched zip and friends I quickly tried with scripts/miscellaneous/zip.m from the stable branch swapped into place but that didn't make a difference. So it must be something outside zip/... that affects the zip.gzip/bzip2 functions.

Philip Nienhuis <philipnienhuis>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by philipnienhuis (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-03-31 philipnienhuis StatusWorks For Me Invalid / Not an Octave Bug
        Open/ClosedOpen Closed
    2016-03-31 mtmiller StatusNeed Info Works For Me
    2016-03-31 mtmiller StatusNone Need Info
    2016-03-31 philipnienhuis Item GroupRegression Incorrect Result
        Summaryzip/gzip/bzip2 fail when writing to path with spaces in name zip/gzip/bzip2 fail when run from inside a path with spaces in name

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code