bugGNU Octave - Bugs: bug #53695, tar.m and unpack.m are...


bug #53695: tar.m and unpack.m are incompatible with BSD tar

Submitted by:  Mike Miller <mtmiller>
Submitted on:  Wed 18 Apr 2018 10:49:19 PM UTC  
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  BSD

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Wed 27 Jun 2018 05:04:54 PM UTC, comment #14:

This is more complicated than I expected. Without checking which tar (bsd/gnu) we cannot figure out if the output "a xyz" of tar means
1. the file with name "xyz" was added by bdstar to the archive or
2. the file with name "a xyz" was added by gnutar to the archive

So we have to check what tar we are actually using. The solution attached is not elegant but it works...


(file #44445)

Sebastian <sschoeps>
Sun 24 Jun 2018 09:09:13 PM UTC, comment #13:

@sschoeps: I patched octave to use "gtar" as you suggested, and it worked.

Test is passing now.

Note that this patch is a hack specific to Octave.app on macOS; it is not a general fix for this bug.

Andrew Janke <apjanke>
Sun 24 Jun 2018 04:03:00 PM UTC, comment #12:

I am not working on this at the moment.

I still believe that the best solution is #3, to make the tar.m and unpack.m functions work with any program called 'tar'.

Mike Miller <mtmiller>
Project Administrator
Sun 24 Jun 2018 03:36:32 PM UTC, comment #11:

Is anyone working on it?

I think I can fix the regexp if I redirect stderr. This should work on linux/mac but windows would need to be treated separately because there is no redirection as far as I know.

@Mike: do you have an opinion on how to proceed? We have three options...
1. configure check for gnutar
2. runtime option (like makeinfo)
3. try to fix regexp (with the drawback mentioned above)

@Andrew: for now we can patch unpack.m to use "gtar" instead of "tar" and add gnu-tar as homebrew dependency.

Sebastian <sschoeps>
Sun 24 Jun 2018 06:19:53 AM UTC, comment #10:

Would you consider as an alternative requiring a GNU tar, but doing a configure-time check to locate it, and allowing a user override? As far as I know, GNU tar is available on most systems, even if it has to be installed at a non-default location or prefix (like `/usr/local/bin/gtar`), so it's not an onerous requirement.

I'm working on a macOS installation of GNU Octave and running in to the same problem, since the system-supplied tar on macOS is a BSD tar. It's easy to install GNU Tar as a supplemental tar, but I'd need some way to point GNU Octave at it.

A run time check might be even better, as long as there was a way to supply a run time override that pointed Octave at the right GNU tar. That would make it easier to package Octave for redistribution, to systems which may or may not have GNU tar installed, and may have it installed at different locations. Something like the `makeinfo_program` function that allows run time specification of the texinfo location.

Andrew Janke <apjanke>
Wed 02 May 2018 09:47:10 PM UTC, comment #9:

Incidentally I'm pulling the FreeBSD source code to try to get a non-GNU sed to compare with, see if any of our sed patterns can be made more portable, or if we need to add a sed feature test to configure. Separate bug report forthcoming.

Mike Miller <mtmiller>
Project Administrator
Wed 02 May 2018 09:41:53 PM UTC, comment #8:

I wanted to do the same initially but then I thought that redirecting stderr might break windows compatibility. However, I will think about it tomorrow. It's bed time in Germany :)

Sebastian <sschoeps>
Wed 02 May 2018 09:35:24 PM UTC, comment #7:

No I'm thinking we force stderr to stdout, and we parse the output with the regexp function to get the file list.

Mike Miller <mtmiller>
Project Administrator
Wed 02 May 2018 09:32:17 PM UTC, comment #6:

Ok, I will play around with bsd-tar and probably propose a patch. Would you recommend using a runtime check in the m-file to test if we are using bsd or gnu?

Sebastian <sschoeps>
Wed 02 May 2018 09:15:06 PM UTC, comment #5:

Thanks for the confirmation.

I think it's a little different, since we do require GNU sed in some places to build Octave (which I would also be happy to fix if possible, file a bug?), but we don't put any requirement on the user for a particular sed when using Octave.

This is different as it extends to the end user of the tar/untar/unpack functions. I think it's important to define the impact between the admin/build user and the end user.

Mike Miller <mtmiller>
Project Administrator
Wed 02 May 2018 08:44:56 PM UTC, comment #4:

I think we do require also gnu-sed [1], so it would make sense to require gnu-tar but then we should list it as a dependency and then check for it during configure-time.


[1] https://lists.gnu.org/archive/html/octave-maintainers/2016-09/msg00193.html

Sebastian <sschoeps>
Thu 19 Apr 2018 04:33:46 PM UTC, comment #3:

I have only found bug #43979 and bug #44007, both closed, both about similar test failures, but they were primarily because the programs in question didn't work well with forward-slash-separated file names.

Bug #44007 has some mention of GNU tar vs BSD tar, but it's in context of compiling one or the other within mxe-octave. And that seems to have been ditched in favor of continuing to use a pre-compiled GNU win32 tar. That might be what you are remembering.

I want to make sure the tar, untar, and unpack functions are more robust and compatible with either tar program.

Mike Miller <mtmiller>
Project Administrator
Thu 19 Apr 2018 04:57:41 AM UTC, comment #2:

I think we came across this issue earlier. All that means is there might be an existing bug report, since we didn't actually have a solution for it.

Rik <rik5>
Project Administrator
Wed 18 Apr 2018 10:51:43 PM UTC, comment #1:

The same applies for unpack.m with a tar file.

Mike Miller <mtmiller>
Project Administrator
Wed 18 Apr 2018 10:49:19 PM UTC, original submission:

The tar.m function only works with the output of GNU tar, in particular when parsing the output of the program to return a list of files.

This can be tested and worked on easily enough on a GNU system by installing the 'bsdtar' package and changing the function to invoke 'bsdtar' temporarily.

Ideally the tar.m function (and other functions that call tar) can be made to work with all variants of the tar utility.

With bsdtar on Debian:

Two key differences, the '-v' output goes to stderr, and each line starts with a prefix of 'a' before the file name.

Confirmed test failure with the buildbot workers running on macOS.

Mike Miller <mtmiller>
Project Administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #44445:  tar.m added by sschoeps (5KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by sschoeps (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mtmiller (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-06-27 sschoeps Attached File- => Added tar.m, #44445
    2018-04-18 mtmiller Summarytar.m is incompatible with BSD tar => tar.m and unpack.m are incompatible with BSD tar

    Back to the top

    Powered by Savane 3.4