Wed 18 Mar 2015 06:06:33 PM UTC, comment #57:
Right, one more thank you.
Closing reports
|
Wed 18 Mar 2015 05:54:23 PM UTC, comment #56:
I reviewed the patch, made a few changes, and pushed it. Go ahead and close the two linked bug reports.
I believe this leaves only one more test, complex.tst, which is still failing on the Windows side.
|
Wed 18 Mar 2015 05:41:12 PM UTC, comment #55:
(Who pushed it? I saw it in the log. Thx)
Can I close this report + that of bug #43979?
|
Tue 17 Mar 2015 08:04:04 PM UTC, comment #54:
OK here's the fix for both tar.m and unpack.m (see bug #43979) as these are closely related.
Both functions require _w2mpth_.m that converts the Windows path to MSYS style.
In unpack.m, the last remaining bug I saw was fixed with make_absolute_filename() - that leaves Windows filesep ("\") intact while canonicalize_file_name() converted them to "/" (which broke the tests on Windows).
I tested the fixes on Windows and Linux.
@Rik:
Will you please review and if OK, push the patch? Thanks
(file #33384)
|
Sun 15 Mar 2015 10:07:33 PM UTC, comment #53:
it would be good in main stream gnu tar one day compiled ok for mingw.
I will try to continue looking at tar, and see if I can get some changes that allow it to compile that can be provided to one day make it happen.
|
Sun 15 Mar 2015 08:44:02 PM UTC, comment #52:
OK Rik, I'll polish my original patch & make one for unpack as well.
I agree it is better to concentrate on getting a 4.0.0 release soon than go to great lengths to fix fairly ancient bugs by fixing upstream bugs as well (I still think there's a little job for the mingw tar developers).
It may be a little later this week before I have something acceptable (busy week ahead)
|
Sun 15 Mar 2015 08:36:25 PM UTC, comment #51:
I think we should concentrate on using the existing tar (msys-tar) and use Philip's patch to adjust the path to something it accepts.
The other two tar choices have shortcomings. gnuwintar we can't get to cross-compile and bsdtar spits out a different format of filelist which would require post-processing.
|
Sun 15 Mar 2015 07:34:07 PM UTC, comment #50:
No luck here with getting gnuwintar to compile under mxe. Since its such an old version, its probally going to be easier to make a current tar work, or a modified bsdtar.
|
Sat 14 Mar 2015 11:57:50 PM UTC, comment #49:
I took a quick look at gnuwin32tar - its going to take some manipulation to get it to cross compile
|
Sat 14 Mar 2015 11:04:13 PM UTC, comment #48:
In the mean time I found out why gnuwin32 tar.exe doesn't work in Octave:
Indeed it depends on its very old libiconv-2.dll and refuses to run with the one built by MXE (that is ~12 years newer). OTOH Octave refuses to start with the old (2003) libiconv-2.dll (obviously).
Maybe if we can build our own gnuwin32 tar.exe the whole unpack.m/tar.m issue is solved.
|
Sat 14 Mar 2015 09:39:38 PM UTC, comment #47:
Sorry, my mistake.
The was another tar.m file in my working directory.
I removed it, and it is OK now .
|
Fri 13 Mar 2015 06:54:03 PM UTC, comment #46:
@Avinoam: The problems with directory changes after running 'test tar' are still fixed for me. Maybe there is cruft in your system?
|
Fri 13 Mar 2015 05:40:12 PM UTC, comment #45:
@Rik / comment #40:
An intriguing thing is that gnuwin32's tar works fine in MSYS and cmd32, but not in the "shell" process spawned by Octave's system() command (it gives return code 127). That is an MSYS shell too, isn't it?
Maybe it is confused because of another libiconv-2.dll (built by mxe), libintl-2.dll was from gnuwin32 as well. But with its own libiconv-2.dll it didn't work either.
Another intriguing thing is that tar does work fine when invoked by pkg.m (....)
So, I'll go on with the ispc() variant? (it did work for me). I'll start then with polishing my patch from comment #2.
Hopefully I can get a little further with it this weekend, depending on my/my families' occupations. Plus, I have a busy week ahead.
@ comment #43
Initially with my patch from comment #2 I saw no directory changes (and test tar worked fine) but later on I noted it again. Also intriguing.
|
Thu 12 Mar 2015 08:15:34 PM UTC, comment #44:
I'll have to recompile in order to test so it won't be until this evening that I have a good idea whether I can reproduce the problem.
I do know that the Octave code base is pretty unsettled right now. I'm having lots of odd behaviour on both Linux and Windows where sometimes thing work and other times I get segfaults.
|
Thu 12 Mar 2015 07:40:50 PM UTC, comment #43:
@Rik, comment #30:
Checking version http://hg.savannah.gnu.org/hgweb/octave/rev/065ccb9a1324,
test tar still changes working directory.
It's even strange, because in some directories it does change
the working directory, and in other it doesn't.
|
Wed 11 Mar 2015 11:12:14 PM UTC, comment #42:
Alright, I give up. Why don't we use ispc() to craft a special tar command for Windows as Philip suggested.
|
Wed 11 Mar 2015 09:40:33 PM UTC, comment #41:
gnu tar doesn't seem to want to compile for mingw without some modifications.
So far, it looks like its been issues that using more gnulib functionality will fix.
|
Wed 11 Mar 2015 09:33:34 PM UTC, comment #40:
I think we need GNU tar, rather than GNU Win32 tar. I just tried gnuwin32 as well and it doesn't seem to like drive letters.
|
Wed 11 Mar 2015 04:42:56 PM UTC, comment #39:
Hmm spoke too early. Just dropping gnuwin32 tar.exe + libint2.dll into OCTAVE_HOME/bin doesn't make tar work.
Unfortunately I'm running out of time - Friday I have more time for this.
|
Wed 11 Mar 2015 04:21:18 PM UTC, comment #38:
@comment #36:
It seems that gnuwin32 tar [1] can do the job fine, although the binary dates back to 2003.
ISTR that gnuwin32 binary utils were bundled with earlier Octave releases (3.2, 3.4 ?).
So that looks to be the final solution.....
Neither "test tar" nor "test unpack" complain about "cannot connect to c:/....." with gnuwin32 tar
|
Wed 11 Mar 2015 06:34:01 AM UTC, comment #37:
@Rik / comment #34:
Indeed, just tar, see comment #1.
That's why 2 ispc()'s wouldn' t hurt much IMO
|
Wed 11 Mar 2015 12:42:21 AM UTC, comment #36:
gnu win32 tar [1], or gnu tar [2]?
[1] http://gnuwin32.sourceforge.net/packages/gtar.htm
[2] http://www.gnu.org/software/tar/
|
Tue 10 Mar 2015 10:34:14 PM UTC, comment #35:
After substituting GNU tar I get very close. The problem now is that the output filelist has CR/NL as the line separator and the m-file is coded to assume just NL. I can work around that by modifying the ostrsplit call. So really, it comes back to whether we can get a tar that works with native separators and returns output on stdout. GNU tar seems to fit the bill.
|
Tue 10 Mar 2015 10:08:54 PM UTC, comment #34:
@Philip: I thought that this problem was generalizable to most of the utilities, like gzip, tar, bzip2, that we might use. Is it really only tar that has this problem? Or have we not tested for the path separator bug in the other utilities?
I just tried gzip and bzip2 and they only accept the native, 'c:\', notation rather than the MinGW syntax. Is it possible to get GNU tar? I have that on my Windows system and it seems to work with native notation.
|
Tue 10 Mar 2015 09:41:37 PM UTC, comment #33:
@Rik, comment #31:
AFAICS only tar.m and unpack.m would require ispc() checks.
I'm not aware of other places in Octave's code where a tar binary is called directly.
So I think it may still be easier to use the patch I made in comment #2, to simply overhaul the file / directory path before feeding it to msys tar.exe. That'll avoid having to maintain our own version of tar/bsdtar, and will preserve some independence of mxe builds.
For tar.m the fix is already there; for unpack.m (related) I think I've spotted a good location for the fix.
|
Tue 10 Mar 2015 09:28:32 PM UTC, comment #32:
@JohnD, comment #28:
The superior, but involved, solution would be to fix the original tar to accept Windows-style path names and donate that back to mingw, rather than fix bsdtar to copy output to stdout instead of stderr (and strip the file attributes).
Anyway if you think it is easy to adapt bsdtar then it might be a good temporary solution. I feel somehow we'll get on slippery territory that way. Be aware that bsdtar's core code could be in a library - that's the main difference with the self-contained msys (GNU) tar.
The msys/mingw folks do not seem very content with Windows idiosyncrasies (see comment "#8). Nevertheless I suppose it might be good to bug them again; I find it a shame that such a basic utility like tar isn't adapted to the host platform.
I'll think about that, = bringing it up again upstream.
Anyway I'm sorry that you do 90 % of the hard work of this bug fix while I only bring up more problems :-)
@Rik, comment #27: maybe what you see is due to the spaces in the path name?
|
Tue 10 Mar 2015 09:22:40 PM UTC, comment #31:
I would really prefer if we could get the utilities to behave the same way under Linux and MinGW. Otherwise I foresee a bunch of ispc () checks all throughout the code.
However, just a check, I used
which uses the shell to send STDERR to STDOUT which then can be captured by the system() call in Octave.
|
Tue 10 Mar 2015 09:07:53 PM UTC, comment #30:
I pushed a cset that at least will restore the original working directory when the test fails (http://hg.savannah.gnu.org/hgweb/octave/rev/ee7159d9a330). That was the original bug title.
I've re-titled to reflect the remaining problem.
|
Tue 10 Mar 2015 09:07:24 PM UTC, comment #29:
Looking at the code, its definitely outputting the files in verbose mode to stderr.
|
Tue 10 Mar 2015 08:46:17 PM UTC, comment #28:
@Rik , my tar doesn't seem to have the 'f1' issue and appears to work ok.
@Philip, I see it outputting the 'a' and does indeed appear to output on all bsdtars in mingw via stderr.
Since we are now 'building our own' tar, does it make sense to change the code of bsdtar to what we want?
|
Tue 10 Mar 2015 08:35:31 PM UTC, comment #27:
I get even crappier behavior out of tar. It refuses to recognize the filename 'f1', although if I rename the file to something longer it works fine. This was discovered in Octave, but I verified it on the command line.
|
Tue 10 Mar 2015 07:46:58 PM UTC, comment #26:
To illustrate the differences between bsdtar and msys tar:
msys (GNU) tar:
bsdtar:
It turns out bsdtar echoes the file list to stderr rather than stdout, apparently with archive "flagbits" prepended.
Is there a way to catch stderr using Octave's "system" command?
|
Tue 10 Mar 2015 07:23:58 PM UTC, comment #25:
I have used John's patch. It created bsdtar.exe and all the
relavant dll's, and renamed bsdtar.exe to tar.exe.
All the dll's were copied to the bin directory, but tar.exe
wasn't, so I get the old tar.exe
|
Tue 10 Mar 2015 07:10:34 PM UTC, comment #24:
Hmmmm, it seems using bsdtar may not be such a good idea after all.
Its output options differ from that of tar: it doesn't echo a list of tar'ed files like tar does; at least not in the same way.
That'll make fixing unpack.m and tar.m on Windows a bit hard.
I'll investigate further.
|
Tue 10 Mar 2015 01:32:53 PM UTC, comment #23:
BTW I just renamed msys-base/tar.exe to ..../tar_orig.exe, and deleted usr/home/......./bin/tar.exe and usr/x86_64-mingw32/bin/tar.exe
That way I also get:
|
Tue 10 Mar 2015 01:30:18 PM UTC, comment #22:
I get:
which looks good. Thanks John!
"test unpack" and "test tar" look better now - no more "can't connect to C:....."
Right, back to the original bug(s) then....
(status back to "In progress")
|
Tue 10 Mar 2015 11:51:13 AM UTC, comment #21:
It should be
I did a completely clean build on my system (and made sure the sys folders msys-base and msys-extension were gone), so chances are you are overwriting the new tar with the msys one.
Looking in Makefile.in, the msys-extension folder doesn't get deleted on a make clean.
And in binary-dist-files.mk, anything in msys-base and msys-extension if effectively copied to the dist files, so any old tar.exe there would be copied.
|
Tue 10 Mar 2015 05:12:14 AM UTC, comment #20:
Same result on Win7
|
Tue 10 Mar 2015 04:57:49 AM UTC, comment #19:
With patch applied, and running in Windows XP VM,
If that is the correct version, then I think Philip can start his patch on top of John's.
|
Mon 09 Mar 2015 07:42:15 PM UTC, comment #18:
Package libarchive, already in mxe, has a version of bsd in it that we weren't building.
Enabled in this patch
(file #33304)
|
Mon 09 Mar 2015 10:55:18 AM UTC, comment #17:
I'd say any bsdtar.exe that works in Octave (try
'system ("bsdtar")'
)
mingw32-bsdtar was suggested by a MinGW-dev and I tried that one. Maybe it is the only one that handles Windows-style full file paths, I don't know.
I noted that in MinGW, bsdtar doesn't have as many options as tar. But when I renamed it to tar, it worked with unpack.m and tar.m (but the bugs in those functions still have to be mended).
|
Sun 08 Mar 2015 10:53:31 PM UTC, comment #16:
Looks like there are 3 versions of bsdtar in msys/mingw:
1) basic-bsdtar
2) msys-bsdtar
3) mingw-bsdtar
And then there is msys-tar, which is what we were currently using.
From previous comments, it has to be the mingw32-bsdtar package?
|
Sun 08 Mar 2015 09:48:03 PM UTC, comment #15:
unpack.m call "tar" directly as well.
I was busy fixing the XFAIL in unpack.m (bug #43979) until I hit this "sub"bug and got stalled.
Once (bsd)tar works I'll pick up this bug and the unpack.m one again.
|
Sun 08 Mar 2015 08:56:34 PM UTC, comment #14:
Im creating patch for mxe. I'll post it when I have verified it works.
|
Sun 08 Mar 2015 12:03:11 PM UTC, comment #13:
Octave looks for tar which it calls out via a system() call (see tar.m in scripts/miscellaneous).
Thus, we need to rename bsdtar to just tar.
|
Sun 08 Mar 2015 02:24:33 AM UTC, comment #12:
Since we only install binaries of the msys tools, we should be able to just add bsdtar to the list installed.
So will the octave scripts look for bsdtar.exe, or does it need to be renamed?
|
Sun 08 Mar 2015 01:42:52 AM UTC, comment #11:
@JohnD: Is it possible to add bsdtar to MXE? Seems like the easiest solution is to have truly compatible UNIX utilities on the Windows side and if the default tar isn't doing it then we should use one that does.
|
Sat 14 Feb 2015 01:42:24 PM UTC, comment #10:
I agree that it seems messy to have a lot of ispc() tests all over the Octave code base. Therefore, my vote is to include working versions of any utilities we need with the Windows build.
--Rik
|
Sat 14 Feb 2015 12:28:13 PM UTC, comment #9:
-revisiting-
The MinGW supplied solution to use bsdtar.exe rather than tar.exe gives rise to a few questions:
- Should mxe-octave (cross-)build bsdtar and in the end rename (or copy) the executable to tar.exe? (= the easiest solution for MinGW builds)
- Or should we introduce a stanza in unpack.m to check for windows ("ïf (ispc)") and then invoke bsdtar rather than tar if the if clause returns true? (a bit messy IMO)
Once this issue is resolved I can proceed fixing "test unpack.m".
I'll add Rik & John D. to the CC list, I'll await their opinion.
Rik, John: the underlying issue is that mingw tar doesn't accept file paths like 'c:\full\path\to\file'; but bsdtar (and gzip and bzip2 and zip) do.
AFAICS, apart from this bsdtar is functionally equivalent to tar.
|
Tue 03 Feb 2015 09:59:23 PM UTC, comment #8:
OK I got an answer from the mingw users ML:
I've tested bsdtar.exe and indeed it works fine. BUT I also needed to d/ld libz-1.dll manually, mingw-get didn't supply this.
I don't have time until next weekend to work on this.
Plus, I can't post on the maintainer ML currently, something is broken there.
|
Sun 01 Feb 2015 11:42:06 AM UTC, comment #7:
@Avinoam:
I haven't had any comments yet from the core devs. Not here, not in the maintainers ML.
Everyone is busy, this is a volunteer project, so just be patient please. It'll be fixed sooner or later.
BTW I've just managed to post in the mingw users list; I'll await comments from there as well.
|
Sun 01 Feb 2015 05:55:45 AM UTC, comment #6:
Hi,
the patch works well. Why didn't you push it?
|
Mon 19 Jan 2015 10:25:23 PM UTC, comment #5:
Yes, I saw that; plus two more instances in the examples in the texinfo header.
I'm still awaiting comments from the core devs before I make a better cs.
I haven't had any answers yet on my post in the maintainers ML.
Fixing unpack.m so that the actual xtest in tar.m becomes a real test is much harder than I first thought. The only way out in the current setup is if tar.exe is fixed.
I haven't managed to post in the mingw users list yet as I got some issues registering there :-(
|
Mon 19 Jan 2015 08:58:55 PM UTC, comment #4:
There is a typo in the patch: last lile of __w2mpth__.m
should be
|
Fri 16 Jan 2015 10:01:34 PM UTC, comment #3:
The patch works for me.
Thanks
|
Fri 16 Jan 2015 04:50:31 PM UTC, comment #2:
The attached changeset fixes the "original" bug (= not returning to the original working directory) and makes tar.m run on Windows.
It introduces a new private function _w2mpth_.m as I suppose it'll be needed elsewhere as well.
As to the failing xtest, that suffers from more or less the same issue, I'll post in the maintainers ML.
Again, I'd like core devs to have a look at the fix.
(file #32841)
|
Fri 16 Jan 2015 11:46:08 AM UTC, comment #1:
A few issues in one.
1. Apparently the unwind_protect block doesn't work as advertised; the command to restore the old working directory in the unwind_protect_cleanup block doesn't get executed, that's
why you're left in the temp subdir.
As to why the unwind_protect block fails (the actual bug), at least in Windows, I don't know.
2. The actual test (yet an xtest) fails in tar.m in the "system" command on line 61.
That issue is that mingw tar doesn't like pathnames with a leading Windows style drive letter ("C:\blahblah...") and backslash fileseps, but insists on mingw style leading drive letters ("/c/blahblah...") and forward slash fileseps.
I suppose that's one of those creepy issues in the Windows/Mingw/Linux blend constituting MXE that the core devs (who mostly develop on *nix) just cannot foresee :-)
Maybe this also holds for other archivers (gzip, zip) as well.
Investigating a bit further I get the impression that this might be an old bug that only comes to light now that tests have been made for tar.m in 4.1.0+.
I'll first try to prepare a fix for the Windows side so that the call to "system" in tar.m works in mingw. But there's a risk that it'll break MSVC-compiled MXE Octave as I cannot rule out that tar.exe does accept "C:\" style drive letters there.
If time permits, I'll also try to see if I can fix the "xtest" as well so that it becomes a genuine "test".
Thanks for your report, Avinoam.
|
Wed 14 Jan 2015 09:02:04 PM UTC, original submission:
"test tar" changes working directory to temporary directory in MXE-Octave ver 4.1.0+
|