bugmake - Bugs: bug #40241, CreateProcess failure with unixy...

 
 

bug #40241: CreateProcess failure with unixy paths

Submitter:  Mike Hommey <glandium>
Submitted:  Fri 11 Oct 2013 11:54:37 AM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  Fixed Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Component Version:  4.0 Operating System:  MS Windows
Fixed Release:  4.1 Triage Status:  Verified
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 22 Oct 2013 03:57:54 PM UTC, comment #19: 

OK, thanks.  I committed the changes to the repository, and I'm closing this bug.

Eli Zaretskii <eliz>
Group Member
Tue 22 Oct 2013 10:04:06 AM UTC, comment #18: 

From several tests, it does seem to work, despite it not following BATCH_MODE_ONLY_SHELL, which is probably due to sh.exe not screwing up with quotes with unixy paths.

Mike Hommey <glandium>
Tue 22 Oct 2013 09:01:30 AM UTC, comment #17: 

MSYS Make 3.82.90 from the URL below does work well with -jN:

  https://sourceforge.net/projects/mingwbuilds/files/external-binary-packages/

I use that build all the time.

Yes, using single quotes are also a possibility.

Eventually, I need to know whether the last patch I suggested is OK for your use case.  If it is, I will commit those changes to the Make repository.

Eli Zaretskii <eliz>
Group Member
Tue 22 Oct 2013 08:44:18 AM UTC, comment #16: 

Using single quotes seems to be another workaround.

Mike Hommey <glandium>
Tue 22 Oct 2013 08:35:54 AM UTC, comment #15: 

3.8x deadlocks with -jN.

As for paths, it's a complex issue. We can't reliably use msys paths everywhere (and for many reasons, we don't want to use more msys paths)

Mike Hommey <glandium>
Tue 22 Oct 2013 08:22:17 AM UTC, comment #14: 

Indeed, that's what I see as well.

Can you use the MSYS file name notation instead, as in "/c/full/path/to/python.exe"?  My testing indicates that the problem disappears when this format of file names is used.

Btw, these are exactly the problems that convinced me long ago to use only MSYS Make when I need MSYS Bash to be the shell in Make.  (It is true that there's no MSYS Make 4.0 yet, but do you really need 4.0?  3.82.90 is available.)

Eli Zaretskii <eliz>
Group Member
Tue 22 Oct 2013 07:41:04 AM UTC, comment #13: 

Err, I was not looking at the right thing. It is bash not handling quotes properly:

CreateProcess(C:\mozilla-build\msys\bin\sh.exe,C:/mozilla-build/msys/bin/sh.exe -c "c:/full/path/to/python.exe foo.py \"a\" b d",...)

Mike Hommey <glandium>
Tue 22 Oct 2013 07:26:26 AM UTC, comment #12: 

Ah no, I'm hitting it again without BATCH_MODE_ONLY_SHELL.

Here is a small testcase:

$ cat > foo.py <<EOF
import sys
print sys.argv
EOF

$ cat > Makefile <<EOF
foo:
        c:/full/path/to/python.exe foo.py "a" b c
EOF

$ make
["foo.py", "a b", "c"]

This does not happen if using "python" instead of "c:/full/path/to/python.exe".

... which turns out to be a make bug when not using the shell, contrary to what i thought. It happens to be fixed by BATCH_MODE_ONLY_SHELL because it forces the use of a shell in most cases and unbreaks things.

Mike Hommey <glandium>
Tue 22 Oct 2013 04:55:45 AM UTC, comment #11: 

Why not msys make? First, because i haven't found an msys make 4.0 build, and using msys make has its own quirks that are best avoided entirely.

Why "/usr/bin/touch"? Well, it's not actually /usr/bin/touch, it's mostly /usr/bin/install or others, and that's because that's what comes out of configure.

Interestingly, I'm testing your patch with BATCH_MODE_ONLY_SHELL undefined, and so far, it builds fine, but I'm only half-way through.

Mike Hommey <glandium>
Tue 22 Oct 2013 04:23:40 AM UTC, comment #10: 

I actually tested the patch with MSYS Bash.  Can you show me one or two examples of it mishandling quotes?  I use MSYS Bash quite a lot (although not from the native Make), and never saw any problems with quotes.

And, btw, if you use MSYS Bash, why not use MSYS Make as well?  Then all your problems with file names will be gone.

Also, I don't quite understand why you need to use something like "/usr/bin/touch" and not just "touch" (assuming commands are on PATH).

IOW, I wonder whether the use case you presented is really justified, and moreover justifies changes in Make.  I imagine you already have to cope with quite a few quirks given your peculiar combinations of tools; perhaps one more simple quirk will remove this problem without any need to change Make?

As for your proposed patch: I understand that it works for you.  But your use case is quite marginal and borders on situations that the native Windows build of Make is not supposed to support.  If such behavior is to affect everybody, it better be right.  Your patch makes decisions at the wrong place, where it is not yet known what the first word on the command means; it could very well not be the command to execute, but something else entirely.  That is why I suggested the alternative change in the place where Make knows positively that it is looking for an executable file.

As for your question about the number of people using commands that start with a forward slash: I don't know, but I can easily imagine that this number is not smaller than the number of people having your specific problem.  Moreover, while unlikely, using such file names in the native Windows port of Make is a legitimate use case that I cannot possibly dismiss, sorry.

Eli Zaretskii <eliz>
Group Member
Tue 22 Oct 2013 03:48:45 AM UTC, comment #9: 

I'm using a msys bash shell, and i'm having problems with it unless I set BATCH_MODE_ONLY_SHELL.

AFAICT, the patch I attached makes things use BATCH_MODE_ONLY_SHELL as I'd expect, so it works for me. I don't know why you don't want to hook at that level. How many people do you think will be using commands starting with a forward slash that actually point to existing paths on the current drive?

Mike Hommey <glandium>
Tue 22 Oct 2013 02:58:47 AM UTC, comment #8: 

What does "doesn't handle quotes right" mean?  It's a Unixy shell, right?  What kind of shell are you using?

BATCH_MODE_ONLY_SHELL cannot be catered to at this stage, it's something considered at a higher level.  But that higher level doesn't know whether the command is a file reachable by following the filesystem.

So if you cannot use the proposed patch, I guess this problem has no solution.  I can only suggest to use quotes on the command line, which will force Make to use the shell.

Eli Zaretskii <eliz>
Group Member
Mon 21 Oct 2013 09:43:36 PM UTC, comment #7: 

Without testing this patch I know I'll probably have trouble with it, as $SHELL doesn't handle quotes right and I'm setting BATCH_MODE_ONLY_SHELL, which your patch ignores for those.

Mike Hommey <glandium>
Mon 21 Oct 2013 04:44:22 PM UTC, comment #6: 

My patch did had 2 conceptual problems, sorry.

Please try this patch instead.


(file #29434)

Eli Zaretskii <eliz>
Group Member
Mon 21 Oct 2013 02:48:32 AM UTC, comment #5: 

It worked for me, but of course I don't have your exact setup.  I will have another look.

(Don't worry about leaking memory, I just wanted to see if this kind of change fixes your problem.  The actual change will have all this straightened up.)

Thanks for the feedback.

Eli Zaretskii <eliz>
Group Member
Mon 21 Oct 2013 12:23:55 AM UTC, comment #4: 

Besides the fact that your patch would leak the strdup'ed string, it doesn't work:

$ ./Release/make_msvc.net2003.exe -f test.mk --trace
test.mk:2: target 'foo' does not exist
echo foo
foo
/usr/bin/touch foo
/usr/bin/sh: (/w foo: No such file or directory
test.mk:2: recipe for target 'foo' failed
make_msvc.net2003: * [foo] Error 127

(test.mk is the same as in comment #2, but with an additional "echo foo" in the recipe)

Note without the "echo foo", the error message is borked:
$ ./Release/make_msvc.net2003.exe -f test.mk --trace
test.mk:2: target 'foo' does not exist
/usr/bin/touch foo
/usr/bin/sh: Ó<.wÿ: No such file or directory
test.mk:2: recipe for target 'foo' failed
make_msvc.net2003: * [foo] Error 127

Which suggests unintialized memory use or something alike.

Mike Hommey <glandium>
Fri 18 Oct 2013 02:48:37 PM UTC, comment #3: 

Thanks for the test case.

I'm reluctant to "go slow" in this case, because that would change the semantics of the command, as it doesn't have any shell-specific features.

Instead, could you please try the attached patch?


(file #29405)

Eli Zaretskii <eliz>
Group Member
Tue 15 Oct 2013 09:56:07 PM UTC, comment #2: 

Something like:

foo:
        /usr/bin/touch $@

fails because /usr/bin/touch doesn't exist in the filesystem (so, to answer your question, the former).

Mike Hommey <glandium>
Tue 15 Oct 2013 07:41:38 PM UTC, comment #1: 

Can you show a Makefile recipe that fails in this way?

Does it fail because /usr/bin/install is actually mapped to some directory like C:\MSYS\bin, and /usr/bin does not really exist in the filesystem?  Or does it fail because you think CreateProcess chokes on forwards slashes?

Thanks.

Eli Zaretskii <eliz>
Group Member
Fri 11 Oct 2013 11:54:37 AM UTC, original submission:  

When using a MSVC-built make 4.0 with some autoconf/automake-driven projects with a bash shell, some commands retrieved during configure are set with a full unixy path, like /usr/bin/install.

In such cases, the command fails in make because CreateProcess fails.

One way to look at it is that if the command line begins with a forward slash, it's likely to be a unixy path, and it should thus be treated by the shell instead of CreateProcess.

Mike Hommey <glandium>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #29434:  w32_unixy_shell.dif added by eliz (2KiB - application/vnd.ms-excel)
file #29405:  w32_unix_exec_path.dif added by eliz (671B - application/vnd.ms-excel)
file #29354:  diff added by glandium (558B - application/octet-stream - Proposed patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by eliz (Posted a comment)
  • -email is unavailable- added by glandium (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-10-22 eliz StatusNone Fixed
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.1
        Triage StatusNone Verified
    2013-10-21 eliz Attached File- Added w32_unixy_shell.dif, #29434
    2013-10-18 eliz Attached File- Added w32_unix_exec_path.dif, #29405
    2013-10-11 glandium Attached File- Added diff, #29354

    Back to the top

    Powered by Savane 3.13-8ccc.
    Corresponding source code