bugmake - Bugs: bug #15718, Weird behavior of the SHELL...

 
 

bug #15718: Weird behavior of the SHELL variable on Win32

Submitter:  Agent Zhang <agent>
Submitted:  Sun 12 Feb 2006 08:52:46 AM UTC
Votes: 1
 
Severity:  3 - Normal Item Group:  Bug
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  None Operating System:  MS Windows
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 06 Mar 2006 04:51:04 AM UTC, comment #19: 

The real bugs wrt SHELL that still need to be worked on are:

1. When the Windows port searches along PATH for a shell whose name is not cmd, it does not try to append the executable extensions, so SHELL=foo would not find foo.exe

2. If the value of SHELL is not a simple file name, Make does the wrong thing: it decides that the shell cannot be found and uses the default value instead.

3. There's also a mess regarding the unixy_shell flag: it is not set/reset correctly, in some situations, if the Makefile or command line specify a non-default shell.  The correct place to set that flag is in job.c, where the shell is actually used.

The first one is relatively easy to fix, provided that the release won't happen before next weekend.  The second and the third I suggest to postpone until after the release.

Eli Zaretskii <eliz>
Group Member
Sun 05 Mar 2006 08:43:52 PM UTC, comment #18: 

OK, I've applied Eli's patch.

This bug/thread has gotten way out of control: I can't determine what's actually a real problem and what's just a misunderstanding.  Can one or more of the principles followup with a comment clarifying what the real bug is here, or if this should be closed now that this patch is applied?  Thanks!

Paul D. Smith <psmith>
Group administrator
Sat 04 Mar 2006 07:35:33 PM UTC, comment #17: 

Okay, I've found the culprit: setting SHELL from the command-line arguments wasn't handled as setting it from the Makefile.  I think that's a bug (the Unix version doesn't handle SHELL in any special way when it comes from the command line), so I think it should be fixed by the attached patch .


Eli Zaretskii <eliz>
Group Member
Sat 04 Mar 2006 06:17:19 PM UTC, comment #16: 

Thanks.

Does it work if you put

SHELL=cmd

inside the Makefile?  That does work for me.

The fact that it doesn't work with setting SHELL from the command line might mean there's another place in the code where the special treatment of cmd should be added.  Hmmm...

Eli Zaretskii <eliz>
Group Member
Sat 04 Mar 2006 03:19:15 PM UTC, comment #15: 

Hmm...here is the screen-shot:

D:\Agent2002\AISystem1.0\SVK\de\UniSimu\Methodology>win32-make SHELL=cmd test
dmd -quiet binary_tree_set3.d
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

D:\Agent2002\AISystem1.0\SVK\de\UniSimu\Methodology>type Makefile
RM_F = perl -MExtUtils::Command -e rm_f

all: binary_tree_set3.exe palindrome.exe char_stat.exe

%.exe: %.d
        dmd -quiet $<

.PHONY: test clean

test: all
        prove *.t

clean:
        $(RM_F) .exe .obj *.map

D:\Agent2002\AISystem1.0\SVK\de\UniSimu\Methodology>win32-make -v
GNU Make 3.81rc1
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for Windows32

--- END ---

I must admit that this problem doesn't appear very time. I don't know what's going on here.

FYI, I always have Cygwin-shell visible in my PATH environment.

Agent

Agent Zhang <agent>
Sat 04 Mar 2006 02:46:26 PM UTC, comment #14: 

Paul is right: the Windows port does not invoke cmd.exe with the -c switch.  So there's no bug here and nothing to fix.

Here are the gory details:

If you look closer at job.c:construct_command_argv_internal, you will see that, in the `slow' case, if the shell is not a Unixy shell, the code creates a temporary batch file via the call to create_batch_file.  It then writes to that batch file the actual command, but without the shell name and the -c switch (it uses command_ptr which was set past the shell name and -c).  Next, it creates new_argv[] whose zeroth member is the name of the batch file.  This new_arg[] array is then passed (under the name argv[]) to process_easy, which simply calls CreateProcess to run the batch file.  The OS will then invoke the shell to do that.

So there's no -c switch passed to cmd.exe nowhere in sight.

Btw, when you type "cmd -c echo hello", cmd.exe doesn't ignore that, it actually invokes another copy of cmd.exe (that's why you see the Windows banner).  Type "exit" to return to your original shell.

If you still think there's a bug related to working with cmd.exe in a Makefile, please show a test case where it fails.  AFAIK, the current release candidate code supports cmd.exe with no problems.

Eli Zaretskii <eliz>
Group Member
Sat 04 Mar 2006 02:08:16 PM UTC, comment #13: 

Well, you can't assume that just because on UNIX make forcibly adds a -c to every $(SHELL) invocation, that it also does the same thing in the other ports.  There's a lot of special code to handle different types of "shell" on Windows; Eli etc. would have to say for sure because it's tricky to track the code with all the ifdefs, but it's quite possible the Windows cmd.exe ports do something different.

As you say, it'd be surprising if this failed so completely.  Can you show a misbehavior related to this using real make commands?  If not I'm inclined to think the cmd.exe -c is not an issue.

Paul D. Smith <psmith>
Group administrator
Sat 04 Mar 2006 11:59:32 AM UTC, comment #12: 


> P.S. In general, I think that setting SHELL to anything but a
> shell, either cmd.exe or some version of Unixy shell, is
> playing with fire


If I'm not misunderstanding you, the Win32 build of GNU make 3.81rc is supposed to work with cmd.exe, right? But surprisingly it does NOT work at all on Windows XP!

I'm shocked when I see the following (I'm sure you all will too):

D:\>cmd /C "echo hello"
hello

D:\>cmd /c "echo hello"
hello

D:\>cmd -c "echo hello"
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

D:\>

Apparently WinXP's cmd.exe unfortunately ignores the `-c' option silently. As Paul pointed out earlier, "(in GNU make) there's no way to change the argument from '-c' to something else (or nothing)."

I'm still wondering why I didn't figure it out earlier and why Eli hasn't noticed that yet. Maybe what I was using was Windows 2000 Professional which kindly regards the `-c' option?

No clue. :(

I have no idea whether make 3.81 has already released or not. Hopefully Eli and Paul will fix this annoying bug before the BIG release for the last four years.

Thanks!
Agent

Agent Zhang <agent>
Mon 13 Feb 2006 10:40:05 PM UTC, comment #11: 

Well, Perl is not a simple-minded shell with simple quoting rules, as one example.  The entire point of SHELL is that you can set it to a command which can handle lines that are as complex as you like, and with a completely arbitrary syntax.  Maybe the SHELL is a Java interpreter, or Python or Ruby, or something like that.

Because make invokes the command in SHELL using exec() directly, the value passed to that command doesn't have to undergo shell quoting at all (at least not in UNIX), so it can have totally different syntax and semantics than the regular shell.

Paul D. Smith <psmith>
Group administrator
Mon 13 Feb 2006 10:09:44 PM UTC, comment #10: 

I'm not sure it's a bug in the Windows port that an arbitrary value of SHELL does not immediately "goto slow"; after all, it's hard to imagine radically different quoting rules for the relatively simple-minded shells we have on WIndows.

But I will think about this some more when I find time to work on this issue.

Eli Zaretskii <eliz>
Group Member
Mon 13 Feb 2006 08:42:33 PM UTC, comment #9: 

Just a note: changing SHELL shouldn't be dangerous in the way you mention, Eli.  On UNIX, at any rate, if SHELL is not set to an acknowledged value for a Bourne shell then make doesn't even try to use the fast path: all invocations of commands are sent to the SHELL command via the slow path.  So, issues of quoting, etc. are not relevant: the whole thing is handed to the shell regardless and whatever quoting rules apply there will be followed.  Note that, since you changed SHELL, it's obviously up to you to ensure that your commands work properly!  If that's not the case in Windows (and from what you've written here it doesn't appear to be) then that's a bug.

The only real problem for changing the shell is the one I mentioned in comment #3: there's no way to change the argument from "-c" to something else (or nothing).  So if the program you want to use for SHELL doesn't grok -c you have to wrap it in something that does.

Paul D. Smith <psmith>
Group administrator
Mon 13 Feb 2006 08:30:08 PM UTC, comment #8: 

Here's a response to Comments #2 and #5:

It's not true that ``SHELL variable assignment within Makefile doesn't work for Win32 port of GNU make''.  It's just that it's a bit tricky, and also your test Makefile was too simple to force Make to invoke the shell.  Try this Makefile instead:

SHELL = perl.exe

all: ; echo 'hello

Note: the unmatched single quote is intentional (it forces Make to invoke the shell)!

Also note that the .exe extension is currently required, since Make looks for the shell's file name without appending executable extensions (I think this is a misfeature, so I will try to fix it soon).

With these changes in the Makefile, I see with the latest beta of Make 3.81 the same behavior as you see in the Cygwin build.

P.S. In general, I think that setting SHELL to anything but a shell, either cmd.exe or some version of Unixy shell, is playing with fire, and not only on Windows: GNU Make assumes too much about the shell (it knows about characters that need special treatment and commands that are built into the shell, for starters).  Using a program such as Perl that violates these assumptions will generally cause things to fall apart.

Children, don't try that at home!

Eli Zaretskii <eliz>
Group Member
Mon 13 Feb 2006 08:13:27 PM UTC, comment #7: 

Let me clarify: Comment #6 is for Doug Konrad.

Eli Zaretskii <eliz>
Group Member
Mon 13 Feb 2006 07:59:27 PM UTC, comment #6: 

Please try the pretest version of Make 3.81 (beta4).  This problem should be fixed in that version.

Eli Zaretskii <eliz>
Group Member
Mon 13 Feb 2006 05:47:18 PM UTC, comment #5: 

I was just trying to get the native W32 build to work, as I'm upgrading our build systems from 3.74 to 3.80. We use cmd.exe almost exclusively, and the W32 build broke the system because it refused to use cmd.exe.

I examined the source, and in variable.c, the logic for DOS file handling depends on _MSDOS_ which is true for the DJGPP build, but not the native W32 build. The native W32 build uses WINDOWS32 for this purpose and has separate logic.

Doug Konrad <dkonrad>
Mon 13 Feb 2006 10:11:05 AM UTC, comment #4: 

Hi, Paul~

Everything you said is very true. I'm a Perl programmer and I know what `perl foo' and `perl -e foo' do. And I know GNU make always sends `-c' to the shell as well. Furthermore, I also quite agree the idea of SHELLFLAGS and the current workaround by writing a wrapper around the perl interpreter, and etc, etc, etc.

But, unfortunately, you seem to miss my point in [comment #2]. Sorry, I didn't make my point clear enough. I think I have to rephrase it here:

  • SHELL variable assignment within Makefile doesn't work for Win32 port of GNU make. That's the bug.


  • SHELL variable assignment on the command line does work for Win32 port of GNU make. That's what we expect


In the screen-shot given in my previous [comment #2], cygwin-make did the right thing although it complains.

BUT, win32-make did NOT do the right thing. win32-make should have complained but it didn't. win32-make apparently did not even try to use `perl' as the shell at all. It must looked back to my Cygwin sh.exe or something instead of `perl', the value of the SHELL varaible in the Makefile.

That's all. :=)

This bug really has nothing to do with `perl'. `perl' is just an arbitrary sample value, just as what Eli said in [comment #1].

For another example demonstrating this bug, please take a look at my [comment #3] for [bug #15720]. Thank you for your patience. :P

Respectfully,

Agent

Agent Zhang <agent>
Mon 13 Feb 2006 09:06:04 AM UTC, comment #3: 

If you want to have Perl execute a program on the command line you have to give it the -e option (see the Perl manual).  Running "perl foo" causes Perl to try to execute the file named "foo".  If you want to run a Perl script on the command line, which is what make wants to do, then you have to use -e.

Now the bad news: make doesn't support this.  Make always invokes its commands using the syntax "$(SHELL) -c <script>", and there's no way to change that -c to anything else.  This is something I've long considered fixing, by adding a SHELLFLAGS variable or something that could be overridden by the makefile--even in UNIX without changing SHELL this is handy; some people would prefer to invoke /bin/sh with the -e option as well as the -c option.

The only way to get around it right now is write a little script wrapper that takes arguments of the form make provides, then invokes Perl with the proper translation of those arguments.

Paul D. Smith <psmith>
Group administrator
Mon 13 Feb 2006 05:50:09 AM UTC, comment #2: 

Hi, Eli~

To be honest, I got very happy when I heard from you that `SHELL = perl' works for Win32 make. But sadly it doesn't, at least for version 3.81 beta4, as witnessed below:

C:\Temp>type Makefile
SHELL = perl

all: ; echo 'hello'

C:\Temp>win32-make
echo 'hello'
hello

C:\Temp>cygwin-make
echo 'hello'
Can't open perl script "echo 'hello'": No such file or directory
cygwin-make: * [all] Error 2

Although cygwin-make complained to me, it worked just as expected. no?

Sorry, I am too lazy to check in the C source since sometimes source code can be misleading IMHO. :=)

Does I miss something obvious? Does this sound stupid? Anyway, I could be wrong. :P

Thanks!
Agent

Agent Zhang <agent>
Sun 12 Feb 2006 08:22:31 PM UTC, comment #1: 

As far as I could see from the sources, the Windows port currently requires that the value of SHELL be a simple file name.  That is, it cannot be a command with arguments.

So you can use Perl as the value of SHELL, but only thusly:

    SHELL = perl
or
    SHELL = d:/foo/bar/perl.exe

I will add to my TODO to look into lifting this limitation.

Eli Zaretskii <eliz>
Group Member
Sun 12 Feb 2006 08:52:46 AM UTC, original submission:  

I'm using Win32 native build of GNU make 3.81 beta4. The SHELL varaible behaves very differently from my Cygwin build (and perhaps other *NIX builds). Here's a sample Makefile that demonstrates the problem:

all: ; pwd
SHELL = perl C:\agent\script\sh

It seems to me that on Win32 `make' will silently ignore the value of `SHELL' unless the first word of its value contains  'sh' or 'cmd' (no?). Is it a intentional design decision or is it a bug?

To be honest, it will be very helpful if I can redirect SHELL to arbitrary programs on Win32. :=)

Agent

Agent Zhang <agent>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #1622:  shell.dif added by eliz (662B - application/vnd.ms-excel - Patch for setting SHELL= on Windows from the command line)
file #1993:  Makefile added by agent (34B - application/octet-stream - Example Makefile)

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There is 1 vote so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

Only logged-in users can vote.

 

Follow 4 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2006-03-04 eliz Attached File- Added shell.dif, #3471
2006-02-13 dkonrad Carbon-Copy- Added dkonrad
2006-02-13 agent Attached File- Added Makefile, #3399
2006-02-12 agent Carbon-Copy- Added -email is unavailable-

Back to the top

Powered by Savane 3.13-758e.
Corresponding source code