bugmake - Bugs: bug #53201, Target runs incorrect command when...

 
 

bug #53201: Target runs incorrect command when shebang line exceeds kernel limit

Submitted by:  Casey McGinty <cmcginty>
Submitted on:  Wed 21 Feb 2018 07:35:13 AM UTC  
 
Severity:  3 - Normal Item Group:  Bug
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  3.81 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 08 Apr 2018 10:57:29 PM UTC, comment #4:

I don't understand the issue being reported. It would be greatly illuminating if you could provide an example of the extra-long shebang line for me to examine. I've tried for a bit to reproduce the problem and cannot. It's not clear from your report whether it's the path to the interpreter which is too long, or whether it's an argument to the interpreter which is too long.

The situation cannot be as simple as you suggest. First, the exec operation that make invokes is in a forked process (obviously) so even if we were to try to catch the ENOENT error, we couldn't do anything about it in the main make process. It doesn't know anything about failures during exec in its child process (it only knows when the child process exits, whether it had an error or not).

Second, make itself never tries to parse the shebang line in a script. It merely runs the program as provided to it. Make uses the execvp() function in the forked process to invoke the command and this function performs PATH lookup then invokes execve() (according to the GNU/Linux man page, these functions are wrappers around execve()).

If the way make is invoking processes and the way the shell invoke them are different then I can only put it down to differences in the way that the shell and the execvp() function in GNU libc treat interpreter scripts. I'm not sure I can see what GNU make should do about this difference.

Paul D. Smith <psmith>
Project Administrator
Thu 22 Feb 2018 01:12:28 PM UTC, comment #3:

Sorry, I was thinking of a different kernel limit but my second observation, that the difference observed has to do with fast path optimization, remains. Working around it by setting SHELL=//bin/sh in the makefile is probably safer than .ONESHELL. Again, discussion relating to the fix per se belongs to Paul.

David Boyce <boyski>
Thu 22 Feb 2018 07:06:23 AM UTC, comment #2:

See http://man7.org/linux/man-pages/man2/execve.2.html

A maximum line length of 127 characters is allowed for the first line in an interpreter script.

My kernel is Linux 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

This is a production system and can not be updated, however I don't see how that would help.

The kernel define is still set to 127:
https://github.com/torvalds/linux/blob/master/include/uapi/linux/binfmts.h#L19

If you think this is incorrect, please explain where my misunderstanding is.

Casey McGinty <cmcginty>
Wed 21 Feb 2018 01:06:40 PM UTC, comment #1:

I think a couple of facts here may need correcting though the issue is quite real. I suspect you are not running into a kernel limit since modern Linux kernels have no such limit (at one time the limit was 131K, then doubled to 262K, then removed altogether). However, the mere fact that the kernel imposes no limit doesn't mean individual utilities, in particular the shell, have the same property.

Most likely the limit being exceeded belongs to bash. The so-called fast-path optimization in make skips the shell and execs the recipe directly if it contains no shell special characters, thus in your original test case the shell isn't used which explains the variance. Use of .ONESHELL suppresses fast-path causing the shell to come back in and fail properly.

There are other ways of suppressing fast path to test this thesis; try appending a semicolon to the recipe line or overriding the SHELL variable. One of my favorite tricks is SHELL=//bin/sh which addresses the same executable while convincing make it's different, thus disabling fast path.

I have no wisdom to offer regarding the fix. Your solution sounds fine but that belongs to Paul.

David Boyce <boyski>
Wed 21 Feb 2018 07:35:13 AM UTC, original submission:

I have a Python script (in ./env/bin/pip) generated by virtualenv with a very long shebang line. The virtualenv is active and the PATH env contains "./env/bin;...." before all other paths. When I run "pip" from the system shell I see the following error:

```
... /bin/python: can't open file 'pip': [Errno 2] No such file or directory
```

This error is expected because the shebang line is beyond the limit allowed by the kernel.

However when the same command runs in Make, it does NOT fail. Instead, it succeeds by running /bin/pip. This is very unusual since the local script `./env/bin/pip` is first in the PATH . The problem with running the system pip is that this should never happen as the local environment thinks it is inside of Python virtualenv. By running the system pip, we are tainting the system Python runtime environment with unwanted packages.

Some debugging with strace revealed exactly what Make is doing:

```
[pid 8578] execve("/var/lib/jenkins/workspace/..../env/bin/pip", ["pip", "list"], [/* 26 vars */]) = -1 ENOENT (No such file or directory)
[pid 8578] execve("/sbin/pip", ["pip", "list"], [/* 26 vars */]) = -1 ENOENT (No such file or directory)
[pid 8578] execve("/bin/pip", ["pip", "list"], [/* 26 vars */] <unfinished ...>
```

Note that adding the .ONESHELL directive resolve the issue and causes Make returns the expected error string above.

I think the proper fix is to stat() the file before accepting the ENOENT error from execve. If the file exists and execve still returns an ENOENT error, Make should fail the target and return the error to the user.

Casey McGinty <cmcginty>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by boyski (Posted a comment)
  • -email is unavailable- added by cmcginty (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    No changes have been made to this item

    Back to the top


    Powered by Savane 3.3