bugmake - Bugs: bug #45763, Split args along MAX_ARG_STRLEN...

 
 

bug #45763: Split args along MAX_ARG_STRLEN for linux/posix

Submitter:  None
Submitted:  Fri 14 Aug 2015 07:06:31 PM UTC
Votes: 2
 
Severity:  3 - Normal Item Group:  Bug
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.1 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 25 Nov 2020 05:42:39 PM UTC, comment #3: 

I've attempted to update the patch so that it compiles successfully on current master and added a test. Commit message follows:
--8<--
When presented with a very long command line (as is common when linking
a large number of files with absolute paths in a deep subdirectory),
make fails to execute the command as it doesn't split the command line
to fit within the limits.

This is based on a fix for Debian bug 688601[1] by Adam Conrad applied
by Manoj Srivastava originally applied for Debian make-dfsg 4.0-5 in
2014 but dropped in 2018 (it seems under the incorrect assumption that
it had been accepted upstream.)

I've tweaked Adam's original patch so that it compiles successfully with
-Werror on top of current master. This required:

  • moving the eval_line declaration to the top of the block, so I moved

  the macros too

  • using a const variable when iterating over the shell
  • adding a cast to avoid a signed/unsigned mismatch.


As suggested in the Savannah bug report[2], I've added a test case that
fails without the rest of the patch. I'm not sure what the consequences
of running the test on non-POSIX targets would be and whether it needs
marking as an expected failure.

  • src/job.c (construct_command_argv_internal): support running commands

longer than MAX_ARG_STRLEN

  • tests/scripts/features/long_command_line: add test for such a command
  • configure.ac: check for now-required sys/user.h and linux/binfmts.h

headers

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=688601
[2] https://savannah.gnu.org/bugs/?45763#comment2

Mike Crowe <mac>
Thu 16 May 2019 04:04:59 PM UTC, comment #2: 

1) the patch could do with a test. 

2) grpc hits a similar problem, see https://github.com/grpc/grpc/issues/14844

Dan Kegel <dank>
Wed 08 Feb 2017 10:02:33 PM UTC, comment #1: 

I've rebased the patch to the latest git master as of this writing.

Keshav Kini <kini>
Fri 14 Aug 2015 07:06:31 PM UTC, original submission:  

When you run a make command where one of the targets has a sufficiently large enough arg list, you get:

make: execvp: /bin/sh: Argument list too long

This is because in GNU make they don't split arguments to /bin/sh in their $(shell) directive, so it's just /bin/sh -c "everything". This is running into MAX_ARG_STRLEN, and the debian patch definitely fixes it.

On debian they patch this to split along the pagesize so you can execute things of arbitrary length arguments. I've attached the patch that debian uses in their current package of make (4.0). My limits.h has things like stacksize set to unlimited, which means the following:

; getconf ARG_MAX
4611686018427387903
; cat /usr/include/linux/limits.h | grep ARG_MAX
#define ARG_MAX       131072    /* # bytes of args + environ for exec() */
; grep -HIins MAX_ARG_STRLEN /usr/include/linux/binfmts.h
/usr/include/linux/binfmts.h:10: * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to
/usr/include/linux/binfmts.h:14:#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
; cat > test.c
#include <stdio.h>
#include <unistd.h>
int
main() {
    printf("ARG_MAX: %ld\n", sysconf(_SC_ARG_MAX));
    printf("PAGESIZE: %ld\n", sysconf(_SC_PAGESIZE));
}
^D
; gcc test.c && ./a.out
ARG_MAX: 4611686018427387903
PAGESIZE: 4096
;

Steps to reproduce:
Create a makefile with the following contents:

---
ALL_FILES := $(shell / -type f)
ECHO_FILES := echo $(ALL_FILES) | fmt -1

all: $(shell /bin/bash $(ECHO_FILES))
ls /
---

Name it 'makefile.test' or something, and then run:

make -f makefile.test all

And you'll see it print the error message.

Fix:

Debian has a patch that fixes this behavior, see my attachment.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50343:  0001-SV-45763-Fix-large-command-line-on-POSIX-systems.patch added by mac (6KiB - text/x-patch - Updated patch with test case)
file #39706:  0001-Patch-to-fix-large-command-line.patch added by kini (4KiB - text/x-patch - Rebased patch to latest git master (2017-02-08))
file #34646:  0009-handle_excessive_command_length-Patch-to-fix-large-c.patch added by None (4KiB - text/x-patch - debian patch to make for long argument)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by bertsky (Voted in favor of this item)
  • -email is unavailable- added by mac (Posted a comment)
  • -email is unavailable- added by dank (Posted a comment)
  • -email is unavailable- added by kini (Voted in favor of this item)
  • -email is unavailable- added by kini (Updated the item)
  •  

    There are 2 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-06-26 bertsky Carbon-Copy- Added bertsky
    2020-11-25 mac Attached File- Added 0001-SV-45763-Fix-large-command-line-on-POSIX-systems.patch, #50343
    2017-02-09 kini Carbon-Copy- Added kini
    2017-02-08 kini Attached File- Added 0001-Patch-to-fix-large-command-line.patch, #39706
    2015-08-14 None Attached File- Added 0009-handle_excessive_command_length-Patch-to-fix-large-c.patch, #34646

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code