bugmake - Bugs: bug #61409, The code used to create...

 
 

bug #61409: The code used to create sub-processes on Windows triggers the spawnve() issue

Submitter:  Liviu Ionescu <ilg>
Submitted:  Mon 01 Nov 2021 09:13:09 AM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.3 Operating System:  MS Windows
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 29 Nov 2021 01:30:36 PM UTC, comment #8: 


> Your suggestion to modify the Make's own environment also won't work well, because Make invokes programs asynchronously, i.e. it doesn't wait for the child to exit, and so modifying Make's environment will affect Make itself


I'm not sure this is accurate, my understanding is that the environment is copied to the child, so once the child is created, the parent can further change its own environment unhindered.

> report this bug to Microsoft


I'm currently investigating on how to report this to Microsoft, but, as I said, even if Microsoft fixes the bug, millions of older installs will fail.

> and to MinGW64 folks, and hope that they fix it in a future release.


The folks at Mingw-w64 helped with the detailed diagnose. There is not much they can do. The only thing that is guaranteed to work is to always create the children processes with a NULL environment, and this is up to the application.

Remember, the BusyBox sh.exe, which was the first UCRT program to fail when invoked by make.exe, so the same sh.exe binary works just fine when invoked from cmd.exe. I'd bet that cmd.exe creates the process with a NULL environment.

So, when I report this to Microsoft, they might very well reply that cmd.exe works, and if make.exe doesn't, I should report this to GNU make folks. ;-)



Liviu Ionescu <ilg>
Mon 29 Nov 2021 01:16:41 PM UTC, comment #7: 

It is unrealistic to ask Make not to modify the environment of the programs it invokes, because some of the GNU Make features absolutely require to put stuff into the child environment.  Your suggestion to modify the Make's own environment also won't work well, because Make invokes programs asynchronously, i.e. it doesn't wait for the child to exit, and so modifying Make's environment will affect Make itself.

So my recommendation is to report this bug to Microsoft and to MinGW64 folks, and hope that they fix it in a future release.  Or maybe a less painful workaround could be found.

Of course, the final decision is that of Paul, not mine.

Eli Zaretskii <eliz>
Group Member
Mon 29 Nov 2021 12:53:47 AM UTC, comment #6: 

Now I have a slightly better understanding of the problem, although I had no time to experiment anything yet.

The real problem is in Microsoft UCRT, which has a bug in the code managing the environment.

Unfortunately, even if Microsoft will fix the bug tomorrow, we still have to deal with millions of Windows instances which exhibit the bug. And in this respect, all programs that spawn children processes should consider workarounds, including GNU make.

If my understanding is correct (based on the explanations I received from someone much more knowledgeable than myself), the problem is related to the way the environment is understood by Microsoft.

---
Internally, the "environment" in a Windows process consists of a little more than what you see in the standard C envp array. If you run GetEnvironmentStringsA/W and iterate over it, you'll see the normal environment variables, but also a few other ones, like these:

=::=::\
=C:=C:\code\ms-spawn-issue\sources\spawn
=ExitCode=00000000

These are special vars that start with an '=' - apparently the include information about things like which directory is the "current directory" on each drive letter.

When UCRT's spawn() tries to create a process, it wants to include these extra variables from the current environment (from GetEnvironmentStringsA/W) on top of the variables you provide in the 'envp' parameter.

When make starts its subprocesses, it does that by calling CreateProcess() and passing in a string with the desired environment, but it doesn't know/care about this kind of extra variables. So processes launched by make are missing these extra variables in GetEnvironmentStringsA/W.

When launching a process with CreateProcess() and passing NULL to the environment parameter, or passing NULL to the envp parameter to spawn, there's no issue (the current exact environment is passed on). But if UCRT's spawn is called in an environment that is missing these extra variables, it crashes.
---

So, if make creates sub-processes with a custom environment (i.e. does not pass NULL to inherit the parent), those processes cannot use UCRT to create sub-processes.

The same applies to sh.exe which spawns, for example, the compiler, to the compiler who spawns the internal steps, and so on.

The workaround is relatively simple. The only known way that is guaranteed to work is to pass a NULL environment, which means to inherit the environment from the parent.

This means that instead of creating a custom environment, the parent must temporarily adjust its own environment to match the desired one, spawn the process, and restore the initial environment.

I did not check the code used by make, but if it passes a custom environment and not a NULL, its children are affected by this Windows bug.

Regarding the question related to UCRT vs MSVCRT, the use of MSVCRT was deprecated about 10 years ago when UCRT was introduced. mingw-w64 already implements UCRT for some time, and it is very likely to become the default in future mingw-w64 releases, since the MSVCRT code has many limitations and is no longer maintained. All recent binary toolchain distributions (both clang and gcc) produce binaries the use UCRT.

Thus the suggestion to stick to MSVCRT is not realistic.

Sure, you can close the issue, but this will not solve the problem.


Liviu Ionescu <ilg>
Sun 28 Nov 2021 08:46:52 PM UTC, comment #5: 

Reading the busybox issue thread here https://github.com/rmyorston/busybox-w32/issues/234 it appears that the problem really is not related to GNU make, although I can't understand why the failure is only seen if the program is invoked from GNU make and not from other tools or directly from the shell.

In any event, I'm tempted to close this issue.  @ilg please comment.

Paul D. Smith <psmith>
Group administrator
Sun 28 Nov 2021 08:18:40 PM UTC, comment #4: 

If Make invokes other programs, and those programs have problems because the use UCRT, how is that a problem of Make?

What I meant by my comment about MinGW was that as long as Make doesn't use the 'spawnve' implementation from UCRT, Make itself is not directly affected by that problem.

Eli Zaretskii <eliz>
Group Member
Sun 28 Nov 2021 08:10:53 PM UTC, comment #3: 

I agree that GNU make shouldn't / can't fix a problem in the MS runtime.  The question is, whether there's a way GNU make could work around or avoid the problem (and how annoying that would be).

I'm not sure I understand the comment that as long as MinGW doesn't use UCRT, it's a theoretical problem; surely GNU make on Windows is used to invoke other kinds of programs, that aren't built with MinGW?  Or did I misunderstand what you meant?

Liviu: you didn't provide any information on how you compiled GNU make; can you specify?  Did you use MSVC with the default build_w32.bat file?  Did you use GCC with build_w32.bat?  Something else?  Maybe that doesn't matter WRT this issue.

Paul D. Smith <psmith>
Group administrator
Sun 28 Nov 2021 07:47:49 PM UTC, comment #2: 

AFAIU, this problem only happens when using the UCRT runtime, which is not what MinGW uses.  So, as long as MinGW doesn't use UCRT, this is largely a theoretical problem for GNU Make on Windows.  If and when MinGW64 will start using UCRT as its runtime, the problem should be reported to them, so that they provide some workaround.

IOW, I don't think it should be the GNU Make's business to fix a problem in the MS runtime.

Eli Zaretskii <eliz>
Group Member
Sun 28 Nov 2021 07:38:26 PM UTC, comment #1: 

Unfortunately I have no experience in Windows.  Someone who has such experience will need to track this down and figure out what's going wrong.  I don't know if Eli has time or energy for this or not.

Paul D. Smith <psmith>
Group administrator
Mon 01 Nov 2021 09:13:09 AM UTC, original submission:  

Although I cannot tag this report as a direct bug in make, it is a major contributing factor.

It seems that the implementation of 'spawnve()' in the modern Microsoft UCRT API has a problem, and in certain conditions, calling the 'spawn*e()' functions with a non-NULL environment, crashes with Error -1073741819 which is 0xC0000005, or ERROR_ACCESS_DENIED.

Unfortunately these conditions are met by GNU make, and some programs invoked by 'make.exe' that try to create sub-processes with 'spawn*e()', fail.

Since 'spawnve()' is also called by BusyBox in its 'sh.exe' implementation, this is a major issue, because it affects most common build tools.

I documented this issue in a separate project:

- https://github.com/xpack-dev-tools/ms-spawn-issue


Since ideally make should be able to start any sub-processes, including those who use the unreliable 'spawnve()', it would be useful to review the Windows specific code and try to avoid this problem.

A good source of inspiration might be Ninja Build code, which does not trigger the condition that affects 'spawnve()' calls.

My knowledge of Windows is not good enough to provide a patch myself, but I did my best to diagnose this issue, and I offer my help for further testing.


Liviu Ionescu <ilg>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by eliz (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by ilg (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code