bugmake - Bugs: bug #52922, -Otarget does not function...

 
 

bug #52922: -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

Submitted by:  John Hood <cgull>
Submitted on:  Thu 18 Jan 2018 02:35:34 AM UTC  
 
Severity:  3 - Normal Item Group:  Bug
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.2.1 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 19 Jan 2018 10:01:14 PM UTC, comment #2:

I spent a while longer looking at this, and being reminded of how messy UNIX locking is.

  • On FreeBSD, none of fcntl-locks, flock(), and lockf() work on pipes.
  • POSIX reminds me that write() is atomic on regular files. If make were to carefully accumulate and concatenate output for stdout and stderr into buffers, and then write() entire buffers, make might not need locking at all for regular files. Of course, buffers of unbounded size would be a problem here.
  • Network file systems have varying support for locking.

I think having either a lock file or a POSIX semaphore to share with children is probably the best way to handle this. This loses the redirected-children-dont-lock property, but in output_dump() make is pretty much just copying the temporary output and doing fairly simple sprintfs for its own output. That should go fairly fast most of the time, and I'd guess concurrency wouldn't be a major problem, except for really pathological cases such as very large numbers of targets or very large output from targets.

John Hood <cgull>
Thu 18 Jan 2018 01:13:08 PM UTC, comment #1:

Yes, I'm aware of this as I've seen it happen on MacOS systems myself. Thanks for filing the bug, as I never got around to it.

I've been considering different ways to address the issue including using a lock file on systems where locking of pipes fails. Using a file that all makes have open is itself not an issue, because the parent make can simply open the file and leave it open when invoking submakes. We just need to create a way for the parent to notify the submakes which FD to use for locking. We already do similar things for, for example, the jobserver pipes so this is not breaking new ground.

However, using stdout has the nice feature of automatically working correctly if you redirect output (for example, if you invoke a submake and redirect it's output to a file--that submake doesn't need to participate in the same output locking that the parent does and since the stdout is a different file descriptor that just magically happens).

So far I haven't found the time to sit down and really think about how this could work correctly using a lock file instead of stdout, or what extra effort make needs to go to. Maybe this enhancement (using a separate lock file for submakes that are redirected) is something that simply won't be supported on those systems as the effort is not worth the reward.

Paul D. Smith <psmith>
Project Administrator
Thu 18 Jan 2018 02:35:34 AM UTC, original submission:

FreeBSD 11.1, GNU Make version 4.2.1 (installed from FreeBSD packages; the distribution patches are small and don't seem relevant here).

On this system, if GNU Make is invoked with options '-j32 -Otarget' and standard output is redirected to a pipe, Make prints the following errors:

jhood@jhood_lfs:/tmp 0 $ gmake -j32 -Otarget | cat
fcntl(): Bad file descriptor
touch foo
fcntl(): Bad file descriptor
touch bar
jhood@jhood_lfs:/tmp 0 $ uname -a

This is because FreeBSD only supports fcntl(..., F_SETLKW, ...) on a few types of files, including regular files and ttys. Pipes and sockets are not supported.

Locking for other than regular files is not required for POSIX compliance, see http://pubs.opengroup.org/onlinepubs/9699919799/

I think the impact is fairly minimal. The error message appears, and locking doesn't happen, but since GNU make gathers output and writes it more atomically in this mode, -Otarget should still mostly work.

This isn't a problem on Linux, which does seem to support F_SETLKW on pipes and sockets at least. I do see the problem on macOS, and I would expect that the problem exists on most/all BSD-derived systems and probably various other POSIX environments as well.

I don't see an obvious easy solution; I think this is intended to work with child invocations of make, and so it's best to use a file that all of them already have open, like stdout. BSD systems could probably use flock() instead, but choosing flock() over fcntl() in Configure would require figuring out which is the better implementation on a given system.

John Hood <cgull>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by cgull (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    No changes have been made to this item

    Back to the top


    Powered by Savane 3.3