mainThe GNU Bourne-Again SHell - Support: sr #110910, Starting a coproc closes process...

 
 

sr #110910: Starting a coproc closes process substitution file descriptors

Submitter:  Hal Blackburn <hal>
Submitted:  Sat 29 Jul 2023 10:31:21 AM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  None
Open/Closed:  Open Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 04 Aug 2023 05:53:40 AM UTC, comment #8: 

It's easy to do! Thanks very much for your help & fixing this, I look forward to the patch.

Hal Blackburn <hal>
Thu 03 Aug 2023 07:15:27 PM UTC, comment #7: 

Yes, I think we were talking past each other. That will teach me to read all the way to the end! :-)

I see the problem and I'll fix it in the next devel branch push. Thanks for the report.

Chet Ramey <chet>
Group administrator
Thu 03 Aug 2023 04:10:08 PM UTC, comment #6: 

Ah, I think we are misunderstanding each other. This is what I would expect, but my experience is that they are closed in the parent! In my example program the cat "${input_file:?}" line runs in the parent, but it fails to cat the psub FD only after the parent has started a coproc.

Hal Blackburn <hal>
Thu 03 Aug 2023 03:49:13 PM UTC, comment #5: 

Starting a subshell does only close the pipe file descriptors in the child. The parent still holds open file descriptors to the other end of the pipe.

Chet Ramey <chet>
Group administrator
Thu 03 Aug 2023 03:26:17 PM UTC, comment #4: 

Thanks. I must be missing something, but wouldn't it be safe to only close the FDs in the child, leaving them open in the parent?

If I can do this manually in the script, couldn't bash do this internally automatically when starting a coproc?

Hal Blackburn <hal>
Thu 03 Aug 2023 02:57:03 PM UTC, comment #3: 

Hi. The difference between a coproc and command substitution, for instance, is that the command substitution is synchronous -- the parent waits for the child to finish and collects its output before going on. There's no danger that the subshell will run in parallel with the parent for an unbounded period and hold a file descriptor the parent is expecting to manage by itself.

coprocs are bound to the shell process, yes, but they're asynchronous. There can be use cases where the coproc runs and the parent comes back to use that file descriptor occasionally, when it needs the service the coproc is supposed to provide. A file descriptor open to an unrelated process inherited by the long-running coproc can prevent the process substitution from terminating (or the shell getting EOF when the process substitution terminates, as I said previously).

It's just safer.

You need to manage the file descriptors in the parent process because that is the process that opened them and because the coproc closes the file descriptors as part of the subshell initialization, before executing the command in the coproc, so you have lost the handle to those open file descriptors before you run any user code.

Chet Ramey <chet>
Group administrator
Thu 03 Aug 2023 02:34:25 PM UTC, comment #2: 

Thanks for explaining, Chet.

I'm a little confused by this behaviour, as forking a subshell in another way, e.g. via $() doesn't close the process substitution FDs in the parent, and I would expect the lifetime of the coproc subshell to be bounded to the parent process, so I wouldn't expect it to be unsafe for it to have references.

In my case I'm only using the process substitution FDs in the parent process, not inside the coproc subshell. I was able to work around it like you describe, by duping the psub FDs before starting the coproc, then re-assigning them back to the original FDs after starting it. It seems like it shouldn't be necessary to do this in the parent process.

Hal Blackburn <hal>
Thu 03 Aug 2023 02:14:47 PM UTC, comment #1: 

Yes, subshells close any process substitution file descriptors they inherit from a parent shell, and the file descriptors are set to close-on-exec to handle the other cases.

They're pipes, so a child process that inherits a file descriptor open on a pipe that it's not prepared to manage (or isn't aware of) will leave it open and prevent the parent from getting EOF or completely closing the pipe.

If you want to use a process substitution file descriptor in a subshell, you can dup the file descriptor to another one and manage it manually. You have to be careful to appropriately close it, though.

Chet Ramey <chet>
Group administrator
Sat 29 Jul 2023 10:31:21 AM UTC, original submission:  

When using coproc, Bash seems to close any open FDs from 63 downwards. Specifically if 63, 62... are in use from process substitution, they are closed when coproc is started. Here's an example demonstrating this:


#!/usr/bin/env bash

function coproc_fn() {
  input_file=${1:?}
  echo "input_file=${input_file@Q}" >&2

  if [[ ${run_coproc:-} ]]; then
    coproc example( : )
    echo "coproc FDs: ${example[@]@Q}" >&2
  fi

  cat "${input_file:?}"
}

echo Without coproc
coproc_fn <(echo hi)
echo
echo With coproc
run_coproc=1 coproc_fn <(echo hi)


Output:


$ bash coproc-close-fds.bash
Without coproc
input_file='/dev/fd/63'
hi

With coproc
input_file='/dev/fd/63'
coproc FDs: '62' '59'
cat: /dev/fd/63: No such file or directory


Tested with GNU bash, version 5.2.15(1)-release (aarch64-unknown-linux-gnu)

Hal Blackburn <hal>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by chet (Posted a comment)
  • -email is unavailable- added by hal (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-08-03 chet StatusInvalid Done
    2023-08-03 chet StatusNone Invalid

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code