mainThe GNU Bourne-Again SHell - Support: sr #109541, "set -e", "set -o...

 
 

sr #109541: "set -e", "set -o pipefail" and "shopt -s lastpipe" do not like each other

Submitter:  Björn Kautler <vampire0>
Submitted:  Thu 09 Aug 2018 04:31:02 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  None
Open/Closed:  Open Operating System:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 24 Aug 2018 08:40:10 PM UTC, comment #3: 

Sorry, that was sloppy on my part. The lastpipe code didn't properly freeze and unfreeze the jobs list to deal with more than one external command in the loop body.

When the true pipeline finished, having run the right side in the current shell because of the `lastpipe' option, it unfroze the jobs list, allowing jobs to be removed. Job 0, having already completed, is removed. The cat command then runs as a new job 0, and succeeds, overwriting the status associated with job 0. This overwriting confuses the lastpipe code, which causes it to wait for an invalid pid (exit status 127) and assume that this is really a pipeline, and because of the pipefail option, combines the two exit statuses. That gives it the impression that the cat fails, which causes the loop to break (it uses the status from read -r) and return failure (it uses the status of the last command executed in the function body -- cat).

When the while loop fails, the -e option causes the shell to exit. Since the lastpipe option causes that loop to be run in the parent shell, the script exits.

The fix is, as I said, to freeze the jobs list until all the commands in the loop body have completed. You can't just unfreeze the jobs list unconditionally in execute_pipeline -- what if it's being called recursively?

Chet Ramey <chet>
Group administrator
Fri 24 Aug 2018 01:17:06 PM UTC, comment #2: 

A little understanding question if you allow.
You said 'the result from "cat" overwrites the return code(s) from the outer pipeline'.
But shouldn't the `cat <<< ""` exit successfully?

Björn Kautler <vampire0>
Fri 17 Aug 2018 08:04:48 PM UTC, comment #1: 

Thanks for the report. This is the result of a race condition -- with everything included, the result from "cat" overwrites the return code(s) from the outer pipeline. We have to keep the jobs list from being changed until all the processes spawned by the outer pipeline have completed.

This will be fixed in the devel branch with the next push.

Chet Ramey <chet>
Group administrator
Thu 09 Aug 2018 04:31:02 PM UTC, original submission:  

The following script will exit with code 127 after the while loop and before the declare with 'read -r' being the last command printed by 'set -x'.


#!/usr/bin/env bash

set -x
set -e
set -o pipefail
shopt -s lastpipe

echo | while read -r; do
        true | true
        cat <<< ""
done
declare -p PIPESTATUS





Removing only 'set -e' from the original script will exit with code 0 and print 'declare -a PIPESTATUS=([0]="1")'




Removing only 'set -o pipefail' from the original script will exit with code 0 and print 'declare -a PIPESTATUS=([0]="1")'




Removing only 'shopt -s lastpipe' from the original script will exit with code 0 and print 'declare -a PIPESTATUS=([0]="0" [1]="0")'




Removing only 'true | true' from the original script will exit with code 0 and print 'declare -a PIPESTATUS=([0]="0" [1]="0")'




Removing only 'cat <<< ""' from the original script will exit with code 0 and print 'declare -a PIPESTATUS=([0]="0" [1]="0")'




Changing the script to


#!/usr/bin/env bash

set -x
set -e
set -o pipefail
shopt -s lastpipe

while read -r; do
        true | true
        cat <<< ""
done <<< ""
declare -p PIPESTATUS


will exit with code 0 and print 'declare -a PIPESTATUS=([0]="1")'




My guess is, that it is somehow like
- because of 'lastpipe` the while is executed in the current shell, setting 'PIPESTATUS' to '(1)' for the last read call that aborts the while loop in the current shell
- because of 'pipefail' and seeing the `echo | while` it evaluates the 'PIPESTATUS', sees the '1' and fails
- because of the 'set -e' it aborts the script as it falsely considers the pipe to have failed somewhere

I actually wonder why commenting out either of the two nested commands 'fixes' the failure though.

Björn Kautler <vampire0>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by chet (Posted a comment)
  • -email is unavailable- added by vampire0 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-08-17 chet StatusNone Done

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code