This will not work because wait does not wait for >() to complete:

_filterwrap() {
    { "$@" } 2>(_myfilter >&2) | _myfilter
}

This might work:

error() { _exit=$((_exit+1)); if too many errors: exit; }

_tagger() {
  {
    rm -f "$1"
    # This will fail for either /1 or /2
    rmdir "$2" 2>/dev/null
    perl -pe 's/^/tag:\t/'
  } < "$1"
}

_taggerwrap() {
  _dir=`mktemp -d`
  mkfifo "$_dir"/1 "$_dir"/2
  _tagger "$_dir"/1 "$_dir" >&1 &
  _tagger "$_dir"/2 "$_dir" >&2 &
  { "$@" } >"$_dir"/1 2>"$_dir"/2
  wait
}

_taggerwrap "mycommand" "my" "arg" "||" "error"

Keeps $foo

    _temp_dir=$(mktemp -d)
    mkfifo "$_temp_dir"/out "$_temp_dir"/err
    myfilter outfilter >&1 <"$_temp_dir"/out &
    myfilter errfilter >&2 <"$_temp_dir"/err &
    {
        rm -r "$_temp_dir"
        myprogram 11 || foo=$?.7;
    } 1>"$_temp_dir"/out 2>"$_temp_dir"/err
    wait
    echo Foo:$foo
    echo Done $exit $_exit $myvar

This may be helpful for _tagger:

exec 3>&1 4>&2

# Run mycmd and redirect stdout to myprog1 and stderr to myprog2
{
  { myprogram 2>&1 1>&3 3>&- 4>&1 | myfilter err 3>&- 4>&-; } 2>&1 1>&4 | myfilter out
} 3>&1 4>&2

# Close file descriptors
exec 3>&- 4>&-

Instead of having users use $_exit, predefine:

    error() {...}

If users want to use --halt-on-error or $? then they can simply call 'error' if their program fails. And (more importantly) they can refuse to do so if they want to focus on speed.

For error handling:

    parallel --fast 'myprogram {} || error'

Without error handling:

    parallel --fast myprogram {}

error() should Do The Right thing:

• Increase $_exit.
• If $_exit is too big (compared to --halt) then exit and tell GNU Parallel to halt.
• Add exit code to a temporary job log.

By using error() GNU Parallel can extend the function to do more in the future.

The same idea can be used to setup a job:

    parallel --fast 'setup; myprogram {}'

'setup' should then set $PARALLEL_SEQ and similar, that can be done in shell, but may not be needed for --fast jobs. Maybe it should be called 'start'?

Downgrade to dash:

    # 0.348s in dash
    { mycmd; } || exit=$((exit + 1))
    # 1.93s in bash
    { mycmd; } || exit=$((exit + 1))

Build a script that removes itself as the first command.

bash-parallel is fast

Combine the ideas of bash-parallel with GNU Parallel is much as it is possible.

Generate all jobs.

Put them in files that are passed to X bash workers.

-g will work, but only print after a worker is done with all jobs.

-k also.

--rr will make it possible not to generate all jobs at a start, but the workers will have to read jobs on stdin.

Can --tag be made to work with some wrapper? If the tag is fixed - probably yes.

--latest-line will not work on a per-job-basis, but will be OK for per-job-slot-basis.

--trc will not work.

--delay will not work.

--files will not work per job.

--results will not work.

--line-buffer will work.

--load will not work.

--limit will not work.

--memfree will not work.


How can we deal with exit codes?

Can --joblog, --retries, --halt work?