bugGNU Parallel - Bugs: bug #64955, parallel --fast for short lived...


bug #64955: parallel --fast for short lived jobs

Submitter:  None
Submitted:  Fri 01 Dec 2023 11:59:10 AM UTC
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Tue 02 Jan 2024 01:30:26 PM UTC, comment #12: 

This will not work because wait does not wait for >() to complete:

_filterwrap() {
    { "$@" } 2>(_myfilter >&2) | _myfilter

This might work:

error() { _exit=$((_exit+1)); if too many errors: exit; }

_tagger() {
    rm -f "$1"
    # This will fail for either /1 or /2
    rmdir "$2" 2>/dev/null
    perl -pe 's/^/tag:\t/'
  } < "$1"

_taggerwrap() {
  _dir=`mktemp -d`
  mkfifo "$_dir"/1 "$_dir"/2
  _tagger "$_dir"/1 "$_dir" >&1 &
  _tagger "$_dir"/2 "$_dir" >&2 &
  { "$@" } >"$_dir"/1 2>"$_dir"/2

_taggerwrap "mycommand" "my" "arg" "||" "error"

Tue 02 Jan 2024 01:15:13 PM UTC, comment #11: 

_myfilter() {
  perl -pe 's/^/tag:\t/'
_myfilter_fall_back() {
    rm -f "$1"
    # This will fail for either /1 or /2
    rmdir "$2" 2>/dev/null
  } < "$1"

if true >(true) ; then
  # >() works
  _filterwrap() {
    { "$@" } 2>(_myfilter >&2) | _myfilter
  # fall back
  _filterwrap() {
    _dir=`mktemp -d`
    mkfifo "$_dir"/1 "$_dir"/2
    _myfilter_fall_back "$_dir"/1 "$_dir" >&1 &
    _myfilter_fall_back "$_dir"/2 "$_dir" >&2 &
    { "$@" } >"$_dir"/1 2>"$_dir"/2

Thu 21 Dec 2023 10:56:25 PM UTC, comment #10: 

Keeps $foo

    _temp_dir=$(mktemp -d)
    mkfifo "$_temp_dir"/out "$_temp_dir"/err
    myfilter outfilter >&1 <"$_temp_dir"/out &
    myfilter errfilter >&2 <"$_temp_dir"/err &
        rm -r "$_temp_dir"
        myprogram 11 || foo=$?.7;
    } 1>"$_temp_dir"/out 2>"$_temp_dir"/err
    echo Foo:$foo
    echo Done $exit $_exit $myvar

Ole Tange <tange>
Group administrator
Thu 21 Dec 2023 07:04:11 PM UTC, comment #9: 


    exec 3>&1
    myprogram 11 2>&1 >&3 | myfilter ferr >&2
    exec 3>&-
} | myfilter fout

Ole Tange <tange>
Group administrator
Thu 21 Dec 2023 05:55:23 PM UTC, comment #8: 

This may be helpful for _tagger:

exec 3>&1 4>&2

# Run mycmd and redirect stdout to myprog1 and stderr to myprog2
  { myprogram 2>&1 1>&3 3>&- 4>&1 | myfilter err 3>&- 4>&-; } 2>&1 1>&4 | myfilter out
} 3>&1 4>&2

# Close file descriptors
exec 3>&- 4>&-

Thu 21 Dec 2023 03:21:10 PM UTC, comment #7: 

--tag should do something like:

    _tagger() { (rm $1; $awk) < $1; }
    error() { _exit=$((_exit+1)); }

    mkfifo fifo1 fifo2
    _tagger fifo1 >&1 &
    _tagger fifo2 >&2 &
    { myprogram arg || error; } > fifo1 2>fifo2

The $awk should be computed and will probably only work for fixed strings.

Thu 21 Dec 2023 02:59:35 PM UTC, comment #6: 

Instead of having users use $_exit, predefine:

    error() {...}

If users want to use --halt-on-error or $? then they can simply call 'error' if their program fails. And (more importantly) they can refuse to do so if they want to focus on speed.

For error handling:

    parallel --fast 'myprogram {} || error'

Without error handling:

    parallel --fast myprogram {}

error() should Do The Right thing:

  • Increase $_exit.
  • If $_exit is too big (compared to --halt) then exit and tell GNU Parallel to halt.
  • Add exit code to a temporary job log.

By using error() GNU Parallel can extend the function to do more in the future.

The same idea can be used to setup a job:

    parallel --fast 'setup; myprogram {}'

'setup' should then set $PARALLEL_SEQ and similar, that can be done in shell, but may not be needed for --fast jobs. Maybe it should be called 'start'?

Wed 20 Dec 2023 08:48:22 PM UTC, comment #5: 

--fast should be as fast as possible at all costs: Disable anything that takes time.

So downgrade to dash and don't autodetect shell: Tell in documentation to set PARALLEL_SHELL to override, and tell cost. PARALLEL_SHELL=auto should turn on auto detect.

Do not register exit codes: Tell users to use

      { mycmd; } || _exit=$((_exit + 1))

and we add 'exit $_exit' to the end of each chunk.

--ungroup: Can be undone with --group.

Ole Tange <tange>
Group administrator
Mon 18 Dec 2023 10:54:50 AM UTC, comment #4: 

Downgrade to dash:

    # 0.348s in dash
    { mycmd; } || exit=$((exit + 1))
    # 1.93s in bash
    { mycmd; } || exit=$((exit + 1))

Sat 02 Dec 2023 02:01:53 PM UTC, comment #3: 

If the input sources are not a pipe (i.e. not stdin), then:

  • generate all commands
  • split the commands into n lists where n = jobslots * k
  • run these lists similar to --pipe --cat

By having k * jobslots, if one list has all the slow jobs, the other jobslots can make up for this. How do we determine a good k?

It will also make --keep-order simple.

Ole Tange <tange>
Group administrator
Sat 02 Dec 2023 01:01:58 PM UTC, comment #2: 

Build a script that removes itself as the first command.

Ole Tange <tange>
Group administrator
Sat 02 Dec 2023 12:59:39 PM UTC, comment #1: 

Bash prefers reading from a file - not a pipe:

$ time parallel --pipe-part --block -10 --cat bash {} :::: 1M|wc
1000000 1000000 4889400

real    0m2.697s
user    0m11.542s
sys     0m5.670s

$ time parallel --pipe-part --block -10 --fifo  bash {} :::: 1M|wc
1000000 1000000 4889400

real    0m13.448s
user    0m18.346s
sys     1m14.599s

Ole Tange <tange>
Group administrator
Fri 01 Dec 2023 11:59:10 AM UTC, original submission:  

bash-parallel is fast

Combine the ideas of bash-parallel with GNU Parallel is much as it is possible.

Generate all jobs.

Put them in files that are passed to X bash workers.

-g will work, but only print after a worker is done with all jobs.

-k also.

--rr will make it possible not to generate all jobs at a start, but the workers will have to read jobs on stdin.

Can --tag be made to work with some wrapper? If the tag is fixed - probably yes.

--latest-line will not work on a per-job-basis, but will be OK for per-job-slot-basis.

--trc will not work.

--delay will not work.

--files will not work per job.

--results will not work.

--line-buffer will work.

--load will not work.

--limit will not work.

--memfree will not work.

How can we deal with exit codes?

Can --joblog, --retries, --halt work?



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-12-20 tange Summarycombine bash-parallel with GNU Parallel parallel --fast for short lived jobs

    Back to the top

    Powered by Savane 3.13-04b1.
    Corresponding source code