GNU Parallel - News: GNU Parallel 20110205 (FOSDEM release) released

GNU Parallel 20110205 (FOSDEM release) released

Item posted by Ole Tange <tange> on Mon 07 Feb 2011 10:15:14 PM UTC.

GNU Parallel 20110205 (the FOSDEM release) has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/

This is a major release as the --pipe option introduces a new way to work. GNU Parallel has so far been similar to xargs, with --pipe it becomes somewhat similar to tee. To learn about --pipe see the example section for uses of --pipe.

But rest assured: No old functionality is changed.

If you want GNU Parallel to be part of your favourite distribution contact the people maintaining the distribution (complaining on Twitter is not enough).

New in this release:

• --pipe splits piped data into blocks. Each block is piped to a program for processing. The piping and the programs will be run in parallel. Useful if the data you want to process is data for a program and not arguments.

• --blocksize sets the blocksize in bytes for --pipe. The blocksize is approximate. It can deviate as much as the size of one record. Default is 1M.

• --recstart sets the string matching the start of a record. Default is "".

• --recend sets the string matching the end of a record. Default is '\n'. To specify none use --recend "". If both --recstart and --recend are set, the end of a record must be followed immediately by a start of a record. This is useful if either recend or recstart can occur in the middle of a record.

• --remove-rec-sep removes the string matched by --recstart and --recend.

• --regexp will make GNU Parallel treat --recstart and --recend as regular expressions.

• --output-as-files will put the output of the programs into files and instead of giving the output GNU Parallel will output the name of these files.

• -N if used with --pipe sets the number of records to read.

• New video showing --pipe at http://www.youtube.com/watch?v=1ntxT-47VPA

• GNU Parallel was presented at FOSDEM.

• Article in USENIX Magazine ;login: (print) http://www.usenix.org/publications/login/2011-02/

• GNU Parallel is now on ohloh.net. Thanks to Wim Muskee. https://www.ohloh.net/p/gnu-parallel

• Advanced recursive example. Thanks to Ruarí Ødegaard. http://my.opera.com/ruario/blog/2011/01/24/editing-debian-packages-more-fun-with-gnu

• Small example on using GNU Parallel through 0install (German). http://forum.ubuntuusers.de/topic/gnu-parallel-mit-zero-install-kurzinfo-kein-wi/

• Bug fixes and man page updates.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job is can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

You can find more about GNU Parallel at: http://www.gnu.org/software/parallel/

Watch the intro video on http://www.youtube.com/watch?v=OpaiGYxkSuQ or at http://tinyogg.com/watch/TORaR/ and http://tinyogg.com/watch/hfxKj/