Thu 28 Nov 2013 10:37:15 AM UTC, original submission:
The manual states:
GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU parallel as input for other programs.
However this doesn't seem to be the case.
I am using parallel to process gzipped log files.
The output I see depends on the number of jobs I specify in the -j parameter.
Example -- copy and pasted from terminal:
[davteam@lo3uppaldbs001 05]$ ls 2013-05-04*gz | /home/aclegan/parallel_bug/parallel-20131022/src/parallel --gnu -j 8 zcat > test8.txt
[davteam@lo3uppaldbs001 05]$ wc -l test8.txt
37327858 test8.txt
[davteam@lo3uppaldbs001 05]$ ls 2013-05-04*gz | /home/aclegan/parallel_bug/parallel-20131022/src/parallel --gnu -j 4 zcat > test4.txt
[davteam@lo3uppaldbs001 05]$ wc -l test4.txt
52038007 test4.txt
These are archives -- the files are not changing.
I can't easily share the files, as they contain personally identifying information. I've tried to reproduce this with synthetic test data, but I can't.
The files are not huge -- between ~ 10 to 150 MB each.
I have checked using the --dry-run parameter, and the output looks identical each time:
zcat 2013-05-04-00.gz
zcat 2013-05-04-01.gz
zcat 2013-05-04-02.gz
zcat 2013-05-04-03.gz
zcat 2013-05-04-04.gz
zcat 2013-05-04-05.gz
zcat 2013-05-04-06.gz
zcat 2013-05-04-07.gz
zcat 2013-05-04-08.gz
zcat 2013-05-04-09.gz
zcat 2013-05-04-10.gz
zcat 2013-05-04-11.gz
zcat 2013-05-04-12.gz
zcat 2013-05-04-13.gz
zcat 2013-05-04-14.gz
zcat 2013-05-04-15.gz
zcat 2013-05-04-16.gz
zcat 2013-05-04-17.gz
zcat 2013-05-04-18.gz
zcat 2013-05-04-19.gz
zcat 2013-05-04-20.gz
zcat 2013-05-04-21.gz
zcat 2013-05-04-22.gz
zcat 2013-05-04-23.gz
These are hourly log rotations over the course of one day.
Additional note:
I used the following commands to check which hours were present in the output files:
[davteam@lo3uppaldbs001 05]$ cut -c 1-13 test4.txt|sort -u
and
[davteam@lo3uppaldbs001 05]$ cut -c 1-13 test8.txt|sort -u
The output in both cases showed that every hours was represented:
2013-05-03 23
2013-05-04 00
2013-05-04 01
2013-05-04 02
2013-05-04 03
2013-05-04 04
2013-05-04 05
2013-05-04 06
2013-05-04 07
2013-05-04 08
2013-05-04 09
2013-05-04 10
2013-05-04 11
2013-05-04 12
2013-05-04 13
2013-05-04 14
2013-05-04 15
2013-05-04 16
2013-05-04 17
2013-05-04 18
2013-05-04 19
2013-05-04 20
2013-05-04 21
2013-05-04 22
2013-05-04 23
(plus a few from the previous date which happens due to a lag in our log processing pipeline)
So each file is getting at least partially processed by zcat in my parallel command.
|