bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #31858, joblog file

 
 

bug #31858: joblog file

Submitted by:  None
Submitted on:  Mon 13 Dec 2010 08:54:37 AM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Need Info
Privacy: PublicAssigned to: None
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 20 Jan 2011 10:18:55 AM UTC, comment #3:

Very use full.

I can now see at the Runtime that I should only use castor if I have less than 100 files. The fun thing is that elvis and frank has more CPU power than castor, but the bandwith to/from elvis and frank is very poor.

Seq Host Starttime Runtime Send Receive Command
1 castor 1295516288 4 1852724 9916 convert SNV39370.JPG -geometry 100 1SNV39370.JPG
18 castor 1295516292 4 1850267 9495 convert SNV39387.JPG -geometry 100 1SNV39387.JPG
19 castor 1295516296 3 1854421 9948 convert SNV39388.JPG -geometry 100 1SNV39388.JPG
20 castor 1295516299 4 1836749 9903 convert SNV39389.JPG -geometry 100 1SNV39389.JPG
21 castor 1295516303 4 1852724 9916 convert SNV39390.JPG -geometry 100 1SNV39390.JPG
22 castor 1295516307 3 1853651 9744 convert SNV39391.JPG -geometry 100 1SNV39391.JPG
23 castor 1295516310 4 1905065 9878 convert SNV39392.JPG -geometry 100 1SNV39392.JPG
24 castor 1295516314 4 1841061 9619 convert SNV39393.JPG -geometry 100 1SNV39393.JPG
25 castor 1295516318 4 1859844 10233 convert SNV39394.JPG -geometry 100 1SNV39394.JPG
26 castor 1295516322 3 1871707 9612 convert SNV39395.JPG -geometry 100 1SNV39395.JPG
27 castor 1295516325 4 1856193 9321 convert SNV39396.JPG -geometry 100 1SNV39396.JPG
28 castor 1295516329 4 1850267 9495 convert SNV39397.JPG -geometry 100 1SNV39397.JPG
29 castor 1295516333 3 1854421 9948 convert SNV39398.JPG -geometry 100 1SNV39398.JPG
30 castor 1295516336 4 1836749 9903 convert SNV39399.JPG -geometry 100 1SNV39399.JPG
12 elvis 1295516288 139 1853651 9744 convert SNV39381.JPG -geometry 100 1SNV39381.JPG
10 elvis 1295516288 139 1836749 9903 convert SNV39379.JPG -geometry 100 1SNV39379.JPG
14 elvis 1295516288 140 1841061 9619 convert SNV39383.JPG -geometry 100 1SNV39383.JPG
2 frank 1295516288 140 1853651 9744 convert SNV39371.JPG -geometry 100 1SNV39371.JPG
13 elvis 1295516288 140 1905065 9878 convert SNV39382.JPG -geometry 100 1SNV39382.JPG
11 elvis 1295516288 140 1852724 9916 convert SNV39380.JPG -geometry 100 1SNV39380.JPG
3 frank 1295516288 141 1905065 9878 convert SNV39372.JPG -geometry 100 1SNV39372.JPG
7 frank 1295516288 141 1856193 9321 convert SNV39376.JPG -geometry 100 1SNV39376.JPG
8 frank 1295516288 141 1850267 9495 convert SNV39377.JPG -geometry 100 1SNV39377.JPG
17 elvis 1295516288 142 1856193 9321 convert SNV39386.JPG -geometry 100 1SNV39386.JPG
5 frank 1295516288 142 1859844 10233 convert SNV39374.JPG -geometry 100 1SNV39374.JPG
6 frank 1295516288 142 1871707 9612 convert SNV39375.JPG -geometry 100 1SNV39375.JPG
9 frank 1295516288 142 1854421 9948 convert SNV39378.JPG -geometry 100 1SNV39378.JPG
16 elvis 1295516288 142 1871707 9612 convert SNV39385.JPG -geometry 100 1SNV39385.JPG
4 frank 1295516288 142 1841061 9619 convert SNV39373.JPG -geometry 100 1SNV39373.JPG
15 elvis 1295516288 143 1859844 10233 convert SNV39384.JPG -geometry 100 1SNV39384.JPG

Hans Schou <chlor>
Wed 19 Jan 2011 10:12:15 AM UTC, comment #2:

[48454a3] implements --joblog as:

Seq Host Starttime Runtime Send Receive Command

Ole Tange <tange>
Project Administrator
Mon 13 Dec 2010 10:32:40 AM UTC, comment #1:

Please elaborate on the fields input and output.

What if the input is a hostname? A directory name? A user name? A fifo or device? A part of a filename (e.g. the {} of /foo/{}/thumb.jpg)?

If the input is a filename then we can stat the file and find the size of that.

If the output is --return'ed then we can find the size of that. But if the job creates file locally that we do not know about, how should we find the value for output?

Ole Tange <tange>
Project Administrator
Mon 13 Dec 2010 08:54:37 AM UTC, original submission:

Feature request.

I would like to have a logfile for debug purpose which in detail show the executions of all jobs on all servers.

Example:
--joblog <file name>

Columns:
host: host name as specified in the serverlist
seconds: Number of seconds the job has run
start: Time when job was started, ISO-8601 strict
input: Number of bytes input (maybe in MegaByte)
output: Number of bytes output

Output example:
----------------------
host seconds start cpuusage input output
foo.com 17 2010-12-31T23:00 23 9876 5679
----------------------

If TAB is used as delimiter it is easy to use 'cut' to handle the output file. Which server ran most jobs?
cut -f1 < job.log | sort | uniq -c
Order by start time:
sort -k3 < job.log

/hans

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by chlor (Posted a comment)
  • -unavailable- added by tange (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 19 Jan 2011 10:12:15 AM UTCtangeOpen/ClosedOpen=>Closed
    Mon 13 Dec 2010 10:32:40 AM UTCtangeSeverity3 - Normal=>1 - Wish
      StatusNone=>Need Info

    Back to the top


    Powered by Savane 3.1-cleanup1