bugGNU Parallel - Bugs: bug #61410, merge sort --lb output

 
 

bug #61410: merge sort --lb output

Submitter:  None
Submitted:  Mon 01 Nov 2021 01:36:56 PM UTC
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       

 

Sat 14 May 2022 02:13:42 AM UTC, comment #1: 

No.

Make instead a program (partsort) to pipe into:

    ... | partsort -n 100

partsort will buffer 100 lines, read one line and output the smallest.

Ole Tange <tange>
Project Administrator
Mon 01 Nov 2021 01:36:56 PM UTC, original submission:  

Will it make sense to have a merge sort of output?

Assume the output is:

2015-10-27T08:34:02.447894577Z Printing line 0
2015-10-27T08:34:07.456426850Z Printing line 0
2015-10-27T08:34:02.447962742Z Printing line 1
2015-10-27T08:34:07.456426850Z Printing line 1
2015-10-27T08:34:02.447971906Z Printing line 2
2015-10-27T08:34:07.456426850Z Printing line 2
2015-10-27T08:34:02.447971990Z Printing line 3
2015-10-27T08:34:07.456426850Z Printing line 3
2015-10-27T08:34:02.447972073Z Printing line 4
2015-10-27T08:34:02.448410164Z Printing line 4

(0-4 is job ID).

It would make sense to get it sorted by time:

2015-10-27T08:34:02.447894577Z Printing line 0
2015-10-27T08:34:02.447962742Z Printing line 1
2015-10-27T08:34:02.447971906Z Printing line 2
2015-10-27T08:34:02.447971990Z Printing line 3
2015-10-27T08:34:02.447972073Z Printing line 4
2015-10-27T08:34:02.448410164Z Printing line 4
2015-10-27T08:34:07.456426850Z Printing line 0
2015-10-27T08:34:07.456426850Z Printing line 1
2015-10-27T08:34:07.456426850Z Printing line 2
2015-10-27T08:34:07.456426850Z Printing line 3

This can be done on the fly, if we know the output of each job is sorted so that we can run a merge sort:

1. read 1 line from each job
2. print the lowest line of all the jobs
3. read another line from that job
4. goto 2 unless all are EOF

Problem: It will only look at the currently running jobs, so it only makes sense to use if #jobs <= #jobslots.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    No changes have been made to this item

    Back to the top


    Powered by Savane 3.9