bugGNU Parallel - Bugs: bug #66540, parallel cat an input file...

 
 

bug #66540: parallel cat an input file occasionally produces corrupted records

Submitter:  None
Submitted:  Sat 07 Dec 2024 02:33:26 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Works For Me
Privacy:  Public Assigned to:  tange
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 16 Dec 2024 09:42:57 PM UTC, comment #1: 

Thank you for a good error report.

If it is OK with you I will be referring to this as an example of an excellent error report.

You have made it very easy for me to download OSBoxes/Ubuntu20.04 and OSBoxes/Ubuntu24.04 and test your code on that.

Unfortunately I cannot reproduce the error you see.

That tells us, that there is something in your environment that is different from the virtual machines I downloaded.

Next step is therefore for you to download a virtual machine, change the environment on that so that you can reproduce the error and give me a copy of the modified VM.

Ole Tange <tange>
Group administrator
Sat 07 Dec 2024 02:33:26 PM UTC, original submission:  

I've encountered an issue while using GNU Parallel to process a large JSONL file containing very long lines. The command I'm using is:

```bash
parallel -a data.jsonl -j 64 --pipepart --block -1 -k "python process.py"
```

The goal is to process each JSON line in parallel using a Python script. However, I've noticed that occasionally, the Python script encounters json.decoder.JSONDecodeError, indicating corrupted JSON data.

*Observations:*

  • *Input data integrity:* The original JSONL file is verified to be well-formatted.
  • *Discrepancy in input and output:* When the Python script fails, the line causing the error is indeed malformed. This suggests that the data is being corrupted during the parallel processing.
  • *Reproducibility:* To isolate the issue, I've created a simplified script (`run_tests.sh`) that merely `cat`s the input file using `parallel` and compares the output with the original using SHA256 sums. The script demonstrates that the output is not always identical to the input, even when using a simple `cat` command.
  • *Environment:*

  * GNU Parallel 20241122
  * Ubuntu 20.04
  * Perl 5.30.0 and 5.40.0
  * Bash and Zsh
  * Locale: en_US.UTF-8

  • *Reproducible test case:* I've created a GitHub repository containing the `run_tests.sh` script and a sample input file. Running `bash run_tests.sh test.input 1000` will execute 1000 tests and report any discrepancies.


Script and test inputs to reproduce the bug: https://github.com/pluiez/reproduce-gnu-parallel-bug

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-12-16 tange StatusNone Works For Me
        Assigned toNone tange

    Back to the top

    Powered by Savane 3.14-04e1.
    Corresponding source code