bugGNU Parallel - Bugs: bug #66955, --pipepart with .gz

 
 

bug #66955: --pipepart with .gz

Submitter:  None
Submitted:  Thu 27 Mar 2025 02:31:14 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
* Mandatory Fields

Post a Comment

Add a New Comment Rich Markup
   

Discussion

Jump to the original submission

Fri 10 Oct 2025 11:41:34 AM UTC, comment #10: 

Zstd men kræver pakket med t2sz

https://pypi.org/ ... ource=chatgpt.com

Ole Tange <tange>
Group administrator
Fri 10 Oct 2025 11:38:14 AM UTC, comment #9: 
Ole Tange <tange>
Group administrator
Fri 10 Oct 2025 11:12:27 AM UTC, comment #8: 

Pixz instead produces a collection of smaller blocks which makes random access to the original data possible.

https://githu ... om/vasi/pixz/

Ole Tange <tange>
Group administrator
Fri 10 Oct 2025 11:02:37 AM UTC, comment #7: 

zstd --seekable (kræver ny zstd og komprimeret med --seekable).

https://github.co ... m/rorosen/zeekstd

Ole Tange <tange>
Group administrator
Thu 24 Apr 2025 11:43:00 AM UTC, comment #6: 

We need a way to tell different index formats apart: bgzip.idx

    ./bgzip -r out.gz # Generate out.gz.gzi
    ./bgzip -b 10000 out.gz|head -c 10

bgzip only works on files created with bgzip - probably not useful.

Anonymous
Thu 24 Apr 2025 11:20:46 AM UTC, comment #5: 
Anonymous
Fri 28 Mar 2025 07:56:31 AM UTC, comment #4: 

seems to wotk with bzip2 https://github.co ... indexed_bzip2.git

Ole Tange <tange>
Group administrator
Thu 27 Mar 2025 08:33:05 PM UTC, comment #3: 

With few changes this will work:

https://github.co ... zseek/tree/master

Anonymous
Thu 27 Mar 2025 06:07:52 PM UTC, comment #2: 
Anonymous
Thu 27 Mar 2025 02:47:21 PM UTC, comment #1: 

Generating the index will be as slow as decompressing everything, so keep the index, so it can be reused.

Search for the index as:

  • foo.gzidx
  • $CACHE/md5sum($dir)/foo.gzidx

If not found generate it:

    Warning: Generating index foo.gzidx

If not possible to save as foo.gzidx, save as $CACHE/md5sum($dir)/foo.gzidx

Anonymous
Thu 27 Mar 2025 02:31:14 PM UTC, original submission:  

Use https://pypi.org/ ... ect/indexed-gzip/


#!/usr/bin/env python3
import argparse
import indexed_gzip

def main():
    parser = argparse.ArgumentParser(
        description="Opret et index for en gzip-fil med indexed_gzip."
    )
    parser.add_argument("gzfile", help="Sti til gzip-fil (f.eks. my.gz)")
    parser.add_argument("indexfile", help="Sti til output index-fil (f.eks. my.gzidx)")
    args = parser.parse_args()

    # Åbn gzip-filen med indexed_gzip og opbyg indeks
    with indexed_gzip.IndexedGzipFile(args.gzfile) as f:
        f.export_index(args.indexfile)
    print(f"Index-fil skrevet til: {args.indexfile}")

if __name__ == "__main__":
    main()


gzip-extract


#!/usr/bin/env python3
import argparse
import sys
import indexed_gzip

def parse_number(s):
    """
    Konverter et tal angivet som streng til et heltal.
    Understøtter suffixerne K (1024) og M (1024*1024).
    """
    if s[-1] in ('M', 'm'):
        return int(float(s[:-1]) * 1024 * 1024)
    elif s[-1] in ('K', 'k'):
        return int(float(s[:-1]) * 1024)
    else:
        return int(s)

def parse_range(r):
    """
    Forventet format: "start-end", f.eks. "10M-12M".
    """
    parts = r.split('-')
    if len(parts) != 2:
        raise ValueError(f"Ugyldigt range-format: {r}")
    start = parse_number(parts[0])
    end = parse_number(parts[1])
    if end < start:
        raise ValueError("End skal være større end eller lig med start")
    return start, end

def main():
    parser = argparse.ArgumentParser(
        description="Udtræk specifikke byte-ranges fra en gzip-fil vha. en index-fil."
    )
    parser.add_argument("--index", required=True, help="Sti til index-fil (f.eks. my.gzidx)")
    parser.add_argument("gzfile", help="Sti til gzip-fil (f.eks. my.gz)")
    parser.add_argument("--range", action="append", required=True,
                        help="Byte-range at udtrække i formatet start-end (f.eks. 10M-12M)")
    args = parser.parse_args()

    # Parse alle angivne ranges
    ranges = [parse_range(r) for r in args.range]

    # Åbn gzip-filen og importer indexet
    with indexed_gzip.IndexedGzipFile(args.gzfile) as f:
        f.import_index(args.index)
        # For hvert range: seek til startposition og læs det specificerede antal bytes.
        for start, end in ranges:
            length = end - start
            f.seek(start)
            data = f.read(length)
            sys.stdout.buffer.write(data)

if __name__ == "__main__":
    main()


Anonymous

 

Attached Files

This item currently has no attached files.

(Note: upload size limit is set to 4.0MiB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    History

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.16-ed84.
    Corresponding source code