bugGNU Parallel - Bugs: bug #67591, zextract

 
 

bug #67591: zextract

Submitter:  Ole Tange <tange>
Submitted:  Wed 08 Oct 2025 12:58:24 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
* Mandatory Fields

Post a Comment

Add a New Comment Rich Markup
   

Discussion

Wed 15 Oct 2025 09:12:17 AM UTC, comment #2: 

To be able to split a full file into even sized chunks it also needs --quiet and --size.

It might also be handy to generate a separate index of line numbers even if this means decompressing the full file once.

For every 1000 lines: dense.append(packed(byte positions of lines)), sparse.append(size of dense so far).

print total lines, sparse, dense > line.index

Lookup checks if wanted < total lines, finds the sparse entry that links to the dense entry. Unpacks the dense entry and returns the byte position.

Ole Tange <tange>
Group administrator
Wed 08 Oct 2025 01:12:21 PM UTC, comment #1: 
Ole Tange <tange>
Group administrator
Wed 08 Oct 2025 12:58:24 PM UTC, original submission:  



=pod

=head1 NAME

zextract - Extract specific byte ranges from compressed files

=head1 SYNOPSIS

B<zextract> [I<range>...] [B<--idx> I<index_file_or_auto>]
I<compressed_file>...

=head1 DESCRIPTION

B<zextract> is a utility for extracting specific byte ranges from one
or more compressed files. It supports indexing for faster access and
can operate on multiple files at once. By default, it uses automatic
indexing (B<--idx auto>), creating or updating the index as needed.

Ranges are specified in a human-friendly format. You can use the
following formats for ranges:
=over 4

=item I<start>-I<end>

Extract bytes from I<start> to I<end>. Both I<start> and I<end> are
included in the extracted range.

=item I<start>+I<length>

Extract I<length> bytes starting at I<start>.

=item I<start>-

Extract bytes from I<start> to the end of the file.

=item -I<end>

Extract bytes from the beginning of the file to I<end>.

=back

Numbers in ranges can be postfixed by k, m, g, t, or p (1000^x) or K,
M, G, T, or P (1024^x) to specify larger byte ranges. For example, 1M
is 1024^2 bytes, and 1m is 1000^2 bytes.

=head1 OPTIONS

=over 4

=item B<--idx> I<index_file_or_auto>

Specifies the index file to use for extraction. If set to B<auto>
(default), the index will be built automatically and saved as
I<compressed_file>.zex in the same directory as the compressed file.
If the directory is read-only, the index will be saved in the XDG
cache directory (default: F<$XDG_CACHE_HOME/zextract> or
F<$HOME/.cache/zextract>). If the index is not found or is older than
the compressed file, it will be recreated.

=back

=head1 ARGUMENTS

=over 4

=item I<range>

One or more byte ranges to extract. If no ranges are provided, only
the index will be created or updated.

=item I<compressed_file>

One or more compressed files to extract from or create an index for.
Wildcards (e.g., B<*.gz>) are supported.

=back

=head1 EXAMPLES

=over 4

=item Extract bytes 100 to 200 (inclusive) from myfile.gz:

  zextract 100-200 myfile.gz

=item Extract 1 megabyte starting at byte 100 from myfile.gz:

  zextract 100+1M myfile.gz

=item Extract bytes 100 to the end of the file from myfile.gz:

  zextract 100- myfile.gz

=item Extract the first 100 kilobytes from myfile.gz:

  zextract -100K myfile.gz

=item Extract ranges 100-200 and 1MB-2MB from myfile.gz:

  zextract 100-200 1M-2M myfile.gz

=item Create or update the index for foo.gz without extracting any data:

  zextract foo.gz

=back

=head1 CACHE DIRECTORY

If the directory containing the compressed file is read-only,
B<zextract> will save the index file in the XDG cache directory
(default: F<$XDG_CACHE_HOME/zextract> or F<$HOME/.cache/zextract>).
The cache directory will be created if it does not exist.

Index files are stored with a unique path derived from the full,
normalized path of the compressed file (as returned by B<readlink -f>).
The structure is:

  $XDG_CACHE_HOME/zextract/<first-letters>/<8-char-hash>_<original-filename>.zex

Where:
=over 4

=item B<< <first-letters> >>

A subdirectory path constructed from the first letter of each
directory component in the normalized path. For example:

  /tmp/tmp.88s8/my/sub/dir/sub/file.gz
  → t/t/m/s/d/s

=item B<< <8-char-hash> >>

A 32-bit (8-character hex) SHA-256 hash of the full normalized path,
ensuring uniqueness and collision resistance for up to millions of
files.

=item B<< <original-filename> >>

The original filename of the compressed file, without modification.

=back

This structure ensures no collisions between files with the same name
in different directories, while keeping the cache organized and
efficient.

=head1 EXAMPLES

=over 4

=item Original path: F</home/user/dir/sub/file.gz>

Cache path: F<$XDG_CACHE_HOME/zextract/h/u/d/s/a1b2c3d4_file.gz.zex>

=item Original path: F</tmp/tmp.88s8/my/sub/dir/sub/file.gz>

Cache path: F<$XDG_CACHE_HOME/zextract/t/t/m/s/d/s/a1b2c3d4_file.gz.zex>

=back

=head1 SEE ALSO

B<gzip>(1), B<bzip2>(1), B<lzma>(1), B<xdg-user-dirs>(1)

=head1 AUTHOR


=cut


Ole Tange <tange>
Group administrator

 

Attached Files

This item currently has no attached files.

(Note: upload size limit is set to 16MiB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    History

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.16.
    Corresponding source code