bugGNU tar - Bugs: bug #66116, GNU tar should default to...

 
 

bug #66116: GNU tar should default to --sort=name

Submitter:  Deven T. Corzine <deven>
Submitted:  Wed 21 Aug 2024 01:44:10 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 21 Aug 2024 02:20:34 AM UTC, comment #1: 

Another group of similar feed files ended up 50% smaller after compression, just because of sorting the filenames!

$ ls -l 130001-137621.tar.random.xz 130001-137621.tar.sorted.xz
-rw-r--r-- 1 root root 9006972 Aug 20 19:57 130001-137621.tar.random.xz
-rw-r--r-- 1 root root 4509824 Aug 20 20:00 130001-137621.tar.sorted.xz
$ xz -l 130001-137621.tar.random.xz 130001-137621.tar.sorted.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1       1  8,795.9 KiB  3,329.7 MiB  0.003  CRC64   130001-137621.tar.random.xz
    1       1  4,404.1 KiB  3,329.7 MiB  0.001  CRC64   130001-137621.tar.sorted.xz
-------------------------------------------------------------------------------
    2       2     12.9 MiB  6,659.5 MiB  0.002  CRC64   2 files


Deven T. Corzine <deven>
Wed 21 Aug 2024 01:44:10 AM UTC, original submission:  

(I originally wrote this as a StackExchange answer and it occurred to me that I should probably submit it here!)

The order of files in a tar archive might appear unimportant, but it can actually make a significant difference.

I was just archiving a large number of highly-redundant feed files on an NFS filesystem, and compressed the tar file with "xz -9e" for maximum compression.  For a directory containing 18.8 GB of feed files, the "*.tar.xz" compressed archive ended up being 144.5 MB.  Excellent compression ratio!

After this, I created a SquashFS filesystem of the same directory of feed files, removed the original files, loopback mounted the SquashFS filesystem, and created another compressed archive exactly the same way as before.  This time, the "*.tar.xz" compressed archive containing the same 18.8 GB of feed files ended up being 93.3 MB, over 35% smaller than the original archive!

Investigating this unexpected difference, I discovered that NFS was returning the filenames in random order, while SquashFS returns the filenames in sorted order by default.  This ended up creating a tar file which was better optimized for compression, grouping similar data closer together, allowing xz to perform substantially better.  I could have achieved the same result from the NFS filesystem the first time by adding "--sort=name" to the GNU tar command, if I had realized how much it would help.

Moral of the story?  GNU tar should really default to "--sort=name" instead of "--sort=none" to archive the files in a nicer order and potentially optimize the tar file for better compression!

Deven T. Corzine <deven>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by deven (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.14-650d.
    Corresponding source code