bugfindutils - Bugs: bug #35254, Compress sort's temporary files

 
 

bug #35254: Compress sort's temporary files

Submitter:  Andreas Metzler <ametzler>
Submitted:  Fri 06 Jan 2012 06:37:07 PM UTC
 
Category:  updatedb Severity:  1 - Wish
Item Group:  None Status:  Need Info
Privacy:  Public Assigned to:  jay
Originator Name:  Attila Kinali Open/Closed:  Open
Release:  None Fixed Release:  None
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       

 

( Jump to the original submission )

Thu 22 Sep 2022 12:16:29 PM UTC, comment #6: 

comment #5:

> If PRUNEPATHS contains /storage/backup, yet the db contains files from there, then there's something wrong.
> Please check what the effective find(1) command looks like.


Oops, please ignore this comment; it turns out that on my system, "locate" is actually "plocate". Sorry!

Julian Gilbey <juliangilbey>
Wed 21 Sep 2022 06:49:08 PM UTC, comment #5: 

If PRUNEPATHS contains /storage/backup, yet the db contains files from there, then there's something wrong.
Please check what the effective find(1) command looks like.

Bernhard Voelker <berny>
Project Administrator
Wed 21 Sep 2022 09:56:14 AM UTC, comment #4: 

On my system, it's using around 4 GB of temporary files.

One thing I've noticed is that these files contain filenames from directories which are meant to be pruned: my PRUNEPATHS includes /storage/backup, but the /var/tmp/sortXXXXXXX files contain lots (all?) of the files in this hierarchy, which is both excessively space-consuming and time-consuming (updatedb took about 2 hours to run).  It may be that excluding files under PRUNEPATHS at file search time would make a significant difference to this issue.

Julian Gilbey <juliangilbey>
Wed 04 Sep 2019 08:57:16 PM UTC, comment #3: 

If that is sort(1) of the GNU coreutils, then one could set the
environment variable TMPDIR to let the tool write the temporary
data somewhere else:

See <https://www.gnu.org/software/coreutils/sort>:

If the environment variable TMPDIR is set, sort uses its
value as the directory for temporary files instead of /tmp.
The --temporary-directory (-T) option in turn overrides the
environment variable.

Bernhard Voelker <berny>
Project Administrator
Sun 01 Sep 2019 02:14:22 PM UTC, comment #2: 

I'd like to get an idea how many users would benefit from such a change.  Is it more than one? 

James Youngman <jay>
Project Administrator
Wed 25 Sep 2013 09:06:47 PM UTC, comment #1: 

This isn't a good use for a configure check, since the result is not compiled into the program.   That is, a change to the system sort command could make the choice made by "configure" incorrect after the findutils binaries have all been built.   Having said this, there's probably no easily available alternative approach other than trying to use the --compress-program option to find out if it works.

James Youngman <jay>
Project Administrator
Fri 06 Jan 2012 06:37:07 PM UTC, original submission:  

Hello,

this is http://bugs.debian.org/654860 submitted by Attila Kinali:

------------------
Hi,

I am running locate/updatedb on my fileserver which contains a large amount of files (several milion). When running updatedb in such an enviroment, its subprocess sort will create a large amount (1.5GB in my case) of temporary data in /var/tmp. If /var/tmp is on a rather small partition, this will cause the partition to be completely filled up.

The solution is to tell sort to use gzip on its temporary data, which results in a decrease of temp data size by a factor of 10. The performance impact on updatedb is negligible (less than 1%).

                        Attila Kinali
------------------

Attila suggests to simply use "sort --compress-program=gzip ...", I guess a clean patch would need to doublecheck at ./configure whether sort actually supports this option.

Andreas Metzler <ametzler>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by juliangilbey (Posted a comment)
  • -email is unavailable- added by berny (Posted a comment)
  • -email is unavailable- added by jay (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-09-01 jay StatusNone Need Info
        Assigned toNone jay
    2013-09-25 jay Severity3 - Normal 1 - Wish
    2012-01-06 ametzler Carbon-CopyRemoved 20807 -

    Back to the top


    Powered by Savane 3.9