buggrep - Bugs: bug #34020, RFE: Improve handling of sparse...


You are not allowed to post comments on this tracker with your current authentification level.

bug #34020: RFE: Improve handling of sparse files

Submitted by:  Jaroslav ┼ákarvada <yarda>
Submitted on:  Tue Aug 16 08:21:43 2011  
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: None
Privacy: PublicAssigned to: None
Open/Closed: Open

Wed Dec 21 19:01:10 2011, comment #2:

How about the following idea instead (it is simpler
and should be more efficient too):

Change 'grep' so that by default it ignores all NULs
in binary input files. POSIX allows this behavior, and it
is typically far more useful than what grep does now.
People who really want to examine the NULs can use
the -a option as before.

If we see a file with holes, treat it as binary, and
ignore the holes by default.

Paul Eggert <eggert>
Project Member
Wed Dec 21 15:57:45 2011, comment #1:

This is not a bug. The problem is not the sparseness of the file. The problem is that grep works on lines, and lines are separated by LF characters. So, a file that has 2GB of NULLs still has a single line.

This bug in particular is not fixable when grep's input is a pipe (so you cannot rewind the input when printing).

grep used to have some old code that used mmap to read the file. We ripped that out because in normal cases it actually had worse performance than read(), but in this case it had an advantage: the kernel has the possibility to swap out (and back in) to the sparse file. It would still require 100% CPU and can fail on 32-bit machines due to the small address space.

However, care must be taken for code like this:

echo foo > bar
(grep foo; cat) < bar

Here, the "cat" command must output nothing. IIUC the old mmap code did not advance the file pointer, so that the "cat" would print again the whole contents of the input file.

Paolo Bonzini <bonzini>
Tue Aug 16 08:21:43 2011, original submission:

Handling of sparse files should be improved, e.g the following example is handled by sed, but not by grep:

# cat /proc/meminfo | grep Total
MemTotal: 1017616 kB
SwapTotal: 2064380 kB
VmallocTotal: 34359738367 kB
HugePages_Total: 0

# dd if=/dev/zero of=/tmp/img bs=1 count=0 seek=40G

# grep a /tmp/img
grep: memory exhausted
# echo $?

# sed -n '/a/p' /tmp/img
# echo $?

Original ticket:

Jaroslav Škarvada <yarda>


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by eggert (Posted a comment)
  • -unavailable- added by bonzini (Posted a comment)
  • -unavailable- added by rwmj
  • -unavailable- added by yarda (Submitted the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue Aug 16 10:03:25 2011rwmjCarbon-Copy-=>Added -unavailable-

    Back to the top

    Powered by Savane 3.1-cleanup