buggrep - Bugs: bug #34020, RFE: Improve handling of sparse...


You are not allowed to post comments on this tracker with your current authentication level.

bug #34020: RFE: Improve handling of sparse files

Submitted by:  Jaroslav ┼ákarvada <yarda>
Submitted on:  Tue 16 Aug 2011 08:21:43 AM UTC  
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Wed 21 Dec 2011 07:01:10 PM UTC, comment #2: 

How about the following idea instead (it is simpler
and should be more efficient too):

Change 'grep' so that by default it ignores all NULs
in binary input files.  POSIX allows this behavior, and it
is typically far more useful than what grep does now.
People who really want to examine the NULs can use
the -a option as before.

If we see a file with holes, treat it as binary, and
ignore the holes by default.

Paul Eggert <eggert>
Project Member
Wed 21 Dec 2011 03:57:45 PM UTC, comment #1: 

This is not a bug.  The problem is not the sparseness of the file.  The problem is that grep works on lines, and lines are separated by LF characters.  So, a file that has 2GB of NULLs still has a single line.

This bug in particular is not fixable when grep's input is a pipe (so you cannot rewind the input when printing).

grep used to have some old code that used mmap to read the file.  We ripped that out because in normal cases it actually had worse performance than read(), but in this case it had an advantage: the kernel has the possibility to swap out (and back in) to the sparse file.  It would still require 100% CPU and can fail on 32-bit machines due to the small address space.

However, care must be taken for code like this:

   echo foo > bar
   (grep foo; cat) < bar

Here, the "cat" command must output nothing.  IIUC the old mmap code did not advance the file pointer, so that the "cat" would print again the whole contents of the input file.

Paolo Bonzini <bonzini>
Tue 16 Aug 2011 08:21:43 AM UTC, original submission:  

Handling of sparse files should be improved, e.g the following example is handled by sed, but not by grep:

# cat /proc/meminfo | grep Total
MemTotal:        1017616 kB
SwapTotal:       2064380 kB
VmallocTotal:   34359738367 kB
HugePages_Total:       0

# dd if=/dev/zero of=/tmp/img bs=1 count=0 seek=40G

# grep a /tmp/img
grep: memory exhausted
# echo $?

# sed -n '/a/p' /tmp/img
# echo $?

Original ticket:

Jaroslav Škarvada <yarda>


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by eggert (Posted a comment)
  • -email is unavailable- added by bonzini (Posted a comment)
  • -email is unavailable- added by rwmj
  • -email is unavailable- added by yarda (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-08-16 rwmj Carbon-Copy- => Added -email is unavailable-

    Back to the top

    Powered by Savane 3.7