buggrep - Bugs: bug #34020, RFE: Improve handling of sparse...

 
 

You are not allowed to post comments on this tracker with your current authentification level.

bug #34020: RFE: Improve handling of sparse files

Submitted by:  Jaroslav ┼ákarvada <yarda>
Submitted on:  Tue 16 Aug 2011 08:21:43 AM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: None
Privacy: PublicAssigned to: None
Open/Closed: Open

Wed 21 Dec 2011 07:01:10 PM UTC, comment #2:

How about the following idea instead (it is simpler
and should be more efficient too):

Change 'grep' so that by default it ignores all NULs
in binary input files. POSIX allows this behavior, and it
is typically far more useful than what grep does now.
People who really want to examine the NULs can use
the -a option as before.

If we see a file with holes, treat it as binary, and
ignore the holes by default.

Paul Eggert <eggert>
Project Member
Wed 21 Dec 2011 03:57:45 PM UTC, comment #1:

This is not a bug. The problem is not the sparseness of the file. The problem is that grep works on lines, and lines are separated by LF characters. So, a file that has 2GB of NULLs still has a single line.

This bug in particular is not fixable when grep's input is a pipe (so you cannot rewind the input when printing).

grep used to have some old code that used mmap to read the file. We ripped that out because in normal cases it actually had worse performance than read(), but in this case it had an advantage: the kernel has the possibility to swap out (and back in) to the sparse file. It would still require 100% CPU and can fail on 32-bit machines due to the small address space.

However, care must be taken for code like this:

echo foo > bar
(grep foo; cat) < bar

Here, the "cat" command must output nothing. IIUC the old mmap code did not advance the file pointer, so that the "cat" would print again the whole contents of the input file.

Paolo Bonzini <bonzini>
Tue 16 Aug 2011 08:21:43 AM UTC, original submission:

Handling of sparse files should be improved, e.g the following example is handled by sed, but not by grep:

# cat /proc/meminfo | grep Total
MemTotal: 1017616 kB
SwapTotal: 2064380 kB
VmallocTotal: 34359738367 kB
HugePages_Total: 0

# dd if=/dev/zero of=/tmp/img bs=1 count=0 seek=40G

# grep a /tmp/img
grep: memory exhausted
# echo $?
2

# sed -n '/a/p' /tmp/img
# echo $?
0

Original ticket:
http://bugzilla.redhat.com/show_bug.cgi?id=729052

Jaroslav Škarvada <yarda>

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by eggert (Posted a comment)
  • -unavailable- added by bonzini (Posted a comment)
  • -unavailable- added by rwmj
  • -unavailable- added by yarda (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 16 Aug 2011 10:03:25 AM UTCrwmjCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup