bugfindutils - Bugs: bug #46815, problem when testing file size

 
 

bug #46815: problem when testing file size

Submitted by:  None
Submitted on:  Tue 05 Jan 2016 06:08:51 AM UTC  
 
Category: documentationSeverity: 3 - Normal
Item Group: NoneStatus: Need Info
Privacy: PublicAssigned to: James Youngman <jay>
Originator Name: Originator Email: -unavailable-
Open/Closed: OpenRelease: None
Fixed Release: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 25 Apr 2016 06:35:26 PM UTC, comment #6:

There's no strong reason to add -filesize N because -size Nc has the same effect. The major reason would be if -size Nc is just too complicated for an ordinary user to figure out.

Dale Worley <worley>
Sat 23 Apr 2016 07:37:39 AM UTC, comment #5:

I also find this very counter-intuitive.
A -filesize instead of -size sounds ok to me it it is documented well.
Even if -size was documented well, I don't think it should behave like this without providing an alternative like -filesize, but that's like only my opinion. A simple normal user.

ukuvbu oibws <xyzdragon>
Sat 09 Jan 2016 03:41:14 PM UTC, comment #4:

Yes, both Sebastian and Dale are correct. The 4.1.7 version of the manual page is much less clear, though consistent with the current behaviour:

-size n[bckw]
File uses n units of space. The units are 512-byte blocks by default or if `b' follows n, bytes if `c' follows n, kilobytes if `k' follows n, or 2-byte words if `w' follows n. The size does
not count indirect blocks, but it does count blocks in sparse files that are not actually allocated.

The current and previous code behaves similarly. Here is the 4.1.7 code for illustration:

boolean
pred_size (pathname, stat_buf, pred_ptr)
char *pathname;
struct stat *stat_buf;
struct predicate *pred_ptr;
{
unsigned long f_val;

f_val = (stat_buf->st_size + pred_ptr->args.size.blocksize - 1)
/ pred_ptr->args.size.blocksize;
switch (pred_ptr->args.size.kind)
{
case COMP_GT:
if (f_val > pred_ptr->args.size.size)
return (true);
break;
case COMP_LT:
if (f_val < pred_ptr->args.size.size)
return (true);
break;
case COMP_EQ:
if (f_val == pred_ptr->args.size.size)
return (true);
break;
}
return (false);
}

As you can see, this does round up.

Let's go over to take a quick look at the POSIX requirements for the -size test at http://pubs.opengroup.org/onlinepubs/009695399/utilities/find.html :-

-size n[c]
The primary shall evaluate as true if the file size in bytes, divided by 512 and rounded up to the next integer, is n. If n is followed by the character 'c', the size shall be in bytes.

So POSIX requires -size -1 should be false for a 500-byte file, and it also introduces a suffix c for bytes.

As is quite common for GNU tools, every chance to make a potentially-useful extension is eventually taken. So in this case the introduction of alternative suffixes beyond the mandatory "c". The "k" suffix denotes units of 1024 bytes, for example. It's not suprising that somebody thought that the behavior for dealing in k should be quite similar to that for dealing in units of 512 bytes (especially if they were using one of the several systems where the system block size for things like ls -s is in fact 1024 bytes).

But this is clearly surprising for unit suffixes like m, for which it is pretty clear that the user is not thinking in terms of how many blocks the file occupies on the storage layer. So the existing behavior is kind of understandable, but it is obviously confusing for most users.

The canonical version of the bug report on this, one might say, is https://savannah.gnu.org/bugs/?12162. There are others (see https://savannah.gnu.org/bugs/?group=findutils&func=browse&set=custom&msort=0&status_id[=3&resolution_id[]=0&submitted_by[]=0&assigned_to[]=0&category_id[]=0&bug_group_id[]=0&severity[]=0&summary[]=-size&details][]=&advsrch=0&msort=0&chunksz=50&spamscore=5&report_id=101&sumORdet=0&morder=severity%3C&sumOrdet=0&order=date#results).

I think that particular discussion got side-tracked, at the end, by the introduction of time tests. The problems of rounding with those tests have largely been obviated by the introduction of tests like -newermt, where the timestamp is specified directly in absolute not relative terms (and no rounding occurs).

I've been in favour of providing a more sensible test for a long time, the problem has always been how to spell the new usage and describe its semantics. The use of > and < prefixes is attractive, but doomed by the use of those characters by the shell. Yes, the user could quote them to avoid redirection, but this would clearly be a source of confusion for less experienced users.

The alternatives that seem attractive to me are Nigel McNie's proposal to use a new test, '-filesize' or something along the lines of Martin Steigerwald's three-word variant (i.e. -size lt 20M being the sane, no-rounding, replacement for -size -20M).

Both of those options have the nice property that they're likely POSIX-compliant in the sense that POSIX provides no required meaning for those constructs (though -filesize is I suppose more obviously a GNU extension).

Let's re-open the discussion about what to call the "sane" alternative to -size, and implement it this time.

James Youngman <jay>
Project AdministratorIn charge of this item.
Fri 08 Jan 2016 07:57:27 PM UTC, comment #3:

I have a very strong suspicion that the current code behaves "correctly", that is, in the way that "find" has always worked. The original -size meaning "Does the file occupy N (512-byte) disk blocks?". (Of course, usually people would use "-size -N" or "-size +N".) So the code can't be changed without causing a lot of scripts to fail.

That being said, the documentation should be made clearer. If you read the text carefully, you can tell that it means what the code does. But the new user isn't likely to read it correctly. On my system, the text is:

File uses n units of space. The following suffixes can be used:

The current text is reportedly:

File uses n units of space, rounding up. The following suffixes can be used:

A better version is:

File uses exactly n of the specified units of space, rounding up any fractional unit. The following suffixes can be used to specify the unit:

Do you think there is any ambiguity in that text?

Dale Worley <worley>
Fri 08 Jan 2016 07:57:04 AM UTC, comment #2:

Hi James,

Thanks for your quick response!

Before I decided to submit this report I checked out the current version of GNU findutils as recommended on the project's website. The code for comparing file sizes, which I've quoted in the report, is nearly 16 years old. Hence I wasn't sure which version to choose from the drop-down list because the oldest selectable one (4.1.7) has been released one year after the commit and so all versions are affected.

Sorry that I didn't read the description you've quoted from the manpage. The file has been changed only 5 days before my bug report - I didn't noticed the change.

I still wonder about the way GNU find compares file sizes. Rounding up an one byte sized file to one gigabyte, for example, doesn't seem to be the way an average user expects. I discussed this topic with various persons before I opened the ticket. Everyone was very surprised that GNU find rounds up the size to the specified unit. I also tested the find implementations of FreeBSD and Busybox. As you can see both commands don't round up:

I would be glad if someone could state a reasonable use-case where rounding up makes sense, because I can't find one.

Kind regards & many thanks in advance,

Sebastian

Anonymous
Tue 05 Jan 2016 05:14:54 PM UTC, comment #1:

You didn't state which version of find you are using. But to quote from the current version of the manual page:

-size n[cwbkMG]
File uses n units of space, rounding up. The following suffixes can be used:
...
The + and - prefixes signify greater than and less than, as usual. Bear in mind that the size is rounded up to the next unit. Therefore -size -1M is not equivalent to -size -1048576c. The
former only matches empty files, the latter matches files from 1 to 1,048,575 bytes.

James Youngman <jay>
Project AdministratorIn charge of this item.
Tue 05 Jan 2016 06:08:51 AM UTC, original submission:

Hi,

I noticed a strange behaviour in GNU find when testing the file
size. Let me explain the found problem with a small example:

$ dd if=/dev/zero of=foo.1k bs=1024 count=1
$ dd if=/dev/zero of=foo.24k bs=1024 count=24
$ dd if=/dev/zero of=foo.1M bs=1024 count=1024

$ # this should only find foo.1M:
$ find . -size 1M
.
./foo.24k
./foo.1k
./foo.1M

$ # this one works correctly:
$ find . -size 24k
./foo.24k

$ # all files less than 1M should be found but result is empty:
$ find . -size -1M

$ # all files are found:
$ find . -size -2M
.
./foo.24k
./foo.1k
./foo.1M

$ # a different unit also solves the issue:
$ find . -size -1024k
.
./foo.24k
./foo.1k

The problem is how pred_size() calculates the file size. When
searching for a file with a size of 1M and processing the test
files we created previously f_val is always 1:

f_val = ((1024 / 1048576) + (1024 % 1048576 != 0)) = 1
f_val = ((24576 / 1048576) + (24576 % 1048576 != 0)) = 1
f_val = ((24576 / 1048576) + (24576 % 1048576 != 0)) = 1

I fixed the problem on my system by applying this patch:

diff --git a/find/pred.c b/find/pred.c
index d633ab9..73abbec 100644
--- a/find/pred.c
+++ b/find/pred.c
@@ -956,12 +956,12 @@ bool
pred_size (const char pathname, struct stat stat_buf, struct predicate *pred_ptr)
{
uintmax_t f_val;

(void) pathname;
- f_val = ((stat_buf->st_size / pred_ptr->args.size.blocksize)
- + (stat_buf->st_size % pred_ptr->args.size.blocksize != 0));
+ f_val = stat_buf->st_size / pred_ptr->args.size.blocksize;
+
switch (pred_ptr->args.size.kind)
{
case COMP_GT:
if (f_val > pred_ptr->args.size.size)
return (true);

Kind regards,

Sebastian

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by xyzdragon (Posted a comment)
  • -unavailable- added by jay (Proposed a solution at https://savannah.gnu.org/bugs/?12162)
  • -unavailable- added by jay (Proposed a solution at https://savannah.gnu.org/bugs/?12162)
  • -unavailable- added by worley (Posted a comment)
  • -unavailable- added by jay (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 10 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 09 Jan 2016 03:41:14 PM UTCjayStatusInvalid=>Need Info
      Open/ClosedClosed=>Open
    Sat 09 Jan 2016 03:39:30 PM UTCjayCarbon-CopyRemoved -unavailable-=>-
    Sat 09 Jan 2016 03:39:19 PM UTCjayCarbon-Copy-=>Added msteamix
    Sat 09 Jan 2016 03:37:33 PM UTCjayCarbon-Copy-=>Added nigel
    Sat 09 Jan 2016 03:36:35 PM UTCjayCarbon-Copy-=>Added -unavailable-
    Tue 05 Jan 2016 05:14:54 PM UTCjayCategoryfind=>documentation
      StatusNone=>Invalid
      Assigned toNone=>jay
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1