bugfindutils - Bugs: bug #46815, problem when testing file size

 
 

bug #46815: problem when testing file size

Submitter:  None
Submitted:  Tue 05 Jan 2016 06:08:51 AM UTC
   
 
Category:  documentation Severity:  3 - Normal
Item Group:  None Status:  Need Info
Privacy:  Public Assigned to:  jay
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  None
Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 23 Sep 2018 01:46:20 PM UTC, comment #8: 

The existing behaviour is user-surprising.

POSIX specifies nly (IIRC) the rouding behaviour when dealing with sizes lacking a suffix (i.e. implicitly, blocks).  GNU find's current behaviour generalizes this rounding behaviour, but many people find this surprising.

However, it's proabably impossible to change it now without breaking user proggrams.  So the best answer is likely to be an extension or alternative usage of some kind.

This is been discussed at some length in https://savannah.gnu.org/bugs/index.php?12162

James Youngman <jay>
Group administrator
Thu 06 Sep 2018 01:17:59 PM UTC, comment #7: 

I'm a decades-long professional programmer, and even I just found the present behavior of rounding up very surprising, which is why I'm here posting.  If I can get confused, an ordinary user has no chance, and may well miss the fact that the data they thought they might see is simply missing from the output.  If the semantics of "-size -1M" are to be preserved as-is, I believe it's definitely worthwhile having a non-surprising alternative.  Nobody wants to type "-size -1048576c" instead of "-size -1M".

Anonymous
Mon 25 Apr 2016 06:35:26 PM UTC, comment #6: 

There's no strong reason to add -filesize N because -size Nc has the same effect.  The major reason would be if -size Nc is just too complicated for an ordinary user to figure out.

Dale Worley <worley>
Sat 23 Apr 2016 07:37:39 AM UTC, comment #5: 

I also find this very counter-intuitive.
A -filesize instead of -size sounds ok to me it it is documented well.
Even if -size was documented well, I don't think it should behave like this without providing an alternative like -filesize, but that's like only my opinion. A simple normal user.

ukuvbu oibws <xyzdragon>
Sat 09 Jan 2016 03:41:14 PM UTC, comment #4: 

Yes, both Sebastian and Dale are correct.  The 4.1.7 version of the manual page is much less clear, though consistent with the current behaviour:

-size n[bckw]
File uses n units of space.  The units are 512-byte blocks by default or if `b' follows n, bytes if `c' follows n, kilobytes if `k' follows n, or 2-byte words if `w' follows n.  The size does
not count indirect blocks, but it does count blocks in sparse files that are not actually allocated.

The current and previous code behaves similarly.  Here is the 4.1.7 code for illustration:

boolean
pred_size (pathname, stat_buf, pred_ptr)
     char *pathname;
     struct stat *stat_buf;
     struct predicate *pred_ptr;
{
  unsigned long f_val;

  f_val = (stat_buf->st_size + pred_ptr->args.size.blocksize - 1)
    / pred_ptr->args.size.blocksize;
  switch (pred_ptr->args.size.kind)
    {
    case COMP_GT:
      if (f_val > pred_ptr->args.size.size)
return (true);
      break;
    case COMP_LT:
      if (f_val < pred_ptr->args.size.size)
return (true);
      break;
    case COMP_EQ:
      if (f_val == pred_ptr->args.size.size)
return (true);
      break;
    }
  return (false);
}

As you can see, this does round up.

Let's go over to take a quick look at the POSIX requirements for the -size test at http://pubs.opengroup.org/onlinepubs/009695399/utilities/find.html :-

-size  n[c]
The primary shall evaluate as true if the file size in bytes, divided by 512 and rounded up to the next integer, is n. If n is followed by the character 'c', the size shall be in bytes.

So POSIX requires -size -1 should be false for a 500-byte file, and it also introduces a suffix c for bytes.

As is quite common for GNU tools, every chance to make a potentially-useful extension is eventually taken.  So in this case the introduction of alternative suffixes beyond the mandatory "c".  The "k" suffix denotes units of 1024 bytes, for example.  It's not suprising that somebody thought that the behavior for dealing in k should be quite similar to that for dealing in units of 512 bytes (especially if they were using one of the several systems where the system block size for things like ls -s is in fact 1024 bytes).

But this is clearly surprising for unit suffixes like m, for which it is pretty clear that the user is not thinking in terms of how many blocks the file occupies on the storage layer.  So the existing behavior is kind of understandable, but it is obviously confusing for most users.

The canonical version of the bug report on this, one might say, is https://savannah.gnu.org/bugs/?12162.  There are others (see https://savannah.gnu.org/bugs/?group=findutils&func=browse&set=custom&msort=0&status_id[=3&resolution_id[]=0&submitted_by[]=0&assigned_to[]=0&category_id[]=0&bug_group_id[]=0&severity[]=0&summary[]=-size&details][]=&advsrch=0&msort=0&chunksz=50&spamscore=5&report_id=101&sumORdet=0&morder=severity%3C&sumOrdet=0&order=date#results).

I think that particular discussion got side-tracked, at the end, by the introduction of time tests.  The problems of rounding with those tests have largely been obviated by the introduction of tests like -newermt, where the timestamp is specified directly in absolute not relative terms (and no rounding occurs).

I've been in favour of providing a more sensible test for a long time, the problem has always been how to spell the new usage and describe its semantics.   The use of > and < prefixes is attractive, but doomed by the use of those characters by the shell.  Yes, the user could quote them to avoid redirection, but this would clearly be a source of confusion for less experienced users.

The alternatives that seem attractive to me are Nigel McNie's proposal to use a new test, '-filesize' or something along the lines of Martin Steigerwald's three-word variant (i.e. -size lt 20M being the sane, no-rounding, replacement for -size -20M).

Both of those options have the nice property that they're likely POSIX-compliant in the sense that POSIX provides no required meaning for those constructs (though -filesize is I suppose more obviously a GNU extension).

Let's re-open the discussion about what to call the "sane" alternative to -size, and implement it this time.

James Youngman <jay>
Group administrator
Fri 08 Jan 2016 07:57:27 PM UTC, comment #3: 

I have a very strong suspicion that the current code behaves "correctly", that is, in the way that "find" has always worked.  The original -size meaning "Does the file occupy N (512-byte) disk blocks?".  (Of course, usually people would use "-size -N" or "-size +N".)  So the code can't be changed without causing a lot of scripts to fail.

That being said, the documentation should be made clearer.  If you read the text carefully, you can tell that it means what the code does.  But the new user isn't likely to read it correctly.  On my system, the text is:

File uses n units of space.  The following suffixes can be used:

The current text is reportedly:

File uses n units of space, rounding up. The following suffixes can be used:

A better version is:

File uses exactly n of the specified units of space, rounding up any fractional unit.  The following suffixes can be used to specify the unit:

Do you think there is any ambiguity in that text?

Dale Worley <worley>
Fri 08 Jan 2016 07:57:04 AM UTC, comment #2: 

Hi James,

Thanks for your quick response!

Before I decided to submit this report I checked out the current version of GNU findutils as recommended on the project's website. The code for comparing file sizes, which I've quoted in the report, is nearly 16 years old. Hence I wasn't sure which version to choose from the drop-down list because the oldest selectable one (4.1.7) has been released one year after the commit and so all versions are affected.

Sorry that I didn't read the description you've quoted from the manpage. The file has been changed only 5 days before my bug report - I didn't noticed the change.

I still wonder about the way GNU find compares file sizes. Rounding up an one byte sized file to one gigabyte, for example, doesn't seem to be the way an average user expects. I discussed this topic with various persons before I opened the ticket. Everyone was very surprised that GNU find rounds up the size to the specified unit. I also tested the find implementations of FreeBSD and Busybox. As you can see both commands don't round up:


I would be glad if someone could state a reasonable use-case where rounding up makes sense, because I can't find one.


Kind regards & many thanks in advance,

Sebastian

Anonymous
Tue 05 Jan 2016 05:14:54 PM UTC, comment #1: 

You didn't state which version of find you are using.   But to quote from the current version of the manual page:

-size n[cwbkMG]
File uses n units of space, rounding up.  The following suffixes can be used:
...
The + and - prefixes signify greater than and less than, as usual.  Bear in mind that the size is rounded up to the next unit. Therefore -size -1M is not equivalent to -size  -1048576c.   The
former only matches empty files, the latter matches files from 1 to 1,048,575 bytes.


James Youngman <jay>
Group administrator
Tue 05 Jan 2016 06:08:51 AM UTC, original submission:  

Hi,

I noticed a strange behaviour in GNU find when testing the file
size. Let me explain the found problem with a small example:

$ dd if=/dev/zero of=foo.1k bs=1024 count=1
$ dd if=/dev/zero of=foo.24k bs=1024 count=24
$ dd if=/dev/zero of=foo.1M bs=1024 count=1024

$ # this should only find foo.1M:
$ find . -size 1M
.
./foo.24k
./foo.1k
./foo.1M

$ # this one works correctly:
$ find . -size 24k
./foo.24k

$ # all files less than 1M should be found but result is empty:
$ find . -size -1M

$ # all files are found:
$ find . -size -2M
.
./foo.24k
./foo.1k
./foo.1M

$ # a different unit also solves the issue:
$ find . -size -1024k
.
./foo.24k
./foo.1k

The problem is how pred_size() calculates the file size. When
searching for a file with a size of 1M and processing the test
files we created previously f_val is always 1:

f_val = ((1024 / 1048576)  + (1024 % 1048576 != 0))  = 1
f_val = ((24576 / 1048576) + (24576 % 1048576 != 0)) = 1
f_val = ((24576 / 1048576) + (24576 % 1048576 != 0)) = 1

I fixed the problem on my system by applying this patch:

diff --git a/find/pred.c b/find/pred.c
index d633ab9..73abbec 100644
--- a/find/pred.c
+++ b/find/pred.c
@@ -956,12 +956,12 @@ bool
 pred_size (const char *pathname, struct stat *stat_buf, struct predicate *pred_ptr)
 {
   uintmax_t f_val;

   (void) pathname;
-  f_val = ((stat_buf->st_size / pred_ptr->args.size.blocksize)
-    + (stat_buf->st_size % pred_ptr->args.size.blocksize != 0));
+  f_val = stat_buf->st_size / pred_ptr->args.size.blocksize;
+
   switch (pred_ptr->args.size.kind)
     {
     case COMP_GT:
       if (f_val > pred_ptr->args.size.size)
  return (true);


Kind regards,

Sebastian

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by xyzdragon (Posted a comment)
  • -email is unavailable- added by jay (Proposed a solution at https://savannah.gnu.org/bugs/?12162)
  • -email is unavailable- added by jay (Proposed a solution at https://savannah.gnu.org/bugs/?12162)
  • -email is unavailable- added by worley (Posted a comment)
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-01-09 jay StatusInvalid Need Info
        Open/ClosedClosed Open
    2016-01-09 jay Carbon-CopyRemoved -email is unavailable- -
    2016-01-09 jay Carbon-Copy- Added msteamix
    2016-01-09 jay Carbon-Copy- Added nigel
    2016-01-09 jay Carbon-Copy- Added -email is unavailable-
    2016-01-05 jay Categoryfind documentation
        StatusNone Invalid
        Assigned toNone jay
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code