bugfindutils - Bugs: bug #51506, Better support for data processing...

 
 

bug #51506: Better support for data processing with basenames

Submitter:  Markus Elfring <elfring>
Submitted:  Wed 19 Jul 2017 01:06:47 PM UTC
   
 
Category:  find Severity:  1 - Wish
Item Group:  None Status:  Wont Fix
Privacy:  Public Assigned to:  jay
Originator Name:  Open/Closed:  Closed
Release:  4.6.0 Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 25 Jul 2017 07:33:56 AM UTC, comment #16: 


> I agree with James - there's no need to add yet more knobs to findutils' configure.


How can any patch make a difference for which you would care also (despite of concerns for special program configurations) then?


> …, you'll still be free to use your patch locally.


It seems that this approach will be better for a while until the change acceptance can be adjusted a bit more.

Markus Elfring <elfring>
Mon 24 Jul 2017 11:17:02 AM UTC, comment #15: 

Submit patches, and we'll review them.  Until then, you are wasting more developer time arguing for the theoretical, than you are saving by optimizing any programs.

I agree with James - there's no need to add yet more knobs to findutils' configure. If you submit a patch that makes a difference, we'll enable it by default; if your patch doesn't make a difference, we'll let you know why it doesn't make the cut for inclusion upstream. But as this is free software, you'll still be free to use your patch locally.

Eric Blake <ericb>
Group administrator
Mon 24 Jul 2017 07:40:30 AM UTC, comment #14: 


> Every additional option is a burden in maintenance …


A few parameters can cause also run time effects which may be unwanted for special use cases.
I would find it nicer if known settings (besides the default one) could be selected to adjust the build a bit more for specific software needs.

Markus Elfring <elfring>
Sun 23 Jul 2017 08:26:50 PM UTC, comment #13: 

I recommend using an approach such as the one suggested by Bernhard.   

We're not going to change the behaviour of -print because this would make the resulting executable fail to comply with POSIX.  We're not going to create/build an an alternate executable with a different meaning of -print because distributions won't include it anyway (they don't include the existing alternate binary "oldfind" which we already have).

Consider "find" from a user-interface point of view.  It already has too many options, primitives, tests etc.   The manual (both Texinfo and man) is large.  Every additional option is a burden in maintenance (of tests, of the implementation code, and of documentation).  For these reasons we apply a high bar to the creation of new features.   Higher still for features whose underlying use-case can easily be achieved by the use of existing features of find.

James Youngman <jay>
Group administrator
Thu 20 Jul 2017 05:29:17 PM UTC, comment #12: 


> … warrants a separate ~200k binary.


I imagine that there are further build configuration possibilities to reduce the size of the executable file.


> exec find "$@" -printf "%f\n"


Thanks for your suggestion.

Such a command can also occasionally work.

Markus Elfring <elfring>
Thu 20 Jul 2017 05:09:56 PM UTC, comment #11: 


> I suggest to consider the choice between additional variants for the program “find”.


I don't think this marginal feature to fall back to basenames for -print
(or when no other action is specified) warrants a separate ~200k binary.
I propose to create a script like this and save as "$HOME/bin/findbasenames"
(or maybe in /usr/local/bin):

  #!/bin/sh
  exec find "$@" -printf "%f\n"

Have a nice day,
Berny

Bernhard Voelker <berny>
Group administrator
Thu 20 Jul 2017 12:04:59 PM UTC, comment #10: 


> You mean to change the behavior of the default -print action (or when omitted)
> to strip off leading directories?


Yes.


> This would break almost all existing scripts,


Not directly.

I suggest to consider the choice between additional variants for the program “find”.


> and future scripts would not be portable any more


Corresponding users have got the same choice possibilities for portability preferences.


> because it would depend on the mood of the admin to
> build it either way.


I guess that risks will remain for software build misconfigurations as usual, won't they?

Markus Elfring <elfring>
Thu 20 Jul 2017 11:51:52 AM UTC, comment #9: 


> [...] introduce a corresponding build configuration parameter ...


You mean to change the behavior of the default -print action (or when omitted)
to strip off leading directories?
This would break almost all existing scripts, and future scripts would not
be portable any more because it would depend on the mood of the admin to
build it either way.  Therefore, no.

Bernhard Voelker <berny>
Group administrator
Thu 20 Jul 2017 11:24:39 AM UTC, comment #8: 


> ... because no one ever complained about a performance / resource issue with the directories part not stripped off
> (because there is the "%f" option), …


  • Is the software situation still different for called commands?
  • How do you think about the idea to introduce a corresponding build configuration parameter so that the desired output function can be selected there?
Markus Elfring <elfring>
Thu 20 Jul 2017 09:45:03 AM UTC, comment #7: 


> How many test cases check already run time consequences for
> handling directory specifications there?


Most probably Zero ... because no one ever complained about a
performance / resource issue with the directories part not
stripped off (because there is the "%f" option), and I still
don't see that such an issue would exist.

Bernhard Voelker <berny>
Group administrator
Thu 20 Jul 2017 09:38:32 AM UTC, comment #6: 


> … a new action like e.g. -printbase makes any difference in any case


  • I imagine that such a “parameter alias” could be easier to remember.
  • But it would also mean that there is an additional string which would need a corresponding check in the affected source code.
  • Which setting will switch off the passing of prefixes to called commands?



> A patch for a new test case demonstrating the deficiency …


How many test cases check already run time consequences for handling directory specifications there?

Markus Elfring <elfring>
Thu 20 Jul 2017 09:17:47 AM UTC, comment #5: 


> So the specification of such a command line parameter would also be
> a bit too much extra data ...


I heavily doubt that abbreviating the 14 bytes of "-printf '%f\n'"
on the command line to a new action like e.g. -printbase makes any
difference in any case (neither in the trivial case with an empty
directory nor when diving into many hundred-thousands of subdirectories).

> ... (besides the internal parsing structures)
> under the view of special efficiency expectations.


Please point to the piece of code which you consider being inefficient.
A patch for a new test case demonstrating the deficiency ("test driven
development") would be much appreciated.

Bernhard Voelker <berny>
Group administrator
Thu 20 Jul 2017 08:20:35 AM UTC, comment #4: 


>For optimization issues, people usually come with a certain, reproducible case to point out a bottleneck.


I suggest a software adjustment where I can also imagine that you might not interpret its impact in significant ways because a specific functionality is already available.


>With that (only), we could start working on it.


I disagree here. - I dared to point a place out where it can happen that a bit more data processing would be performed so far than required.

  • Can it occasionally help with software efficiency to work a bit less at the beginning?


  • How much can it matter to avoid additional copies for directory names (or just prefixes)?




>… what is wrong with "-printf '%f\n'"?


A process category exists which would use a fixed output function there. So the specification of such a command line parameter would also be a bit too much extra data (besides the internal parsing structures) under the view of special efficiency expectations.
The corresponding processes would depend on a built-in system configuration for file searches.

Markus Elfring <elfring>
Wed 19 Jul 2017 09:21:34 PM UTC, comment #3: 


> Which system test would be representative to show that reduced file names
> can result in nicer run time characteristics because of less data transfer?


Whom do you ask?  For optimization issues, people usually come with
a certain, reproducible case to point out a bottleneck.  With that (only),
we could start working on it.

After all, this sounds more like the discussion to strip off the
leading "./" part when doing a bare "find" ... which is usually for
cosmetic reasons in end-user output.  But even then, this could
be avoided with "-printf '%P\n'".

Back to Eric's question: what is wrong with "-printf '%f\n'"?

Bernhard Voelker <berny>
Group administrator
Wed 19 Jul 2017 08:33:10 PM UTC, comment #2: 


>What resources are you envisioning that '-printf %f\\n' uses that would not be used by adding a new option, …


A bit of parsing could also be omitted for a string like “-printf %f\\n” if the desired output function could be selected by a build configuration parameter instead.


>… script that demonstrates such a need?


I guess that my data processing goals might not be prominent enough so far for this aspect.
Some developers are still used to the detail that a function like “basename()” needs to be called more often because extra prefixes are provided by default.

  • Does the hint to the “benchmark” indicate eventually a rejection just because corresponding software development efforts will not be accepted?
  • Which system test would be representative to show that reduced file names can result in nicer run time characteristics because of less data transfer?
Markus Elfring <elfring>
Wed 19 Jul 2017 07:46:25 PM UTC, comment #1: 

What resources are you envisioning that '-printf %f\\n' uses that would not be used by adding a new option, when balanced against the pain of adding, documenting, and maintaining a new option, for something that is already such a corner case that you had to lead out with "I image that there is a class of programs" instead of actually documenting a command line sequence or script that demonstrates such a need?

You are asking for a premature optimization.  Until you can demonstrate a concrete use case where find's -printf %f is not fast enough for your needs, I see no reason to even try to bloat find with something that merely repeats what -printf %f already supports.

-exec does NOT omit leading directory names, only -execdir (and -okdir) do that.  But there really isn't any much reason to add more than those actions.

I don't see how this is a valid request for change, so much as an idle musing of wondering if it is even possible to optimize something that wouldn't get used in the first place.  The first rule of optimization: present a benchmark showing that the existing approach has a bottleneck.  You haven't done that yet.

Eric Blake <ericb>
Group administrator
Wed 19 Jul 2017 01:06:47 PM UTC, original submission:  

I imagine that there is a class of programs which prefer to start their data processing with basenames. They will operate on file names without referencing directories from their input data at the beginning. So I am looking for ways that the copy of a found directory could be occasionally omitted from a search result.

  1. The parameter “-printf %f\\n” is already supported. Its data processing needs also an amount of computation resources. How do you think about to let an other program version work without it because of a system configuration where the applied output function can be a built-in setting?
  2. The parameters “-exec” and “-execdir” are also supported. Can an other argument (or another built-in setting) trigger the omission of a directory name from the passed command line parameters?
Markus Elfring <elfring>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by berny (Posted a comment)
  • -email is unavailable- added by ericb (Posted a comment)
  • -email is unavailable- added by elfring (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-07-23 jay Severity3 - Normal 1 - Wish
        StatusNone Wont Fix
        Assigned toNone jay
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code