bugmake - Bugs: bug #51309, Determination of a file list from...

 
 

bug #51309: Determination of a file list from a single folder without changing the working directory

Submitter:  Markus Elfring <elfring>
Submitted:  Sun 25 Jun 2017 05:53:27 PM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  Wont Fix Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Component Version:  4.2.1 Operating System:  Any
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 20 Jul 2017 09:04:36 AM UTC, comment #10: 

Would you like to compare run time characteristics a bit more with an approach similar to “$(shell find ~/Projekte/Bau/OCamlbuild -name '*.txt' -printf "%f ")”?

Markus Elfring <elfring>
Sun 02 Jul 2017 08:14:31 PM UTC, comment #9: 

I suggested also a comparison of two algorithms.

  • A simple time measurement can also show test results for which you might not be looking for.
  • The usage of data structures like list and trees has got also different run time characteristics. Each of them can have a corresponding application domain.
  • Which test environments would you find representative?



> From this I conclude that the performance impact of an additional notdir function is negligible and there's no performance advantage to avoiding it.


It might matter more under other circumstances. (The current software can be fast enough to some degree.)

Markus Elfring <elfring>
Sun 02 Jul 2017 07:45:35 PM UTC, comment #8: 

If, for example, you can show an example of notdir adding significantly to the amount of time the make program takes.

To test this I created a makefile like this:


foo = ...

all: ; : $(words $(foo))


where the "..." represents 6000 paths hardcoded into the makefile.  I then ran "time make":


$ time make
: 6000

real    0m0.004s
user    0m0.000s
sys     0m0.000s


Then I modified the makefile to add a notdir like this:


foo = $(notdir ...)

all: ; : $(words $(foo))


where the "..." is the same as last time.  I got this result:


$ time make
: 6000

real    0m0.004s
user    0m0.000s
sys     0m0.000s


Same time.  This means that the amount of time needed to run notdir on 6000 paths is less than 1/1000th of a second.  From this I conclude that the performance impact of an additional notdir function is negligible and there's no performance advantage to avoiding it.

Paul D. Smith <psmith>
Group administrator
Sun 02 Jul 2017 04:42:13 PM UTC, comment #7: 


> (a) you prefer to do it this way rather than the currently supported method of post-processing the wildcard results, …


Yes.


>(b) there's a significant performance benefit to avoiding the extra processing.


I find that the difference can be eventually measured.

We have got different opinions about the circumstances when it would be significant enough.


> … adding new features in a backward- and POSIX-compatible way is difficult due …


How many software developers would dare to extend corresponding programming interfaces so that they can be better reused by the make function library?

I am curious if this story could ever turn into a more promising “maybe” thing.

Markus Elfring <elfring>
Sun 02 Jul 2017 04:31:59 PM UTC, comment #6: 

A theoretical argument is not what Paul meant by "actual results".  Have you measured the time spent adding and removing the directory names in some realistic case?  If not, then please do not advocate for making Make bigger..

Martin Dorey <mdorey>
Sun 02 Jul 2017 04:24:29 PM UTC, comment #5: 

I understand what you are suggesting.

But I don't see any need to add new capabilities for this when the existing features of GNU make can already give these results.

The only reasons for it that I can see would be (a) you prefer to do it this way rather than the currently supported method of post-processing the wildcard results, or (b) there's a significant performance benefit to avoiding the extra processing.  For (a), I'm not inclined to create new capabilities just for this reason.  For (b), I seriously doubt that there will be any measurable performance difference in a real makefile.

If you can show an example where there is a significant performance difference then we can discuss that.

To be clear, my attitudes on makefile syntax are that (1) it's already complex and there are already a lot of features, complexities, and "gotchas", (2) adding new features in a backward- and POSIX-compatible way is difficult due to the relatively free-form syntax of makefiles, and (3) features combine in ways that are multiplicative, not additive, which greatly increases the potential for incorrect interactions, bugs, and the testing needed to support them.

Thus, my initial answer to adding new capabilities is almost always going to be "no".  It requires a clear improvement in power (not just syntactic sugar) and a reasonable syntax that makes sense and doesn't break lots of existing makefiles to clear that hurdle and get to "maybe" or "yes".

Paul D. Smith <psmith>
Group administrator
Sun 02 Jul 2017 03:53:17 PM UTC, comment #4: 

I repeat my argument from 26. June:
https://lists.gnu.org/archive/html/bug-make/2017-06/msg00198.html

If the desired directory on which a file filter should be applied could be selected before, the file names would not contain references to this directory and another function call is therefore unnecessary to remove it from the returned list elements.

Markus Elfring <elfring>
Sun 02 Jul 2017 03:38:03 PM UTC, comment #3: 

There's no question that


FOO := $(notdir $(wildcard Bau/*.txt))


is more efficient than:


~/Projekte> $(cd Bau && echo "echo $(ls *txt)")


because the latter involves starting a whole new shell (two, technically) while the former is handled entirely within the make process with no fork/exec operation required.

If you can show actual results where adding the notdir makes a significant performance difference in a makefile environment we can reconsider, but I'm not interested in adding new features duplicating existing capabilities without proof that there's a real need.

Paul D. Smith <psmith>
Group administrator
Sun 02 Jul 2017 03:27:53 PM UTC, comment #2: 

I disagree with the run time characteristics of the mentioned functionality for my special use case.


~/Projekte> $(cd Bau && echo "echo $(ls *txt)")
MOTD.txt



Can a similar data processing operation be performed faster by advanced make functions?

Markus Elfring <elfring>
Sun 02 Jul 2017 03:08:44 PM UTC, comment #1: 

The current capabilities (using notdir to remove the entire directory, or using patsubst etc. to remove partial directories) is sufficient.  Nothing new is needed here.

Paul D. Smith <psmith>
Group administrator
Sun 25 Jun 2017 05:53:27 PM UTC, original submission:  

The function “wildcard” is supported by the make software.
Corresponding file patterns work also in combination with directory names. Such combinations are occasionally useful. The query result will contain them as usual.

I came along the need to query a single folder without changing the working directory so that only file names (without folder names) will be in the result.
Can another software extension become useful for the function library to make such data processing more efficient?

Markus Elfring <elfring>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mdorey (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by elfring (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-07-02 psmith StatusNone Wont Fix
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code