bugmake - Bugs: bug #59318, Add option ('-g') of generating...

 
 

bug #59318: Add option ('-g') of generating build graph for updated goals

Submitter:  Jonas Danielsson <jonasdn>
Submitted:  Thu 22 Oct 2020 08:00:08 AM UTC
Votes: 30
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.3 Operating System:  None
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 15 Nov 2020 07:02:46 PM UTC, comment #4: 

comment #3:

> Hi!
>
> Nice proposal. The Remake project (http://bashdb.sourceforge.net/remake/) already implements something like this. We even tried it on our project.'

Hi, thanks!

I just tried remake. The --profile option feels like  a superset of this function yes! Browsing the resulting graph via kcachegrind is a nice experience!

>
> The problem I found with that feature is that the resulting graph can be incomprehensibly large (millions of nodes for large projects) and therefore not very useful in reality. In the end, we figured out that the easiest way to ensure that we can identify why something is being remade is to add basic logging to all make runs (--debug=b). This shows very minimal details, does not distort the logs too much but provides the vital info required to identify the root cause in most cases.
>


Yes, I agree with this. This feature might be most useful for finding out result for specific targets or for smaller projects.
At least if you want to create a full graph and visualize it all at once.

Another way to go is what kcachegrind does, with the remake case. And another tool that is taskexp (and depexp before it) in the Yocto project.
They allow you to search and browse targets, and follow certain paths that you are interested in.
With an explorer using a well-thought UI you could create a nice experience from a dot graph as well I believe.

> On the other hand, such a graph can be really useful to discover dependencies and use it to optimize the build. Of course, that requires proper tooling for the graph handling.
>


Agreed.

> Have you tried the Remake implementation? Did you find it useful in practice?


I tried it for the first time tonight. I feel it could be useful!
I might say that an "advantage" with my implementation is that it is smaller in ambition :) I tried to keep the code and the outputs as non-invasive as I could. And I believe other tools could be used to interpret the resulting dot graph.

Thanks for your comment!

Jonas Danielsson <jonasdn>
Sat 14 Nov 2020 04:47:22 PM UTC, comment #3: 

Hi!

Nice proposal. The Remake project (http://bashdb.sourceforge.net/remake/) already implements something like this. We even tried it on our project.

The problem I found with that feature is that the resulting graph can be incomprehensibly large (millions of nodes for large projects) and therefore not very useful in reality. In the end, we figured out that the easiest way to ensure that we can identify why something is being remade is to add basic logging to all make runs (--debug=b). This shows very minimal details, does not distort the logs too much but provides the vital info required to identify the root cause in most cases.

On the other hand, such a graph can be really useful to discover dependencies and use it to optimize the build. Of course, that requires proper tooling for the graph handling.

Have you tried the Remake implementation? Did you find it useful in practice?

Andras Kovi <allprog>
Fri 13 Nov 2020 01:17:49 PM UTC, comment #2: 

Does anyone have any opinions about this enhancement proposal?


Jonas Danielsson <jonasdn>
Wed 28 Oct 2020 08:25:52 AM UTC, comment #1: 
Jonas Danielsson <jonasdn>
Thu 22 Oct 2020 08:00:08 AM UTC, original submission:  

Hi!

I know this enhancement has been discussed before for the GNU Make project. But I would like to modestly propose complementing the powerful --print-database option with something that is a bit easier to grasp at a glance.

A graph visualisation can be helpful in order to get a better understanding of the workings of a GNU Make based project. Particularly if you are debugging why something keeps rebuilding each pass or if you are very new to a projekt.

All the information that can be gleaned from this graph output is already present in the output of --print-database. However I strive to present it more succinct. Only the targets that were modified in the goal is present and the relationship between targets and prerequisite is hopefully more clear.
   
The output from the goal graph is stored in files called goal.dot in the starting directory.
   
The hope is also that this can be paired with already existing viewers and explorers, since the dot language is well known (29 years!).

This is something that can be used for instance when trying to figure out why something keeps rebuilding / updating on a second run of Make. Or for figuring out the dependency chain of a particular goal. As a first approach to get a hint, and then perhaps using --print-database to learn more.
   
An example (rebuilding GNU Make):
      $ touch src/file.c
      $ AM_MAKEFLAGS="-g" make -g
      Making all in lib
      make[1]: Entering directory '/home/jonas/sandbox/mymake/lib'
      /home/jonas/sandbox/mymake/./make -g all-recursive
      make[2]: Entering directory '/home/jonas/sandbox/mymake/lib'
      make[3]: Entering directory '/home/jonas/sandbox/mymake/lib'
      make[3]: Nothing to be done for 'all-am'.
      make[3]: Leaving directory '/home/jonas/sandbox/mymake/lib'
      make[2]: Writing dependency graph to '/home/jonas/sandbox/mymake/lib/all-recursive.dot'
      make[2]: Leaving directory '/home/jonas/sandbox/mymake/lib'
      make[1]: Writing dependency graph to '/home/jonas/sandbox/mymake/lib/all.dot'
      make[1]: Leaving directory '/home/jonas/sandbox/mymake/lib'
      Making all in po
      make[1]: Entering directory '/home/jonas/sandbox/mymake/po'
      make[1]: Nothing to be done for 'all'.
      make[1]: Leaving directory '/home/jonas/sandbox/mymake/po'
      Making all in doc
      make[1]: Entering directory '/home/jonas/sandbox/mymake/doc'
      make[1]: Nothing to be done for 'all'.
      make[1]: Leaving directory '/home/jonas/sandbox/mymake/doc'
      make[1]: Entering directory '/home/jonas/sandbox/mymake'
   
      [...]
      [gcc building]
      [...]
   
      make[1]: Writing dependency graph to '/home/jonas/sandbox/mymake/all-am.dot'
      make[1]: Leaving directory '/home/jonas/sandbox/mymake'
      make: Writing dependency graph to '/home/jonas/sandbox/mymake/all.dot'

$ cat all-am.dot
      digraph all-am {
      "make" -> "src/file.o"
      "all-am" -> "make"
      "all-am"
      }
   
      $ cat all.dot
      digraph all {
      "all" -> "all-recursive" [label="forced: PHONY prerequisite"]
      "all"
      }

PNG versions of these graphs (generated with dot -Tpng) can be found as attachments.

Jonas Danielsson <jonasdn>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50115:  0001-Add-g-opt-for-graph-output-of-modified-targets.patch added by jonasdn (10KiB - text/x-patch - A file can be a dependency to different targets, make patch reflect that)
file #50069:  0001-Add-g-opt-for-graph-output-of-modified-targets.patch added by jonasdn (10KiB - text/x-patch - Updated version 23 oct)
file #50046:  all.png added by jonasdn (12KiB - image/png)
file #50047:  all-am.png added by jonasdn (11KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by botmotion (Voted in favor of this item)
  • -email is unavailable- added by allprog (Posted a comment)
  • -email is unavailable- added by jonasdn (Submitted the item)
  • -email is unavailable- added by jonasdn
  •  

    There are 30 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-04-15 botmotion Carbon-Copy- Added botmotion
    2020-10-26 jonasdn Attached File- Added 0001-Add-g-opt-for-graph-output-of-modified-targets.patch, #50115
    2020-10-23 jonasdn Attached File- Added 0001-Add-g-opt-for-graph-output-of-modified-targets.patch, #50069
    2020-10-22 jonasdn Attached File- Added 0001-Add-g-opt-for-graph-output-of-modified-targets.patch, #50048
    2020-10-22 jonasdn Attached File- Added 0001-Add-g-opt-for-graph-output-of-modified-targets.patch, #50045
        Attached File- Added all.png, #50046
        Attached File- Added all-am.png, #50047
        Carbon-Copy- Added jonasdn

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code