bugmake - Bugs: bug #45244, memory consumption increases with...

 
 

bug #45244: memory consumption increases with runtime

Submitted by:  None
Submitted on:  Tue 02 Jun 2015 09:37:32 PM UTC  
Votes:  20  
 
Severity:  3 - Normal Item Group:  Bug
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.1 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 02 Jun 2015 09:37:32 PM UTC, original submission:

While executing, make allocates memory to hold the automatic variables for each target. This memory is not free'd after the target is built, so it just accumulates until exit.

In my use case, the variables $+, $^, and $? are responsible for most of the memory allocation because they list all of the target's prerequisites. My use case has the triple-whammy of long prerequisite names and many prerequisites on my many targets.

To reproduce the issue, please find the script "makefile_generator.rb" attached. This creates an example makefile to demonstrate the problem.

Commands:
makefile_generator.rb 250 300 .1 (creates 300 targets with names 250 characters long, each does "sleep .1")
valgrind --tool=massif make all
ms_print massif.out.*

I've also attached massif_report.txt generated using these steps. (My report comes from make 4.0, but I've observed the problem in 4.1 as well.)

The graph obviously shows the memory consumption increasing throughout the run.

In the final sample, you can see that 90.35% of the memory was allocated via xstrdup. Most of these allocations come from 3 places:
| | | ->29.95% (11,181,817B) 0x405A6C: set_file_variables (commands.c:231)
| | | ->29.95% (11,181,817B) 0x405C84: set_file_variables (commands.c:315)
| | | ->29.95% (11,181,817B) 0x405CC9: set_file_variables (commands.c:318)

These correspond to the places where automatic variables are created:
$+ here: http://git.savannah.gnu.org/cgit/make.git/tree/commands.c?id=4.0#n231
$^ here: http://git.savannah.gnu.org/cgit/make.git/tree/commands.c?id=4.0#n315
$? here: http://git.savannah.gnu.org/cgit/make.git/tree/commands.c?id=4.0#n318

I couldn't figure out exactly where it would be safe to free this memory, but this modification seems to work:

--- a/remake.c
+++ b/remake.c
@@ -876,6 +876,13 @@ notice_finished_file (struct file *file)
file->command_state = cs_finished;
file->updated = 1;

+ /* I'm assuming that since this file is finished, we won't need
+ * its variables anymore...*/
+ if (file->variables)
+ free_variable_set (file->variables);
+ if (file->pat_variables)
+ free_variable_set (file->pat_variables);
+
if (touch_flag
/* The update status will be:
us_success if 0 or more commands (+ or ${MAKE}) were run and won;

I did some quick-and-dirty benchmarking, and this change doesn't seem to cost much extra CPU time, either.

Thanks,
mblythe

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #34149:  makefile_generator.rb added by None (374B - application/octet-stream)
file #34150:  massif_report.txt added by None (157KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mblythe
  • -email is unavailable- added by mblythe
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 20 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    Tue 09 Jun 2015 04:56:14 PM UTC mblythe Carbon-Copy- => Added mblythe
        Carbon-Copy- => Added -email is unavailable-
    Tue 02 Jun 2015 09:37:32 PM UTC None Attached File- => Added makefile_generator.rb, #34149
        Attached File- => Added massif_report.txt, #34150

    Back to the top


    Powered by Savane 3.3