bugmake - Bugs: bug #58734, gmake does not check for the...

 
 

bug #58734: gmake does not check for the existence of a file before complaining it is missing

Submitter:  Jörg Schilling <schily>
Submitted:  Wed 08 Jul 2020 02:59:25 PM UTC
   
 
Severity:  3 - Normal Item Group:  None
Status:  Duplicate Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  None Operating System:  None
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 30 Nov 2020 02:08:31 PM UTC, comment #11: 

I understand the lack of time.

As I mentioned I've implemented and pushed a fix for the directory caching issue.  When I follow the directions here as best as I could, the build succeeded and didn't throw any warnings or errors with this change installed.

So, I believe this issue to be resolved.

Paul D. Smith <psmith>
Group administrator
Mon 30 Nov 2020 12:30:52 PM UTC, comment #10: 

I am sorry, but there are problems in gmake that only show
up with large or complex makefiles. I cannot give simpler
examples as I cannot spend an infinite amount of time in trying
to find a smaller example.

I however already explained to you how you could make gmake
compatible to make in this area. Since even the classic make
did cache file time stamps, it is obvious that things can
be done in a compatible way.

- You may cache whatever you like as long as you do not
  break things that are part of the basic functionality of
  make since 1976. So do just not make the wrong conclusions
  from cached time stamps.

- If a cached time stamp results in the assumption that
  the related file is up to date, this is OK. This may be
  wrong if the file was removed meanwhile, but I believe
  this is not a problem.

- If a file was the target of a rule command that did exit()
  with code zero, assume "younger than any file" without
  checking the file time stamp again.

- If the cached time results in the assumption that the
  file is out of date or missing, check the time stamp
  again. This is what seems to be missing in gmake.

Just a note: assuming a file is out of date and rerunning the
rule command is more expensive than calling stat() for the file.
The cases where you need to rerun stat() are rare and thus do
not affect the overall performance.

Claiming that the file does not exist (this is what happens
with gmake) is incompatible to the behavior of make. If you
are familiar with the source code of gmake, it should be
obvious how to fix it. Just call stat() if you believe that
a file does not exist.

Jörg Schilling <schily>
Sun 29 Nov 2020 11:13:52 PM UTC, comment #9: 

The reproduction case here is too complicated for me to have the time to really understand.  If you wanted to provide a simpler repro case I'd be happy to look at it.  Also, it's helpful if you use LC_ALL=C when running make so that error messages can be looked up in the source code.

However I ran this repro case using the latest version which has a fix for bug #41273 and it appeared to work correctly, so I'm closing this as a duplicate.

Thanks for reporting!

Paul D. Smith <psmith>
Group administrator
Wed 22 Jul 2020 06:09:07 AM UTC, comment #8: 

Sorry for the typo, the command in step 8 of course should be:

  find . -type l -exec rm {} +

Jörg Schilling <schily>
Tue 21 Jul 2020 02:32:20 PM UTC, comment #7: 

You are mistaken. My makefile describes enough of the files to allow any non-buggy make implementation to work correctly.

If gmake does not check the state of a file before it claims that this file is missing, then gmake is broken.

Steps to reproduce:

1) download the current state of schilytools from: https://sourceforge.net/projects/schilytools/files/

 The currently up to date file is: https://sourceforge.net/projects/schilytools/files/schily-2020-07-18.tar.bz2

2) unpack the archive and chdir to "schily-2020-07-18"

3) call "make" and wait for the compilation to finish

4) call "make install" to get smake and dmake installed in /opt/schily/bin

5) add /opt/schily/bin to your PATH

6) chdir to libshedit

7) install the attachec gmake-fail.mk

8) remove symlinks and OBJ by calling:

   find . -type d-exec rm {} +

9) call "smake -f gmake\-fail.mk"

and see:

smake -f gmake-fail.mk
        ==> MAKING SYMLINKS in .
...

10) repeat step 8)

11) call "dmake -f gmake-fail.mk"

and see:

dmake -f gmake\-fail.mk  
        ==> MAKING SYMLINKS in .
...

12) repeat setp 8)

13) call gmake -f gmake-fail.mk GMAKE_NOWARN=true

        ==> MAKING SYMLINKS in .
gmake: * Keine Regel vorhanden, um das Ziel »inputc.c«,
  benötigt von »OBJ/x86_64-linux-gcc/inputc.o«, zu erstellen.  Schluss.

So gmake is the only make implementation that complains here.

(file #49528)

Jörg Schilling <schily>
Fri 10 Jul 2020 03:47:54 PM UTC, comment #6: 


> I forgot to mention, that this happens in parallel and in serial mode. So your assumption is wrong.


I said, running the command twice can only happen with parallel builds enabled.  I stand behind that statement as correct.  I was careful to make a distinction between that and the problem of files not existing.

> I am also not going to add unneeded hidden files when there is already a file that is created as a result of the command that has been run.


I explained this but just realized that message went to the mailing list, not to the bug:

If you want parallel builds to work correctly you must "introduce unneeded files".  This is a deficiency in the POSIX standard (it does not provide any way to explain to make that a single invocation of a recipe generates multiple output files) and if you want to write portable makefiles that will also work properly with parallelism enabled then there's no other way to solve that problem.

As a result of introducing the hidden files to fix the parallel builds issue, that will also fix the caching problem.

In short, if you ensure that your makefile works properly with parallel builds then you won't have a caching problem.  This is why the caching problem is not a huge issue for every user: virtually every GNU make user these days enables parallel mode, and they want their makefile to work properly with it.

> It would be so easy just to re-check the time stamp of every cached file that is believed to be out of date or missing.


The only purpose of the cache is reducing the number of stat() calls make has to do.  Obviously once make determines that a target is already up to date, it doesn't have to check it again.  If you say that for all other files make should always re-check, then there's no reason to cache.

If you're saying something different, such as as long as make has not run a command that might possibly change some timestamp that it can cache the mod time ... well ... that's pretty much exactly what I said in my list of possible enhancements (again this went to the mailing list unfortunately).

> Time stamp checking is also faster than re-running a command for a related rule.


Of course!  But it's not faster than not checking the timestamp in the first place, which is what the directory cache is for.  If your makefile describes the target and prerequisite relationships completely, without omitting any relevant information, then the directory cache provides performance improvements.  If you don't, then it can do the wrong thing.

As I've written already, it's clear from experience that some makefiles don't provide sufficient information and the cache should be modified to address that.

Paul D. Smith <psmith>
Group administrator
Fri 10 Jul 2020 09:23:42 AM UTC, comment #5: 

I forgot to mention, that this happens in parallel and in serial mode. So your assumption is wrong.

Caching is not a bad thing if it helps to speed up things. I did not make related tests and I am not going to doubt here because this would put effort on the wrong target.

I am also not going to add unneeded hidden files when there is already a file that is created as a result of the command that has been run.

The main problem for this bug is that gmake does not implement the cache the right way.

If you like to use caching the right way, you may do so whenever this caching contains information that leads to the assumption that a target is up to date.

The gmake command failed not because of caching in general but because of the fact that gmake does not correct invalid cache content.

Please keep in mind that the current implementation of gmake causes gmake to stop working with a wrong error message. If you run gmake again, it works further on, up to the next similar problem while other make implementations just continue to work with the first call.

This behavior of gmake is annoying and slows down the compilation.

It would be so easy just to re-check the time stamp of every cached file that is believed to be out of date or missing. Time stamp checking is also faster than re-running a command for a related rule.

A time stamp in "make" may have three values (the last one is a range):

- File dos not exist. This may become invalid if cached.

- Younger than any file. This will never become invalid for the lifetime of make.

- A specific time stamp. This may become invalid and rechecking is faster than running a command again. In special as only the time stamp of a single file needs to be checked in such a case.

Just a note: I've seen commands that have been run twice as a result of not verifying the third case in gmake.

BTW: some of the problems I have with gmake would go away if there was not an unlucky combination of contradictory behavior.

Jörg Schilling <schily>
Thu 09 Jul 2020 02:18:36 PM UTC, comment #4: 

This is one of the gmake bugs that does not seem to exist with trivial makefiles. This is why I am apparently unable to give a small example to repeat that problem. If you really need a makefile to repeat the problem, I can give you a modified makefile that would cause the problem in a larger project that yu would need to fetch for the test (this is the schilytools project).

What I can say to help fixing the problem is that truss(1) shows that gmake calls stat(2) as a block on a larger list of files at startup and at that time correctly finds that the file in question does not exist.

Then it executes some rules that have the side effect to create the file in question (a symlink to a source file).

Gmake however never again calls stat(2) on that file but incorrectly claims that the file does not exist, eventhough it exists at the time it is really needed.

Please note that the fact that there is no explicit rule for that  file, is caused by a workaround for another gmake bug that would cause a specific command to be called 4x concurrently, resulting in overwritten results.

BTW: I write portable software, so I cannot use a non portable feature (e.g. your &: proposal) within a leaf makefile.

Jörg Schilling <schily>
Thu 09 Jul 2020 04:15:21 AM UTC, comment #3: 


> directory caching


... which I was convinced was flawed in:

https://lists.gnu.org/archive/html/bug-make/2016-10/msg00019.html

I bet that's not the problem here, but, until we see the example Dmitry requested...

Anonymous
Wed 08 Jul 2020 05:03:36 PM UTC, comment #2: 


> either the program is run twice


This can only happen if parallel mode is enabled, and is the correct behavior assuming you've written your rule like this:


one two: three ; touch one two


This means the same thing as:


one: three ; touch one two
two: three ; touch one two


(this interpretation is required by the POSIX standard).  If parallelism is enabled make may see that two is out of date before one finishes updating it, and so it will run the rule twice.

In serial mode this should never happen (I've never seen it happen): if you see it please give an example.

> or a depending rule that needs the created file incorrectly complains that the file does not exist.


I've never seen this happen either: please provide an example.  The only possible reason this could happen would be due to directory caching but I don't think so.

If you want to declare a rule that builds multiple outputs, and you have GNU make 4.3 or above, you can use the new "&:" separator to tell make that one invocation will build both targets:


one two &: three ; touch one two


Paul D. Smith <psmith>
Group administrator
Wed 08 Jul 2020 04:13:32 PM UTC, comment #1: 

Can you please attach a makefile that demonstrates the issue?

Dmitry Goncharov <dgoncharov>
Wed 08 Jul 2020 02:59:25 PM UTC, original submission:  

A file that has been created by a program that has more than one output file is not handled correctly by gmake.

Depending on how the rules are written, either the program is run twice or a depending rule that needs the created file incorrectly complains that the file does not exist.

gmake should check for the time stamp of a file whenever it actually needs the file and believes it does not exist or is outdated.

Jörg Schilling <schily>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49528:  gmake-fail.mk added by schily (2KiB - text/x-makefile)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by dgoncharov (Posted a comment)
  • -email is unavailable- added by schily (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-11-29 psmith StatusNone Duplicate
        Assigned toNone psmith
        Open/ClosedOpen Closed
    2020-07-21 schily Attached File- Added gmake-fail.mk, #49528

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code