bugmake - Bugs: bug #58734, gmake does not check for the...

 
 

bug #58734: gmake does not check for the existence of a file before complaining it is missing

Submitted by:  Jörg Schilling <schily>
Submitted on:  Wed 08 Jul 2020 02:59:25 PM UTC  
 
Severity:  3 - Normal Item Group:  None
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  None Operating System:  None
Fixed Release:  None Triage Status:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Wed 22 Jul 2020 06:09:07 AM UTC, comment #8: 

Sorry for the typo, the command in step 8 of course should be:

  find . -type l -exec rm {} +

Jörg Schilling <schily>
Tue 21 Jul 2020 02:32:20 PM UTC, comment #7: 

You are mistaken. My makefile describes enough of the files to allow any non-buggy make implementation to work correctly.

If gmake does not check the state of a file before it claims that this file is missing, then gmake is broken.

Steps to reproduce:

1) download the current state of schilytools from: https://sourceforge.net/projects/schilytools/files/

 The currently up to date file is: https://sourceforge.net/projects/schilytools/files/schily-2020-07-18.tar.bz2

2) unpack the archive and chdir to "schily-2020-07-18"

3) call "make" and wait for the compilation to finish

4) call "make install" to get smake and dmake installed in /opt/schily/bin

5) add /opt/schily/bin to your PATH

6) chdir to libshedit

7) install the attachec gmake-fail.mk

8) remove symlinks and OBJ by calling:

   find . -type d-exec rm {} +

9) call "smake -f gmake\-fail.mk"

and see:

smake -f gmake-fail.mk
        ==> MAKING SYMLINKS in .
...

10) repeat step 8)

11) call "dmake -f gmake-fail.mk"

and see:

dmake -f gmake\-fail.mk  
        ==> MAKING SYMLINKS in .
...

12) repeat setp 8)

13) call gmake -f gmake-fail.mk GMAKE_NOWARN=true

        ==> MAKING SYMLINKS in .
gmake: *** Keine Regel vorhanden, um das Ziel »inputc.c«,
  benötigt von »OBJ/x86_64-linux-gcc/inputc.o«, zu erstellen.  Schluss.

So gmake is the only make implementation that complains here.

(file #49528)

Jörg Schilling <schily>
Fri 10 Jul 2020 03:47:54 PM UTC, comment #6: 

> I forgot to mention, that this happens in parallel and in serial mode. So your assumption is wrong.


I said, running the command twice can only happen with parallel builds enabled.  I stand behind that statement as correct.  I was careful to make a distinction between that and the problem of files not existing.

> I am also not going to add unneeded hidden files when there is already a file that is created as a result of the command that has been run.


I explained this but just realized that message went to the mailing list, not to the bug:

If you want parallel builds to work correctly you must "introduce unneeded files".  This is a deficiency in the POSIX standard (it does not provide any way to explain to make that a single invocation of a recipe generates multiple output files) and if you want to write portable makefiles that will also work properly with parallelism enabled then there's no other way to solve that problem.

As a result of introducing the hidden files to fix the parallel builds issue, that will also fix the caching problem.

In short, if you ensure that your makefile works properly with parallel builds then you won't have a caching problem.  This is why the caching problem is not a huge issue for every user: virtually every GNU make user these days enables parallel mode, and they want their makefile to work properly with it.

> It would be so easy just to re-check the time stamp of every cached file that is believed to be out of date or missing.


The only purpose of the cache is reducing the number of stat() calls make has to do.  Obviously once make determines that a target is already up to date, it doesn't have to check it again.  If you say that for all other files make should always re-check, then there's no reason to cache.

If you're saying something different, such as as long as make has not run a command that might possibly change some timestamp that it can cache the mod time ... well ... that's pretty much exactly what I said in my list of possible enhancements (again this went to the mailing list unfortunately).

> Time stamp checking is also faster than re-running a command for a related rule.


Of course!  But it's not faster than not checking the timestamp in the first place, which is what the directory cache is for.  If your makefile describes the target and prerequisite relationships completely, without omitting any relevant information, then the directory cache provides performance improvements.  If you don't, then it can do the wrong thing.

As I've written already, it's clear from experience that some makefiles don't provide sufficient information and the cache should be modified to address that.

Paul D. Smith <psmith>
Project Administrator
Fri 10 Jul 2020 09:23:42 AM UTC, comment #5: 

I forgot to mention, that this happens in parallel and in serial mode. So your assumption is wrong.

Caching is not a bad thing if it helps to speed up things. I did not make related tests and I am not going to doubt here because this would put effort on the wrong target.

I am also not going to add unneeded hidden files when there is already a file that is created as a result of the command that has been run.

The main problem for this bug is that gmake does not implement the cache the right way.

If you like to use caching the right way, you may do so whenever this caching contains information that leads to the assumption that a target is up to date.

The gmake command failed not because of caching in general but because of the fact that gmake does not correct invalid cache content.

Please keep in mind that the current implementation of gmake causes gmake to stop working with a wrong error message. If you run gmake again, it works further on, up to the next similar problem while other make implementations just continue to work with the first call.

This behavior of gmake is annoying and slows down the compilation.

It would be so easy just to re-check the time stamp of every cached file that is believed to be out of date or missing. Time stamp checking is also faster than re-running a command for a related rule.

A time stamp in "make" may have three values (the last one is a range):

- File dos not exist. This may become invalid if cached.

- Younger than any file. This will never become invalid for the lifetime of make.

- A specific time stamp. This may become invalid and rechecking is faster than running a command again. In special as only the time stamp of a single file needs to be checked in such a case.

Just a note: I've seen commands that have been run twice as a result of not verifying the third case in gmake.

BTW: some of the problems I have with gmake would go away if there was not an unlucky combination of contradictory behavior.

Jörg Schilling <schily>
Thu 09 Jul 2020 02:18:36 PM UTC, comment #4: 

This is one of the gmake bugs that does not seem to exist with trivial makefiles. This is why I am apparently unable to give a small example to repeat that problem. If you really need a makefile to repeat the problem, I can give you a modified makefile that would cause the problem in a larger project that yu would need to fetch for the test (this is the schilytools project).

What I can say to help fixing the problem is that truss(1) shows that gmake calls stat(2) as a block on a larger list of files at startup and at that time correctly finds that the file in question does not exist.

Then it executes some rules that have the side effect to create the file in question (a symlink to a source file).

Gmake however never again calls stat(2) on that file but incorrectly claims that the file does not exist, eventhough it exists at the time it is really needed.

Please note that the fact that there is no explicit rule for that  file, is caused by a workaround for another gmake bug that would cause a specific command to be called 4x concurrently, resulting in overwritten results.

BTW: I write portable software, so I cannot use a non portable feature (e.g. your &: proposal) within a leaf makefile.

Jörg Schilling <schily>
Thu 09 Jul 2020 04:15:21 AM UTC, comment #3: 

> directory caching


... which I was convinced was flawed in:

https://lists.gnu.org/archive/html/bug-make/2016-10/msg00019.html

I bet that's not the problem here, but, until we see the example Dmitry requested...

Anonymous
Wed 08 Jul 2020 05:03:36 PM UTC, comment #2: 

> either the program is run twice


This can only happen if parallel mode is enabled, and is the correct behavior assuming you've written your rule like this:

one two: three ; touch one two

This means the same thing as:

one: three ; touch one two
two: three ; touch one two

(this interpretation is required by the POSIX standard).  If parallelism is enabled make may see that two is out of date before one finishes updating it, and so it will run the rule twice.

In serial mode this should never happen (I've never seen it happen): if you see it please give an example.

> or a depending rule that needs the created file incorrectly complains that the file does not exist.


I've never seen this happen either: please provide an example.  The only possible reason this could happen would be due to directory caching but I don't think so.

If you want to declare a rule that builds multiple outputs, and you have GNU make 4.3 or above, you can use the new "&:" separator to tell make that one invocation will build both targets:

one two &: three ; touch one two

Paul D. Smith <psmith>
Project Administrator
Wed 08 Jul 2020 04:13:32 PM UTC, comment #1: 

Can you please attach a makefile that demonstrates the issue?

Dmitry Goncharov <dgoncharov>
Wed 08 Jul 2020 02:59:25 PM UTC, original submission:  

A file that has been created by a program that has more than one output file is not handled correctly by gmake.

Depending on how the rules are written, either the program is run twice or a depending rule that needs the created file incorrectly complains that the file does not exist.

gmake should check for the time stamp of a file whenever it actually needs the file and believes it does not exist or is outdated.

Jörg Schilling <schily>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49528:  gmake-fail.mk added by schily (2KiB - text/x-makefile)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by dgoncharov (Posted a comment)
  • -email is unavailable- added by schily (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-07-21 schily Attached File- => Added gmake-fail.mk, #49528

    Back to the top


    Powered by Savane 3.5