bugmake - Bugs: bug #63157, Unlink temporary files.

 
 

bug #63157: Unlink temporary files.

Submitter:  Dmitry Goncharov <dgoncharov>
Submitted:  Wed 05 Oct 2022 02:24:38 AM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  4.4 Operating System:  Any
Fixed Release:  4.4 Triage Status:  Medium Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 20 Oct 2022 10:42:05 PM UTC, comment #26: 

Ach, it did reproduce for me, with or without Dmitry's modification.  I was just looking in the wrong place, despite Frank providing copy-and-paste instructions on where to look.  Perhaps I'd just read about MAKE_TMPDIR and assumed that they'd end up in the nice clean directory I'd made for them, as they do with:

 MAKE_TMPDIR=$(pwd) make -j2 -O 2>&1 | :

Sorry for the mislead.

Martin Dorey <mdorey>
Thu 20 Oct 2022 09:56:39 PM UTC, comment #25: 

This may not always reproduce, because there is a race between make writing its output to the pipe and the reader exiting.
Sigpipe is sent when the reader exits before make is able to write its output.
However, if you change it like this

echo 'all:; sleep 2' | make -f- -j2 -O |:

it'll reproduce.

Thanks for you report, Frank.
i opened https://savannah.gnu.org/bugs/index.php?63248 and attached a patch.

Dmitry Goncharov <dgoncharov>
Thu 20 Oct 2022 03:15:25 AM UTC, comment #24: 


comment #23:

> Frank wrote [
>
> I now found a way to reproduce it quite easily on my system.
> As I suspected, it seems to be SIGPIPE:


> % rm -f /tmp/G*
> % cat Makefile
> all: ; sleep 1
> % make -j2 -O 2>&1 | :
> % ls -l /tmp/G*
> prw------- 1 frank frank 0 19. Okt 22:42 /tmp/GMfifo370973
> -rw------- 1 frank frank 0 19. Okt 22:42 /tmp/GmFkJIuq
> %


>
> ].  That looked like just the ticket but sadly it doesn't reproduce it for me, on amd64 Debian Stretch.


I just reproduced it on a Linux 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux machine. So it doesn't seem to be the version.

Anyway, strace confirms that make is killed by SIGPIPE, so if you catch it (or ignore it and then handle or ignore stdout/stderr write errors), it should work.

Frank Heckenbach <frank>
Wed 19 Oct 2022 11:39:52 PM UTC, comment #23: 

Frank wrote [

I now found a way to reproduce it quite easily on my system.
As I suspected, it seems to be SIGPIPE:

% rm -f /tmp/G*
% cat Makefile
all: ; sleep 1
% make -j2 -O 2>&1 | :
% ls -l /tmp/G*
prw------- 1 frank frank 0 19. Okt 22:42 /tmp/GMfifo370973
-rw------- 1 frank frank 0 19. Okt 22:42 /tmp/GmFkJIuq
%


].  That looked like just the ticket but sadly it doesn't reproduce it for me, on amd64 Debian Stretch.

Martin Dorey <mdorey>
Wed 19 Oct 2022 12:50:08 PM UTC, comment #22: 

Frank, can you please tell us how to reproduce?

Dmitry Goncharov <dgoncharov>
Wed 19 Oct 2022 12:27:12 PM UTC, comment #21: 

The only places leftover files may happen is (a) on Windows, only when (b) you kill GNU make with ^C or similar.

Any other leftover files should not happen.  Please provide details of the names of the files that are left over and which circumstances you see them in.  We do have tests that verify that temp files are removed so maybe those tests are not complete.

Paul D. Smith <psmith>
Group administrator
Wed 19 Oct 2022 05:39:40 AM UTC, comment #20: 

Unfortunately, I still get leftover temp files and fifos with 4.3.91.

If I understand you right, it's not worth investigating it further now, is it?

Just one guess: Since I pipe make's output to my tools, that might be the cause. So you might want to make SIGPIPE a FATAL_SIG too, or ignore SIGPIPE and make sure write errors to stdout and stderr are handled.

Frank Heckenbach <frank>
Sat 15 Oct 2022 02:42:30 PM UTC, comment #19: 

It's probably not worth spending a lot of time ruminating on these things.  The reality is that currently make does many non-handler-safe things in its signal handler, and there are Savannah bugs about hangs etc. while catching SIGINT which prove it if you don't want to check the code.

The goal of my rewrite (slated for the next release) is to rework the signal handling so that the handler does nothing but set a variable (volatile sig_atomic_t) and then all the actual handling of the signal will be done in "normal code".

As I'm sure you're aware this means a lot of careful changes to avoid race conditions etc. but once complete these issues will not be relevant.

Paul D. Smith <psmith>
Group administrator
Sat 15 Oct 2022 07:44:22 AM UTC, comment #18: 


comment #17:

> Paul, if you decide to proceed with changes in comment 11, temp_stdin_fileno should probably be of type volatile sig_atomic_t, rather than int.


Are you sure that's big enough? According to https://cplusplus.com/reference/cstdint/ it might not be.

Frank Heckenbach <frank>
Fri 14 Oct 2022 06:18:45 PM UTC, comment #17: 

i think, it is fine (maybe even good) if the changes in comment 11 are abandoned. The whole critical section only lasts while make is reading and parsing the temp stdin file. The duration of this critical section is likely to be a fraction of compilation time of even one source file.
Paul, if you decide to proceed with changes in comment 11, temp_stdin_fileno should probably be of type volatile sig_atomic_t, rather than int.

Dmitry Goncharov <dgoncharov>
Wed 12 Oct 2022 07:31:05 PM UTC, comment #16: 

Sorry I should have been more clear.  When I said "non-regression changes" I was specifically referring to the extra effort needed to get files to be deleted on Windows, as in comment #11

That change is related to the temporary stdin file, not to the jobserver, and treatment of that hasn't changed since 4.3 on either Windows or POSIX, I don't think.  The already-existing behavior is sufficient to remove these files on POSIX.  The extra changes are only needed to remove them on Windows.

I do have Dmitry's patch already mostly applied, but needed some changes.

Paul D. Smith <psmith>
Group administrator
Wed 12 Oct 2022 07:11:28 PM UTC, comment #15: 

FWIW, I do consider this a regression, since these things weren't even created before.

Now, after some weeks of trying make 4.3.90, I get a number of empty files and named pipes lying around -- probably from aborted make runs, but such is life (sometimes you realise a mistake while compiling and abort it).

I hope this can be fixed in the next release, otherwise I'll have to go back to 4.3, I'm afraid.

Frank Heckenbach <frank>
Thu 06 Oct 2022 03:11:11 PM UTC, comment #14: 

Thanks, I will look at this within the next few days.

Note, I'm not too concerned with non-regression changes for signal handling: the fact is that our current signal handling is totally broken (see the 5-6 issues related to things not dying properly on SIGINT for example) and fixing this was the last thing I was hoping to get done for this release but had to leave it only mostly-complete as it was taking too long.  I will revisit this after the release is done.

The goal is to modify our signal handling so it does nothing but set a notification that the signal was received, then all handling of the signal will happen in the main code not in the handler so that these types of issues can be avoided.

The problem of course is that doing this without race conditions is the classic tricky problem in POSIX programming so some things need to be reworked.

Paul D. Smith <psmith>
Group administrator
Thu 06 Oct 2022 12:08:39 AM UTC, comment #13: 

Thank you, Eli.

Dmitry Goncharov <dgoncharov>
Wed 05 Oct 2022 06:20:17 PM UTC, comment #12: 

Thanks, the diffs look good (although I didn't actually test them).

Eli Zaretskii <eliz>
Group Member
Wed 05 Oct 2022 06:07:36 PM UTC, comment #11: 


> If temp_stdin is fclosed, then there's no need to call close on its fileno.


Right. We have to take care of the critical section between fopen and fclose.

The reasoning is the following.
1. The patch works on unices.
2. On windows the patch works if the file is closed.
3. eval_makefile uses fopen and then fclose to open and close the file.
4. On windowns the file is not deleted, if a signal arrives after the file is opened and before the file is closed by eval_makefile.

One option to handle this critical section on windows is to set the fd of the file to a file scope variable and then close this fd in temp_stdin_unlink.
Something like this (Not tested on windows).



diff --git a/src/main.c b/src/main.c
index 129dd661..cf87a6a0 100644
--- a/src/main.c
+++ b/src/main.c
@@ -306,6 +306,7 @@ unsigned long command_count = 1;
 /* Remember the location of the name of the batch file from stdin.  */

 static int stdin_offset = -1;
+static int temp_stdin_fileno = -1;

 ^L
 /* The usage output.  We write it this way to make life easier for the
@@ -3714,12 +3715,33 @@ clean_jobserver (int status)
     }
 }

+void
+temp_stdin_set_fileno (const char *name, FILE *f)
+{
+  if (temp_stdin_fileno == -1 && f &&
+      stdin_offset >= 0 && streq (makefiles->list[stdin_offset], name))
+    temp_stdin_fileno = fileno (f);
+}
+
+void
+temp_stdin_reset_fileno (const char *name)
+{
+  if (temp_stdin_fileno >= 0 &&
+      stdin_offset >= 0 && streq (makefiles->list[stdin_offset], name))
+    temp_stdin_fileno = -1;
+}
+
 void
 temp_stdin_unlink ()
 {
   /* This function is called from a signal handler.
    * Keep async-signal-safe.  */
   /* Get rid of a temp file from reading a makefile from stdin.  */
+#ifdef WINDOWS32
+  /* Windows needs the file to be closed for unlink to succeed.  */
+  if (temp_stdin_fileno >= 0 && close (temp_stdin_fileno) == 0)
+    temp_stdin_fileno = -1;
+#endif
   if (stdin_offset >= 0 && unlink (makefiles->list[stdin_offset]) == 0)
     stdin_offset = -1;
 }
diff --git a/src/makeint.h b/src/makeint.h
index d0e0fab3..cf2975c6 100644
--- a/src/makeint.h
+++ b/src/makeint.h
@@ -530,6 +530,8 @@ void out_of_memory () NORETURN;
                                  (_f), (_n), (_s))

 void decode_env_switches (const char*, size_t line);
+void temp_stdin_set_fileno (const char *name, FILE *f);
+void temp_stdin_reset_fileno (const char *name);
 void temp_stdin_unlink ();
 void die (int) NORETURN;
 void pfatal_with_name (const char *) NORETURN;
diff --git a/src/read.c b/src/read.c
index 79315503..b8e7baed 100644
--- a/src/read.c
+++ b/src/read.c
@@ -353,6 +353,7 @@ eval_makefile (const char *filename, unsigned short flags)
   errno = 0;
   ENULLLOOP (ebuf.fp, fopen (filename, "r"));
   deps->error = errno;
+  temp_stdin_set_fileno (filename, ebuf.fp);

   /* Check for unrecoverable errors: out of mem or FILE slots.  */
   switch (deps->error)
@@ -441,6 +442,7 @@ eval_makefile (const char *filename, unsigned short flags)
   reading_file = curfile;

   fclose (ebuf.fp);
+  temp_stdin_reset_fileno (filename);

   free (ebuf.bufstart);
   free_alloca ();


There is still a smaller critical section. If a signal arrives between fopen and temp_stdin_set_fileno, the file is not deleted. Signals can be blocked during this smaller critical section. But i think, that's an overkill.

> What do you mean by "We won't be able to use fclose, which is okay, since make is exiting."?


Inside temp_stdin_unlink make can only do async signal safe calls. This rules out fclose. Make can call close, because close is async signal safe. Calling close, rather than fclose, leaves the internal FILE bookeeping messed up, which is okay, because make is exiting.

Dmitry Goncharov <dgoncharov>
Wed 05 Oct 2022 04:39:39 PM UTC, comment #10: 

If temp_stdin is fclosed, then there's no need to call close on its fileno.

I don't think I follow your reasoning.  Could you tell more details, including pointers to the source code?  What do you mean by "We won't be able to use fclose, which is okay, since make is exiting."?

In general, yes, a file must be closed before we try to unlink it, otherwise the deletion will fail and the file will not be deleted.

Eli Zaretskii <eliz>
Group Member
Wed 05 Oct 2022 02:41:51 PM UTC, comment #9: 

In the case of temp_stdin eval_makefile fcloses this file.
We'll need to store temp_stdin_fileno in a file scope variable (next to stdin_offset) and pass it to close in temp_stdin_unlink.
We won't be able to use fclose, which is okay, since make is exiting.

Dmitry Goncharov <dgoncharov>
Wed 05 Oct 2022 02:24:21 PM UTC, comment #8: 

we can call close before unlink

Dmitry Goncharov <dgoncharov>
Wed 05 Oct 2022 02:23:08 PM UTC, comment #7: 

close is async signal safe and we call close before unlink.
Should make also calls close before unlink in all other places where make unlinks this (or any other) file?

Dmitry Goncharov <dgoncharov>
Wed 05 Oct 2022 02:14:13 PM UTC, comment #6: 


> If the file is opened at the time of unlink, only file's name is removed from the directory and the file itself stays. When make exits the last reference to the file is gone and the file is deleted.


That's what happens on Unix.  On Windows, the unlink call will simply fail (with EACCES or somesuch), and the file will remain.

Eli Zaretskii <eliz>
Group Member
Wed 05 Oct 2022 02:06:53 PM UTC, comment #5: 


> This should be tested on MS-Windows.  A file can only be deleted on MS-Windows if it isn't open by any program, and the patch (AFAICT) doesn't make sure the files are closed before unlinking them.  I'm especially worried about the temp_stdin_unlink part.


If the file is opened at the time of unlink, only file's name is removed from the directory and the file itself stays. When make exits the last reference to the file is gone and the file is deleted.


> (Btw, why does the patch use `unlink` instead of a more standard `remove`?)


unlink is used, rather than remove, to stay uniform with the rest of the code, which uses unlink.

Dmitry Goncharov <dgoncharov>
Wed 05 Oct 2022 06:07:39 AM UTC, comment #4: 

This should be tested on MS-Windows.  A file can only be deleted on MS-Windows if it isn't open by any program, and the patch (AFAICT) doesn't make sure the files are closed before unlinking them.  I'm especially worried about the temp_stdin_unlink part.

(Btw, why does the patch use `unlink` instead of a more standard `remove`?)

Eli Zaretskii <eliz>
Group Member
Wed 05 Oct 2022 03:05:40 AM UTC, comment #3: 

Also, tested presence of PPID variable in ksh, bash, ash, zsh, sh.
The only shell that i encountered that lacks PPID is tcsh (likely csh as well).
The test uses the default shell. This means that test will likely fail on a system where /bin/sh is csh or tcsh.

Dmitry Goncharov <dgoncharov>
Wed 05 Oct 2022 02:39:04 AM UTC, comment #2: 

Tested this patch on linux, sun and aix. All 64 bit.
This patch adds jobserver_unlink stub on windows. As far as i can visually validate, this stub should compile fine. However, have not tested this on windows.

Dmitry Goncharov <dgoncharov>
Wed 05 Oct 2022 02:25:51 AM UTC, comment #1: 

Make fails to unlink temporary files upon arrival of a fatal signal. Specifically, the jobserver fifo, output sync lock and the stdin temp file.

Dmitry Goncharov <dgoncharov>
Wed 05 Oct 2022 02:24:38 AM UTC, original submission:  

.

Dmitry Goncharov <dgoncharov>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #53792:  sv63157_fix.diff added by dgoncharov (4KiB - text/x-patch)
file #53793:  sv63157_test.diff added by dgoncharov (3KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mdorey (Posted a comment)
  • -email is unavailable- added by frank (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by eliz (Posted a comment)
  • -email is unavailable- added by dgoncharov (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-10-18 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.4
        Triage StatusNone Medium Effort
    2022-10-05 dgoncharov Attached File- Added sv63157_fix.diff, #53792
        Attached File- Added sv63157_test.diff, #53793

    Back to the top

    Powered by Savane 3.13-da57.
    Corresponding source code