Thanks, Paul. This indeed fixes the infinite recursion on reiserfs.

The same effect can be seen on a filesystem which supports O_TMPFILE, if TMPDIR is set to a non existing directory.

The current make enters infinite recursion on a filesystem that does not support O_TMPFILE.

$ cat makefile
all:; $(info hello, world)
$ # current make from master
$ ~/src/gmake/make/m64/make -Orecurse
Segmentation fault (core dumped)

This is the relevant part of the calls stack (many lines omitted)

...
    #240 0x55df2f9322ca in setup_tmpfile ../src/output.c:212
    #241 0x55df2f93356f in output_start ../src/output.c:348
    #242 0x55df2f933695 in outputs ../src/output.c:364
    #243 0x55df2f934d01 in fatal ../src/output.c:481
    #244 0x55df2f934f2e in pfatal_with_name ../src/output.c:501
    #245 0x55df2f9892b9 in os_anontmp ../src/posixos.c:846
    #246 0x55df2f930f60 in get_tmpfd ../src/misc.c:637
    #247 0x55df2f93226e in output_tmpfd ../src/output.c:189
    #248 0x55df2f9322ca in setup_tmpfile ../src/output.c:212

This patch has two parts.

1. A change in output.c avoids infinite recursion. With the change in output.c alone make no longer crashes. Instead make exits with an error message when the attempt to open a temp file fails.

2. A change in posixos.c allows make to failover to tmpfile + dup when open with O_TMPFILE fails. This change is required on any filesystem where O_TMPFILE is not supported. E.g. the test above was performed on reiserfs.