bugmake - Bugs: bug #15719, Test suite not reliable for...

 
 

bug #15719: Test suite not reliable for parallel build support

Submitter:  Agent Zhang <agent>
Submitted:  Sun 12 Feb 2006 09:22:02 AM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  None Operating System:  Any
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 02 Jan 2023 09:58:44 PM UTC, comment #10: 

A lot of work has been done on the test suite to try to ensure that the tests of parallelism work.  A helper utility was added that allows a recipe to wait for some file to be created before exiting, which helps us ensure that things happen in the correct order even when run in parallel.

Paul D. Smith <psmith>
Group administrator
Wed 12 Dec 2012 03:54:52 PM UTC, comment #9: 

With cvs HEAD (as of 2012-12-09), I've seen TEST #11 ("thing1,thing2 start,end") sometimes failing, even without so much load on the Linux build machine, where the diff output shows /missing/ lines.

Extracting this test and running in a loop with strace (test script attached) shows the situation when output lines get lost:

When those two shells running the "echo thingX start; sleep 1; echo thingX end" command do the write(1, "thingX ...") syscall (because echo being a shell-builtin) at the same time (on different cores), one of the line is overwritten by the other one.
However, I've seen this happening when the output is redirected to a file only.

So indeed the tests seem fragile, for these two reasons:
1) It does expect the output lines from concurrent processes in a particular order.
2) It does expect the output lines from concurrent processes redirected to one stream to not overwrite each other.

But I've failed to find a spec about how the content is expected to be merged when multiple processes concurrently write to a single file descriptor...

(file #27084)

Michael Haubenwallner <haubi>
Mon 14 Nov 2011 12:47:24 AM UTC, comment #8: 

Troy Runkel has provided an implementation of the jobserver feature that works on Windows systems.  I don't know if this will work for Cygwin or not.

However it appears that the issue here is really more about the test suite not being reliable due to reliance on sleep invocations, so I'm going to modify the Summary line and retarget this bug.

Paul D. Smith <psmith>
Group administrator
Fri 02 Jun 2006 10:30:40 AM UTC, comment #7: 

This test is extremely fragile.

We're seeing erratic failures in our (SUSE) build service as well. The load of the build service is highly unpredictable, so is the speed of the used machine.

So this is definitively not a Cygwin problem. AFACS the test is somewhat broken by design, because I uses assumptions you cannot make in a loaded system. Decreasing the sleep time will only make the test fail more likely due to environmental disturbances. Increasing the sleep time will make it more robust. No, this will not hide make bugs in this particular test.

Matthias Hopf <mshopf>
Fri 03 Mar 2006 01:52:24 AM UTC, comment #6: 

Agent Zhang wrote:

>I'm guessing it's probably not a bug in GNU make, but a "bug" in the test suit itself. AFAIK, there's no easy way to overcome this problem, hence it's mostly the user's responsibility not to run the test suit either on very slow machines or when the system load is pretty high.


It don't think it is a problem with the test suite: it is a problem with make running on the W32 platform.

It is always a temptation to 'tweak' a test until the software passes - but this is a very dangerous thing to do! It may give the developers  a sigh of relief but it misleads the hapless user into trusting that results are assured when they are, in fact not.

It is better to create tests that TRY TO FAIL, rather than those which try to succeed.

In the case of this bug, I have a very fast machine (AMD Athlon 64 X2 4400+) and it is pretty easy for me to make the test fail, at least 20% of the time.

I think the best solution, if this bug CANNOT be fixed (and I would wager it cannot) is to reduce the sleep time of the test to INCREASE likelihood of failure.

If the bug cannot be truly fixed, under any amount of load - the best solution would be to completely disable the -j feature when running on W32 platform so a user never gets 'surprised' when something fails under load, etc.

Either a feature should work RELIABLY, or not at all, in my opinion.

greg keranen <gkeranen>
Thu 02 Mar 2006 06:51:46 AM UTC, comment #5: 

greg~

In my first mail to Paul, I also mentioned the Cygwin erratic failures in features/parallelism. However, at that time I was using GNU make 3.80. With GNU make 3.81beta4, it seems to me it's harder to reproduce the random failures since verstion 3.81beta4 runs a bit faster. :=)

> It is only TEST #5 which fails, but with somewhat different
> output; my guess is that creating load conditions, running
> other applications like video, while the test is running,
> helps reproduce the failure.


In fact, the tests in features/parallelism is somewhat fragile especially for very slow machine (or fast machine but with heavy load). They make extensive use of "sleep 1", "sleep 2", and etc., to do synchronization and assume that the user's machine runs fast enough. For this reason, it's not a Cygwin-specific problem IMHO.

> It does NOT work reliably under Cygwin, in my experience.


I'm guessing it's probably not a bug in GNU make, but a "bug" in the test suit itself. AFAIK, there's no easy way to overcome this problem, hence it's mostly the user's responsibility not to run the test suit either on very slow machines or when the system load is pretty high. Yes, it's not an ultimate solution. :/

Regards,
Agent

Agent Zhang <agent>
Wed 01 Mar 2006 09:55:46 PM UTC, comment #4: 

I don't know if my experience of failure under Cygwin should be considered a separate bug, since it seems that you all seem to be saying that parallelism works under Cygwin but not under cmd.exe

It does NOT work reliably under Cygwin, in my experience.

I have so far reproduced the success of all 6 tests about 4 times and the failure twice:
features/parallelism .................................... FAILED (5/6 passed)

It is only TEST #5 which fails, but with somewhat different output; my guess is that creating load conditions, running other applications like video, while the test is running, helps reproduce the failure.

I don't pretend to understand the deeper issues here but my guess is that there is nothing you can do about it since Cygwin depends ultimately on windows threading which is probably somewhat uncontrollable, at this level.

Comments from the test script concerning this case:
    # Michael Matz <matz@suse.de> reported a bug where if make is running in
    # parallel without -k and two jobs die in a row, but not too close to each
    # other, then make will quit without waiting for the rest of the jobs to die.

--- BEGIN FILE:  work/features/parallelism.diff.4
* work/features/parallelism.base.4 Wed Mar  1 13:03:39 2006
--- work/features/parallelism.log.4 Wed Mar  1 13:03:39 2006
*************
* 4,8 **
  Fail
  make: * [fail.2] Error 1
  Fail
- make: * [fail.3] Error 1
  Ok done
--- 4,8 ----
  Fail
  make: * [fail.2] Error 1
  Fail
  Ok done
+ make: * [fail.3] Error 1

--- EOF

--- BEGIN FILE:  work/features/parallelism.diff.4
* work/features/parallelism.base.4 Wed Mar  1 13:45:39 2006
--- work/features/parallelism.log.4 Wed Mar  1 13:45:39 2006
*************
* 1,8 **
  Fail
  make: * [fail.1] Error 1
  make: * Waiting for unfinished jobs....
- Fail
  make: * [fail.2] Error 1
  Fail
- make: * [fail.3] Error 1
  Ok done
--- 1,8 ----
  Fail
+ Fail
  make: * [fail.1] Error 1
  make: * Waiting for unfinished jobs....
  make: * [fail.2] Error 1
  Fail
  Ok done
+ make: * [fail.3] Error 1

--- EOF

--- TEST ENVIRONMENT:
GNU Make 3.81rc1 on CYGWIN_NT-5.1 x2 1.5.19(0.150/4/2) i686
NOTE: I compiled make from sources, using gcc under cygwin/bash
--- SYSTEM:
OS: Windows XP (SP2; +all current patches)
CPU: AMD Athlon 64 X2 4400+
Board: ASUS A8N-SLI Premium (nVidia nForce 4 SLI chipset)

greg keranen <gkeranen>
Wed 01 Mar 2006 08:58:00 PM UTC, comment #3: 

Just FYI, I have found erratic results under Cygwin:
features/parallelism sometimes reports succeeds, sometimes fails.

$parallel_jobs = 1;

[greg@x2 tests]$ make -j 2 -f /dev/null
make: * No targets.  Stop.

--- TEST ENVIRONMENT:
GNU Make 3.81rc1 on CYGWIN_NT-5.1 x2 1.5.19(0.150/4/2) i686

--- SYSTEM:
OS: Windows XP (SP2; +all current patches)
CPU:    AMD Athlon 64 X2 4400+
Board:  ASUS A8N-SLI Premium (nVidia nForce 4 SLI chipset)

greg keranen <gkeranen>
Mon 13 Feb 2006 09:47:32 PM UTC, comment #2: 

I'm recategorizing this as an enhancement and changing the title to reflect the issue.

Paul D. Smith <psmith>
Group administrator
Sun 12 Feb 2006 08:14:41 PM UTC, comment #1: 

This is expected: for now, the native Windows build doesn't support the job server method of parallelism, and without that support, Make always passes "-j 1" to sub-Makes, because it has no way of knowing how many jobs are run by the sub-Make.  You will see it in the sources.

If you modify the test Makefile to pass an explicit "-j 4" switch to the sub-Makes, the test will work as expected.

In other words, this test, as written, cannot succeed in a port of Make that doesn't support the job server feature.

(I have on my TODO to add job server support to the Windows port, but that has to wait for a rainy day, and probably won't happen before 3.81 is released.)

Eli Zaretskii <eliz>
Group Member
Sun 12 Feb 2006 09:22:02 AM UTC, original submission:  

This is a problem specific to native Win32 build of GNU make 3.81 beta4. Cygwin and *NIX builds of 3.81 beta4 works fine.

The test case that demonstrates this bug is already contained in GNU make 3.81 beta4's source distribution. It is the third test in tests\scripts\features\parallelism. :=)

Because currently the whole test suit breaks horribly on Win32, I think it's wise to repeat the test case here:

The content of the sample Makefile is

--- BEGIN ---

recurse: ; @$(MAKE) --no-print-directory -f MAKEFILE INC=yes all
all: 1 2; @echo success

INC = no
ifeq ($(INC),yes)
-include 1.inc 2.inc
endif

1.inc: ; @echo ONE.inc; sleep 2; echo TWO.inc; echo '1: ; @echo ONE; sleep 2; echo TWO' > $@
2.inc: ; @sleep 1; echo THREE.inc; echo '2: ; @sleep 1; echo THREE' > $@

--- END ---

When invoking Cygwin build of make 3.81 beta4, we can get what we expect:

C:\Temp>cygwin-make -j4
ONE.inc
THREE.inc
TWO.inc
ONE
THREE
TWO
success

But if we invoke Win32 build of make, we will get

C:\Temp\tmp>del *.inc

C:\Temp\tmp>win32-make -j4
THREE.inc
ONE.inc
TWO.inc
ONE
TWO
THREE
success

Apparently, the execution got serialized. :(

FYI, My Win32 build of GNU make 3.81 beta4 passes other tests in `features/parallelism'.

Regards,
Agent

Agent Zhang <agent>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #27084:  makebug-15719-test11.sh added by haubi (451B - application/x-shellscript - Test for multiple processes concurrently writing to one single redirected stream)
file #1986:  Makefile added by agent (312B - application/octet-stream - Sample Makefile that demonstrates the bug)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by haubi (Updated the item)
  • -email is unavailable- added by psmith (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-02 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
    2012-12-12 haubi Attached File- Added makebug-15719-test11.sh, #27084
    2011-11-14 psmith Operating SystemMS Windows Any
        SummaryMake jobserver feature not supported on WINDOWS32 platforms Test suite not reliable for parallel build support
    2006-06-02 mshopf Carbon-Copy- Added mshopf
    2006-02-13 psmith Item GroupBug Enhancement
        SummaryParallelism with included files in recursive calling becomes serialized on Win32 Make jobserver feature not supported on WINDOWS32 platforms
    2006-02-12 agent Attached File- Added Makefile, #3392
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code