bugmake - Bugs: bug #15719, Test suite not reliable for...

 
 

bug #15719: Test suite not reliable for parallel build support

Submitted by:  Agent Zhang <agent>
Submitted on:  Sun 12 Feb 2006 09:22:02 AM UTC  
 
Severity: 3 - NormalItem Group: Enhancement
Status: NonePrivacy: Public
Assigned to: NoneOpen/Closed: Open
Component Version: NoneOperating System: Any
Fixed Release: NoneTriage Status: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 12 Dec 2012 03:54:52 PM UTC, comment #9:

With cvs HEAD (as of 2012-12-09), I've seen TEST #11 ("thing1,thing2 start,end") sometimes failing, even without so much load on the Linux build machine, where the diff output shows /missing/ lines.

Extracting this test and running in a loop with strace (test script attached) shows the situation when output lines get lost:

When those two shells running the "echo thingX start; sleep 1; echo thingX end" command do the write(1, "thingX ...") syscall (because echo being a shell-builtin) at the same time (on different cores), one of the line is overwritten by the other one.
However, I've seen this happening when the output is redirected to a file only.

So indeed the tests seem fragile, for these two reasons:
1) It does expect the output lines from concurrent processes in a particular order.
2) It does expect the output lines from concurrent processes redirected to one stream to not overwrite each other.

But I've failed to find a spec about how the content is expected to be merged when multiple processes concurrently write to a single file descriptor...

(file #27084)

Michael Haubenwallner <haubi>
Mon 14 Nov 2011 12:47:24 AM UTC, comment #8:

Troy Runkel has provided an implementation of the jobserver feature that works on Windows systems. I don't know if this will work for Cygwin or not.

However it appears that the issue here is really more about the test suite not being reliable due to reliance on sleep invocations, so I'm going to modify the Summary line and retarget this bug.

Paul D. Smith <psmith>
Project Administrator
Fri 02 Jun 2006 10:30:40 AM UTC, comment #7:

This test is extremely fragile.

We're seeing erratic failures in our (SUSE) build service as well. The load of the build service is highly unpredictable, so is the speed of the used machine.

So this is definitively not a Cygwin problem. AFACS the test is somewhat broken by design, because I uses assumptions you cannot make in a loaded system. Decreasing the sleep time will only make the test fail more likely due to environmental disturbances. Increasing the sleep time will make it more robust. No, this will not hide make bugs in this particular test.

Matthias Hopf <mshopf>
Fri 03 Mar 2006 01:52:24 AM UTC, comment #6:

Agent Zhang wrote:

>I'm guessing it's probably not a bug in GNU make, but a "bug" in the test suit itself. AFAIK, there's no easy way to overcome this problem, hence it's mostly the user's responsibility not to run the test suit either on very slow machines or when the system load is pretty high.


It don't think it is a problem with the test suite: it is a problem with make running on the W32 platform.

It is always a temptation to 'tweak' a test until the software passes - but this is a very dangerous thing to do! It may give the developers a sigh of relief but it misleads the hapless user into trusting that results are assured when they are, in fact not.

It is better to create tests that TRY TO FAIL, rather than those which try to succeed.

In the case of this bug, I have a very fast machine (AMD Athlon 64 X2 4400+) and it is pretty easy for me to make the test fail, at least 20% of the time.

I think the best solution, if this bug CANNOT be fixed (and I would wager it cannot) is to reduce the sleep time of the test to INCREASE likelihood of failure.

If the bug cannot be truly fixed, under any amount of load - the best solution would be to completely disable the -j feature when running on W32 platform so a user never gets 'surprised' when something fails under load, etc.

Either a feature should work RELIABLY, or not at all, in my opinion.

greg keranen <gkeranen>
Thu 02 Mar 2006 06:51:46 AM UTC, comment #5:

greg~

In my first mail to Paul, I also mentioned the Cygwin erratic failures in features/parallelism. However, at that time I was using GNU make 3.80. With GNU make 3.81beta4, it seems to me it's harder to reproduce the random failures since verstion 3.81beta4 runs a bit faster. :=)

> It is only TEST #5 which fails, but with somewhat different
> output; my guess is that creating load conditions, running
> other applications like video, while the test is running,
> helps reproduce the failure.


In fact, the tests in features/parallelism is somewhat fragile especially for very slow machine (or fast machine but with heavy load). They make extensive use of "sleep 1", "sleep 2", and etc., to do synchronization and assume that the user's machine runs fast enough. For this reason, it's not a Cygwin-specific problem IMHO.

> It does NOT work reliably under Cygwin, in my experience.


I'm guessing it's probably not a bug in GNU make, but a "bug" in the test suit itself. AFAIK, there's no easy way to overcome this problem, hence it's mostly the user's responsibility not to run the test suit either on very slow machines or when the system load is pretty high. Yes, it's not an ultimate solution. :/

Regards,
Agent

Agent Zhang <agent>
Wed 01 Mar 2006 09:55:46 PM UTC, comment #4:

I don't know if my experience of failure under Cygwin should be considered a separate bug, since it seems that you all seem to be saying that parallelism works under Cygwin but not under cmd.exe

It does NOT work reliably under Cygwin, in my experience.

I have so far reproduced the success of all 6 tests about 4 times and the failure twice:
features/parallelism .................................... FAILED (5/6 passed)

It is only TEST #5 which fails, but with somewhat different output; my guess is that creating load conditions, running other applications like video, while the test is running, helps reproduce the failure.

I don't pretend to understand the deeper issues here but my guess is that there is nothing you can do about it since Cygwin depends ultimately on windows threading which is probably somewhat uncontrollable, at this level.

Comments from the test script concerning this case:
# Michael Matz <matz@suse.de> reported a bug where if make is running in
# parallel without -k and two jobs die in a row, but not too close to each
# other, then make will quit without waiting for the rest of the jobs to die.

--- BEGIN FILE: work/features/parallelism.diff.4

      • work/features/parallelism.base.4 Wed Mar 1 13:03:39 2006

--- work/features/parallelism.log.4 Wed Mar 1 13:03:39 2006
***************

      • 4,8 ****

Fail
make: *** [fail.2] Error 1
Fail
- make: *** [fail.3] Error 1
Ok done
--- 4,8 ----
Fail
make: *** [fail.2] Error 1
Fail
Ok done
+ make: *** [fail.3] Error 1

--- EOF

--- BEGIN FILE: work/features/parallelism.diff.4

      • work/features/parallelism.base.4 Wed Mar 1 13:45:39 2006

--- work/features/parallelism.log.4 Wed Mar 1 13:45:39 2006
***************

      • 1,8 ****

Fail
make: *** [fail.1] Error 1
make: *** Waiting for unfinished jobs....
- Fail
make: *** [fail.2] Error 1
Fail
- make: *** [fail.3] Error 1
Ok done
--- 1,8 ----
Fail
+ Fail
make: *** [fail.1] Error 1
make: *** Waiting for unfinished jobs....
make: *** [fail.2] Error 1
Fail
Ok done
+ make: *** [fail.3] Error 1

--- EOF

--- TEST ENVIRONMENT:
GNU Make 3.81rc1 on CYGWIN_NT-5.1 x2 1.5.19(0.150/4/2) i686
NOTE: I compiled make from sources, using gcc under cygwin/bash
--- SYSTEM:
OS: Windows XP (SP2; +all current patches)
CPU: AMD Athlon 64 X2 4400+
Board: ASUS A8N-SLI Premium (nVidia nForce 4 SLI chipset)

greg keranen <gkeranen>
Wed 01 Mar 2006 08:58:00 PM UTC, comment #3:

Just FYI, I have found erratic results under Cygwin:
features/parallelism sometimes reports succeeds, sometimes fails.

$parallel_jobs = 1;

[greg@x2 tests]$ make -j 2 -f /dev/null
make: *** No targets. Stop.

--- TEST ENVIRONMENT:
GNU Make 3.81rc1 on CYGWIN_NT-5.1 x2 1.5.19(0.150/4/2) i686

--- SYSTEM:
OS: Windows XP (SP2; +all current patches)
CPU: AMD Athlon 64 X2 4400+
Board: ASUS A8N-SLI Premium (nVidia nForce 4 SLI chipset)

greg keranen <gkeranen>
Mon 13 Feb 2006 09:47:32 PM UTC, comment #2:

I'm recategorizing this as an enhancement and changing the title to reflect the issue.

Paul D. Smith <psmith>
Project Administrator
Sun 12 Feb 2006 08:14:41 PM UTC, comment #1:

This is expected: for now, the native Windows build doesn't support the job server method of parallelism, and without that support, Make always passes "-j 1" to sub-Makes, because it has no way of knowing how many jobs are run by the sub-Make. You will see it in the sources.

If you modify the test Makefile to pass an explicit "-j 4" switch to the sub-Makes, the test will work as expected.

In other words, this test, as written, cannot succeed in a port of Make that doesn't support the job server feature.

(I have on my TODO to add job server support to the Windows port, but that has to wait for a rainy day, and probably won't happen before 3.81 is released.)

Eli Zaretskii <eliz>
Project Member
Sun 12 Feb 2006 09:22:02 AM UTC, original submission:

This is a problem specific to native Win32 build of GNU make 3.81 beta4. Cygwin and *NIX builds of 3.81 beta4 works fine.

The test case that demonstrates this bug is already contained in GNU make 3.81 beta4's source distribution. It is the third test in tests\scripts\features\parallelism. :=)

Because currently the whole test suit breaks horribly on Win32, I think it's wise to repeat the test case here:

The content of the sample Makefile is

--- BEGIN ---

recurse: ; @$(MAKE) --no-print-directory -f MAKEFILE INC=yes all
all: 1 2; @echo success

INC = no
ifeq ($(INC),yes)
-include 1.inc 2.inc
endif

1.inc: ; @echo ONE.inc; sleep 2; echo TWO.inc; echo '1: ; @echo ONE; sleep 2; echo TWO' > $@
2.inc: ; @sleep 1; echo THREE.inc; echo '2: ; @sleep 1; echo THREE' > $@

--- END ---

When invoking Cygwin build of make 3.81 beta4, we can get what we expect:

C:\Temp>cygwin-make -j4
ONE.inc
THREE.inc
TWO.inc
ONE
THREE
TWO
success

But if we invoke Win32 build of make, we will get

C:\Temp\tmp>del *.inc

C:\Temp\tmp>win32-make -j4
THREE.inc
ONE.inc
TWO.inc
ONE
TWO
THREE
success

Apparently, the execution got serialized. :(

FYI, My Win32 build of GNU make 3.81 beta4 passes other tests in `features/parallelism'.

Regards,
Agent

Agent Zhang <agent>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #27084:  makebug-15719-test11.sh added by haubi (451B - application/x-shellscript - Test for multiple processes concurrently writing to one single redirected stream)
file #1986:  Makefile added by agent (312B - application/octet-stream - Sample Makefile that demonstrates the bug)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by haubi (Updated the item)
  • -unavailable- added by psmith (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 8 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 12 Dec 2012 03:54:52 PM UTChaubiAttached File-=>Added makebug-15719-test11.sh, #27084
    Mon 14 Nov 2011 12:47:24 AM UTCpsmithOperating SystemMS Windows=>Any
      SummaryMake jobserver feature not supported on WINDOWS32 platforms=>Test suite not reliable for parallel build support
    Fri 02 Jun 2006 10:30:40 AM UTCmshopfCarbon-Copy-=>Added mshopf
    Mon 13 Feb 2006 09:47:32 PM UTCpsmithItem GroupBug=>Enhancement
      SummaryParallelism with included files in recursive calling becomes serialized on Win32=>Make jobserver feature not supported on WINDOWS32 platforms
    Sun 12 Feb 2006 09:22:02 AM UTCagentAttached File-=>Added Makefile, #3392
      Carbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup1