bugGNU Octave - Bugs: bug #58393, (buildbot) Tests are marked as...

 
 

bug #58393: (buildbot) Tests are marked as "pass" if Octave crashes

Submitted by:  Markus Mützel <mmuetzel>
Submitted on:  Mon 18 May 2020 03:02:18 PM UTC  
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  other Operating System:  GNU/Linux

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 30 May 2020 02:59:57 PM UTC, comment #3: 

I didn't check the repository or I would have noticed that you already caught that error.

I believe indentation matters in python. Does the attached patch make a difference?

(file #49187)

Markus Mützel <mmuetzel>
Project Member
Tue 26 May 2020 05:36:29 PM UTC, comment #2: 

There is a typo in the code snippet in comment #0. It should read "errLineReceived", not "errLineRecived":
http://docs.buildbot.net/current/developer/cls-logobserver.html#buildbot.process.logobserver.LogLineObserver

Sorry for that.

Not sure this will make any difference however.

Markus Mützel <mmuetzel>
Project Member
Tue 26 May 2020 04:25:22 PM UTC, comment #1: 

Since May 13 when I last restarted the buildbot master, we have been running with this change and it unfortunately doesn't seem to flag the segfaults as failures.

I'm also not an expert in buildbot and I don't know what change to make.

BTW, I noticed that my workers were at version 2.4 and 2.7 but the master was at 1.1.1.  I updated all to 2.7 this morning.  I think I was ultimately successful with that, but it required more time that it probably should have.

John W. Eaton <jwe>
Project Administrator
Mon 18 May 2020 03:02:18 PM UTC, original submission:  

Like already reported on the mailing list a while ago:
https://octave.1599824.n4.nabble.com/buildbots-False-pass-results-for-segmentation-fault-in-test-td4695266.html

Quoting here:

> It looks like a segmentation fault in the last step of running the test suite on the build bots is still marked as "success" if there were no failed test up to that point. E.g.:
> http://buildbot.octave.org:8010/#/builders/12/builds/1525/steps/6/logs/stdio
>
> Maybe any output to stderr in that step should result in it to be marked as "failure"?


And a potential fix (but keep in mind that I don't know much about buildbot configuration):

diff -r e02079ee69d1 master.cfg
--- a/master.cfg Fri Feb 08 15:58:06 2019 +0000
+++ b/master.cfg Mon Jan 13 19:44:13 2020 +0100
@@ -324,6 +324,7 @@
     self.passed = 0
     self.failed = 0
     self.warnings = 0
+    self.errors = 0
     self.summary_found = False
     self.complete = False

@@ -352,6 +353,11 @@
           self.rc = FAILURE
           self.failed += num

+  def errLineRecived (self, line)
+    self.total += 1
+    self.errors += 1
+    self.rc = FAILURE
+

class octave_test (Test):

@@ -365,7 +371,7 @@
       passed = self.observer.total - self.observer.failed

       self.setTestResults (total = self.observer.total,
-                           failed = self.observer.failed,
+                           failed = self.observer.failed + self.observer.errors,
                            passed = self.observer.passed,
                            warnings = self.observer.warnings)

Markus Mützel <mmuetzel>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-30 mmuetzel Attached File- => Added bug58393_logline_observer_err.patch, #49187
    2020-05-18 rik5 Carbon-CopyRemoved 72865 => -
    2020-05-18 rik5 StatusNone => Confirmed

    Back to the top


    Powered by Savane 3.5