bugGNU Octave - Bugs: bug #53186, Without osmesa it is not possible...

 
 

bug #53186: Without osmesa it is not possible to print/getframe invisible figures

Submitted by:  Pantxo Diribarne <pantxo>
Submitted on:  Sun 18 Feb 2018 10:41:22 PM UTC  
 
Category:  Plotting with OpenGL Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  dev Operating System:  Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Mon 12 Mar 2018 09:16:46 AM UTC, comment #45:

Ok, now I can reproduce the test failure, I filed bug #53328.

Pantxo Diribarne <pantxo>
Project Member
Sat 10 Mar 2018 11:36:31 PM UTC, comment #44:

I see the same as Dmitri, if I run another CPU-intensive process at the same time, this test is more likely to fail on my system.

Mike Miller <mtmiller>
Project Administrator
Sat 10 Mar 2018 11:32:09 PM UTC, comment #43:

I do not see any warnings (with multiple failures).
From whatever testing I did, it looks to me that failure more likely to occur
if you do some reasonably heavy I/O at the same time as you run the test. The CPU load does not seem to matter that much.
Essentially if I run the test from #41 and at the same time do "make -j8" or "make check" I get more failures (I managed to get 20 max). Doing the same test on mearly idle machine gives only one or 2 (or none). That may explain why buildbots have those failures regularly.

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Sat 10 Mar 2018 08:34:21 PM UTC, comment #42:

I can't reproduce the test failure on my system but I attached a patch that adds some warnings here and there. Could you apply this patch and look at what warning is triggered (if any) when the test fails?

(file #43506)

Pantxo Diribarne <pantxo>
Project Member
Fri 09 Mar 2018 10:44:07 PM UTC, comment #41:

I think there could be a race condition. There is certainly something wrong. Testing in the GUI with

shows that there are always 5-8 failures on my system.

A separate bug report should be filed since this may have nothing to do with getframe, but more with how the GUI thread and the Octave thread interact.

Rik <rik5>
Project Administrator
Fri 09 Mar 2018 06:33:51 PM UTC, comment #40:

If you look at buildbots this failure test

happens quite often.
LIBGL_ALWAYS_SOFTWARE=1 perhaps make it worse (more often).

I was able to reproduce on a desktop getting about 2 failures per
1000 repeats. On buildbots it happens more often.
I wonder if there is some race condition going on (e.g. figure() does not have enough time to setup properties which are getting read by get())...
In any case it does not look like a high priority issue, but the problem is there.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 03 Mar 2018 08:43:14 PM UTC, comment #39:

This change "fixes" bug #44478 too. Congratulations !

ederag <ederag>
Fri 02 Mar 2018 04:14:08 PM UTC, comment #38:

So Fedora's clang "make test" failed again at getframe.
I checked the pictures and they are fine. Also if I do
test getframe interactively it passes as well.
Also Fedora's lto "make test" passed this time.

So there is some randomness here. Perhaps I should set
LIBGL_ALWAYS_SOFTWARE=1 again.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 02 Mar 2018 03:18:50 PM UTC, comment #37:

Those are buildbots errors. They also seems to be kind of random -- some buildbots (on the same computer) dis not report the errors.

I should be able to check fedora's build bot docs later today.

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Fri 02 Mar 2018 03:03:09 PM UTC, comment #36:

@Dmitri: Are the figures of the manual generated correctly? If so this would point to a problem with readPixels not working correctly with bound framebuffer object.

Pantxo Diribarne <pantxo>
Project Member
Fri 02 Mar 2018 02:41:07 PM UTC, comment #35:

That might have some problems:

and

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 02 Mar 2018 12:51:12 PM UTC, comment #34:

@Mike: I updated the test condition to be HAVE_QOFFSCREENSURFACE

http://hg.savannah.gnu.org/hgweb/octave/rev/735175432a20

Pantxo Diribarne <pantxo>
Project Member
Fri 02 Mar 2018 03:17:00 AM UTC, comment #33:

Should the %!tests in scripts/image/getframe.m be updated to work with the new configuration?

Mike Miller <mtmiller>
Project Administrator
Fri 02 Mar 2018 02:07:24 AM UTC, comment #32:

@Dmitri: Qt 5.5 here.

@Rik: Wow, major config cleanup. I reconfigured with no options, cleaned out the old build and rebuilt without any problems.

Thanks everyone.

Dan Sebald <sebald>
Thu 01 Mar 2018 08:20:46 PM UTC, comment #31:

I overhauled configure.ac in this cset (http://hg.savannah.gnu.org/hgweb/octave/rev/8a92f442c4a2). As part of that effort I disabled OSMesa by default, but it can be used by passing the '--with-osmesa' argument to configure.

Rik <rik5>
Project Administrator
Wed 28 Feb 2018 10:38:12 PM UTC, comment #30:

Good idea to announce the demise of FLTK. The sooner
the better.

Michael Godfrey <godfrey>
Project Member
Wed 28 Feb 2018 10:14:44 PM UTC, comment #29:

This should be a step in that direction. Once configure is re-jiggered, OSMesa won't be used by default which will put FLTK at a disadvantage and encourage more users to just use qt.

Rik <rik5>
Project Administrator
Wed 28 Feb 2018 10:05:58 PM UTC, comment #28:

Since OSMesa doesn't seem to play well with anything except a straight Mesa-based libGL, it seems that we should take steps to remove support for OSMesa from Octave.

I hope that we will also be able to remove the FLTK widget soon.

John W. Eaton <jwe>
Project Administrator
Wed 28 Feb 2018 09:44:15 PM UTC, comment #27:

@Dmitri: if Qt>=5.1 it is now the default for building the doc, so there is no need to use --without-osmesa for that purpose.

Pantxo Diribarne <pantxo>
Project Member
Wed 28 Feb 2018 09:31:41 PM UTC, comment #26:

@Dan. What is your Qt version? If it is > 5.1 you may want to
try --without-osmesa and it should use Qt facilities to build documentation. I do not need LD_PRELOAD of mesa libs for this anymore.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 28 Feb 2018 09:19:12 PM UTC, comment #25:

@Pantxo "BTW, why do you use LD_PRELOAD?"

The LD_PRELOAD was the way to make the documentation build complete successfully for the nVidia driver users:

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGL.so make -j4

Without it, OSMesa failed with nonvisible images created in the doc/interpreter, e.g., doc/interpreter/voronoi.eps. That being the case, every time "make" was re-run it would go through trying to build the documentation again and fail at the figure generation.

The other way was to configure out documentation or osmesa, something like:

configure --without-osmesa --no-recursion

That has drawbacks too.

http://octave.1599824.n4.nabble.com/Verbose-make-output-for-documentation-tp4671907p4671911.html

https://savannah.gnu.org/bugs/index.php?44478

Dan Sebald <sebald>
Wed 28 Feb 2018 08:44:08 PM UTC, comment #24:

@Dan:

>> But, again, why it worked prior to the recent changes and not now, I don't know. Has conditional initialization of the frame buffer been added somewhere?


What worked ? Before cset 45470049a43f there was no explicit use of framebuffer objects in Octave code. It looks like swrast does not do well with this opengl extension.

>> For what it is worth, the following little OpenGL test program from one of those links fails: ...


Well, at the very least, it tells us that it is not Octave specific, that it is not Qt specific and that it is not a good idea to do this LD_PRELOAD :-). BTW, why do you use LD_PRELOAD?

Pantxo Diribarne <pantxo>
Project Member
Wed 28 Feb 2018 08:19:39 PM UTC, comment #23:

Yes, I see what is happening now. The warning message from the OCTAVE_CHECK_LIB macro is only issued if --with-osmesa is being used, but in fact is not present.

Rik <rik5>
Project Administrator
Wed 28 Feb 2018 08:17:22 PM UTC, comment #22:

I removed osmesa-devel from Fedora's buildbot. You can see the message in recent configure log e.g:

http://buildbot.octave.org:8010/builders/gcc-fedora/builds/1348/steps/configure/logs/stdio

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 28 Feb 2018 08:15:10 PM UTC, comment #21:

I just configuring using '--without-osmesa' and this tip

and the warning I get is

which is generic and probably fine. The message you get "Offscreen rendering with OpenGL will be disabled" is still in configure.ac, but I don't see it being executed.

In any case, the default should be not to build in osmesa if octave_cv_qt_opengl_os_ok says the newer alternative is available, unless the user has explicitly used --with-osmesa.

Rik <rik5>
Project Administrator
Wed 28 Feb 2018 08:09:23 PM UTC, comment #20:

It's strange. The test you posted in #18 doesn't fail for some accounts and does fail on other accounts only when I use

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGL.so

as the preload. I assume that libGL.so is swrast (swrast is in the process of being replaced by a different driver by OpenGL). It is a bad way for swrast to fail--to just hang like that.

But why this fails on some accounts and not others...it must have something to do with (well, obviously) the frame buffer (i.e., fbConfig) not being set up. But, again, why it worked prior to the recent changes and not now, I don't know. Has conditional initialization of the frame buffer been added somewhere?

For what it is worth, the following little OpenGL test program from one of those links fails:

when without the preload the gears program runs fine.

The nVidia driver seems to work fine for building (i.e., without the LD_PRELOAD in my case) so that's good. Apparently nVidia is setting up the frame buffer correctly. swrast is deprecated, but obviously still hanging around some Linux bundles. That's the only issue, i.e., that some bundle maintainer will have a hard time building Octave because, say for example on Fedora systems where they disallow non-open software the swrast driver is still in service. For me, I'm happy with nVidia working.

So, the only thing I can suggest is that maybe the changes that have been made no longer set up the frame buffer and maybe somewhere at the initialization of Octave we simply need to always do such a thing.

Dan Sebald <sebald>
Wed 28 Feb 2018 07:56:52 PM UTC, comment #19:

Should --without-osmesa be made a default option?
(We want to encorage people to use the recent Qt. The unfortunate ones who stuck with the older version would have to specify explicitly --with-osmesa and be aware of all the problems it brings.)

Also, at the moment configure still reports:

configure: WARNING: OSMesa library not found. Offscreen rendering with OpenGL will be disabled.

(which is not true with Qt > 5.9)

May be those should be a different bug...

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 28 Feb 2018 10:17:23 AM UTC, comment #18:

@Dan: looks like we are not going to get rid of those opengl driver issues so easily ... I see much people reporting the same issue with other software (see e.g. [1, 2]). Could you play with the proposed workarounds (generally reinstalling nVidia drivers) and report back?

Also, you say the process is "frozen". Does the following hang Octave started with LD_PRELOAD...?

[1] https://askubuntu.com/questions/834254/steam-libgl-error-no-matching-fbconfigs-or-visuals-found-libgl-error-failed-t
[2] https://askubuntu.com/questions/541343/problems-with-libgl-fbconfigs-swrast-through-each-update

Pantxo Diribarne <pantxo>
Project Member
Wed 28 Feb 2018 02:33:16 AM UTC, comment #17:

Just want to note that I've found I can now build without any special driver pre-install (nVidia) and the figures in the doc/interpreter directory look correct. However, if I do use a driver pre-install, the build hangs:

At this point the build process is frozen and one cannot break out. Looking up the process ID and kill -9 is required.

If this is simply a case of don't-do-that-anymore, that's fine, so long as occasional builders with nVidia drivers realize that (e.g., Linux bundle maintainers).

Dan Sebald <sebald>
Tue 27 Feb 2018 09:21:21 AM UTC, comment #16:

@Rik: Many thanks for reviewing.

Pantxo Diribarne <pantxo>
Project Member
Tue 27 Feb 2018 02:19:51 AM UTC, comment #15:

Oops -- it was a stable branch.

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Tue 27 Feb 2018 02:17:12 AM UTC, comment #14:

Your hg id is too old and does not include this patch. Try updating to 45470049a43f.

Rik <rik5>
Project Administrator
Tue 27 Feb 2018 02:15:10 AM UTC, comment #13:

OK it fell back to gnuplot for docs...

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Tue 27 Feb 2018 01:58:05 AM UTC, comment #12:

I am still not sure what is happening:

_octave_config_info_ ("hg_id")
ans = d3a79cb734d2

This all seems in accord with
scripts/plot/util/private/__opengl_print__.m

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Tue 27 Feb 2018 01:39:15 AM UTC, comment #11:

It built docs fine. (Also I did not have to preload mesa libs.)

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Tue 27 Feb 2018 01:36:10 AM UTC, comment #10:

I assume this is just an incorrect warning from configure, but that actual offscreen rendering works correctly as long as Qt > 5.1?

That would imply adjusting this section in configure.ac

Could be as simple as checking the variable octave_cv_qt_opengl_os_ok and changing the message based on the result.

Rik <rik5>
Project Administrator
Tue 27 Feb 2018 01:09:05 AM UTC, comment #9:

running configure w/o OSmesa devel still results in:

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Tue 27 Feb 2018 01:00:06 AM UTC, comment #8:

I made a few small changes. I capitalized the first word of each sentence in the check-in message as is traditional for English syntax. I also put the new macro OCTAVE_CHECK_QT_OPENGL_OFFSCREEN_OK in alphabetical order within acinclude.m4. Finally, instead of the comment "## do nothing" I described what is actually happening with "## Use qt with QOffscreenSurface for plot". Everything was checked in here http://hg.savannah.gnu.org/hgweb/octave/rev/45470049a43f.

Marking as fixed and closing report.

Rik <rik5>
Project Administrator
Fri 23 Feb 2018 01:35:56 PM UTC, comment #7:

I finally had configure pick qt4 (needed to temporarily invert the preference order in configure.ac) and there was a warning, which I fixed, but everything seems to work as expected with and without osmesa. I attached an updated cset which I think is now ok for review.

(file #43391)

Pantxo Diribarne <pantxo>
Project Member
Fri 23 Feb 2018 11:25:23 AM UTC, comment #6:

I attached an updated patch with configure checks and feature tests for print and getframe. I verified that it allows building the manual even without osmesa support, but I was not able to check how things works with qt4:

was not enough to force configure to pick qt4 on my system even though

returns

(file #43390)

Pantxo Diribarne <pantxo>
Project Member
Wed 21 Feb 2018 09:56:52 AM UTC, comment #5:

@Rik: thanks for testing, it's very good news. Since Octave is always built with Qt > 5.1 on Windows, this means we could drop this long standing goal of having osmesa work. This also means that Octave documentation can be built without osmesa support.

I'll rework the patch to take care of configure checks and all cases when a figure is invisible:

  • Qt >= 5.1 : print OK, getframe OK,
  • Qt < 5.1 + osmesa: print OK, getframe OK,
  • Qt < 5.1 + no osmesa: print ERROR unless the figure has been visible once, getframe ERROR unless the figure has been visible once,
  • FLTK + osmesa: print OK, getframe OK,
  • FLTK + no osmesa: print ERROR, getframe ERROR.
Pantxo Diribarne <pantxo>
Project Member
Wed 21 Feb 2018 12:10:27 AM UTC, comment #4:

I applied the patch on top of

and then built a version with MXE. It seems to work for both this bug, and for bug #53087 (tested that code too).

Rik <rik5>
Project Administrator
Mon 19 Feb 2018 09:49:43 PM UTC, comment #3:

I ran the attached modifed dump_plot_demos (a single invisible figure is used for all plots) in order to compare figures generated with osmesa's context and with Qt's offscreen context:

  • no evident visual difference
  • timings are comparable
  • timings are sometimes much smaller than with a visible figure

(file #43364)

Pantxo Diribarne <pantxo>
Project Member
Mon 19 Feb 2018 09:58:25 AM UTC, comment #2:

I also forgot to mention that the cset also fixes bug #53087 for me.

Pantxo Diribarne <pantxo>
Project Member
Mon 19 Feb 2018 09:47:15 AM UTC, comment #1:

I forgot to mention that it is already possible to print an invisible Qt figure if it has been visible once. So for testing properly one should use something like:

Pantxo Diribarne <pantxo>
Project Member
Sun 18 Feb 2018 10:41:22 PM UTC, original submission:

I attached a very preliminary cset (no configure time checks at least ...) that makes "getframe" and "print" work without osmesa when figures are invible. Basically, using QOffscreenSurface (Qt5.1+), we can create a valid opengl context without the need of a visible QOpenGLWidget.

I also made use (for getframe) of QOpenGLFramebufferObject, in order to have suitably sized render buffers when the figure is invisible.

Pantxo Diribarne <pantxo>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #43506:  warn_getframe.patch added by pantxo (2KiB - text/x-patch)
file #43391:  qt_print_invible3.patch added by pantxo (18KiB - text/x-patch)
file #43390:  qt_print_invible2.patch added by pantxo (14KiB - text/x-patch)
file #43364:  dump_plot_demos.m added by pantxo (385KiB - text/x-objcsrc)
file #43351:  qt_print_invible.patch added by pantxo (9KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ederag (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by godfrey (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by pantxo (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-03-10 pantxo Attached File- => Added warn_getframe.patch, #43506
    2018-02-28 pantxo StatusIn Progress => Fixed
        Open/ClosedOpen => Closed
    2018-02-28 rik5 StatusFixed => In Progress
        Open/ClosedClosed => Open
    2018-02-27 rik5 StatusPatch Reviewed => Fixed
        Open/ClosedOpen => Closed
    2018-02-23 pantxo Attached File- => Added qt_print_invible3.patch, #43391
    2018-02-23 pantxo Attached File- => Added qt_print_invible2.patch, #43390
    2018-02-21 rik5 StatusNone => Patch Reviewed
    2018-02-19 pantxo Attached File- => Added dump_plot_demos.m, #43364
    2018-02-18 pantxo Attached File- => Added qt_print_invible.patch, #43351

    Back to the top


    Powered by Savane 3.4