I agree, not exactly fixed, but the issue is still resolved since osmesa is no longer required.

I don't know what the status of the Nvidia/OSMesa/swrast situation is.  Years ago I built OSMesa and the OpenGL stuff from scratch, was able to make changes, submit bug reports to the maintainers of the various drivers, etc.  I don't want to dig that deep again.

But the problem still seems to be an issue on my relatively up-to-date Mint, in particular the build process.  Perhaps there are some build work-arounds that avoid OSMesa and the issue altogether.  I'm happy to use that, but I suspect that generally speaking users who are just compiling won't like doing too much special.

With that in mind, and particular to address the issue raised in

https://savannah.gnu.org/bugs/?func=detailitem&item_id=44478#comment112

I've written a little bash script called "octave-olp" that is analogous to "octave-cli" but it does a few things:

1) Main issue: searches for the possibility that an Nvidia driver is active, and if there is a possibility octave-olp will search for an alternative generic driver to pre-load.  Since octave-olp is intended for off-line printing, what does it matter if the spiffy Nvidia driver is used?  I do see some figures in octave.pdf for voronoi (Figure 30.1) so it must be work (and I did a full rebuild with just the "configure" sans any preload on my part).

2) I'll make this a second point for importance: I've confirmed that Nvidia driver and generic driver are found on my system.  That is, I wondered if the system utility I'm using will only recognize the generic driver if the generic driver had been used prior (i.e., it only finds "installed" libraries).  Perhaps it is because at power up the OS loads the generic driver before even knowing details about the graphics hardware and a custom driver; then the library remains resident in the proper tables.  On the other hand, maybe there is an issue on systems that don't use any sort of generic driver at power-up.  That would be the fly in the ointment here, i.e., there is a generic driver present but it isn't found because it wasn't used prior.  I left some commented-out alternative OS utilities in the script file in case there is an issue.

3) This may not be a solution for Mac X OS (or windows).  Does the Nvidia driver problem exist for those systems?  Hmm, that has got me thinking.  I've placed "octave-olp" in the "figure-generating-commands", but if Windows can't run octave-olp, then what?  Is it possible to make a script that looks like bash on Unix and some other kind of fall-through script file on Windows?

4) Systems for which there is no Nvidia driver present, or for systems where there is no generic driver present, either there is no pre-load (former) or the loaded $GRAPHICS_LIB is empty "".  In both cases, the fall-back should be just like running "octave-cli --norc --silent --no-history" with the additional inputs.

5) The octave-olp is meant to work in both the build environment and the installed environment.  (See top of the bash script.)  If I run octave-olp after the fact and try plotting some figure, I see

linux@ ~ $ octave-olp
octave:1> plot(1:50)
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
octave:2>

Oh well, that's the way it goes.

As far as I'm concerned, this approach is a good enough solution if the online print works.  It solves the Nvidia driver build problem and at least gives a quick path for Nvidia users to create off-line figures.  I don't know what else beyond that one could expect from Octave.

(file #42760)

I saw the added OpenGL query yesterday and rebuilt.  So that is a realtime test that can be used.  Although the library info is not displayed when the figure is not visible, I would think that info could be gotten in any case by creating an unused context at launch.

I'm building OSMesa from scratch and still have the problem we're seeing.

I wouldn't mind flashing during building, be it temporary, but the reason most other people looking for a similar solution to their projects is that they are building on a system that has no graphics, e.g., server.

Ubuntu 14.04 and Linux Mint 17 users should either compile their own OSMesa or avoid using OSMesa with Octave altogether. Any results you get on either of those distributions are likely to be skewed by a broken libosmesa6 package. See https://bugs.launchpad.net/bugs/1551028

I'm hoping this reduces confusion, not adds to it. This has nothing to do with whether nvidia is used or not, I'm just noting here that OSMesa on these distributions will not work in Octave, unless you upgrade to Ubuntu 16.04 or Linux Mint 18 (not yet released).

Slow is better than the documentation failing every time one tries to build Octave.  I realize the bug is more about _osmesa_print_ crashing than the documentation, but disabling that test for the short term is an easy solution.  Long term there is going to need to be some work.

I'm going to try and see if I can use GLEW to at least recognize the presence of framebuffer support.  If that can be done, then that is an easy runtime conditional that can be used anywhere in Octave, be it _osmesa_print_, to check whether we should expect malfunction if OSMesa OpenGL commands are used.

This result

https://savannah.gnu.org/bugs/?44478#comment93

shows that your sample program will work when just OSMesa is linked.  For the Octave sample we've been using:

[launch octave]

graphics_toolkit ('qt')
figure visible off
sombrero
print invisible.pdf

isn't it the case that the only OpenGL context created will be

OSMesaCreateContext()
OSMesaMakeCurrent()

?  Of course, if the user were to make this visible it would crash...or wait, would it even link?  Well, for octave-os, one could switch around -lOSMesa -lGL (as done in a previous comment).  The point is, the user will never use octave-os...it would be only for creating the documentation and nothing else.

Dynamically loading would probably be a solution.  Not a real good one, I suppose.  It would mean deactivating the current graphics context altogether.

The standalone program approach would work, but I'm more partial to the framebuffers approach for a couple reasons, after thinking about this a bit.  First, the framebuffers is pretty seemless and doesn't require any type of context switching.  Second, and probably the bigger reason, is that framebuffers seems more OpenGL standard-friendly, so if there is any entity that is likely to change direction on this it is Mesa.  They may well decide at some point to take OSMesa and shape it into a framebuffer mold.  If that does happen, then Octave won't have to change anything because it would find framebuffer support in the future Mesa driver and not call OSMesaCreateContext() which will have to have come from an older version of Mesa at that point.

In the short term, if anything, what makes sense to me is simply creating a version of Octave that only links in -lOSMesa and not -lGL used for creating the documentation at build time.  Otherwise, the consequence of this problem is small because 99% of the time I would guess the user has the figure visible when attempting to print, and that works.  It's only when off-screen rendering is needed for documentation that this is critical.  It would be pretty easy to have two slightly different links.  And then if framebuffers or something else proves to be robust, take out that build command that links only -lOSMesa to create an octave-os.

Instead of using framebuffers, my thought was to write a standalone program that links only with OSMesa and not GL.  Then this external program acts as a filter, accepting the graphics properties that define the plot and generating PostScript (or whatever) to be read by Octave.  The standalone program can use Octave libraries, it just can't to be linked with libGL or the GUI libraries (which would indirectly cause libGL to be linked).

That quote from Brian on freedesktop.org seems definitive.

Instead of getting OSMesa and Nvidia libgl to work together, should the focus of this issue change to detecting that the condition exists and avoiding it if it does?

I understand what you've done as far as linking.  However, as far as I understand, OpenGL routines are slightly different.  All the glGetIntegerv(), etc. are not directly callable functions from the application.  One has to create a valid context and then the OpenGL routines are mapped somehow so they can be called by the application.

I would think for OpenGL linking order shouldn't matter in theory (but of course, that's currently the case).

  OSMesaContext ctx = OSMesaCreateContextExt (OSMESA_RGBA, 16, 0, 0, NULL);

should know the GL routines to map into the context.

Anyway, I have Mesa (with OSMesa) compiling, so I'm able to make modifications to OSMesa, print out variables, etc. under the various scenarios.  If you have something in mind I should try, let me know.

All I can add right now is a confirmation that the OSMesaCreateContext() seems to go through the same program flow no matter the order of library linking.  But, when it is -lGL -lOSMesa it's clear that the Mesa glGetIntegerv is not being called.

I can't find init in the OSMesa code (there's an __init_), so I'm assuming that _init is present in all library files for dynamic loading or something.  OK, that one is out as a possible conflict.  I think I'll just have to build OSMesa and debug...hopefully is isn't a involved as the swrast and gallium drivers.

In fact, the OpenGL video does act flaky.  Here is the result for John's second example:

sebald@ ~/octave/opengl_nvidia $ g++ osmesa-fltk-foo-mod.cc -lGL -lOSMesa -lfltk -lfltk_gl
sebald@ ~/octave/opengl_nvidia $ a.out
GL_DEPTH_BITS: 24
GL_STENCIL_BITS: 0
GL_DEPTH_BITS: 1361941888
GL_STENCIL_BITS: 32766
sebald@ ~/octave/opengl_nvidia $ g++ osmesa-fltk-foo-mod.cc -lOSMesa -lGL -lfltk -lfltk_gl
sebald@ ~/octave/opengl_nvidia $ a.out
Segmentation fault

In the first case -lGL -lOSMesa, the FLTK X appears in a window and I close the window and it continues.  In the second case, there's a blip of a window appearing in the upper left corner, it disappears and Seg-fault appears.

I think there are some inadvertently shared variables and or functions between nVidia and Mesa.  That, or both started from a similar code base and programmers didn't bother to change variable/function names.

Any ideas?  All I can think of is to compare the global name's in the two libraries for what might be common, or compile OSMesa attempting to iteratively debug where exactly OSMesa is failing in the test programs and Octave.

Is the order of -l options important?  For example, is the first listed library searched first to resolve symbols?  In that case, I would hope that

g++ osmesa-foo-mod.cc -lOSMesa -lGL

might work.

Yes, I think the multiple function names is the issue.  Check this out:

sebald@ ~/octave/opengl_nvidia $ g++ osmesa-foo-mod.cc -lGL -lOSMesa
sebald@ ~/octave/opengl_nvidia $ a.out
CTX: 0x7fad95c9b010
GL_DEPTH_BITS: 0
GL_STENCIL_BITS: 0
sebald@ ~/octave/opengl_nvidia $ g++ osmesa-foo-mod.cc -lOSMesa
sebald@ ~/octave/opengl_nvidia $ a.out
CTX: 0x7f28624d6010
GL_DEPTH_BITS: 16
GL_STENCIL_BITS: 0

I didn't even have to link in the OpenGL library to make the glGetIntegerv function work properly.  Whether that means the whole OSMesa will function properly, I don't know.  But, is it possible to do a partial link and resolve the function calls in the _osmesa_print_.o object using -lOSMesa and then the rest of Octave using -lGL?  Then combine?

Responding to comment #90, wouldn't having two identical function names in two different libraries produce some sort of warning or error during the link phase?  Or if it doesn't, is there a way to turn on verbose messages during the link to make that happen?

If this is really about the link process rather than the code then it might be good to take jwe's example and separate it out into a compile stage command and a link command so that you could focus on the piece that was actually the problem.

"It works fine for me when linked with Mesa libs (GL and OSMesa), but not when linked with the nvida OpenGL lib."

Do you think the issue is simply that OSMesa only works well with Mesa GL because Mesa GL has knowledge of OSMesa's operation somehow?  Something that another library (in this case nVidia) can't know about?  If so, perhaps it is futile trying to solve this through coding, but instead maybe get lucky during the Octave config process to point to the proper Mesa GL library when linking, if it is present rather than just -lGL.

There is the OpenGL extensions I mentioned the other day.  (But that may be limited to nVidia).  Any other approach available?

https://www.opengl.org/discussion_boards/showthread.php/165359-Offscreen-rendering

http://stackoverflow.com/questions/12157646/how-to-render-offscreen-on-opengl

It seems to me the "figure visible off" should have a work-around because there otherwise is a functioning OpenGL window environment.  But in the case of generating documentation, I'm assuming folks want to use open GL and no external OpenGL windowing.

How about some version of Octave octave-os (off-screen) that doesn't link in -lGL but links in -lOSMesa that can be used for documentation generation?

I've got it: -lfltk -lfltk_gl

Well, I'm trying the GLEW route and right now I'm basically trying glewInit() after each of the context creations:

GLEW is fine.
GL_DEPTH_BITS: 24
GL_STENCIL_BITS: 0
GLEW init failed: Missing GL version
GL_DEPTH_BITS: -597588432
GL_STENCIL_BITS: 32764

Here's a link to what might be an explanation for the missing GL version:

http://stackoverflow.com/questions/23451435/glewinit-crashing-segfault-after-creating-osmesa-off-screen-mesa-context

Raises more questions than answers.

What are the link flags for FLTK?

Mangling the mesa symbols won't help because other libraries that we use call OpenGL functions without using the mangled names.

Your sample program is exhibiting the same problem with invalid context.  I run this and see nonsensical numbers as well.  If I make the change:

--- osmesa-foo.cc        2016-05-11 12:47:27.675647015 -0500
+++ osmesa-foo-mod.cc        2016-05-11 13:03:55.287656681 -0500
@@ -31,7 +31,7 @@
   if (! OSMesaMakeCurrent (ctx, buffer, GL_UNSIGNED_BYTE, Width, Height))
     error ("__osmesa_print__: OSMesaMakeCurrent failed!");

-  GLint z, s;
+  GLint z=0, s=0;
   glGetIntegerv (GL_DEPTH_BITS, &z);
   glGetIntegerv (GL_STENCIL_BITS, &s);

I see

GL_DEPTH_BITS: 0
GL_STENCIL_BITS: 0

It's just that in the more complicated Octave/multi-thread setting we get a seg fault rather than "no action".

But your program could be a good debugging environment because it compiles quickly and may give more traceable debugging.

Either OSMesaCreateContextExt() is not working properly, even though the return value is telling us it is.  Or, the context is constructed correctly and the switch OSMesaMakeCurrent() is not correct.  If I had to guess, I say the former (i.e., creating the context) is what fails.

The thing I'm wondering about is how the OpenGL environment is initialized, or if it needs to be done at all.  The OpenGL documentation is very vague on this saying that it is system dependent and suggesting that we should just use a window manager (e.g., Qt?) which will have done the necessary work.

In that regard, is there something that needs to be done at the start of this sample program (call some function, include some library)?

For example, take a look at this example where someone tries to call glGetIntegerv() without having even created a context:

http://stackoverflow.com/questions/6594214/glgetintegerv-returning-garbage-value

The person then adds some "glut" type of code at the front which does the context creation (apparently) for a window.  There is some magic going on there which I don't fully comprehend yet.  Is there some type of initialization that needs to be done for the nVidia OpenGL library?

A DC is a device context:

http://stackoverflow.com/questions/878452/how-do-opengl-contexts-and-device-contexts-work#comment686293_878502

"The threading issue is different - when you make a DC, you need to always use that thread for all calls into that DC. In general, this means one graphics/rendering thread per device context (ie: per window). – Reed Copsey May 18 '09 at 18:39"

I meant to finish the previous comment with:

But at least this can be used as a debugging aid when trying to diagnose why someone is having trouble with OpenGL graphics.  Now we can ask people to generate a figure and look at these properties.

This is such a PITA.  It is possible that this is a thread issue.  If the OpenGL context was created in one thread, it might not be available to the other thread where Octave is running.  When you call glGetIntegerv it probably tries to look for the configuration value at an offset from where the context is located in memory.  If the context is 0 in the current thread then this could lead to the segfault.

And, with the nvidia drivers installed, /usr/lib/x86_64-linux-gnu/libGL.so.1 ultimately points to /usr/lib/x86_64-linux-gnu/nvidia/legacy-340xx/libGL.so.340.96, so the crash is happening inside the nvidia OpenGL implementation.

So, how can the following cause a segfault?

  GLint z, s;
  glGetIntegerv (GL_DEPTH_BITS, &z);

Still there in 4.0.1-rc4

  libinterp/dldfcn/__osmesa_print__.cc-tst ....................panic: Segmentation fault -- stopping myself...
octave exited with signal 11

LD_PRELOAD=/usr/lib64/libGL.so make check
  libinterp/corefcn/givens.cc-tst ............................. PASS      6/6
  libinterp/corefcn/graphics.cc-tst ...........................libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
...
 PASS     38/38
  PASS     12999
  FAIL         0
  XFAIL       25
  SKIPPED     77

The link isn't helping because the problme is not witht he nvidia drivers but with mesa

This seems more related
https://answers.launchpad.net/ubuntu/+source/xorg-lts-transitional/+question/259177

This trick worked before but not anymore. there seems ot be something broken in libosmesa6 in Ubuntu 15.10

export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGL.so
export LIBGL_DEBUG=verbose
glxgears

libGL: screen 0 does not appear to be DRI2 capable
libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
libGL: Can't open configuration file /home/juanpi/.drirc: No such file or directory.
libGL: Can't open configuration file /home/juanpi/.drirc: No such file or directory.
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  154 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  35
  Current serial number in output stream:  37

Still there in 4.0.1-rc1

make check
  libinterp/dldfcn/__osmesa_print__.cc-tst ....................panic: Segmentation fault -- stopping myself...
octave exited with signal 11

LD_PRELOAD=/usr/lib64/libGL.so make check
  libinterp/corefcn/graphics.cc-tst ...........................libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
...
  PASS     12917
  FAIL         0
  XFAIL       14
  SKIPPED     77

@Juan: Are you sure the Nvidia libGL.so gets loaded and not the mesa one?

I remove Ubuntu from title because this affects all distros, not only ubuntu. See bug #45719 for a report on arch.

Until now the easiest workaround is to install nouveau drivers or use the LD_PRELOAD from comment #57.

Since the propriatary Nvidias drivers can't work with OSMesa per design we should detect if Nvidia drivers are loaded on runtime and disable OSMesa in this case.

I just tried building from the 4.0.0 release (actually, two commits after the 4.0.0 tag on the stable branch) on Ubuntu 14.10 32 bits.

The problem is still here, exactly as before.

The LD_PRELOAD trick still works as reported below.

It also works here:

Ubuntu vivid, nvidia-346

@Julien: Im able to use OSMesa with FLTK if I preload the mesa libGL.so. Can you confirm that this also works for you?

export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGL.so

my testscript:

graphics_toolkit fltk
figure ("visible", "off")
plot (1:10)
print ("-depsc", "out.eps")

This works with --no-gui, --no-gui-libs and the GUI with FLTK. When using Qt I get

libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
o-num: ../../src/xcb_io.c:274: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
panic: Aborted -- stopping myself...
attempting to save variables to 'octave-workspace'...
save to 'octave-workspace' complete
Abgebrochen

@Julien: Do you also use the proprietary nvidia legacy 304 driver? Using the Nvidia installer or via Debian/Ubuntu repos?

@Rik: Making the OpenGL context current is the responsibility of the underlying toolkit (Qt or FLTK). The toolkit takes care of creating the context and making it current in the current thread, so you can start issuing OpenGL commands. In Qt, you only override QGLWidget::paintGL; as doc says in http://doc.qt.io/qt-4.8/qglwidget.html#paintGL the context has been made current for you. The same system exists for FLTK (as in other UI toolkit).

About the question concerning FLTK and OSMesa, I don't know. Looking at the code, _osmesa__print_ creates its own context and makes it current before issuing other OpenGL commands. I'm not familiar with OSMesa, but I'd expect it to be independent of the toolkit used. My point was that your simple example might not show the same type of error as reported initially in this bug report.

@Andreas: The point is a current context in the current thread. When you use Qt, you create a current context in Qt's eventloop thread, not in octave thread. When you run simple_gl_oct, there is no valid context in octave thread. When you use FLTK, the eventloop is the octave thread, so after the "sombrero" call, there is a current context in the octave thread, the one that was used to plot the figure.

@Michael: How are we accomplishing things now?  The code in gl-render.cc has init_gl_context which sets up the context and also does the same sort of calls to glGetIntegerv that are causing a segfault.  I can't easily trace how this function is being called from the Octave thread versus the Qt thread.

This does open up another testing line.  If this is about threading and FLTK is within Octave's thread then will this work?

/run-octave -f --no-gui
graphics_toolkit fltk
sombrero
set (gcf, "visible", "off")
__osmesa_print__ (gcf, "tst.eps", "eps")

Thanks for testing.  I think Octave must be doing something wrong in the interaction with Qt and OpenGL.  It sure looks like when we call glGetInterv we are sampling a random memory location.  Sometimes this just gives garbage data, like 32,550.  And at other times the memory location is outside the memory range allocated to the process and the kernel intercepts the access and produces a segfault.

Thanks Juan.

I'm attaching a patch that builds the simple_gl_oct function into core Octave rather than having it as a dynamically linked function.  Somebody with an Nvidia driver can apply it, compile, and test as shown below.

What I notice is that I get strange results for the Qt toolkit even without the Nvidia driver.  This makes me think that maybe there is something else Octave is not doing when it initializes OpenGL through Qt.

See below for a series of test commands and the results on my computer.

graphics_toolkit qt
plot (1:10)
simple_gl_oct
ans =  32610
close all
graphics_toolkit fltk
plot (1:10)
simple_gl_oct
ans =  24

24 makes sense, 32610 does not.



(file #33615)

@Andreas: Does ordinary plotting in core Octave work with the NVidia drivers?

I notice, obviously, that both _osmesa_print_.cc and your simple example are dynamically loaded.  Maybe there are compile/linker flags that we are not setting correctly?

The first two items of Nvidia's known problems list are

OpenGL and dlopen()

    There are some issues with older versions of the glibc dynamic loader (e.g., the version that shipped with Red Hat Linux 7.2) and applications such as Quake3 and Radiant, that use dlopen(). See Chapter 7, Frequently Asked Questions for more details.

Interaction with pthreads

    Single-threaded applications that use dlopen() to load NVIDIA's libGL library, and then use dlopen() to load any other library that is linked against libpthread will crash in libGL. This does not happen in NVIDIA's new ELF TLS OpenGL libraries (see Chapter 5, Listing of Installed Components for a description of the ELF TLS OpenGL libraries). Possible workarounds for this problem are:

      1)  Load the library that is linked with libpthread before loading libGL.

      2)  Link the application with libpthread.

I can't say whether your system has an old glibc.  But for pthreads, everything seems okay.  When I do 'make clean; make V=1' I see that the lines for core Octave have -pthread and so do the dynamically loaded modules in the dldfcn/ directory.

Maybe, there is a problem with the fact that octave is itself just a small executable that exec's itself out of the way.  Instead of ./run-octave you could do 'make install' and then try directly running octave-cli-4.0.0-rc2 and your test program.

The log of make octave was removed form the upload. Here it is

@Juan: Can you include the log from running make?  I want to see the order of the -L flags when _octave_mesa_.cc is getting compiled.  From the segfault backtrace it is clearly getting linked against the Nvidia libGL which is probably not right.

Can you also get the log from running make on the osdemos?  That will tell us the correct order of flags.

@Juan: Can you build a version of octave with debugging symbols and minimal optimization and then run the simplified code from comment #33.

Here is what I use for this purpose which happens to assume a tcsh shell:

setenv CFLAGS "-g -O0 -pipe"
setenv CXXFLAGS "${CFLAGS}"
setenv FFLAGS "${CFLAGS}"

./configure --prefix=/usr/local --disable-jit |& tee myconfig.log

Also, when you run make, turn on verbose reporting and record the log.

make V=1 |& tee mymake.log

It may be that the linker flags are not set correctly so that the wrong library is being preferred.

Finally, to run

./run-octave -g -f --no-gui
# at gdb prompt
run
# at octave prompt
sombrero
__osmesa_print__ (1)
# at gdb prompt after segfault
bt

I expect the backtrace to show that it is the glEnable call that leads to the segfault, but I'm wondering if it will give a clue as to which library it is going in to when it fails.

Juan Pablo, have you build Octave using LLVM? This is just a wild guess but if I read http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/mesa_10.3.0-0ubuntu3.dsc it looks like the libosmesa in Ubuntu 14.10 was built with llvm-3.5. There is a "classic" OSMesa and a gallium llvmpipe state tracker. Can this conflict with the nvidia driver?

If you've built with llvm, can you try a "--disable-jit" built?
Perhaps ypu want to try classic libosmesa (http://www.paraview.org/Wiki/ParaView/ParaView_And_Mesa_3D#Installing_Classic_OSMesa).

@Juan: Could you try the minimal__osmesa_print__.cc file I'm attaching?  Copy it to the libinter/dldfcn directory and rename it to _osmesa_print_.cc.

It avoids using OCTAVE_LOCAL_BUFFER and directly calls malloc to get a buffer.  It also prints out the buffer pointer.  According to OSMESA this must be 4-byte aligned.  Maybe OCTAVE_LOCAL_BUFFER is not doing that.

To test:

sombrero
__osmesa_print__ (gcf)

(file #33343)

Hi Juan Pablo, you get a segfault even if you remove the opengl_renderer and/or glps_renderer instantiation and the call of the draw method?

@Rik, I was wrong when I said that I couldn't execute

test __osmesa_print__.cc

The savannah system has problems managing the character

_

when it is written as normal text instead of in a verbatim environment, so when I copy and paste your instructions I got the "not exist in path" message.

Well, I still obtain

./run-octave -f --no-gui
octave:1> test __osmesa_print__.cc
panic: Segmentation fault -- stopping myself...
Segmentation fault

Rik,

Thanks for the instructions.
I tried them in my system as well. I still get the segmentation fault

> test _osmesa_print_.cc

panic: Segmentation fault -- stopping myself...
Segmentation fault (core dumped)

I tried the modification in the mercurial repository as well and got the same result.

I dive into the code and compare it with the demo (see comment #12) that runs on my computer without a problem.

There is no need to Enable the GL_*TEST to retrieve the values of GL_DEPTH_BITS, GL_STENCIL_BITS and GL_ACCUM_RED_BITS.

And I do not think it is a good idea to disable them before doing the actual rendering.

I checked that replacing the call to OCTAVE_LOCAL_BUFFER with the code given in the demo gives the same result.
I also tried to do the retrieval of the values z and s in a local scope, as in the demo... nothing.

If you comment out all this testing part, then the segmentation fault appears later.

I guess the segmentation fault has nothing to do with this chuck of code, but it is something in the configuration of the render that triggers this issue.

We might be barking at the wrong tree.

@Rik, I suppose you mean test osmesa_print instead of test osmesa_print.cc (which results in ????? osmesa_print.cc does not exist in path)

With test osmesa_print I obtain again a segmentation fault (and also running make check)

@Rik, I'm using the 4.0.0-rc1 tarball

Rik, thank you for the explanation

The output for test _osmesa_print_.cc-tst is

octave:1> test __osmesa_print__.cc-tst

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd6756700 (LWP 14291)]
0x00007fffc9ada9a2 in ?? () from /usr/lib/x86_64-linux-gnu/libOSMesa.so.8

The output for bt in gdb is

(gdb) bt
#0  0x00007fffc9ada9a2 in ?? () from /usr/lib/x86_64-linux-gnu/libOSMesa.so.8
#1  0x00007fffc9adc65a in ?? () from /usr/lib/x86_64-linux-gnu/libOSMesa.so.8
#2  0x00007fffc9fbf6f2 in F__osmesa_print__ (args=...) at dldfcn/__osmesa_print__.cc:160
#3  0x00007ffff6ce41f5 in octave_builtin::do_multi_index_op (this=0x7fffd0691f50, nargout=0, args=..., lvalue_list=<optimized out>) at octave-value/ov-builtin.cc:131
Python Exception <type 'exceptions.ValueError'> Cannot find type const std::list<octave_value_list, std::allocator<octave_value_list> >::_Node:
#4  0x00007ffff6ce28f4 in octave_builtin::subsref (this=<optimized out>, type="(", idx=std::list, nargout=0, lvalue_list=lvalue_list@entry=0x0) at octave-value/ov-builtin.cc:63
#5  0x00007ffff6ce38cc in octave_builtin::subsref (this=<optimized out>, type=..., idx=..., nargout=<optimized out>) at octave-value/ov-builtin.cc:46
#6  0x00007ffff6de2bf9 in octave_value::subsref (this=<optimized out>, type=..., idx=..., nargout=<optimized out>) at octave-value/ov.cc:1302
Python Exception <type 'exceptions.ValueError'> Cannot find type const std::list<octave_value_list, std::allocator<octave_value_list> >::_Node:
#7  0x00007ffff6de2c75 in octave_value::subsref (this=this@entry=0x7fffd6753020, type="(", idx=std::list, nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0)
    at octave-value/ov.cc:1313
#8  0x00007ffff6e6adbd in tree_index_expression::rvalue (this=this@entry=0x7fffd04908c0, nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0) at parse-tree/pt-idx.cc:436
#9  0x00007ffff6e6bc33 in rvalue (nargout=0, this=0x7fffd04908c0) at parse-tree/pt-idx.cc:284
#10 tree_index_expression::rvalue1 (this=0x7fffd04908c0, nargout=0) at parse-tree/pt-idx.cc:466
#11 0x00007ffff6e505ad in tree_evaluator::visit_statement (this=0x7ffff79695a0 <std_evaluator>, stmt=...) at parse-tree/pt-eval.cc:744
#12 0x00007ffff6e4fbe1 in tree_evaluator::visit_statement_list (this=0x7ffff79695a0 <std_evaluator>, lst=...) at parse-tree/pt-eval.cc:794
#13 0x00007ffff6ddcbf3 in octave_user_function::do_multi_index_op (this=0x7fffd0561170, nargout=0, _args=..., lvalue_list=0x0) at octave-value/ov-usr-fcn.cc:626
Python Exception <type 'exceptions.ValueError'> Cannot find type const std::list<octave_value_list, std::allocator<octave_value_list> >::_Node:
#14 0x00007ffff6dd8158 in octave_user_function::subsref (this=0x7fffd0561170, type="(", idx=std::list, nargout=0, lvalue_list=lvalue_list@entry=0x0) at octave-value/ov-usr-fcn.cc:438
#15 0x00007ffff6dd835c in octave_user_function::subsref (this=<optimized out>, type=..., idx=..., nargout=<optimized out>) at octave-value/ov-usr-fcn.cc:421
#16 0x00007ffff6de2bf9 in octave_value::subsref (this=<optimized out>, type=..., idx=..., nargout=<optimized out>) at octave-value/ov.cc:1302
Python Exception <type 'exceptions.ValueError'> Cannot find type const std::list<octave_value_list, std::allocator<octave_value_list> >::_Node:
#17 0x00007ffff6de2c75 in octave_value::subsref (this=this@entry=0x7fffd6753ac0, type="(", idx=std::list, nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0)
    at octave-value/ov.cc:1313
#18 0x00007ffff6e6adbd in tree_index_expression::rvalue (this=<optimized out>, nargout=0, lvalue_list=lvalue_list@entry=0x0) at parse-tree/pt-idx.cc:436
#19 0x00007ffff6e6bd4b in tree_index_expression::rvalue (this=<optimized out>, nargout=<optimized out>) at parse-tree/pt-idx.cc:284
#20 0x00007ffff6ea67cb in eval_string (eval_str=" __test__( );", silent=silent@entry=false, parse_status=@0x7fffd6753fd4: 0, nargout=nargout@entry=0) at parse-tree/oct-parse.yy:4845
#21 0x00007ffff6ea6a9d in eval_string (arg=..., silent=silent@entry=false, parse_status=@0x7fffd6753fd4: 0, nargout=nargout@entry=0) at parse-tree/oct-parse.yy:4898
#22 0x00007ffff6ea7202 in Feval (args=..., nargout=0) at parse-tree/oct-parse.yy:4964
#23 0x00007ffff6ce41f5 in octave_builtin::do_multi_index_op (this=0x7fffd010f8b0, nargout=0, args=..., lvalue_list=<optimized out>) at octave-value/ov-builtin.cc:131
Python Exception <type 'exceptions.ValueError'> Cannot find type const std::list<octave_value_list, std::allocator<octave_value_list> >::_Node:
#24 0x00007ffff6ce28f4 in octave_builtin::subsref (this=<optimized out>, type="(", idx=std::list, nargout=0, lvalue_list=lvalue_list@entry=0x0) at octave-value/ov-builtin.cc:63
#25 0x00007ffff6ce38cc in octave_builtin::subsref (this=<optimized out>, type=..., idx=..., nargout=<optimized out>) at octave-value/ov-builtin.cc:46
#26 0x00007ffff6de2bf9 in octave_value::subsref (this=<optimized out>, type=..., idx=..., nargout=<optimized out>) at octave-value/ov.cc:1302
Python Exception <type 'exceptions.ValueError'> Cannot find type const std::list<octave_value_list, std::allocator<octave_value_list> >::_Node:
#27 0x00007ffff6de2c75 in octave_value::subsref (this=this@entry=0x7fffd6754530, type="(", idx=std::list, nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0)
    at octave-value/ov.cc:1313
#28 0x00007ffff6e6adbd in tree_index_expression::rvalue (this=this@entry=0x7fffd051ad60, nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0) at parse-tree/pt-idx.cc:436
#29 0x00007ffff6e6bc33 in rvalue (nargout=0, this=0x7fffd051ad60) at parse-tree/pt-idx.cc:284
#30 tree_index_expression::rvalue1 (this=0x7fffd051ad60, nargout=0) at parse-tree/pt-idx.cc:466
#31 0x00007ffff6e505ad in tree_evaluator::visit_statement (this=0x7ffff79695a0 <std_evaluator>, stmt=...) at parse-tree/pt-eval.cc:744
#32 0x00007ffff6e4fbe1 in tree_evaluator::visit_statement_list (this=0x7ffff79695a0 <std_evaluator>, lst=...) at parse-tree/pt-eval.cc:794
#33 0x00007ffff6e5052a in tree_evaluator::visit_statement (this=0x7ffff79695a0 <std_evaluator>, stmt=...) at parse-tree/pt-eval.cc:718
#34 0x00007ffff6e4fbe1 in tree_evaluator::visit_statement_list (this=0x7ffff79695a0 <std_evaluator>, lst=...) at parse-tree/pt-eval.cc:794
#35 0x00007ffff6e5123c in tree_evaluator::visit_try_catch_command (this=0x7ffff79695a0 <std_evaluator>, cmd=...) at parse-tree/pt-eval.cc:907
#36 0x00007ffff6e5052a in tree_evaluator::visit_statement (this=0x7ffff79695a0 <std_evaluator>, stmt=...) at parse-tree/pt-eval.cc:718
#37 0x00007ffff6e4fbe1 in tree_evaluator::visit_statement_list (this=0x7ffff79695a0 <std_evaluator>, lst=...) at parse-tree/pt-eval.cc:794
#38 0x00007ffff6e5052a in tree_evaluator::visit_statement (this=0x7ffff79695a0 <std_evaluator>, stmt=...) at parse-tree/pt-eval.cc:718
#39 0x00007ffff6e4fbe1 in tree_evaluator::visit_statement_list (this=0x7ffff79695a0 <std_evaluator>, lst=...) at parse-tree/pt-eval.cc:794
#40 0x00007ffff6e530ee in tree_evaluator::visit_unwind_protect_command (this=0x7ffff79695a0 <std_evaluator>, cmd=...) at parse-tree/pt-eval.cc:1045
#41 0x00007ffff6e5052a in tree_evaluator::visit_statement (this=0x7ffff79695a0 <std_evaluator>, stmt=...) at parse-tree/pt-eval.cc:718
---Type <return> to continue, or q <return> to quit---
#42 0x00007ffff6e4fbe1 in tree_evaluator::visit_statement_list (this=0x7ffff79695a0 <std_evaluator>, lst=...) at parse-tree/pt-eval.cc:794
#43 0x00007ffff6e5240a in tree_evaluator::visit_simple_for_command (this=0x7ffff79695a0 <std_evaluator>, cmd=...) at parse-tree/pt-eval.cc:349
#44 0x00007ffff6e5052a in tree_evaluator::visit_statement (this=0x7ffff79695a0 <std_evaluator>, stmt=...) at parse-tree/pt-eval.cc:718
#45 0x00007ffff6e4fbe1 in tree_evaluator::visit_statement_list (this=0x7ffff79695a0 <std_evaluator>, lst=...) at parse-tree/pt-eval.cc:794
#46 0x00007ffff6ddcbf3 in octave_user_function::do_multi_index_op (this=0x7fffd0531f60, nargout=0, _args=..., lvalue_list=0x0) at octave-value/ov-usr-fcn.cc:626
Python Exception <type 'exceptions.ValueError'> Cannot find type const std::list<octave_value_list, std::allocator<octave_value_list> >::_Node:
#47 0x00007ffff6dd8158 in octave_user_function::subsref (this=0x7fffd0531f60, type="(", idx=std::list, nargout=0, lvalue_list=lvalue_list@entry=0x0) at octave-value/ov-usr-fcn.cc:438
#48 0x00007ffff6dd835c in octave_user_function::subsref (this=<optimized out>, type=..., idx=..., nargout=<optimized out>) at octave-value/ov-usr-fcn.cc:421
#49 0x00007ffff6de2bf9 in octave_value::subsref (this=<optimized out>, type=..., idx=..., nargout=<optimized out>) at octave-value/ov.cc:1302
Python Exception <type 'exceptions.ValueError'> Cannot find type const std::list<octave_value_list, std::allocator<octave_value_list> >::_Node:
#50 0x00007ffff6de2c75 in octave_value::subsref (this=this@entry=0x7fffd6755700, type="(", idx=std::list, nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0)
    at octave-value/ov.cc:1313
#51 0x00007ffff6e6adbd in tree_index_expression::rvalue (this=this@entry=0x7fffd01a9ef0, nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0) at parse-tree/pt-idx.cc:436
#52 0x00007ffff6e6bc33 in rvalue (nargout=0, this=0x7fffd01a9ef0) at parse-tree/pt-idx.cc:284
#53 tree_index_expression::rvalue1 (this=0x7fffd01a9ef0, nargout=0) at parse-tree/pt-idx.cc:466
#54 0x00007ffff6e505ad in tree_evaluator::visit_statement (this=0x7ffff79695a0 <std_evaluator>, stmt=...) at parse-tree/pt-eval.cc:744
#55 0x00007ffff6e4fbe1 in tree_evaluator::visit_statement_list (this=0x7ffff79695a0 <std_evaluator>, lst=...) at parse-tree/pt-eval.cc:794
#56 0x00007ffff73a420d in main_loop () at corefcn/toplev.cc:599
#57 0x00007ffff68c3139 in octave_execute_interpreter () at octave.cc:917
#58 0x00007ffff7a783ec in octave_cli_thread::run (this=0x7fffffffb810) at src/octave-gui.cc:80
#59 0x00007ffff298670f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#60 0x00007ffff44ed0a4 in start_thread (arg=0x7fffd6756700) at pthread_create.c:309
#61 0x00007ffff422204d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

I've done the same steps as in comment #10. First of all, I've invoked Octave as

./run-octave -g --no-gui

but gdb is started instead of octave, so I can't run any octave command. I suppose the correct call in comment #10 is
+verbatin+
./run-octave -d --no-gui
-verbatim-
this starts octave printing on the screen LOTS of info

Then, I execute

test __osmesa_print__.cc-tst

this print on screen again LOTS of info lines, but finally octave breaks with

panic: Segmentation fault -- stopping myself...
Segmentation fault

As I explained in my previous comment, I'm using Debian Sid. I've installed the official NVIDIA driver (*.run file from nvidia.com), but I've changed the GL and GLES libraries links to the Debian libgl1-mesa-swx11, libgl1-mesa-swx11-dev, libgles1-mesa and libgles2-mesa.

topo@geo-w540:~/Desktop/octave-4.0.0-rc1$ ls -l /usr/lib/x86_64-linux-gnu/libGL*
-rw-r--r-- 1 root root 7700082 ene 15 15:02 /usr/lib/x86_64-linux-gnu/libGL.a
lrwxrwxrwx 1 root root      17 mar 12 12:39 /usr/lib/x86_64-linux-gnu/libGLESv1_CM.so -> libGLESv1_CM.so.1
lrwxrwxrwx 1 root root      21 mar 12 12:39 /usr/lib/x86_64-linux-gnu/libGLESv1_CM.so.1 -> libGLESv1_CM.so.1.1.0
-rw-r--r-- 1 root root   18232 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libGLESv1_CM.so.1.1.0
-rwxr-xr-x 1 root root   51096 feb 27 15:24 /usr/lib/x86_64-linux-gnu/libGLESv1_CM.so.346.47
lrwxrwxrwx 1 root root      14 mar 12 12:39 /usr/lib/x86_64-linux-gnu/libGLESv2.so -> libGLESv2.so.2
lrwxrwxrwx 1 root root      18 mar 12 12:39 /usr/lib/x86_64-linux-gnu/libGLESv2.so.2 -> libGLESv2.so.2.0.0
-rw-r--r-- 1 root root   26424 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libGLESv2.so.2.0.0
-rwxr-xr-x 1 root root   75416 feb 27 15:24 /usr/lib/x86_64-linux-gnu/libGLESv2.so.346.47
lrwxrwxrwx 1 root root      17 nov 22  2013 /usr/lib/x86_64-linux-gnu/libGLEW.so.1.10 -> libGLEW.so.1.10.0
-rw-r--r-- 1 root root  554088 nov 22  2013 /usr/lib/x86_64-linux-gnu/libGLEW.so.1.10.0
-rw-r--r-- 1 root root     671 feb 27 15:24 /usr/lib/x86_64-linux-gnu/libGL.la
lrwxrwxrwx 1 root root      10 mar 12 12:39 /usr/lib/x86_64-linux-gnu/libGL.so -> libGL.so.1
lrwxrwxrwx 1 root root      14 mar 12 12:39 /usr/lib/x86_64-linux-gnu/libGL.so.1 -> libGL.so.1.6.0
-rw-r--r-- 1 root root 4330248 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libGL.so.1.6.0
-rwxr-xr-x 1 root root 1317432 feb 27 15:24 /usr/lib/x86_64-linux-gnu/libGL.so.346.47
-rw-r--r-- 1 root root  901120 sep 19  2013 /usr/lib/x86_64-linux-gnu/libGLU.a
lrwxrwxrwx 1 root root      15 sep 19  2013 /usr/lib/x86_64-linux-gnu/libGLU.so -> libGLU.so.1.3.1
lrwxrwxrwx 1 root root      15 sep 19  2013 /usr/lib/x86_64-linux-gnu/libGLU.so.1 -> libGLU.so.1.3.1
-rw-r--r-- 1 root root  449144 sep 19  2013 /usr/lib/x86_64-linux-gnu/libGLU.so.1.3.1

The OSMesa libraries I've installed are:

topo@geo-w540:~/Desktop/octave-4.0.0-rc1$ ls -l /usr/lib/x86_64-linux-gnu/libOS*
lrwxrwxrwx 1 root root      18 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libOSMesa.so -> libOSMesa.so.8.0.0
lrwxrwxrwx 1 root root      14 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libOSMesa.so.6 -> libOSMesa.so.8
lrwxrwxrwx 1 root root      18 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libOSMesa.so.8 -> libOSMesa.so.8.0.0
-rw-r--r-- 1 root root 3889448 ene 15 15:02 /usr/lib/x86_64-linux-gnu/libOSMesa.so.8.0.0

In this way, I can use the fltk and qt driver for plotting (the function sombrero() works perfectly), but the test _osmesa_print_.cc-tst call fails as I explained

Grrr, I've had written a long comment, but I don't know why it wasn't complete published. I'll try to reproduce it:

--------------

Then, the GL and GLES libraries point to the ones installed with libgl1-mesa-swx11, libgl1-mesa-swx11-dev, libgles1-mesa and libgles2-mesa and I can use FLTK:

+varbatim+
-rw-r--r-- 1 root root 7700082 ene 15 15:02 /usr/lib/x86_64-linux-gnu/libGL.a
lrwxrwxrwx 1 root root      17 mar 10 15:43 /usr/lib/x86_64-linux-gnu/libGLESv1_CM.so -> libGLESv1_CM.so.1
lrwxrwxrwx 1 root root      21 mar 10 15:43 /usr/lib/x86_64-linux-gnu/libGLESv1_CM.so.1 -> libGLESv1_CM.so.1.1.0
-rw-r--r-- 1 root root   18232 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libGLESv1_CM.so.1.1.0
-rwxr-xr-x 1 root root   51096 feb 27 15:24 /usr/lib/x86_64-linux-gnu/libGLESv1_CM.so.346.47
lrwxrwxrwx 1 root root      14 mar 10 15:43 /usr/lib/x86_64-linux-gnu/libGLESv2.so -> libGLESv2.so.2
lrwxrwxrwx 1 root root      18 mar 10 15:43 /usr/lib/x86_64-linux-gnu/libGLESv2.so.2 -> libGLESv2.so.2.0.0
-rw-r--r-- 1 root root   26424 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libGLESv2.so.2.0.0
-rwxr-xr-x 1 root root   75416 feb 27 15:24 /usr/lib/x86_64-linux-gnu/libGLESv2.so.346.47
-rw-r--r-- 1 root root     671 feb 27 15:24 /usr/lib/x86_64-linux-gnu/libGL.la
lrwxrwxrwx 1 root root      10 mar 10 15:43 /usr/lib/x86_64-linux-gnu/libGL.so -> libGL.so.1
lrwxrwxrwx 1 root root      14 mar 10 15:43 /usr/lib/x86_64-linux-gnu/libGL.so.1 -> libGL.so.1.6.0
-rw-r--r-- 1 root root 4330248 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libGL.so.1.6.0
-rwxr-xr-x 1 root root 1317432 feb 27 15:24 /usr/lib/x86_64-linux-gnu/libGL.so.346.47
-verbatim-

So now I have installed Octave 4.0 and I've installed the osmesa libraries

lrwxrwxrwx 1 root root      18 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libOSMesa.so -> libOSMesa.so.8.0.0
lrwxrwxrwx 1 root root      14 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libOSMesa.so.6 -> libOSMesa.so.8
lrwxrwxrwx 1 root root      18 ene 15 15:01 /usr/lib/x86_64-linux-gnu/libOSMesa.so.8 -> libOSMesa.so.8.0.0
-rw-r--r-- 1 root root 3889448 ene 15 15:02 /usr/lib/x86_64-linux-gnu/libOSMesa.so.8.0.0

Apparently they don't depend on NVIDIA driver, but I obtain the osmesa crash in make check

Well we haven't yet proven that it segfaults outside of Octave. That was what Andy was trying to test with the mesa demos he asked to be run. But those example programs apparently do not cause a segfault using OSMesa with Nvidia.

Thanks to Juan Pablo for confirming, it seems pretty certain now that this segfault is constrained to systems using the Nvidia proprietary drivers.

It occurred to me that this might have something to do with Nvidia's replacement OpenGL libraries (for those that haven't experienced Nvidia, it requires also installing Nvidia's own version of libGL, etc).

Some light searching does yield a few hits on problems with OSMesa and Nvidia.

I have re-built octave-4.0.0-rc1 after removing the nvidia driver and uninstalling self-built OSMesa.
(Ubuntu 14.04 LTS 64 bit ships OSMesa whose version is 10.5.0.)

octave:1> test __osmesa_print__.cc-tst
PASSES 2 out of 2 tests

Thanks all the help!

To Anonymous

I have built osmesa-10.4.6 and built octave against it.

./run-octave -g --no-gui
 :
octave:1> test __osmesa_print__.cc-tst

octave:1> test __osmesa_print__.cc-tst

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd607b700 (LWP 22857)]
0x00007ffff3242569 in glGetIntegerv () from /usr/lib/nvidia-304/libGL.so.1
(gdb) bt
#0  0x00007ffff3242569 in glGetIntegerv () from /usr/lib/nvidia-304/libGL.so.1
#1  0x00007fffc56ac6b2 in F__osmesa_print__ (args=...)
    at ../../octave-4.0.0-rc1/libinterp/dldfcn/__osmesa_print__.cc:160
#2  0x00007ffff6e8055c in octave_builtin::do_multi_index_op (
    this=0x7fffc077d270, nargout=0, args=..., lvalue_list=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-builtin.cc:131
#3  0x00007ffff6e7f791 in octave_builtin::subsref (this=0x7fffc077d270,
    type=..., idx=..., nargout=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-builtin.cc:63
#4  0x00007ffff6e7fe3c in octave_builtin::subsref (this=<optimized out>,
    type=..., idx=..., nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-builtin.cc:46
#5  0x00007ffff6f61399 in octave_value::subsref (this=<optimized out>,
    type=..., idx=..., nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov.cc:1302
#6  0x00007ffff6f61415 in octave_value::subsref (
    this=this@entry=0x7fffd6077fe0, type=..., idx=...,
    nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov.cc:1313
#7  0x00007ffff6fce5d8 in tree_index_expression::rvalue (this=<optimized out>,
    nargout=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:436
#8  0x00007ffff6fcf0cb in tree_index_expression::rvalue (this=<optimized out>,
---Type <return> to continue, or q <return> to quit---
    nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:284
#9  0x00007ffff6fcb61e in tree_index_expression::rvalue1 (this=0x7fffc05f7b50,
    nargout=0) at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:466
#10 0x00007ffff6fc26f8 in tree_evaluator::visit_statement (
    this=<optimized out>, stmt=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:744
#11 0x00007ffff6fc1d41 in tree_evaluator::visit_statement_list (
    this=0x7ffff7977140 <std_evaluator>, lst=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:794
#12 0x00007ffff6f585e3 in octave_user_function::do_multi_index_op (
    this=0x7fffc05688a0, nargout=0, _args=..., lvalue_list=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-usr-fcn.cc:626
#13 0x00007ffff6f54885 in octave_user_function::subsref (this=0x7fffc05688a0,
    type=..., idx=..., nargout=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-usr-fcn.cc:438
#14 0x00007ffff6f54a4c in octave_user_function::subsref (this=<optimized out>,
    type=..., idx=..., nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-usr-fcn.cc:421
#15 0x00007ffff6f61399 in octave_value::subsref (this=<optimized out>,
    type=..., idx=..., nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov.cc:1302
#16 0x00007ffff6f61415 in octave_value::subsref (
---Type <return> to continue, or q <return> to quit---
    this=this@entry=0x7fffd6078aa0, type=..., idx=...,
    nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov.cc:1313
#17 0x00007ffff6fce5d8 in tree_index_expression::rvalue (this=<optimized out>,
    nargout=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:436
#18 0x00007ffff6fcf0cb in tree_index_expression::rvalue (this=<optimized out>,
    nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:284
#19 0x00007ffff6fffc90 in eval_string (eval_str=...,
    silent=silent@entry=false, parse_status=@0x7fffd6078fbc: 0,
    nargout=nargout@entry=0) at parse-tree/oct-parse.yy:4845
#20 0x00007ffff6fffffd in eval_string (arg=..., silent=silent@entry=false,
    parse_status=@0x7fffd6078fbc: 0, nargout=nargout@entry=0)
    at parse-tree/oct-parse.yy:4898
#21 0x00007ffff7000119 in Feval (args=..., nargout=0)
    at parse-tree/oct-parse.yy:4964
#22 0x00007ffff6e8055c in octave_builtin::do_multi_index_op (
    this=0x7fffc010ed60, nargout=0, args=..., lvalue_list=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-builtin.cc:131
#23 0x00007ffff6e7f791 in octave_builtin::subsref (this=0x7fffc010ed60,
    type=..., idx=..., nargout=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-builtin.cc:63
---Type <return> to continue, or q <return> to quit---
#24 0x00007ffff6e7fe3c in octave_builtin::subsref (this=<optimized out>,
    type=..., idx=..., nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-builtin.cc:46
#25 0x00007ffff6f61399 in octave_value::subsref (this=<optimized out>,
    type=..., idx=..., nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov.cc:1302
#26 0x00007ffff6f61415 in octave_value::subsref (
    this=this@entry=0x7fffd6079520, type=..., idx=...,
    nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov.cc:1313
#27 0x00007ffff6fce5d8 in tree_index_expression::rvalue (this=<optimized out>,
    nargout=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:436
#28 0x00007ffff6fcf0cb in tree_index_expression::rvalue (this=<optimized out>,
    nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:284
#29 0x00007ffff6fcb61e in tree_index_expression::rvalue1 (this=0x7fffc05e8c20,
    nargout=0) at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:466
#30 0x00007ffff6fc26f8 in tree_evaluator::visit_statement (
    this=<optimized out>, stmt=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:744
#31 0x00007ffff6fc1d41 in tree_evaluator::visit_statement_list (
    this=0x7ffff7977140 <std_evaluator>, lst=...)
---Type <return> to continue, or q <return> to quit---
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:794
#32 0x00007ffff6fc2635 in tree_evaluator::visit_statement (
    this=0x7ffff7977140 <std_evaluator>, stmt=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:718
#33 0x00007ffff6fc1d41 in tree_evaluator::visit_statement_list (
    this=0x7ffff7977140 <std_evaluator>, lst=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:794
#34 0x00007ffff6fc389a in tree_evaluator::visit_try_catch_command (
    this=0x7ffff7977140 <std_evaluator>, cmd=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:907
#35 0x00007ffff6fc2635 in tree_evaluator::visit_statement (
    this=0x7ffff7977140 <std_evaluator>, stmt=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:718
#36 0x00007ffff6fc1d41 in tree_evaluator::visit_statement_list (
    this=0x7ffff7977140 <std_evaluator>, lst=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:794
#37 0x00007ffff6fc2635 in tree_evaluator::visit_statement (
    this=0x7ffff7977140 <std_evaluator>, stmt=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:718
#38 0x00007ffff6fc1d41 in tree_evaluator::visit_statement_list (
    this=0x7ffff7977140 <std_evaluator>, lst=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:794
#39 0x00007ffff6fc472e in tree_evaluator::visit_unwind_protect_command (
---Type <return> to continue, or q <return> to quit---
    this=0x7ffff7977140 <std_evaluator>, cmd=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:1045
#40 0x00007ffff6fc2635 in tree_evaluator::visit_statement (
    this=0x7ffff7977140 <std_evaluator>, stmt=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:718
#41 0x00007ffff6fc1d41 in tree_evaluator::visit_statement_list (
    this=0x7ffff7977140 <std_evaluator>, lst=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:794
#42 0x00007ffff6fc2cf2 in tree_evaluator::visit_simple_for_command (
    this=0x7ffff7977140 <std_evaluator>, cmd=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:349
#43 0x00007ffff6fc2635 in tree_evaluator::visit_statement (
    this=0x7ffff7977140 <std_evaluator>, stmt=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:718
#44 0x00007ffff6fc1d41 in tree_evaluator::visit_statement_list (
    this=0x7ffff7977140 <std_evaluator>, lst=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:794
#45 0x00007ffff6f585e3 in octave_user_function::do_multi_index_op (
    this=0x7fffc05ffeb0, nargout=0, _args=..., lvalue_list=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-usr-fcn.cc:626
#46 0x00007ffff6f54885 in octave_user_function::subsref (this=0x7fffc05ffeb0,
    type=..., idx=..., nargout=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-usr-fcn.cc:438
---Type <return> to continue, or q <return> to quit---
#47 0x00007ffff6f54a4c in octave_user_function::subsref (this=<optimized out>,
    type=..., idx=..., nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov-usr-fcn.cc:421
#48 0x00007ffff6f61399 in octave_value::subsref (this=<optimized out>,
    type=..., idx=..., nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov.cc:1302
#49 0x00007ffff6f61415 in octave_value::subsref (
    this=this@entry=0x7fffd607a700, type=..., idx=...,
    nargout=nargout@entry=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/octave-value/ov.cc:1313
#50 0x00007ffff6fce5d8 in tree_index_expression::rvalue (this=<optimized out>,
    nargout=0, lvalue_list=lvalue_list@entry=0x0)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:436
#51 0x00007ffff6fcf0cb in tree_index_expression::rvalue (this=<optimized out>,
    nargout=<optimized out>)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:284
#52 0x00007ffff6fcb61e in tree_index_expression::rvalue1 (this=0x7fffc0481160,
    nargout=0) at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-idx.cc:466
#53 0x00007ffff6fc26f8 in tree_evaluator::visit_statement (
    this=<optimized out>, stmt=...)
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:744
#54 0x00007ffff6fc1d41 in tree_evaluator::visit_statement_list (
    this=0x7ffff7977140 <std_evaluator>, lst=...)
---Type <return> to continue, or q <return> to quit---
    at ../../octave-4.0.0-rc1/libinterp/parse-tree/pt-eval.cc:794
#55 0x00007ffff73db4dd in main_loop ()
    at ../../octave-4.0.0-rc1/libinterp/corefcn/toplev.cc:599
#56 0x00007ffff6b8d89e in octave_execute_interpreter ()
    at ../../octave-4.0.0-rc1/libinterp/octave.cc:917
#57 0x00007ffff7a7c2bc in octave_cli_thread::run (this=0x7fffffffb830)
    at ../../octave-4.0.0-rc1/libgui/src/octave-gui.cc:80
#58 0x00007ffff37d432f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#59 0x00007ffff25bd182 in start_thread (arg=0x7fffd607b700)
    at pthread_create.c:312
#60 0x00007ffff62c847d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thanks Mike
First I tried:
Or set LD_LIBRARY_PATH so the mesa libGL.so is found instead of the nvidia version.

$ locate libGL.so
/usr/lib/i386-linux-gnu/mesa/libGL.so.1
/usr/lib/i386-linux-gnu/mesa/libGL.so.1.2.0
/usr/lib/nvidia-304/libGL.so
/usr/lib/nvidia-304/libGL.so.1
/usr/lib/nvidia-304/libGL.so.304.125
/usr/lib/x86_64-linux-gnu/libGL.so
/usr/lib/x86_64-linux-gnu/mesa/libGL.so
/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1
/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0
/usr/lib32/nvidia-304/libGL.so
/usr/lib32/nvidia-304/libGL.so.1
/usr/lib32/nvidia-304/libGL.so.304.125

So I have set LD_LIBRARY_PATH as:

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/mesa/:$LD_LIBRARY_PATH

./run-octave -g --no-gui
octave:1> test __osmesa_print__.cc-tst

octave:1> test __osmesa_print__.cc-tst
[New Thread 0x7fffc0a44700 (LWP 11144)]
[New Thread 0x7fffc0243700 (LWP 11145)]
libGL error: No matching fbConfigs or visuals found
[Thread 0x7fffc0a44700 (LWP 11144) exited]
[Thread 0x7fffc0243700 (LWP 11145) exited]
libGL error: failed to load driver: swrast
[Thread 0x7fffd3fff700 (LWP 10106) exited]
PASSES 2 out of 2 tests

Test passes but

libGL error: No matching fbConfigs or visuals found

appeared. I will check temporally uninstall nvidia driver and execute the test.
 
I

The function that is segfaulting according to gdb is glGetIntegerv and the function prototype says that the variable reference passed in should be of type GLint.  The code in _osmesa_print_.cc is using just int.  It is possible that the C++ int and GLint differ in size so a different amount of memory is being written by the Nvidia routines which should have been compiled against the OpenGL headers.

This is unlikely to be the source of the problem, but was bad practice anyways so I changed it (http://hg.savannah.gnu.org/hgweb/octave/rev/f48e650a68b5).

@Tatsuro, you should be able to build from Mercurial sources on Ubuntu easily.  If you can check whether this helps that would be great.

In text mode (Pressed after CTRL+ALT+F1),

./run-octave -g --no-gui
test _osmesa_print_.cc-tst

was successful.

Thanks for the backtrace on this crash. Can you try running the same test in xvfb-run?

$ sudo apt-get install xvfb
$ xvfb-run ./run-octave -g
>> test __osmesa_print__.cc-tst

This should allow Octave to be run with an X display but without a dependency on your video card.

Or can you switch to the nouveau driver temporarily to see if the crash doesn't occur?