Matlab is a bit vague about the case of "edgecolor" = "flat", but they say this
I interpret the "first vertex of each face (in the positive x and y direction)" to mean the first vertex encountered when heading away from the origin. So 1 < 2. This explains the OpenGL results just fine.

OK, your meshtst.var example is great. I substituted x and y for the input to mesh() and it suggests that the order relationship of amplitude of the position variables as opposed to the order of the position variables is what's used. I.e.,
That is, in this case 1 < 2 so it is the 1 position value that is used. Also, it's still not 100% accurate. Is it Euclidean distance, x^2 + y^2, that is used for comparison? Or some other distance like supnorm, x + y?
Why is it distance from the origin (regardless of the quadrant)? My guess would be that Matlab thought such a definition would look good in the cases of plotting symmetric 2D functions, such as the sombrero and so many other quick yet interesting plots. Otherwise the colors of the mesh/surface wouldn't be symmetric while the function is. Of course, plotting a function symmetric about somewhere other than the origin would come out with nonsymmetric colors.
gnuplot toolkit is using lines with the "palette" setting for these color mesh plots. There is no means of controlling the behavior of that. However, the interesting thing is that there appears to be a new plot element for every line in the mesh. (No wonder the gnuplot plots are somewhat sluggish to initially draw.) That means I might be able to specify an RGB linecolor for individual lines rather than indicating to simply use the palette (i.e., colormap). That opens the possibility for implementing this color blending algorithm in _gnuplot_draw_axes_.m and bypassing the palette altogether for mesh.
I'll look at this in the next few days.

For Matlab, and hence Octave, the vertex with smaller X and Y value sets the color of the segment. I'm not sure which vertex gnuplot is using for color determination. In the example given in comment #23 there are 4 vertices, but only 3 are used for coloring segments.
Vertex (1,1) : colors segments (1,1)>(1,2) and (1,1)>(2,1)
Vertex (2,1) : colors segment (2,1)>(2,2)
Vertex (1,2) : colors segment (1,2)>(2,2)

gnuplot is 1based indexing, so the toolkit adds 1, otherwise it is the same scaling. If one concentrates on the colors of the points of the mesh, they should be the same between OpenGL and gnuplot. We can pick a different formula in gnuplot for blending colors, maybe one will be similar; or maybe it is the ordering of the points that needs to change.

We'll have to see how Matlab behaves, but Octave is calculating the index as
The idx is a zerobased ranged so the idx is [0, num_colors1].

I extracted the relevant segment of the mesh from the demo and created a simpler example. The data and the file to run are attached.
The code in tst_bad_gnup_mesh.m is
This uses a 3color colormap of Red/Green/Blue so the difference is quite easy to see. This might be roundoff error.
(file #42645, file #42646)

@Dan: You're seeing the same thing I am, occasionally line segments are colored differently. Since Matlab is based on OpenGL I think it will behave the same way that the FLTK and Qt toolkits do.
Could someone following this report, who has access to Matlab, try the following code and upload the resulting image?
(file #42644)

I'm not seeing the drastic color difference you are seeing as given in your example fltk/gnuplot toolkit PNGs. I'm attaching screenshots of what I'm seeing and the cyan color is dominant in both FLTK and gnuplot toolkits.
It could be that there is a difference between the gnuplot version you are using and the one I'm using.
So the differences I'm seeing are subtle, occasional differences in a color of a line. I think that is due to the manner in which OpenGL and gnuplot choose (or is configured) to blend colors. There are a half dozen or more ways to construct a color for an element when given the corner colors of a mesh. For example, one could pick one of the corners of the mesh element, choose an average of all corners, so on. See the very last example of the following page:
http://gnuplot.sourceforge.net/demo_5.1/pm3d.html
Do we want to go into that detail? If so, we'd have to reference what Matlab is doing. And then are we assured we can achieve the exact same behavior? I mean, mesh element coloring has far more internal algorithms than the simpler construction of a color map. We could pick this or that blending scheme for both OpenGL and gnuplot and still not get a perfect match (maybe we could, but going to such lengths would mean we might as well match what Matlab is doing as well).
Notice in the fltk_toolkit_trimesh_Screenshot_from_20171213_122744.png file that the title appears in the center of the plot, suggesting that copyobj() didn't work completely correctly. Are you seeing that? If not, it might be an issue with a race condition of when items are updated versus callbacks or something.
(file #42642, file #42643)

After working on bug #52624, colors for trimesh objects seem to be nearly correct. However, I think there may be an offbyone error still present. The following code shows the issue
If you take a look at the figures you can see that, occasionally, one line segment will be a different color. This might have to do with using fix(), floor(), ceil(), or round() in the color computations. And I suppose it might also be a difference between single (OpenGL) and doubles (gnuplot). But I'm guessing there is something in _gnuplot_draw_axes_.m.
I've added png images of the two trimesh plots.
(file #42639, file #42640)

According to the documentation for scatter3, it also uses 36 as the default size.

OK, I follow the logic on that: a marker alpha times wider looks alpha^2 times as massive.
Out of curiosity, what about scatter3() in 3D? Conceptually, a marker could have volume meaning cuberoot is in order, but of course the marker itself is a 2D object in a 3D world, and it is always facing the viewer.

The size simply changed definition. Matlab uses the area of the marker as the size and this is measured in points. The default is 36. The actual marker size is sqrt (sizedata) or 6 points in the default case.

If this is "autosizing" concept (as the color argument seems to use automatic fit across the whole color map), should
scatter (1:100, 1:100, 10*ones(1,100))
scatter (1:100, 1:100, 100*ones(1,100))
these produce a similar size?

I see the change. Is the idea with that size argument that "autosizing" is applied, i.e., the numerical values don't necessarily translated to marker dimensions? Some adjustment might be needed for gnuplot toolkit as it seems there isn't much granularity in the sizes or the size on the larger end of the spectrum is maybe now too small.

I checked in a change that at least gets visual conformance with Matlab scatter, although the performance problem is still unaddressed.

A "scatter" type. That will simplify matters eventually... I'll have a look at the color mapping problem.

I think this is a consequence of how Octave is trying to map to Matlab. In Matlab, scatter produces a Scatter Group Series object. In Octave, we use an hggroup that has the appropriate Scatter Group properties, but the internal implementation is done with patch objects since each scatter object could have a different size and/or color. See http://www.mathworks.com/help/matlab/ref/scatter.html and http://www.mathworks.com/help/matlab/ref/scatterseriesproperties.html.
scatter is very slow, probably as you say, because it has to create so many objects. The function needs an overhaul for other reasons as well as it doesn't produce Matlab compatible results. See bug #48873 where it is the last function that needs work.
Overhauling scatter promises to be a significant coding effort so I don't think it will happen before 4.2.
If there is a temporary fix for gnuplot that would be nice, otherwise it can be addressed when scatter is redone.

This bug has come to my attention. There's been a temporary fix for the gnuplot errors by forcing color values to be uint8 between 0 and 255.
https://savannah.gnu.org/bugs/?49017
Obviously the colors come out incorrect, and I think it is just a matter of mapping the color values to the proper range. We've done some other types of mapping based on cmap and that sort of thing elsewhere.
I'd like to first ask a more general question about how this scatter plot sets up data. I'm a bit surprised that markers go through the "patch" route. I do:
scatter
[break after the second demo with filled circles]
kids = get(gca,'children')
grandkids = get(kids, 'children')
get(grandkids (1))
and these items all have 'tag' property 'patch'. Is _scatter_.m choosing to do this because it is the only way for markers to have colors different for each marker? (gnuplot can handle plotting markers as a function of color.) The ramification of this is that scatter.m seems to be a very slow routine, regardless of toolkit, when all it is doing essentially is plotting a collection of markers. (My guess is that there is some slowness associated with constructing all the plot elements and properties...is this an internal C routine or done as a script?) I'm just trying to think if there is a way to speed things with a change in organization before altering the patch subcode.

With current tip
Neither qt nor gnuplot seems to produce correct output
from
(attached screenshots of both).
gnuplot 5.0.3 (qt terminal).
Dmitri.
(file #38433, file #38434)

Retitlling report. The gnuplot backend now produces the mesh mor or less correctly, but the colors are wrong.

Here is a partial solution, according to comment #7. It
shows the mesh, but doesn't solve the problem with the colors
(overflow and underflows).
Please review. Thanks.
(file #34344)

Hi,
there are few problems with gnuplot plotting of trimesh,
like colormap problems, but according to gnuplot documentation,
pm3d plots surface, and trimesh should plot only the mesh.
To get trimesh working, two lines should be changed
fputs (plot_stream, "set pm3d implicit;\n");
(line 1407) should not be written but
fputs (plot_stream, "set pm3d explicit;\n");
and line 679 "sprintf ("with pm3d");" should be (for mesh only!)
"sprintf ("with lines");"
These changes will not solve the problem shown in comment #6,
but show the mesh lines in demo trimesh
Avinoam

I tried to create a minimalistic example which shows the problem:
Shouldn't this draw a line where the 2 faces are connected (as with fltk)?
sometimes over or undeflows

This is because values of 'color' in go_draw_axes_.m, line 908 (default) is sometimes greater than 255 or negative.

Just some thoughts:
I don't understand where this 5byte hexadecimal color comes from. The alpha channel has been introduced in CVS gnuplot 4.7 and will be part of the 5.0 release (http://gnuplot.info/ReleaseNotes_5_0_rc1.html) so it's possible to use rgb '#AARRGGBB"

I can confirm this issue on linux mint 14 (based on the same version of ubuntu as the OP).
On mac, with octave 3.6.4 or 3.8.0 and gnuplot 4.6 patchlevel 4 (from fink) the problem doesn't occur (even though the resulting patch is essentially white).

I use Gnuplot 4.6.11 (package 4.6.11~precise1 from the standard repository of Ubuntu 12.04)
And this is the error message I get as result of the demo trimesh in octave 3.8.1:

Please add the error message verbatim and the gnuplot version. I can't see an error message an my system.

Here is how to reproduce the error:
Instead of the expected figure window with a funny 3D plot, you'll get several pages of error messages from gnuplot.
The trimesh demo works fine with fltk. But the gnuplot errors are still there with the octave version 3.8.2rc1.
I've observed this under Linux, but I assume it'll be the same for other operating systems.
