OK, I didn't know about fltk and OpenGL.

I checked there is no issue with gnuplot, so the bug is clearly related to OpenGL.

The fltk toolkit does use OpenGL for rendering.

Looks like the known issue that OpenGL only works in single precision. There are several open bug reports about his (can't find them ATM).

A way around is to, yes, try the gnuplot backend:
graphics_toolkit ("gnuplot")

Please report back if that works better for you.

Here is the problematic code:

close all;
clear all;
s = 1e-3;
x = 100 ones(1, 1000) + s randn(1, 1000);
plot(x);

When s is higher than 1e-6, this plots (as expected) a noise around the mean value 100.

When s = 1e-6, this plots a line that is sometimes at the top of the figure, or at the bottom, or there is no line at all. The figure frame is corrupted, etc. (see screenshot1).

When s = 1e-8, there is sometimes no frame in the figure. The line is often not displayed (see screenshot2).

When s <= 1e-10, the plot correctly displays a line in the middle of the figure and the frame is also displayed, as expected.

I was able to reproduce this issue in Octave 4.2.1 (windows), 4.4.1 (windows), 5.1.0 (windows), 5.1.0 (Ubuntu Linux 18.04).

The problem is the same with qt or fltk toolkit. I didn't test with Gnuplot.