@Pantxo: I haven't looke at the code, but your approach sounds right.

This is a feature request.  I've changed the title to reflect the true issue.  I agree, this is a bothersome thing.  The issue is that Octave chooses limits for the plot based on the data, and then tries to divide the axes into five divisions which will result in 6 ticks and ticklabels.

If the axes is particularly small, such as in a subplot, or the labels are particularly long, such as with a large fontsize, then they will overlap.  What the code needs to do is interact with the OpenGL renderer and determine the size ("extent" property) of the ticklabels.  If it determines that they overlap then it has to go back and select fewer ticks.  The issues is that these two pieces of code are in very different parts of Octave so connecting them will not be straightforward.

The current workaround is to manually set ticks and ticklabels.  This isn't all that bad, but is annoying.

This isn't about subplots, it applies to any axes object.  Here is sample code which reveals the problem by shrinking the width of the axes object.

plot (1e3:1e3:1e4, 1:10);
get (gca, 'xtick')
pos = get (gca, 'position');
pos(3) = pos(3) / 4;
set (gca, 'position', pos)
get (gca, 'xtick')

When run in Octave, the ticks are at

ans =

       0    2000    4000    6000    8000   10000

When run in Matlab, the ticks are initially at [1000, 2000, 3000, ..., 10000].  After the axes is shrunk, there are only three ticks at [0, 5000, 10000].