Turns out this is a problem for both Octave and Matlab.  Attached is a very minimal working example, and also shown below.

h = plot (1:10, zeros (1, 10), 'o');
%% Left-hand axes limit is very far below first tick at 1.
xlim ([-1e7, 10]);
%% Setting xtick is slow because number of minor ticks is calculated from
%% difference between xtick(2) - xtick(1) and then applied to the range from
%% axis_limit_low to xtick(1).  In the example given, this is roughly 50e6
%% elements of type double or 400 MB of memory.
set (gca, 'xtick', 1:10);

The problem code is in graphics.cc:calc_ticks_and_lims()

  // minor ticks between, above, and below min and max ticks
  int n = (is_logscale ? 8 : 4);
  double mult_below = (is_logscale ? tmp_ticks(1) / tmp_ticks(0) : 1);
  double mult_above = (is_logscale ? tmp_ticks(n_ticks-1) / tmp_ticks(n_ticks-2)
                       : 1);

  double d_below = (tmp_ticks(1) - tmp_ticks(0)) / mult_below / (n+1);
  int n_below = static_cast<int> (std::floor ((tmp_ticks(0)-lo_lim) / d_below));
  if (n_below < 0)
    n_below = 0;
  int n_between = n * (n_ticks - 1);
  double d_above = (tmp_ticks(n_ticks-1) - tmp_ticks(n_ticks-2)) * mult_above
                   / (n+1);
  int n_above = static_cast<int> (std::floor ((hi_lim-tmp_ticks(n_ticks-1))
                                  / d_above));
  if (n_above < 0)
    n_above = 0;

  Matrix tmp_mticks (1, n_below + n_between + n_above);

The problematic statement is the last one where a Matrix of minor tick values is created which can be very, very large.

For both Octave and Matlab, displaying these minorticks is impossible as they all overlap at the resolution of an ordinary monitor so this is simply an excessive waste of memory.

Attached patch limits the number of minor ticks below or above the actual range of ticks to 1000.  This seems fine to me,



(file #54601)

I extracted a more minimal example and attached tst_xtick_hang.m.

The issue is with a statement

set (gca, 'xticks', ticks);

This seems to cause some near infinite loops in C++ in the function calc_ticks_and_lims().

(file #54596)

It smells like another slice optimization gone wrong.

Dmitri.
--

Consider the following three commands:

octave:26> hist(1:5,10.^(1:10))
Aoctave:27> hist(1:5,10.^(1:11))
error: out of memory or dimension too large for Octave's index type
error: called from
    __bar__ at line 237 column 16
    bar at line 117 column 18
    hist at line 249 column 5
octave:28> hist(1:5,10.^(1:12))

The first one takes about two seconds on my machine.
The second is faster and throws that error.
The third is faster and throws no error, though it has more work to do (one more bin).

After the first command, octave uses 1GB residential memory.
After the second and third it drops down to 180MB.

You may want to disable swap (sudo swapoff -a) before executing the second one. Otherwise your system becomes unresponsive for a while.

They all run very fast an show no error when output is expected:

[a,b] = hist(1:5, 10.^(1:11))

So I think the problem must be with the plotting subsystem.