Re comment #1: That sounds like the behavior of the OOM killer on Linux.
If it is, there is little to nothing Octave can do to prevent becoming a victim of the OOM killer, afaict.

The point when the OOM killer gets active depends on the available and installed memory, the swap, the over-provisioning strategy, ...
It's probably hard to predict it.

Please see https://stackoverflow.com/q/67106887/4183191 for context and details regarding this bug.

In summary, when this algorithm was last changed in 2011 in the repository (https://sourceforge.net/p/octave/communications/ci/83aeb09e7255c3953ac11b56299dab426831b419/), it was changed from a brute-force algorithm to a tree-based one; however, under certain conditions, this seems to result in really large theoretical trees, which exceed Octave's indexing capacity, resulting in a 'dimension too large' error inside huffmandeco.m

Steps to reproduce:

+verbose+
pkg load communications
rand( 'seed', 0 );   Text  = randi( [0,127], 1, 100 );   Fdata = randi( [0,127], 1, 30  );
for i = 0 : 127;   Nlet = length( find( Text == i ) );   p(i + 1) = Nlet / length( Text );   end

Symb       = 0 : 127;
Dict       = huffmandict( Symb, p );        % Create dictionary
[ ~, Idx ] = ismember( Fdata, Symb );       % Data indices from data
Compdata   = huffmanenco( Idx, Dict );      % Encode data indices
Dsig       = huffmandeco( Compdata, Dict ); % Decode Huffman code
Symb( Dsig )                                % Indices to symbols
-verbose-

Output:

+verbose+
error: out of memory or dimension too large for Octave's index type
error: called from
    huffmandeco>dict2tree at line 95 column 19
    huffmandeco at line 55 column 8
    testo at line 9 column 12
-verbose-

Incidentally, as a minor aside, the huffman functions also seem to have a slightly different API from their matlab counterparts, which has also caused problems to users migrating to Octave. See: https://stackoverflow.com/q/66929744/4183191

Also, the huffman functions seem to be getting quite a few hits on stackoverflow lately, not sure why. Perhaps some recently launched nanodegree that uses them is recommending Octave? We saw a huge number of octave questions in stackoverflow relating to Andrew Ng's course ever since that course first went online -- if something analogous is now causing people to interact with the huffman functions, we might reasonably expect that these functions will start getting a lot of attention soon ...