@Rik: Matlab produces a 4-element vector, with a leading 1.3e-33. Our result is now better, since in this test we should recover exactly the original b. Of course, we cannot guarantee this always happens.

What about a left preconditioning like

## Solve for the residues.
d = max (abs (A), [], 2);
r = (diag (d) \ A) \ (B ./ d);

It seems to fix the problem, for me. Of course, it would be better to construct A and B already in a normalized way, but I do not know the details of the algorithm.

Confirmed.  Calling residue_bug with n = 1,2 works fine.  However, with n = 3 there is already a warning printed and the condition number of the matrix is scarily high.

octave:6> debug_on_warning (1)
octave:7> residue_bug (3)
warning: matrix singular to machine precision, rcond = 5.73398e-20
warning: called from
    residue at line 265 column 5
    residue_bug at line 6 column 11
stopped in /home/rik/wip/Projects_Mine/octave-dev/scripts/polynomial/residue.m at line 265
265:   r = A \ B;

The residue algorithm is coded as an m-file so it should be easy to see why A is nearly singlular.  However, I don't understand the algorithm well enough to say how it should be modified or replaced.

Can't figure out how to attach file to comment.  Trying verbatim method.

function residue_bug(n)
  alpha = 100;
  fr = 436;
  N=poly(1i2pifr[-n:-1 1:n]);
  D=poly(-alpha+1i2pifr[-n:-1 1:n]);
  [R, P, K]=residue(N,D)
end

Octave and MATLAB return different results for the following script with argument (n) values of 3 or larger:

function residue_bug(n)
    alpha = 100;
    fr = 436;
    N=poly(1i*2*pi*fr*[-n:-1 1:n]);
    D=poly(-alpha+1i*2*pi*fr*[-n:-1 1:n]);
    [R, P, K]=residue(N,D)
end

Octave displays a warning about a singular matrix and returns results that are different than MATLAB. MATLAB does not display a warning until n is much larger (around 30 or so).  I can normalize the problem so that Octave results are valid for values of n up to about 10, but MATLAB results are then valid for much larger values of n.

I thought that the MATLAB values might be incorrect (even though it does not display a warning), but I computed my ultimate result using a method that does not use residue and it is consistent with the result obtain using MATLAB's residue function.

The basic issue appears to be that whatever method MATLAB uses to compute residue results is much better behaved than the Octave method.