I pushed the change for the preconditioner to the stable branch here (http://hg.savannah.gnu.org/hgweb/octave/rev/8e30ef611002).
It really looks like there should be some additional code to either call polyreduce on the output, or possibly use a tolerance to eliminate leading values that are very small, like 1e33.

@Rik: Matlab produces a 4element vector, with a leading 1.3e33. Our result is now better, since in this test we should recover exactly the original b. Of course, we cannot guarantee this always happens.

I think this is the correct thing to do. However, when I make the change and then run "test residue" I get a failure in the BIST tests.
The issue is that the call
used produce a 4element vector
Now it produces just a 3element vector
Given that this vector represents a polynomial, it seems better that the leading 0 should be removed.
@Tony: Could you check that Matlab produces a 3element vector for the code above?

What about a left preconditioning like
It seems to fix the problem, for me. Of course, it would be better to construct A and B already in a normalized way, but I do not know the details of the algorithm.

In an interesting twist of fate, it looks as if I was one of the original authors of residue.m around 25 years ago (something that I had forgotten). I believe I used a home brew algorithm based on partial fraction expansion. It appears to have evolved significantly (thanks Ben Abbott) since then, but apparently is still not as accurate as whatever MATLAB is doing.

Confirmed. Calling residue_bug with n = 1,2 works fine. However, with n = 3 there is already a warning printed and the condition number of the matrix is scarily high.
The residue algorithm is coded as an mfile so it should be easy to see why A is nearly singlular. However, I don't understand the algorithm well enough to say how it should be modified or replaced.

Sorry. Trying again.

Can't figure out how to attach file to comment. Trying verbatim method.

Could you upload the mfile residue_bug.m? The bug tracker has interpreted your code using markup conventions where an asterisk means bold text. The other possibility is to use the verbatim tag to mark a block of text that should not be interpreted.
Remove the spaces between the '+' or '' and the "verbatim" keyword to have it take effect.

Octave and MATLAB return different results for the following script with argument (n) values of 3 or larger:
function residue_bug(n)
alpha = 100;
fr = 436;
N=poly(1i*2*pi*fr*[n:1 1:n]);
D=poly(alpha+1i*2*pi*fr*[n:1 1:n]);
[R, P, K]=residue(N,D)
end
Octave displays a warning about a singular matrix and returns results that are different than MATLAB. MATLAB does not display a warning until n is much larger (around 30 or so). I can normalize the problem so that Octave results are valid for values of n up to about 10, but MATLAB results are then valid for much larger values of n.
I thought that the MATLAB values might be incorrect (even though it does not display a warning), but I computed my ultimate result using a method that does not use residue and it is consistent with the result obtain using MATLAB's residue function.
The basic issue appears to be that whatever method MATLAB uses to compute residue results is much better behaved than the Octave method.
