I checked in the patch here (https://hg.savannah.gnu.org/hgweb/octave/rev/a018fca707ca).

I made a few small modifications.  In place of tmp1, tmp2, and tmp I used what I hope are more meaningful variable names pos_imag, neg_imag, and is_equal.

In place of using all() and size() together I just used the C++ function size_equal().

-    if (all (size (tmp1) == size (tmp2)))
+    if (size_equal (pos_imag, neg_imag))

For the BIST test, I used commas to clearly delineate where each number stops which can sometimes be ambiguous.  Is "-9 - 3i" a single complex number or is it two numbers, "-9+0i AND 0-3i".

-%! assert (y, [1 +  0i   -9 -  3i   25 + 24i  -17 - 57i  -12 + 36i])
+%! assert (y, [1 + 0i, -9 - 3i, 25 + 24i, -17 - 57i, -12 + 36i]);

Thanks Marco for coming up with a high-performance patch (no isequal call).

This will be a part of the bug fix release 5.2.0 in a few months.

There is probably a bug in cheby2, ok. But poly(p1) is valid syntax and should work. The problem is the use of == instead of the slower isequal. I attach a patch without using isequal.

(file #46448)

This patch fails for zp2tf from the signal package with values that are "nearly" real. Copying from line 84 of share/octave/5.1.0/m/polynomial/poly.m:

octave:14> [z1,p1,k1]=cheby2(5,40,0.1);
octave:15> p1
p1 =
   9.345539872652292e-01 - 1.826618879469199e-01i
   8.372613608092301e-01 - 1.304057652412007e-01i
   7.781404349401341e-01 - 1.537554717468438e-17i
   8.372613608092301e-01 + 1.304057652412007e-01i
   9.345539872652292e-01 + 1.826618879469199e-01i

octave:16> tmp = sort (p1(imag (p1) > 0)) == sort (conj (p1(imag (p1) < 0)))
error: mx_el_eq: nonconformant arguments (op1 is 2x1, op2 is 3x1)
octave:16>

@Markus: thanks.
A patch is included.

(file #45248)

If you take

x = [1+1i,2];
all(sort(x(imag(x)>0))==sort(conj(x(imag(x)<0))))

you get true, which is not correct. How do I check whether sort(x(imag(x)>0)) is the same vector of sort(conj(x(imag(x)<0))))?

Okay, then I think this is fine to work up in to a true changeset and commit it.

That seems to work.  Would the call to eig ever return values which are not exactly complex conjugates such that we need to use a tolerance on the equality test?  In other words,

if (all (abs (sort (x(imag(x)>0)) - sort (conj (x(imag(x)<0)))) < tol))

Yes, but is it on the right track?  Using your example from comment #2 I find that the largest imaginary value is 10 eps which is quite small.  At least there are no horrendous errors like 256i as in the first example.

It seems that sort followed by a test to replace small values with 0 would work.  The algorithm in poly.m is

  for j = 1:n;
    y(2:(j+1)) = y(2:(j+1)) - v(j) .* y(1:j);
  endfor

which has a multiplication and a subtraction per round of the algorithm.  To me that says worst case of 2 eps per cycle or 2 x n x eps for the algorithm.  It's not perfect, but for eps one could use the largest value in v.

epsmax = 2* n * eps (max (abs (v(:))));
idx = abs (imag (y)) < epsmax;
y(idx) = real (y(idx));

Of course, this is starting to look like a fair amount of work so if detecting the conjugate pairs is easier then we could do that as you suggested in comment #0.

The algorithm in poly.m uses a for loop and repeated multiplications.  It could be something as simple as, with the inversion of matrices, using balance() to put the largest factors first, etc.

In this example, if I use sort on the input x then the conjugates are paired and the result of poly is entirely real without spurious imaginary terms.

See the session below

octave:25> n = 3;
octave:26> fr = 436;
octave:27> x = 1i * 2 * pi * fr * [-n:-1 1:n]
x =

     -0.0 - 8218.4i     -0.0 - 5478.9i     -0.0 - 2739.5i      0.0 + 2739.5i      0.0 + 5478.9i      0.0 + 8218.4i

octave:28> N = poly (x)
N =

 Columns 1 through 6:

   1.0000e+00 + 0.0000e+00i   0.0000e+00 - 1.8190e-12i   1.0507e+08 + 0.0000e+00i   0.0000e+00 + 1.8311e-04i   2.7597e+15 + 0.0000e+00i   0.0000e+00 + 2.5600e+02i

 Column 7:

   1.5216e+22 + 0.0000e+00i

octave:29> x2 = sort (x)
x2 =

     -0.0 - 2739.5i      0.0 + 2739.5i     -0.0 - 5478.9i      0.0 + 5478.9i     -0.0 - 8218.4i      0.0 + 8218.4i

octave:30> N2 = poly (x2)
N2 =

   1.0000e+00   0.0000e+00   1.0507e+08   0.0000e+00   2.7597e+15   0.0000e+00   1.5216e+22

octave:31> roots (N2)
ans =

     -0.0 + 8218.4i
     -0.0 - 8218.4i
     -0.0 + 5478.9i
     -0.0 - 5478.9i
     -0.0 + 2739.5i
     -0.0 - 2739.5i

Should we just sort the input vector 'v' after the input validation in poly.m?