Thanks for looking at this again. There is no newcheby2.patch attached, can you try again?

I think adding a call to cplxpair in cheby2 is the correct solution to the cheby2 issue.  I think the change to sftrans was only to address the issue with butter.  I am surprised that the butter example works now without a change to force the complex pairs during the warping process.  But if it works, that is great.

I agree that the issue in butter is not with cplxpair, although I think there are some issues with cplxpair.  I was able to get the butter example to work by playing with sftrans.  In sftrans when the poles and zeros are adapted from low pass with unit cutoff to the proper type (bandpass in this case) a new set of poles and zeros are calculated. By basically forcing these to be pairs, the result will pass cplxpair.

I wrote a function that is like cplxpair, but averages the real and imaginary parts of the pairs, and called that in the bandpass case of sftrans on the b value used to calculate the poles. 

here is the patch to sftrans:

--- ../octave/signal-1.3.2/sftrans.m        2017-01-05 22:10:48.690453045 -0600
+++ sftrans.m        2017-01-07 22:46:39.951910271 -0600
@@ -141,6 +141,7 @@
 ## ----------------  -------------------------  ------------------------
       Sg = Sg * (C/(Fh-Fl))^(z-p);
       b = Sp*((Fh-Fl)/(2*C));
+b=cplxpair_eqlz(b);
       Sp = [b+sqrt(b.^2-Fh*Fl), b-sqrt(b.^2-Fh*Fl)];
       if isempty(Sz)
         Sz = zeros(1,p);

and the cplxpair_eqlz.m is

function ze = cplxpair_eqlz (z, thresh = 100*eps)

  ## interesting for testing: if nargin<2, thresh=1E-3; endif

  if isempty(z)
    ze=[];
  else
    ze = cplxpair(z); # sort complex pairs, real roots at end
    nz = length(z);
    nzrsec = 0;
    i=nz;
    while i && abs(imag(ze(i)))<thresh # determine no. of real values
      ze(i) = real(ze(i));
      nzrsec = nzrsec+1;
      i=i-1;
    endwhile
    nzsect2 = nz-nzrsec;
    if mod(nzsect2,2)~=0
      error('cplxreal: Odd number of complex values!');
    endif
    for i = 1:2:nzsect2-1
      ze(i) = (ze(i) + conj(ze(i+1)))/2;
      ze(i+1) = conj(ze(i));
    endfor
  endif

endfunction

realistically, any time a filter that produces real valued output from real valued input should have all the poles and zeros that are either real, or complex conjugate pairs.  I'm not sure if averaging the real and imaginary parts is better or worse than choosing one of the pair, and conjugating it to get the other member of the pair.

hmmm... verbatim block render fail on the website.

[z,p,k] = cheby2(6,40,[.529150859255659, 491.5561729754713269]*2/1000);

For those looking to fix this and with no access to Matlab (like me), or if you want a working implementation to derive code from, SciPy has free implementations of the same filter design functions, so both the outputs and the algorithms themselves can be analyzed and compared.

>>> from pprint import pprint
>>> from scipy.signal import cheby2
>>> a = cheby2 (6, 40, [.001, .983], btype='bandpass', output='zpk')
>>> (z, p, k) = (list(a[0]), list(a[1]), a[2])
>>>
>>> pprint (z)
[(-0.99866962186808894-0.051565360057387599j),
 (-0.99928680643819134-0.037760806116408051j),
 (-0.99990441719062173-0.013825935146777633j),
 (-0.99990441719062173+0.013825935146777633j),
 (-0.99928680643819134+0.037760806116408051j),
 (-0.99866962186808894+0.051565360057387599j),
 (0.99999539574401186+0.0030345495179681286j),
 (0.99999753249436207+0.0022214871566177065j),
 (0.99999966940456275+0.0008131363754524236j),
 (0.99999966940456275-0.0008131363754524236j),
 (0.99999753249436207-0.0022214871566177065j),
 (0.99999539574401186-0.0030345495179681286j)]
>>>
>>> pprint (p)
[(0.99917623666441646+0.0042929489610568049j),
 (0.99777097697199735+0.0031385196512288016j),
 (0.99696202281637791+0.0011479500050488025j),
 (0.99696202281637791-0.0011479500050488025j),
 (0.99777097697199735-0.0031385196512288016j),
 (0.99917623666441646-0.0042929489610568049j),
 (-0.98362861112405386-0.071960682314650762j),
 (-0.96149154708651674-0.051482086995079612j),
 (-0.94940131856521048-0.018602747339922173j),
 (-0.94940131856521048+0.018602747339922173j),
 (-0.96149154708651674+0.051482086995079612j),
 (-0.98362861112405386+0.071960682314650762j)]
>>>
>>> k
0.89629009720530994

I took a look at the butterworth example, and it appears that in that case, forcing the conjugate pairs to be true conjugate pairs before running sftrans does not solve the issue.

I think in that case, the issue is really in cplxpair, and there is a bug report about it (https://savannah.gnu.org/bugs/?47865). 

I'm trying to follow the discussion there, but it seems like it deals with a couple of different issues.  The discussion there seems to indicate that octaves implementation multiplies the tolerance by the magnitude of the entry, but when I look at the code, It doesn't seem like it does that.  (for determining if the value is real, it does seem like the tolerance is relative, but in the case of the complex values, the check is

[v,idx] = min (abs (z(i+1:p) - conj (z(i))));
if (v > tol)
  error,,,

I think maybe changing the if to

if (v/abs(z(i)) > tol

makes some sense, but it doesn't address the problem with the butter example.

The filter needs to have all the zeros in complex conjugate pairs, which is why zp2sos was complaining.  The zeros are calculated by getting equally spaced values of theta in the range of 0 to pi (not inclusive), spaced apart by pi/n (where n is the order of the filter).

The zeros are then calculated by dividing 1i by the cos(theta).  In the case where n is odd, the purely imaginary zero (at theta=pi/2) is excluded.  This should produce complex conjugate pairs.

I think the issue is that with the small numerical differences between the values of theta that produce conjugate pairs, produce small numerical differences in the cos(), which in turn produce small numerical differences in the conjugate zeros.

It is possible that it would be better that does some sort of average of the values in the conjugate pairs instead of arbitrarily choosing one of the zeros and conjugating it.

Other methods I could think of to get conjugate zeros would be to ensure that the positive and negative values of cos(theta) are of the same magnitude but opposite sign, or perhaps ensuring that the smallest value of theta(1)=pi-theta(n), theta(2)=pi-theta(n-1) ...

the help thread first links back to an older post with the zp2sos cplxpair error, but in that case z,p,k come from butter, not cheby2. 

here's the script that fails with butter:

>> bandHz = [100 500];
>> sampleFreq = 1001;
>> nyquistFreq = 0.5 * sampleFreq;
>> w = bandHz / nyquistFreq;
>> [z,p,k] = butter(2, w);z',p',k
ans =
   1
   1
  -1
  -1
ans =
   0.571788211592795 + 0.293698922713578i
   0.571788211592761 - 0.293698922713564i
  -0.997780778679612 - 0.002216561341880i
  -0.997780778679612 + 0.002216561341880i
k =  0.637776870938282
>> [sos,g] = zp2sos(z,p,k);
error: cplxpair: could not pair all complex numbers
error: called from
    cplxpair at line 118 column 9
    cplxreal at line 52 column 9
    zp2sos at line 78 column 10

in Matlab 2016b, the output produced is:

[z,p,k] = butter(2, w)
z =
     1
     1
    -1
    -1
p =
  0.571788211592765 + 0.293698922713572i
  0.571788211592765 - 0.293698922713572i
 -0.997780778679611 + 0.002216561341880i
 -0.997780778679611 - 0.002216561341880i
k =
   0.637776870938272
>> [sos,g] = zp2sos(z,p,k)
sos =
  Columns 1 through 4
   1.000000000000000  -2.000000000000000   1.000000000000000   1.000000000000000
   1.000000000000000   2.000000000000000   1.000000000000000   1.000000000000000
  Columns 5 through 6
  -1.143576423185530   0.413200816119565
   1.995561557359223   0.995571395446674
g =
   0.637776870938272

the patch on the thread adjusted the way zeros are calculated within cheby2, and the code ran without error except for an ordering issue. Maybe the bug needs to be broadened to butter, and maybe cheby1 and ellip as well?  is there a similarly easy test for those?

just for reference, here's a link to the help thread with some of the technical discussion.
http://octave.1599824.n4.nabble.com/zp2sos-error-cplxpair-could-not-pair-all-complex-numbers-tp4681260.html

I've added the OP and patch author to the bug's cc line.