If 'octave-sqp' is invoked in fmincon, octave_sqp_tolerance is used to call sqp in _octave_sqp_wrapper_.  The last argument below, tolerance, is octave_sqp_tolerance. 

  ## call sqp
  [p_res, objf, info, outp.niter, outp.nobjf, outp.lambda] = ...
      sqp (pin, passed_f, {equc, dequc}, {inequc, dinequc}, -Inf, Inf, niter, tolerance);

octave_sqp_tolerance is not the tolx in fmincon input argument.   To control variable convergence, it appears (I did not test) that fmincon needs options with field octave_sqp_tolerance defined similar to tolx. Without the explicit definition, tolerance=[] is used to call sqp.  This is not what I can find in the fmincon documentation.  

Controlling tolx in fmincon call has no effect on tolx of the Octave sqp call.