Considering that the Jacobian is the gradient as a row vector (in case of a scalar function), how about changing the documentation as follows?

The second argument may also be a 2- or 3-element cell array of function handles. The first element should point to the objective function, the second should point to a function that computes the gradient of the objective function as a column vector, and the third should point to a function that computes the Hessian of the objective function.

[...]

The third and fourth arguments may also be 2-element cell arrays of function handles. The first element should point to the constraint function and the second should point to a function that computes the Jacobian matrix the constraint function (which is the gradient as a row vector in case of a scalar constraint function):

[df(x)/dx_1 df(x)/dx_2 ... df(x)/fx_N]

I think that there is a misunderstanding of the section in question: It aims to give the (correct) mathematical definition of the gradient (I guess to give some background/explanations):

"The first element should point to the constraint function and the second should point to a function that computes the gradient of the constraint function:
            [ d f(x)   d f(x)        d f(x) ]
transpose ( [ ------   -----   ...   ------ ] )
            [  dx_1     dx_2          dx_N  ]



In fact, sqp expects the Gradient for phi (when provided - the Hessian is possible on top) and the Jacobian for g,h (when provided).
This makes sense: phi is a scalar function (hence the Gradient), g/h could be a vector function in order to support multiple constraint functions (hence the Jacobian).

To avoid any possible misunderstanding of the sqp function description (unchanged for more than 12 years), one could simply state that "the second should point to a function that returns the Jacobian matrix" without providing definitions for Gradients (and/or Jacobians) - people using these Octave functions should know what they mean/what they are doing.

This is analogue to the current octave fminunc function description which simply mentions the Jacobian.

comment #1:

> I think that the octave documentation is correct

I don't think so, let me explain please.

> It corresponds to the mathematical derivative convention for vectors depending on several variables. See for example
>
> https://en.wikipedia.org/wiki/Matrix_calculus

Let's look at concrete formulas. In the following I refer to this version:

https://en.wikipedia.org/w/index.php?title=Matrix_calculus&oldid=1256945549

In section "Scope", second formula for nabla-f:

nabla f = [some row vector] transposed

This image/formula does NOT work like Octave does, as demonstrated in my original post:

#g_grad = @(x) transpose([0 0]); # does not work

Further down, in the "vector-by-vector" section, we find a formula that is consistent with Octave's actual behavior:

dy/dx = [dy1/dx1, dy1/dx2, ...;
         dy2/dx1, dy2/dx2, ...;
         ...]

>
> (using "transpose" command is only required when the provided derivative matrix follows the convention described in the link)  
>

The transpose command is not required because Octave expects the derivatives with respect to the second variable (x2) in column 2 (as in the cited matrix above), not row 2, even though the documentation currently wrongly suggests the latter, i.e. Octave's current documentation wrongly says transpose([f/dx_1 f/dx_2 ...]), which means that the derivative with respect to the second variable (x_2) would be in row 2 (which is wrong and should be column 2).

I therefore still believe the documentation is wrong (while you are right that the actual implementation indeed corresponds to the mathematical derivative convention for vectors depending on several variables).

Documentation of sqp(x0, phi, g, h) shows that the second element of g and h should have the derivatives in rows: transpose([f/dx_1 f/dx_2 ...]).

However, it seems to be correct that it's in the columns, i.e. g{2} and h{2} are row vectors (and the derivatives are in different columns) if g{1} or h{1} return a scalar, respectively.

Consider:

phi = @(x) 0;
phi_grad = @(x) [0; 0];
g = @(x) 0;
h = @(x) 0;
g_grad = @(x) [0 0];
h_grad = @(x) [0 0];
#g_grad = @(x) transpose([0 0]); # does not work
#h_grad = @(x) transpose([0 0]); # does not work
sqp([0; 0], {phi, phi_grad}, {g, g_grad}, {h, h_grad})

If I understand right, then the "transpose" in the documentation is wrong and should be removed.