@Rik: Many thanks for this, this is very useful. If it can make it to 4.2, it would be great.
You were entirely right in comment 1: my snippet was just a way to reproduce the issue taken from a larger piece of code and I didn't realize I was introducing another subtlety.

Octave was not allocating new memory on the heap for the dims array when mxSetDimensions was called. I fixed that in this cset (http://hg.savannah.gnu.org/hgweb/octave/rev/e337b8e3592c). Both the original example, and my code which makes a duplicate of the inputs before modifying them, now pass. Marking as fixed and closing report.

I duplicated the array so that I am not working on the constant inputs. I think this is the correct strategy, but it does still fail which means there likely is a problem with mxSetDimensions in Octave.

It may work in Matlab, but the documentation seems to suggest that anything in the prhs array is supposed to be constant and left unchanged. Changing the dimensions of a supposedly const mxArray is probably not good practice.
From the documentation (http://www.mathworks.com/help/matlab/matlab_external/matlabdata.html),
It seems that the correct approach is to duplicate any input array that you may modify. A note on memory management (http://www.mathworks.com/help/matlab/matlab_external/memorymanagementissues.html#f24829) talks about cell arrays

There seems to be an issue if the new dimensions array passed to mxSetDimensions is freed explicitly when called on a cell array. The following piece of code illustrates the problem by trying to reshape an array with two elements in a 1x2 array:
The segfault disappears if the line "mxFree(dim);" is removed. It works as expected in MATLAB. I assume it is valid to free the dimensions array as mxSetDimensions makes a copy of it, if I understand the documentation right:
http://www.mathworks.com/help/matlab/apiref/mxsetdimensions.html
"mxSetDimensions allocates heap space to hold the input size array."
It works for numeric and struct arrays, but not cell arrays.
