I also made the default value of the TrapFlag to be 0 to match Matlab (http://hg.savannah.gnu.org/hgweb/octave/rev/3a8af9d517de).  Previously Octave was behaving as if TrapFlag was set.

Re-titling bug report for remaining issue.

I actually conflated two issues in this bug report: one bug with mexCallMATLAB and a feature request with mexCallMATLABWithTrap.

• bug report concerning mexCallMATLAB:

The integer value returned by mexCallMATLAB is always 1 even if an error has been caught. This can be fixed with the following (this is what mexEvalString does):

diff -r dc4e2203cd15 libinterp/corefcn/mex.cc
--- a/libinterp/corefcn/mex.cc  Mon Aug 29 14:32:22 2016 -0700
+++ b/libinterp/corefcn/mex.cc  Tue Aug 30 14:30:37 2016 +0100
@@ -3072,7 +3072,11 @@
   catch (const octave::execution_exception&)
     {
       if (mex_context->trap_feval_error)
-        recover_from_exception ();
+        {
+          recover_from_exception ();
+
+          execution_error = true;
+        }
       else
         {
           args.resize (0);

• feature request for mexCallMATLABWithTrap:

Given that the try/catch ME currently returns a structure, the following could be used until a complete MException can be returned:

mxArray *mexCallMATLABWithTrap(int nlhs, mxArray *plhs[], int nrhs, mxArray *prhs[], const char *functionName) {
    mxArray *mx = NULL;
    const char **fields = (const char *[]){"identifier", "message", "case", "stack"};
    mexSetTrapFlag(1);
    if (mexCallMATLAB(nlhs, plhs, nrhs, prhs, functionName)) {
        mx = mxCreateStructMatrix(1, 1, 4, fields);
        mxSetFieldByNumber(mx, 0, 0, mxCreateString("MATLAB:error"));
        mxSetFieldByNumber(mx, 0, 1, mxCreateString(functionName));
        mxSetFieldByNumber(mx, 0, 2, mxCreateCellMatrix(0, 0));
        mxSetFieldByNumber(mx, 0, 3, mxCreateStructMatrix(0, 1, 0, NULL));
        return mx;
    }
    else {
        return NULL;
    }
}

It seems that mexCallMATLABWithTrap is not available from the Octave API:
  http://www.mathworks.com/help/matlab/apiref/mexcallmatlabwithtrap.html

I tried instead to use mexCallMATLAB with mexSetTrapFlag, even if its use is deprecated in MATLAB, as it seems to be implemented in Octave:
  http://www.mathworks.com/help/matlab/apiref/mexsettrapflag.html

There seems to be an issue with the integer value returned by mexCallMATLAB though:

#include "mex.h"

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
    int sts;

    mexSetTrapFlag(1); /* or 0 */
    sts = mexCallMATLAB(0, NULL, 1, (mxArray **)prhs, "fileread");
    mexPrintf("status = %d.\n", sts);
}

With mexSetTrapFlag(0), Octave gives:

octave:1> test_mexCallMATLAB('e')
error: fileread: cannot open file
error: called from
    fileread at line 37 column 5

With mexSetTrapFlag(0), MATLAB gives:

>> test_mexCallMATLAB('e')
Error using fileread (line 49)
Could not open file e. No such file or directory.

With mexSetTrapFlag(1), Octave gives:

octave:1> test_mexCallMATLAB('e')
error: fileread: cannot open file
status = 0.

With mexSetTrapFlag(1), MATLAB gives:

>> test_mexCallMATLAB('e')
Error using fileread (line 49)
Could not open file e. No such file or directory.

status = 1.

So it seems that when the Trap Flag is set to true, mexCallMATLAB is not set to a non-zero value when an error is caught.

From what I see, it seems that having an implementation of mexCallMATLABWithTrap might not be too difficult - the main issue being that MException is not implemented: what are the plans regarding this and should it be implemented in M-code or C++?