Since this is a rather weighty topic, should it be shifted to Discourse and leave this bug report as a reminder that something needs to be done?

I don't have access to Matlab to check on MEX functions, but I suspect you're right that no checking is possible unless it is put in to the code itself.

So one solution---which would be consistent across Octave built-in functions, .oct files, and .mex files---would be for the function itself to always check nargin and nargout and issue an error message.  We would want to agree on the standard form of the error message, and probably even make it a standard function in errwarn.cc, so that the same message is reliably produced.

Another approach would be to start tinkering with the DEF macros for function declaration.  Presumably we could figure out a way to gracefully continue to use people's existing code.  I.e., if they were using the old form of the macro the interpreter would assume the equivalent of varargout and varargin and not perform any verification.

I'd want to understand more about the last solution of changing the functions to look like methods before going down that route.

We've identified three possible solutions.  Are there others we should look at before we decide to really examine the merits of each?

Sample code:

[a,b] = newline ()
error: element number 2 undefined in return list
a =

double (a)
ans = 10

The newline() function only produces one output so I would expect that since this is an incorrect way to call the function I would get an error.  This is what Matlab does.  It throws an error about calling newline() with too many output arguments.

Note that the correct error is thrown if the function is defined in an m-file.  Consider the following newline2.m function

function retval = newline2 ()
  retval = "\n";
endfunction

When called from the command line

[a,b] = newline2 ()
error: newline2: function called with too many outputs
error: called from
    newline2

So the problem is only exhibited for built-in C++ functions.  For reference, the code for newline in strfns.cc is

  if (args.length () != 0)
    print_usage ();

  static octave_value_list retval = ovl ("\n");

  return retval;

It would seem that either C++ code needs to explicitly check nargout (which means modifying lots and lots of functions), or we need to communicate to the interpreter via the DEFUN macro how many output arguments are expected and have the interpreter check that.  The second approach also requires lots and lots of changes.