I have been re-writing the syntax highlighter for Vim and came across this issue as well.  How critical is it to support nested block comments?  It really seems like more trouble than it is worth.

I will point out that Matlab does allow nested block comments, I just don't think it is worth the programmer time to support this when the feature is unnecessary (line in a nested comment block are already commented) and confusing.

We currently allow block comment markers to be mixed in any combination.  So '%{' and '#{' are equivalent as are '%}' and '#}'.  This could cause compatibility issues.  For example, there could be some working Matlab code that looks like this:

%{
  block comment
#}
  still a block comment in Matlab
%}

and that will fail in Octave.  This could be solved by keeping track of whether the current block comment started with '#{' or '%{' and requiring the types of the begin and end markers to match.  But even if we did that, the following would work in Matlab but fail in Octave:

%{
  block comment
#{
  nested block comment in Octave but not Matlab
%}
sin (x);   %% some other code that should be executed.

For that problem, I suppose we could require that to be nested, the nested block comments must all be of the same type?  Then the '#{' in the above code would not be recognized as the beginning of a nested block comment.  But that would require more work for the programmer who wants to comment out a block of code.  Now they would need to check the block of code they are commenting for existing block comments, and then use the same style.

A far simpler solution would be to only allow block comments to use '%{' and '%}' instead of also allowing '#{' and '#}'...

I haven't seen any bug reports about this problem occurring in real code, so I'm marking the severity as minor and the priority as low.