In bug #60143, the following behavior of classdef set-methods was observed with the attached classes in Matlab R2020b:

>> a = bug60143_value;
>> a.t = 4
The set function for property 't' must return an instance of class 'bug60143_value'.

>> a.u = 3
method set.u

a =

  bug60143_value with properties:

    t: []
    u: 3

>> b = bug60143_handle;
>> b.t = 2
method set.t
In set function for property 't', the class of the output object is 'double'; the class of the input object is 'bug60143_handle'. They should be the same.

>> b.u = 4
method set.u

b =

  bug60143_handle with properties:

    t: []
    u: 4

The syntax highlighter in its editor marks the set function for u in class `bug60143_value` with an error highlight "Set Methods must have at most one output.". But -- surprisingly -- it still seems to work (ignoring the extra output).

It looks like it is possible that a set function of a handle class returns a different instance of the same class. That's interesting. I didn't know this was possible.

With Octave 7 (hg id 65979d33f147), I see the following:

>> a = bug60143_value;

>> a.t = 4

error: set.t: function called with too many outputs
error: called from
    set.t
>> a.u = 3

method set.u
a =

  bug60143_value object with properties:

      t: [0x0 double]
      u: [1x1 double]

>> b = bug60143_handle;

>> b.t = 2

method set.t
b =

  bug60143_handle object with properties:

      t: [1x1 double]
      u: [0x0 double]

>> b.u = 4
[snip]
method set.u
method set.u
method set.u
method set.u
method set.u
method set.u
method set.u
error: max_recursion_depth exceeded
error: called from
    set.u
    set.u at line 21 column 11

The results with the value class look like they are compatible.

But the behavior of the handle class is different:
It looks like we should not discard the return value of the set-method. One return value seems to be ok. But the type of the return value must match the class.
Additionally, it looks like a set-method should not be able to call itself even if the used object is a different instance of the same class (or whatever the exact condition is).