I now think that this thread should be closed and there is no need to compound transpose operators.
I have found that the current behavior of Octave is related to a typical shiftreduce parser generator like Bison. I have found this blog has a good explanation of such parsers. It is shown that in parsing consecutive postfix operators the precedence rules are ignored.
In the case of Prolog in addition to precedence level the user also should define how operands of a operator are related to the operator and because those relationships are restricted by the language so it operates less flexible than Bison. I have found that Prolog Ciao works as expected but Prolog SWI has a bug and Mathematica works similar to Octave. CafeOBJ has implemented a more restricted precedence capability as of Prolog.
Also in the case of C++ the language hasn't defined any precedence but it can be deduced from rules of the grammar. In other words its grammar hasn't ambiguity. Also all postfix operators have the same level of precedence and all prefix operators have the same level of precedence and both groups have higher level of precedence related to binary operators so in the case of precedence it has less problems than MATLAB or Octave. In other words its precedence relationships is independent of the parsing algorithm.
So I think that my proposed compound operators are suggestions to make the precedence relationship independent of the parsing algorithm but because Bison and its parsing algorithm are sufficiently wellknown there may be no need to introduce such operators to prevent complication of the condition.
Thanks.

In my expriments I have found that Wolfram Mathematica, like Octave, applied postfix operators at list for the example a++[[x]], form inside to outside.
I sent an email to Prof. Theodore S. Norvell an associate professor of Memorial University of Newfoundland because he has designed an algorithm for precedent parsing named precedence climbing and asked him to explain what is the correct meaning of precedence providing the following examples:
So, I have found different programming languages treat precedence as
different concept.
For example if we define an operand "A" and two postfix operators
"OP1" and "OP2" and and setting OP1 to bind to its operand tighter
than OP2 and evaluating the following expressions:
Here are the results of the expressions in some programming languages:
And his kindly response:
Quote from Prof. Norvell
Whether or not a program has a syntax error or not is usually a matter that is decided by a language specification, reference, or standard. So it's not for me to say. For example, in the case of Ciao, they claim to "support ISO Prolog", but they also allow syntactic extensions. (Similar, I guess, to how gcc supports ISO C, but allows extensions.) I'd hope that Ciao has some mode where it rejects any program that the ISO standards says is improper and correctly interprets every program that the ISO standards says is proper. (Similar to gcc with the std=c11 flag.)
Anyway, another question might be why they don't all just accept all your examples? Let's say we have a binary operator # that is in between OP1 and OP2 in precedence.
A # B OP2 OP1
Should it be an error? If not, how do you think it should be parsed?
I don't know how you answered those questions; but for me the interpretation
((A # B) OP2) OP1
makes me uncomfortable, since A # B OP1 would be parsed as A # (B OP1); and the interpretation
A # ((B OP2) OP1)
makes me uncomfortable, since A # B OP2 would be parsed as (A # B) OP2. So, if I were a language designer, I think I'd be inclined to make A # B OP2 OP1 a syntax error. After all, the user can always supply parentheses and these will almost always improve the code's readability.
And, if A # B OP2 OP1 is a syntax error, shouldn't A OP2 OP1 also be a syntax error? Or to put that question another way: can you think of a simple rule that would allow A OP2 OP1, but disallow A # B OP2 OP1?
In my article, I made unary "" have a lower precedence than * and ^. That meant that
 a^2
means (a^2). But that leads to the odd case that a^2 * b means a^((2*b)), so somehow * gained priority over ^. To me that's weird and undesirable. I gave "" lower precedence, not because I think it's a good idea, but because it posed an interesting parsing challenge.
On the other hand suppose we have an ifthenelse construct with low precedence. To me
a ^ if x then c else 2*b
doesn't seem weird at all. But it's not all that different from a^2 * b which does seem weird.
Language design is tricky.
Currently I'm designing a language that has no precedence or parsing issues at all. It saves a lot of mental effort.
End Quote from Prof. Norvell

Oops , A postfix operator cannot be followed by a postfix operator of higher precedence. That means transpose operator should not be followed by call operator.

As in comment 18 I also used the language Prolog (Ciao version) to verify the precedence rules. I defined two postfix operators "pr" ,that means (),and "tr" that means ' and "pr" had higher precedence.
I tested different combination of expressions:
It evaluates three first expressions as:
And for the forth expression it generates error.
You can Try it online!
It verifies the previous experiment that a postfix operator cannot be followed by a postfix operator of lower precedence.

@John W. Eaton
Please read my proposed change to table of precedence in commnet13. After extended discussions I have found it to be the best solution and it is completely compatible with the current behavior of Octave.
= is OK
I think () means the parenthesis operator instead of call operator . As it can be used to group sub expressions: a+b*c ===> a+(b*c)
I agree that compound operators like .^ are the only way to work around otherwise troublesome precedence rules. But I don't know why they have changed its "L to R" order of evaluation to "R to L" ??
Those operators cannot be overloaded because there is no functional form of them and they don't have any name!!
I now have accepted that rearranging is a wrong idea.
I believe that indexing on temporary is a nice thing.
I have proposed 3 compound operators to be documented as I think they are the only way to work around otherwise troublesome precedence rules. No change to the parser is needed.
And that in the documentation the postfix ++/ should have the same level of precedence as of the call operator .

I said it already, and I say it again: to me, precedence is about where parentheses are to be implied, and never about rearranging. Wikipedia (https://en.wikipedia.org/wiki/Order_of_operations) is with me here, and seemingly also jwe.
It seems to me that the only potential issue with unary operators is with "++" and "", which can either be prefix or postfix, and where the precedence table decides which it is for instance here
As an aside, I think that this is a misuse of the precedence table, this is rather a question of parsing (which unfortunately is not documented at all yet). But apart from that, for unary operators precendence is never a question, because it is clear what their argument is, namely the expression to their left (for postfix) or right (prefix), and they are to be evaluated as soon as their argument has been evaluated (this is the difference from binary operators, where you have to decide whether an expression that sits between two binary operators belongs first to the left or to the right). So if I have the expression
I do not need a precedence table for knowing that this is to be understood as
because the expression to the left of ' is just A++. Whatever the precedence table, the converse case (apart from it being a syntax error in this specific case) would be completely counterintuitive  I would be interested in having pointed out any reasonable programming language that does such reorderings.

Mike: for elementwise operators like .^, the expressions (a.^b)' and a.^(b') are not equivalent.

Here is the precedence table for Matlab operators, at least as documented. If actual behavior is shown to be different, then that's a separate issue. Either way, we should aim to be compatible with Matlab, and the documentation should match the code.
https://www.mathworks.com/help/matlab/matlab_prog/operatorprecedence.html
For our discussion, let's begin by ignoring operators that appear in Octave but not in Matlab.
Some things to note:
 '=' is not in the list for Matlab, but then assignment is a statement in Matlab, not an expression, so an assignment doesn't produce a result and can't appear as part of another expression.
 () is listed as having highest precedence and higher than transpose. Does Matlab reorder expressions of the form x'(i) to be x(i)' instead? No, but then indexing is not allowed to apply to temporary results either.
 {} and . are not explicitly listed. Is that just an accidental omission?
 WTF are these operators (#3 in the list)?
Are they included simply to work around otherwise troublesome precedence rules? Can these operators be overloaded (for example, in a classdef class)?
I find the whole idea of rearranging expressions like x'(i) a bit weird.
I don't think we can have operator precedence that differs from Matlab. Does that mean we should not allow indexing of temporary expressions unless they are written as "f(args)(i)" or "(x')(i)" or similar? Completely disallowing indexing of temporary expressions will likely be quite unpopular.

jwe  I haven't looked at the parser yet, I've only been trying to propose a simple change to the docs to match how Octave actually works. Maybe the docs are fine the way they are now, but I think I agree with the OP that somewhere it should be explained that
A.'(m:n)
is evaluated as
(A.')(m:n)
even though the parentheses operator has higher precedence. I don't want to change anything in the implementation of the parser. I just think there should be an explanation as to why that is.
OP  For numeric types, the expressions (A^B)' and A^(B') are equivalent, so I considered it a nonproblem. But yes, for userdefined types that does change the order of operation. Withdrawn.

Mike: to make the code match your change to the precedence table in the manual I think we also need the attached change to octparse.yy. Given the structure of the parser this change may not have any real effect, but it seems to me that the precedence list in the code should match the documentation.
(file #45098)

With your proposed change the transpose operator and the power operator have different precedence. Considering the following expression:
currently it is evaluated as (a^b)' but with your proposed change it should be evaluated as a^(b') . Comment 11 has noted the related problem.
Also I think that postfix ++/ should have the same precedence as call operator because in the expression a++(x) a postfix operator "++" is followed by another postfix operator "()" that has lower precedence and it is syntax error. Is it needed to be discussed in a separate bug ?

I propose the attached change to the precedence table, implementing my suggestion in comment #7. I believe this fixes the operator precedence table to more accurately describe the relationship between the transpose and indexing operators.
(file #45097)

I used The package CafeOBJ * for the verification of the precedence rules. It allows to define postfix operators and to assign different levels of precedence to them.
I defined two postfix operators @ and ' that @ has higher precedence:
Then I used its parser to parse different expressions X''' and X@@@ and X@' :
That produced the following results:
But parsing X'@ generated a syntax error:
It means that:
 A postfix operator cannot be followed by a postfix operator of lower precedence.
 The expression A'(X) cannot be reordered to A(X)'
 Consecutive number of postfix operators that have different levels of precedence don't necessarily work from inside to outside.
What can be done?
1. Generate syntax error and rewrite A'(X) as (A')(X) that is not acceptable.
2. Introducing three compound operators as explained in comment13.
 CafeOBJ is a language for writing formal (i.e. mathematical) specifications of models for wide varieties of software and systems, and verifying properties of them.

It can be concluded from your comment that it is impossible to design a language that only has multiple postfix operators with different level of precedence and if precedence is defined for them it is completely useless and meaningless because as you have noted all unitary operators , regardless of their level of precedence , are applied to their operator from inside to outside.
But I can refer you to papers (1) that have designed algorithms for parsing languages that have postfix,prefix and infix operators of different level of precedence.
Here is the common definitions for precedence and associativity that can be googled:
The operator precedence rules for expression evaluation define the order in which “adjacent” operators of different precedence levels are evaluated.
The operator associativity rules for expression evaluation define the order in which adjacent operators with the same precedence level are evaluated.
So the execution of unary operators from the inside to the outside is related to associativity.
The ++ in the expression a++(3) is postfix and it works similar to languages such as c++. In c++ the postfix operator and the call operator have the same level of precedence.
(1) Aasa A. (1991) Precedences in specifications and implementations of programming languages. In: Maluszyński J., Wirsing M. (eds) Programming Language Implementation and Logic Programming. PLILP 1991. Lecture Notes in Computer Science, vol 528. Springer, Berlin, Heidelberg

No, that was my point in comment #6: precedence is about where parentheses are implicitly assumed. What you are claiming that the present documentation implies corresponds to reordering, which is something different. And there is also a reason why this is so: rules of precedence allow you to be lazy (and also for code to be more readable) by dropping parentheses where they are implied by precedence. So there is something to be gained. Your permutation does not gain anything: when you want
you write it like that, and when you want
you write it otherwise. There is absolutely nothing to be gained when the rules would be understood as you claim (that is, both corresponding to the same operation), neither in terms of laziness (in both cases four characters), and much less in terms of readability.
I think that there is no reasonable language at all that has syntactical rules in some sense comparable to Octave, where unary operators would not work from the inside to the outside. Take the following instance
You claimed that "(3)" has higher precedence, and "++" is applied afterwards. But is this "++" postfix or prefix (that is, would the result be 0 or 1)? After all, it is after one part of its supposed argument, but before the other. Should we now introduce a new concept "medifix"? No, I hope that I have convinced you that your view is perfectly unreasonable.

Having a look at comment2 and considering the following example:
Here A is followed by two postfix operators: ' and () .
Now we should refer to the table of precedence to know which one should be applied first.
The table of precedence says that () has higher precedence so () is applied first and then ' is applied to the result.
According to the current table of precedence it should behave as:
But currently it doesn't take into account precedence of operators and behaves as:
If anything is ambiguous I can provide more clarifications.

I still don't get why you would need the compound operators. I think you haven't made clear yet in your eight comments to date inhowfar the behaviour of
as you describe it (where the description is of course valid, and the described behaviour is intended) isn't correctly specified by the present documentation.

I think here is my final suggestion.
Three new operators will be introduced in the documentation:
Where {t/c...} means consecutive number of transpose/ctranspose operators, and those three operators take the same level of precedence as of the call operator. For example in the expression:
the consecutive number of transpose/ctranspose operators followed by a call operator are regarded as one operator and it behaves as :
Also as a bug is reported in comment8, I think the postfix increment/decrement operators should have the same level of precedence as of the call operator.
Therefor the table of precedence will be changed to :
Thanks to all.

@Michael Leitner Good point!
Now the precedence table can be changed to :
and operator collapsing and virtual operators can be used.

'(a^b)++' just gives an error. But:
Thus () definitely needs higher precedence.

Yes, my guess is that the claim on righttoleft associativity for the postfix inc/dec operators is borrowed from the C++ precedence rules. But that associativity really only matters when combining the inc/dec operators with other unary operators of the same precedence level, like the C++ expressions '*a++' or '!b'. Since '++' and '' can't be combined without using a binary operator of a lower precedence, I don't think we need to claim that they associate from right to left in Octave.
But the Octave postfix inc/dec operators do have higher precedence than the exponent operator. The expression 'a^b++' is not the same as '(a^b)++', which would be the result if they all had equal precedence and left to right associativity.

@Mike Miller
My last proposed comment is the same as your suggestion and after posting my comment I saw your comment. It is OK.

I have found the simplest solution!
Here is the first 3 rows of the current table of precedence:
Other rows of the table are truncated because are unrelated to the current discussion.
If all of them are assigned the same level of precedence the problem is solved.
So all of them should be on the first row:
Now in the expression:
Because the transpose operator and the call operator have the same level of precedence the expression is executed from left to right.
Currently there is a bug about precedence of call/indexing operator over postfix increment/decrement operator that now can be fixed as the current issue.
Consider the expression:
Regarding the current table of precedence it should be evaluated as:
But it is evaluated as:
With the proposed modification to the documentation of the precedence table this bug also can be resolved.
@Michael Leitner
As you I think that "postfix increment and decrement group right to left." is wrong and they group left to right.
Your second point about +++a is the same as c++ behavior. You can see this stackoverflow question. They have noted that it is related to maximum munching rule.

I agree with Michael, in that we should proceed under the assumption that the current behavior is correct and the documentation just needs to be updated. I also agree with Michael that all the verbiage about operator collapsing and virtual operators is way too complex.
I think it would be enough to rearrange the operator precedence table to make it clear that the unary transpose operators have the same precedence as the parentheses and bracket operators, and associate from left to right. Is anything more than that needed?

I was talking about your comment #3, which I read as your suggestion on an improvement of the documentation assuming the behaviour is intended and should stay as it is. Under this assumption my comment is to be understood. My claim is that the statement "Unary operators always group from inside to outside" is both intended and also describes the current behaviour. Yes, it is not explicitly stated in the documentation, thus there is room for improvement.
You are correct, my statement "Unary operators always take precedence before binary operators" ignores the relation between prefix increment/decrement and exponentiation. The fact that the former takes precedence leads to
resulting in an error (because a^b is 1 and not an lvalue) instead of 2 (which would result if the increment was evaluated first). But of course
does not give an error, because here the increment is evaluated first. The reason is that an operator can of course only be evaluated when its arguments have been evaluated, which is a metarule that takes precedence over the rules of precedence. And note that also in the languages cited by you, it is no question that operator precedence is always to be understood as where parentheses are to be set, and not as how operators are to be permuted (they are never permuted). What you imply with your examples corresponds to permuting the operators. So in this light I would even say that the documentation is okay as it is (apart from the grouping of the postfix increment/decrement, which I really do not understand).

"Isn't this unnecessarily complex?"
The rules related to precedence, associativity, side effects and the order of evaluation are unavoidably complex. It is sufficient to have a look at the documentation of the languages, like c++ and java, that have standardization. The point is that the MATLAB has more restricted problems related to Octave because it doesn't allow indexing on a temporary and doesn't have operators like += or ++.
"Unary operators always group from inside to outside"
This statement neglects the precedence of operators. It should be modified to Unary operators always group from inside to outside if they have the same level of precedence.The transpose operator and the call operator don't have the same level of precedence, due to higher precedence of () operator it should be evaluated before ' operator.
"Unary operators always take precedence before binary operators"
The only exception is the power operator that has higher precedence than ‘+’ ‘’ ‘++’ ‘’ ‘~’ ‘!’ operators and has equal precedence related to .' and ' operators.bug 54491
"Is an explicit rule about operator collapsing actually needed?"
Consider the following example:
If we don't use the collapsing rule and only use the compound '() operator because the operators '() and () have higher precedence than ' operator then the expression is translated to:
That is not the intended behavior.
I should correct my previous opinion to:
Collapsing rules say that consecutive transpose operators, regardless of what actually is done internally, should be treated as one operator.
For the builtin numeric types, for the purpose of optimization, we can internally reduce the consecutive transpose operators to 9 new compound types but for the userdefined transpose/ctranspose operators the operators should not be collapsed and should be executed consecutively but the user should be informed that they are treated as one operator and we may have infinite compound operators like ''''() and .'.'''().
Instead of those complicated rules we can respect the current documentation, fix the bug and instead of writing a'(:) we can write (a')(:) .(Changing all instances of such use to parenthesised use)
I think the increment operator may divert the current discussion and it potentially is subject of multiple new bug reports!

Isn't this unnecessarily complex? Is an explicit rule about operator collapsing actually needed? To my view, the only point where the documentation could perhaps be misunderstood is the fact that unary operators always group from inside to outside, and that unary operators always take precedence before binary operators. The second statement follows from the given precedence table, but the first is not explicitly stated.
If we have these rules, then an expression is parsed by first deciding whether a given '+' is unary or binary and so on, then applying all the unary operators onto their objects from inside to outside (where you have only to decide whether the prefix or the postfix has higher precedence), and finally the binary operators according to their precedence and grouping rules.
Two points that I noticed: it says that postfix increment and decrement group right to left. But is this relevant? I do not see a possibility how this could arise in a valid expression, e.g. "a=0;a++++" is not accepted. And anyway it would be nonsense, because grouping from right to left would mean that this should be understood as "a(++(++))".
And the second point: I would have assumed that "+++a" would be valid, and would be just the converse of "++a". But while the second is parsed correctly, for the first expression the parsing seems to result in the increment of "+a", which is not an lvalue. But as it seems that the only documentation of Octave's syntax is the implementation, the parser obviously works correctly by definition.

If the current Octave behavior is intended inform the user the following points:
1. Rules of operator collapsing:
First consecutive operators are collapsed before applying the rules of precedence.
The odd number of consecutive transpose operators reduce to one operator.
The even number of consecutive transpose operators reduce to nothing.
Interleaved transpose and ctranspose operators are arranged then collapsed:
2. Introduce the , virtual , conj operator as:
So A.'' is the same as conj(A) and the operator has the same precedence as the transpose operator.
3. Introduce the following virtual compound operators:
They have the same precedence as call operator and they take transpose of their operand and then apply indexing.
Consider the following example:
Collapsing the operators reduces the expression to:
Both operators .'() and () has the same precedence then the left to right rule applies and .'() executed before ().
All of the changes will be reflected in the documentation and nothing to be done in the source code except that you want to use operator collapsing to perform some sort of optimizations.

Assuming that we have an operand A and two postfix operators OP1 and OP2.
So in the following expression:
If OP1 takes higher precedence over OP2 then OP1 should apply to the operand before OP2.
If OP2 takes higher precedence over OP1 then OP2 should apply to the operand before OP1.
If both of OP1 and OP2 has equal precedence then the left to right rule of MATLAB applies and OP1 should apply to the operand before OP2.
Some interesting results:
The following expressions should produce the same result:
The following expressions should produce the same result:
If the current Octave behavior is the desired one let me to post some workarounds in the following posts.

Thank you for the bug report. This looks like it may be more about clarifying the documentation. Whatever the documentation says, it is not correct that a'(2) and a(2)' should produce the same result.
For example, Octave uses the syntax a.'(:) in many places, which has the effect of performing a transpose and then arranging the result in a single column. This is the intended behavior.
So given that, would you like to describe how you believe the documentation to be incorrect and help suggest improvements?

According to the documentation the call operator takes precedence over transpose/ctranspose operators so considering the following piece of code
Because of higher precedence of the indexing operator it should behave as :
and it should show 3 but it currently behaves as:
that displays the wrong result of 2.
In other words a'(2) and a(2)' should produce the same result.
