bugGNU Octave - Bugs: bug #47676, Cannot apply computed assignment...

 
 

bug #47676: Cannot apply computed assignment to a variable defined after the code was parsed (e.g., in a script)

Submitter:  Carlo de Falco <cdf>
Submitted:  Tue 12 Apr 2016 01:05:52 PM UTC
   
 
Category:  Interpreter Severity:  2 - Minor
Priority:  3 - Low Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Release: 
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 09 Dec 2022 05:24:43 PM UTC, comment #22: 

regarding jwe's last question on this one, the outputs in Matlab 2022b:


>> fun
Error: File: fun.m Line: 3 Column: 3
Using identifier 'xyz' as both a variable and a command is not supported. For more information,
see "How MATLAB Recognizes Command Syntax".

>> xyz = pi

xyz =

    3.1416

>> xyz -pi
Undefined function 'xyz' for input arguments of type 'char'.


in Octave 8.0.1 (hg id: 24bd675bceab), nearly identical behavior:

>> fun
error: parse error near line 3 of file C:\temp\fun.m

  xyz: invalid use of symbol as both variable and command

>>>   xyz -pi;
     ^
>> xyz = pi
xyz = 3.1416
>> xyz -pi
error: variable "xyz" used as function in command style expression
Check whitespace around potential binary operator.



That said, seeing this marked Patch submitted, in addition to the patch no longer applying (some modified file no longer exists) it seems the topic of this report may have been superseded by more recent octave versions and in particular details covered in bug #62552.

seeing as how the original bug was concerned with code producing errors in either scripts/CLI but not the other, and that now seems consistent, closing as fixed.

Nicholas Jankowski <nrjank>
Group Member
Thu 10 May 2018 07:36:53 PM UTC, comment #21: 

Looking at

https://www.mathworks.com/help/matlab/matlab_prog/command-vs-function-syntax.html

it appears that Matlab may do different things depending on whether it is parsing something at the command line or in a function or script, and if it is parsing a function, it may not look to see whether a symbol already appears to be a variable.  Is that correct?  That's not the way I thought Matlab's parser behaved.  Would someone like to try testing that?  For example, does the following return 0 or does it try to call xyz with the argument '-pi'?  Or something else?


function r = fun ()
  xyz = pi;
  xyz -pi;
end


And, at the command line, does


xyz = pi
xyz -pi


perform subtraction or attempt a function call?

In any case, back to Mike's question in comment #18: if we use syntax alone to parse expressions with OP= operators, should we treat them like =, where spacing isn't significant, or like other operators, where it is?

And, should the behavior really be different at the command line?

John W. Eaton <jwe>
Group administrator
Thu 10 May 2018 06:16:18 PM UTC, comment #20: 

Right, if someone has working Matlab code that does "count += 3" and they actually want that to call a count function with string arguments of "+=" and "3".

Mike Miller <mtmiller>
Group Member
Thu 10 May 2018 06:01:12 PM UTC, comment #19: 

I would be OK with that, but if I recall correctly, because those operators don't exist in Matlab, it would be incompatible with Matlab's behavior for the same statement.  So working Matlab code would fail in Octave.

John W. Eaton <jwe>
Group administrator
Thu 10 May 2018 05:42:55 PM UTC, comment #18: 

IMHO, we should change the parser to allow this case to work.

Either parse all assignment operators equivalently, i.e. any assignment or compound assignment operator tags its left operand as a variable. Or at the very least, parse compound assingment operators like other binary operators, if they have space before and after the operator, then it's not a command.

Mike Miller <mtmiller>
Group Member
Thu 16 Jun 2016 06:23:34 AM UTC, comment #17: 

In the future, it would also be good to have the syntax highlighting in the GUI editor show "word command" arguments in the "string" colour.

That will have to wait until the GUI has access to the parse tree (one of my medium term goals).

Lachlan Andrew <lachlan>
Mon 13 Jun 2016 09:48:17 AM UTC, comment #16: 

Oops.  My previous patch had tabs in it. Try this one.

(file #37459)

Lachlan Andrew <lachlan>
Mon 13 Jun 2016 04:40:04 AM UTC, comment #15: 

Here is a patch that warns at parse time if A += B is parsed as A('+=', 'B'), or similar.

I'd be grateful for some feedback, especially on the trade-off between optimisation and readability.

This bug is more serious than I thought.  I thought that it was only triggered if a script is called while another script or function is running.

However, it is also triggered if the variable is created from the command line before a script is run, if an old version of the script is cached.  That means that the script could be run, throwing a warning but not failing due to some conditional code, and then be invoked again and fail without a warning.

(file #37458)

Lachlan Andrew <lachlan>
Wed 08 Jun 2016 07:23:35 AM UTC, comment #14: 

I just thought of another approach.

The parser known whether it is treating the string as a variable or a string, and what the first argument is.  If the first argument starts with something that looks like an operator, it can give a warning that is clearer than "error: Csb(61,_): but Csb has size 1x1".

Some care would be needed with "/" so that "ls /home/joe" doesn't cause a warning.  I'm not sure that much could be done about "ls /home".

This doesn't solve the problem, but it does avoid the worst part: silently giving the wrong result.

Lachlan Andrew <lachlan>
Thu 05 May 2016 07:29:25 AM UTC, comment #13: 

I wouldn't expect an to be thrown error in the case you describe.  However, I would also expect the number to be correct, since JWE's comment #4 suggested that the '/' would be interpreted correctly and Csb would be "called" with no arguments and return the value of the variable.

Did you get an incorrect answer?

Lachlan Andrew <lachlan>
Wed 04 May 2016 09:21:04 AM UTC, comment #12: 

Lachlan,

I tried changing example_1.m to


example_0
a = Csb / 2;


I still don't see any error.
Did you expect this to behave differently?
What did you expect exactly?


Carlo de Falco <cdf>
Group Member
Wed 04 May 2016 02:14:46 AM UTC, comment #11: 

Carlo, I agree that Matlab's response of throwing an error is better than Octave's of silently giving the wrong answer.

If Octave did "proceed with an unexpected value assigned to Csb", then I would consider it very much worse.  However, this issue only affects the printing of the non-semicolon "Csb /2".  If we actually do anything with the value (like using it in an assignment or function call), then I think Octave parses it correctly.

Lachlan Andrew <lachlan>
Thu 14 Apr 2016 03:34:04 PM UTC, comment #10: 

jwe,

Now that I think about it better, probably I can come up
with an example against the "#include" approach myself.

I tried the example lachlan suggested in comment #6 in both
Octave and Matlab and it fails in both cases,
but in different ways:

Matlab

>> example_1
Undefined function or variable 'Csb'.

Error in example_1 (line 2)
Csb /2

>> version

ans =

9.0.0.341360 (R2016a)


Octave

>> example_1
ans =

   0   0

>> version
ans = 4.1.0+



While they both look bad, Matlab's result is slightly better
because at least the code does not proceed with an unexpected
value assigned to Csb.

Can we at least replicate this behaviour somehow?

Carlo de Falco <cdf>
Group Member
Thu 14 Apr 2016 01:31:21 PM UTC, comment #9: 

Lachlan: I don't think reparsing scripts is an option either.

John W. Eaton <jwe>
Group administrator
Thu 14 Apr 2016 01:30:13 PM UTC, comment #8: 

Carlo:  I don't think it would be a good idea to try to do something that works like an include statement without an actual #include (or similar) directive that will make the intent clear.  I don't have time at the moment to provide an example, but if you want one I can probably come up with something.

Note that this problem exists with Matlab too.  The list of operators is just different.  For example, if you modify your script to have


Csb +fact


then I think it will fail because Csb is not recognized as a variable when example_0 is parsed so the statement is converted to Csb ('+fact').  Then when it is evaluated, it can't find a Csb function to call.  But if you write


Csb + fact


it will do what you expect because with this spacing, the arguments look like an expression.

Yes, this is ridiculous.

Yes, we can fix it for Octave so that the OP= operators are handled the same as the OP operators.  All I'm saying is that this will create an incompatibility.  But perhaps it is more useful to recognize the OP= operators as operators (at least in some cases, depending on whitespace, same as other operators) than to ignore them for the sake of compatibility.

I always thought that parsing commands this way was stupid, but I didn't get to vote on what Matlab was going to do.

John W. Eaton <jwe>
Group administrator
Thu 14 Apr 2016 11:31:54 AM UTC, comment #7: 

jwe, lachlan,

I'm not really an expert about parsers
so forgive me if the following suggestion
does not make any sense.

IIUC the issue is that when a script is called
it is parsed completely before executing and
if it contains a call to another script, the
parser does not know what symbols are defined
in the second script when interpreting the first
one.

If this is the issue, would it make any sense to
consider the call to a script from within another
script as an "include" directive and passing to the
parser the full contents of the two combine scripts
to parse together?

BTW, is there any connection between this issue and
bug #38236 ?





Carlo de Falco <cdf>
Group Member
Thu 14 Apr 2016 01:55:50 AM UTC, comment #6: 

Oops -- most of my comment was chopped off.

The gist was that if a script sets  foo = zeros(100,100), then


foo /2


would result in [1,1] with no warning that something was amiss.

As I've argued in the past, giving completely the wrong number with no warning makes Octave untrustworthy.  IMHO, there should be a separate Item Group for this serious class of "non-errors", since "incorrect result" is a catch-all for any incorrect behaviour.

Perhaps the parse tree could flag that something was deemed to be a "function", and this could be checked by subsref of non-function types, either to give a more informative error or ideally to trigger a reparse of the code.

Another alternative would be to explicitly check after each script invocation whether new variables were created.  If so, a reparse could again be triggered (if it is then possible to continue execution from that point).

Lachlan Andrew <lachlan>
Thu 14 Apr 2016 01:19:41 AM UTC, comment #5: 

JWE, your example


foo -bar


is really serious.  If a script sets


foo = ones (100,100);
-verbatim+

then

+verbatim+
foo /2


gives [1,1] -- a totally incorrect result with no errors or warnings raised.  Fortunately the size of the result is not the size of foo and so an astute observer would guess something is wrong -- but what if the size of foo was not known in advance?

To me, this is more than "really unfortunate".  Giving an "incorrect result" without an error is as bad as it gets -- if they occur frequently, no-one can trust results that Octave produces.  I have requested before that we have a separate tracker Item Group for this type of silent but serious error.  I've raised the severity of this bug report; I hope that is OK.

One way to solve the worst bug is to be cautious when using character expressions to index an array.  If a character is used to index an array and the first few characters could be an operator, the file could be re-parsed (somehow remembering where we are up to...) or at least a warning could be given.

Another option would be detect when new variables are added by a script and reparse the file when they are.  That will have a performance penalty, but there are two cases:

1. scripts are not called very often, in which case the cost is low

2. scripts are called often, in which case the seriousness of the bug is high and so even an expensive solution is warranted.


Regarding the issue of Matlab incompatibility, I think that "/=" is the most serious since it could easily be part of a file name.

(file #36921)

Lachlan Andrew <lachlan>
Wed 13 Apr 2016 12:08:36 PM UTC, comment #4: 

I mean that currently, unless foo is already defined as a variable, both Octave and Matlab will parse "foo += bar" as a "command" and then attempt to find a function foo to call with the arguments '+=' and 'bar' (as strings).  If we change Octave to recognize += as an operator in this case, then Octave's behavior will be different from Matlab in this case (and also for "foo +=bar").

Note that the way "commands" are parsed also depends on spacing.  I thought it was documented in the Octave manual, but apparently not.  Note that the following are handled differently (assuming the LHS is not already known to be a variable:


foo - bar  %% binary subtraction, foo is called with 0 args
foo- bar   %% binary subtraction
foo-bar    %% binary subtraction
foo -bar   %% command, foo is called with '-bar' argument


Yes, this is really unfortunate behavior.

Also my comment #2 should have said "NOT restarting your session", so foo.m is not parsed again.  The only thing that changes is that a mycmd function is added to the path.  In a way, this is a good thing, because it would be really bad for the behavior to also depend on what function names happen to be available on the path when a function is parsed.

John W. Eaton <jwe>
Group administrator
Wed 13 Apr 2016 10:47:31 AM UTC, comment #3: 

what do you mean by


foo += bar


will be handled differently in Matlab?


As far as I know "OP=" are not available at all in Matlab ...

Carlo de Falco <cdf>
Group Member
Tue 12 Apr 2016 05:21:16 PM UTC, comment #2: 

The problem is that the parser is interpreting the expression as a command, equivalent to


Csb ('*=', 'fact');


You may have already discovered that if you evaluate example_1 separately before example_0 or if you replace the call to example_1 with the contents of that file, then you will not see the error.  That's because a script file is parsed completely and then executed.  So when parsing example_0 by itself, Octave doesn't know that Csb is a variable so it assumes it can be called as a function.

I suppose we can add a special case for OP= operators so that they may be parsed as expressions even if the LHS is not yet defined as a variable.  However, this introduces a compatibility issue because then things like


foo += bar


will be handled differently in Matlab.

As I recall, checking to see if the symbol on the LHS is a function when parsing the expression is not the solution because the following is supposed to work:

Define in a file with mycmd undefined:


function foo (call_command)
  if (call_command)
    mycmd +arg1
  end
end


Then call it:


foo (0)  %% foo is parsed and there is no error for mycmd being undefined.
foo (1)  %% error for mycmd undefined


Now define


function mycmd (varargin)
  varargin
end


somewhere on the path (restarting your session) and call foo (1) again and it should display the argument to mycmd as a string.

John W. Eaton <jwe>
Group administrator
Tue 12 Apr 2016 04:07:05 PM UTC, comment #1: 

just an update noting that
the same error occurs if other
'OP=' opertaors are used.


Carlo de Falco <cdf>
Group Member
Tue 12 Apr 2016 01:05:52 PM UTC, original submission:  

The attached files demonstrate an error in the interpreter
when applying the multiplication-assignment operator to a
variable defined in a script.

To reproduce the issue, try:

>> example_0
error: Csb(61,_): but Csb has size 1x1
error: called from
    example_0 at line 5 column 1


if the same commands contained in the script example_0
are issued directly on the command line, the problem does
not occur:


>> example_1
>> fact = 1.0;
>> Csb *= fact;
>> Csb
Csb =  11
>>


The same issue is also present in the satble release but the error message is less informative.


Carlo de Falco <cdf>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #36925:  test_comment_6.tgz added by cdf (300B - application/x-tar)
file #36921:  example_2.m added by lachlan (18B - d2l/unknowntype)
file #36908:  example_0.m added by cdf (38B - application/octet-stream)
file #36909:  example_1.m added by cdf (13B - application/octet-stream)

 

Carbon-Copy List
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by lachlan (Updated the item)
  • -email is unavailable- added by cdf
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by cdf (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 18 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-12-09 nrjank StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
    2018-05-10 mtmiller Severity4 - Important 2 - Minor
        Priority5 - Normal 3 - Low
        Item GroupIncorrect Result Unexpected Error or Warning
    2018-05-10 mtmiller Dependencies- bugs #53871 is dependent
    2016-06-13 lachlan Attached File#37458 Removed
    2016-06-13 lachlan Attached File- Added bug_47676_warn_cmd_operator.cset, #37459
    2016-06-13 lachlan Attached File- Added bug_47676_warn_cmd_operator.cset, #37458
        StatusNone Patch Submitted
        SummaryError applying multiplication-assignment operator to variable defined in script Cannot apply computed assignment to a variable defined after the code was parsed (e.g., in a script)
    2016-05-12 rik5 Dependencies- bugs #47909 is dependent
    2016-04-14 cdf Attached File- Added test_comment_6.tgz, #36925
    2016-04-14 lachlan Attached File- Added example_2.m, #36921
        Severity3 - Normal 4 - Important
    2016-04-13 cdf Carbon-Copy- Added -email is unavailable-
    2016-04-12 cdf Attached File- Added example_0.m, #36908
        Attached File- Added example_1.m, #36909

    Back to the top

    Powered by Savane 3.15-e6e5.
    Corresponding source code