bugGNU Octave - Bugs: bug #48428, Support for java.lang.XXX syntax

 
 

bug #48428: Support for java.lang.XXX syntax

Submitter:  Rik <rik5>
Submitted:  Wed 06 Jul 2016 10:59:39 PM UTC
   
 
Category:  Interpreter Severity:  1 - Wish
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 20 Jul 2018 07:58:49 PM UTC, comment #21: 

I don't have answers to all this, but here's some thoughts.

> Given that we currently allow
>   fcn.field
> to call the function fcn and then perform structure field indexing on the result, are you proposing that we continue to allow that, even if there is a +fcn package directory in the load path?


I'm going to defer to your thoughts on this: since I'm an Octave newbie, I'm hesitant to propose any breaking changes to current behavior.

If you prioritize full Matlab compatibility, then yes, I think we probably need to do the four things in your list instead.

> Are you saying that the rules you list in comment #19 should only apply to index sequences that begin with '.' operators?


No - a plain identifier, or an index sequence that begins with (), should be considered a degenerate case of this with 0 .-indexes, and the initial identifier should be checked to see if it names a Java class in the root/default package. Similar to how it could be an Octave classdef name, for calling a constructor.

> Do we ever need to consider more than the current index?


Effectively, yes. Java package names are flat, not hierarchical. That is, if there is a Java package "a.b.c", that does not imply the existence of packages "a" or "a.b". And if they do exist, there is no relationship between them. We could construct those "implied" packages in ov-java, but the JVM won't handle it for us.

> I think we need to recognize '()' followed by '.' and pass both to the octave_struct subsref method. Otherwise, we can't do the right thing for a user-defined subsref function for structs.


I think you're right. The way Matlab does it, as soon as a classdef object with a user-defined subsref() is recognized, then all of the indexing expressions in the entire rest of the chain are passed in to its subsref() method in a single call, and it can do whatever it wants with the entire chain.

>  Calling fcn with no args and indexing teh resulting struct with '(args).field' makes sense, but so does calling fcn(args) and then indexing the result. Should the expression always be 'greedy' and grab as many indices as can possibly make sense?


I think you need to be greedy, at least with () indexes. Otherwise, for a simple "fcn(ix)", how are you going to differentiate between calling fcn with the argument ix versus calling fcn with zero arguments and indexing in to the result? You can always force the other behavior with "(fcn)(ix)".

There's another complication with fcn.field. Classdef names are also constructor names, so they're functions, too. If there is a class named "fcn", should "fcn.field" access a Constant property of the class, or call the zero-arg constructor, and index into the new object? Constant properties can be accessed through classdef instances, so it would still "work", but it seems undesirable to have the instance creation as a side effect.

Andrew Janke <apjanke>
Fri 20 Jul 2018 01:25:35 AM UTC, comment #20: 

Given that we currently allow


fcn.field


to call the function fcn and then perform structure field indexing on the result, are you proposing that we continue to allow that, even if there is a +fcn package directory in the load path?

Or should we be checking whether

  1. fcn is a variable (could hold a classdef or java object)
  2. fcn names a classdef(?) or java object for referencing static members (I don't even know whether that's how these things work)
  3. fcn names a package directory, in which case it limits the scope of additional symbols in the expression
  4. fcn is a regular function to call.  At this point, if we want to be compatible with Matlab, we don't allow any further indexing except '(...)' for function call arguments.


Are you saying that the rules you list in comment #19 should only apply to index sequences that begin with '.' operators?  Do we ever need to consider more than the current index?

I think we could do a better job with the 's(i).f' form of structure array indexing.  Currently, I think we are splitting this indexing operation into 's(i)' and then applying the '.f' field selection to the result.  Instead, I think we need to recognize '()' followed by '.' and pass both to the octave_struct subsref method.  Otherwise, we can't do the right thing for a user-defined subsref function for structs.  I know we don't handle that case now, but it should at least be possible and as far as I can tell, the current code doesn't allow it to work properly.

I find the visit_index_expression function to be way too complicated as it is.  Maybe eliminating some of the Octave extensions to Matlab indexing would help with that.  Maybe we won't have to eliminate them all.  But would doing away with fcn.field really be that bad, especially if we still allowed fcn(args).field?  OTOH, maybe we dan't have that either, because it looks like a struct array indexing operation.

What about a function call that produces a struct array?  If we allow fcn(args).field.  Calling fcn with no args and indexing teh resulting struct with '(args).field' makes sense, but so does calling fcn(args) and then indexing the result.  Should the expression always be 'greedy' and grab as many indices as can possibly make sense?

Ugh.


John W. Eaton <jwe>
Group administrator
Thu 19 Jul 2018 11:08:56 PM UTC, comment #19: 

I forgot to mention what I think the actual resolution algorithm should be. I think it should be:

Given a dot-indexed expression "a.b.c.d.e.f" that is at the start of an indexed expression:

If "a" does not resolve to an Octave variable or function, then:

1. Consider each prefix of the dot-indexed sequence (e.g. "a", "a.b", "a.b.c", "a.b.c.d", ...)
2. If there is a Java class in the JVM's classloaders that exactly matches that fully-qualified name, then it is a Java class reference.
3. Once a class reference is found, keep considering subsequent .-indexes to see if they are inner classes of that class.
4. Once there are no more inner class .-references, you have the Java class reference that stuff should be dispatched on.
5. Consider the single next indexing expression
 a. If it is a ()-index, then that is a constructor call.
 b. If it is a .-index, then that is a static field access or static method call.
 c. If it is a {}-index, that is an error.
 d. If there is no next indexing expression, then it is a zero-argument constructor call.
6. Take the resulting value and evaluate the remaining chained indexings as normal.

5.b.i) If there are multiple overloaded static methods of the same name, choose between them based on their signatures and the input arguments using the existing mechanism for choosing between overloaded non-static Java methods.

This logic would be split between pt-eval, load-path, and ov-java as appropriate.

Examples:

Given a Java class "a.b.c.Foo" with static field "fld" and static method "meth", and without a "bar" static field or method, then:

  • "a.b.c.Foo" is a zero-arg constructor call.
  • "a.b.c.Foo()" is also a zero-arg constructor call.
  • "a.b.c.Foo.fld" is a field access on "fld".
  • "a.b.c.Foo.fld(ix)" is a field access on "fld", whose value is then indexed into using normal Octave Java value ()-indexing.
  • "a.b.c.Foo.meth" calls meth() with zero arguments.
  • "a.b.c.Foo.meth(42)" calls meth() with the argument 42.
  • "a.b.c.Foo.meth(42)(3)" calls meth(42) and then ()-indexes into the returned value using normal Octave Java value ()-indexing (array element access).
  • "a.b.c.Foo.meth(42).bar" calls meth(42) and then .-indexes into the returned value using normal Octave Java value .-indexing (field access).
  • "a.b.c.Foo.bar" raises an error about "no static field or method named 'bar' in class 'a.b.c.Foo'". It does not call the zero-arg "Foo()" constructor and then try to access ".bar" on the resulting object.
  • "a.b" results in an "'a' undefined near line 1 column 1" error, the way it currently does. It does not result in referring to a Java package.
  • "a.b.c" results in an "'a' undefined near line 1 column 1" error, the way it currently does. It does not result in referring to a Java package.


The one about "a.b.c.Foo.bar" not calling the "Foo()" constructor is a bit of a discontinuity in the syntax pattern, but I think it results in more understandable error messages, and avoids some ambiguity. If you really want the 0-arg constructor and then to index into its results, you could call "a.b.c.Foo().bar".

Andrew Janke <apjanke>
Thu 19 Jul 2018 10:22:36 PM UTC, comment #18: 


> Andrew: Yes, that's a place to start, but there really shouldn't be too much code added there for supporting Java.
>
>I think the subsref and subsasgn methods in the octave_java class in ov-java.cc should be doing most of the work.


I agree. The pt-eval stuff just needs to evaluate the indexed expression enough to determine whether there is a Java class reference, and the fully-qualified name of the class. At that point it can hand it off to ov-java.cc to do the actual work/invocation/value construction.

> Is tree_evaluator::visit_index_expression not handling index expressions properly for Java calls like
>
> java.lang.foo.bar
> What needs to be done differently?


visit_index_expression calls evaluate (expr), and by the time the value is returned, from what I can tell, the left-hand parts of the index expression have already been resolved to identifiers, Octave packages, or so on. The determination whether something is a Java reference will need to happen inside that "evaluate (expr)". I think it needs to happen next to the load-path evaluation like Mike Miller says: if the resolution doesn't work by looking on disk for +pkg dirs and M-code files, then it needs to look in the Java classloaders for class definitions.

> What are the name resolution priority rules in Matlab? If I have a file '+org/example/name.m' and 'org/example/name.class', which one takes precedence?


Surprisingly, the Java class takes precedence.

This is done on a per-class, not per-package basis. So if you have some Java classes and M classdef files:


a/b/Class1.class
a/b/Class3.class
+a/+b/Class1.m
+a/+b/Class2.m


then a given function may do:

function foo()
  a.b.Class1  % resolves to Java
  a.b.Class2  % resolves to Matlab/Octave
  a.b.Class3  % resolves to Java
end


So the referent of "a.b" can be multiple things in a single scope. This may take some adjustment in the code.

> Also, in Octave we allow return values to be used as expressions, Matlab does not allow this. So in Octave an expression like 'uname.sysname' resolves to a function call 'uname()', and access of the struct field 'sysname'. It would be nice to preserve that if possible, so something like 'org.example.name' might be a namespaced function, or it might be just a function named 'org'.


I think we could write it either way, and it's just up to you folks to choose what behavior you want in the case where both are defined. Should a function or variable named "org" shadow the "org" namespace, so "org.example.name" is a .-reference performed on the output of calling "org"? Or should the existence of an "org.example.name" class take precedence? Matlab does the latter. Sounds like Octave does the former for Octave classes; it would be consistent to do it that way for Java classes too.


octave:13> system('find .');
.
./a.m
./+a
./+a/+b
./+a/+b/Class2.m
./+a/+b/Class1.m
octave:14> a
ans =  42
octave:15> a.b.Class1
error: scalar cannot be indexed with .
octave:15>


I can see a couple arguments for doing it the latter way:
1. Matlab compatibility.
2. Disambiguation: if namespace-qualification took precedence by default, because Octave allows arbitrary indexing chains, then you could force it to interpret the "a" in "a.b.c" as a variable or function by doing "(a).b.c" or "a().b.c". I don't see an easy syntactic way of disambiguating it to be a namespace qualification if variable/function reference takes precedence by default.

The big argument against this, of course, is that it would be a breaking change for existing Octave users. Not something to be undertaken lightly. So we should probably just do Java support with the current precedence rules, and consider changing them separately if at all.

> It would be unpleasant if we had to give up arbitrary indexing chains and/or chained assignments. [...] [M]aybe now we are seeing why this is impossible for Matlab? I don't know.


I don't see any syntactic reason why we would have to give them up. I think Java class resolution could work just fine in their presence.

I've always assumed Matlab's prohibition on arbitrary indexing chains was a design choice to keep people from making "hard-to-read" complex indexing chains, not a syntactic constraint. I have also never been able to come up with a simple & clear example why it's not possible.

> What about overloading subsref? Does it cause trouble there?


I do not think overloading subsref would cause a problem here, because an overloaded subsref would only be called if part of a dot-indexed expression resolved to a user-defined class. And, if I understand this correctly, that's an either/or with the expression resolving to a Java class reference, so I don't think there's a case for interaction.

And with regards to arbitrary index chains, I don't know of any cases where arbitrary index chains couldn't be converted to subsref arguments; it's a pretty general signature.

Unless you wanted to get fancy and allow user-defined Octave classes to generate objects that would be interpreted as Java package or class references. I had assumed this would just be an internal interpreter feature, at least for now.

> I'm also interested to know if anyone can see this being extensible, for example a user oct file registering its own top-level namespace offering dynamic name resolution.


This idea scares me. :) I like the simplicity of Matlab/Octave code, where it's usually easy to trace a given function's definition back to its source code on disk.

> Actual name resolution probably needs to be worked into the existing code in load-path [...]


I'll look through the load-path code. Thanks.

Andrew Janke <apjanke>
Thu 19 Jul 2018 07:09:03 PM UTC, comment #17: 

It would be unpleasant if we had to give up arbitrary indexing chains and/or chained assignments.  I know people like to be able to do things like


svd(a)(1:n)


for example.  But I think this also makes the interpreter implementation more complicated.  And maybe now we are seeing why this is impossible for Matlab?  I don't know.  I've never been able to come up with a simple and clear example that shows exactly why arbitrary indexing chains aren't possible.  What about overloading subsref?  Does it cause trouble there?

John W. Eaton <jwe>
Group administrator
Thu 19 Jul 2018 06:29:26 PM UTC, comment #16: 

For precedence ordering, see http://www.mathworks.com/help/matlab/matlab_prog/function-precedence-order.html.

From what I see, classes and objects take precedence over ordinary functions within directories (+package).  This can be altered by using the import function, but we don't implement that yet anyways.

Rik <rik5>
Group administrator
Thu 19 Jul 2018 06:16:24 PM UTC, comment #15: 

Actual name resolution probably needs to be worked into the existing code in load-path, which looks for dotted names and attempts to look up '+package' names on the Octave load path.

What are the name resolution priority rules in Matlab? If I have a file '+org/example/name.m' and 'org/example/name.class', which one takes precedence?

Also, in Octave we allow return values to be used as expressions, Matlab does not allow this. So in Octave an expression like 'uname.sysname' resolves to a function call 'uname()', and access of the struct field 'sysname'. It would be nice to preserve that if possible, so something like 'org.example.name' might be a namespaced function, or it might be just a function named 'org'.

I'm also interested to know if anyone can see this being extensible, for example a user oct file registering its own top-level namespace offering dynamic name resolution.

Mike Miller <mtmiller>
Group Member
Thu 19 Jul 2018 03:48:44 PM UTC, comment #14: 

Andrew:  Yes, that's a place to start, but there really shouldn't be too much code added there for supporting Java.

I think the subsref and subsasgn methods in the octave_java class in ov-java.cc should be doing most of the work.  Is tree_evaluator::visit_index_expression not handling index expressions properly for Java calls like


java.lang.foo.bar


What needs to be done differently?

I did some work recently on visit_index_expression and the subsref functions in the octave_value classes and I was looking at this code again yesterday to try to figure out how the next_subsref methods could be eliminated.  So I'd be glad to discuss this with you further, either here, on IRC, or the maintainers mailing list.

John W. Eaton <jwe>
Group administrator
Thu 19 Jul 2018 10:18:48 AM UTC, comment #13: 

Nevermind; found it. It's in libinterp/parse-tree/pt-eval.cc, in tree_evaluator::visit_index_expression.

Andrew Janke <apjanke>
Thu 19 Jul 2018 09:42:27 AM UTC, comment #12: 

I would like to try implementing this, if someone can point me at the place in libinterp where it handles namespace resolution, to get started.

Andrew Janke <apjanke>
Wed 10 Aug 2016 11:56:29 AM UTC, comment #11: 

A, my failure:
This is java code.
In fact, in matlab constructor is without new.
So i cannot know whether an according form of constructor of arrays works in matlab.

Does anyone have a working matlab at hand??

Ernst Reissner <ernstreissner>
Wed 10 Aug 2016 01:10:16 AM UTC, comment #10: 

Is new a keyword in the Matlab language or a function? What happens if you shadow new before trying to create a java array with new in Matlab?

Carnë Draug <carandraug>
Group Member
Tue 09 Aug 2016 11:22:23 PM UTC, comment #9: 

I have forgotten:
seemingly also javaArray can be written
in the form java.lang...
in the matlab docs I found in

http://de.mathworks.com/help/matlab/ref/javaarray.html?searchHighlight=javaArray

the snippet:
A = new PackageName.ClassName[x1]...[xN];

Ernst Reissner <ernstreissner>
Tue 09 Aug 2016 11:07:45 PM UTC, comment #8: 

This thread concentrates on object creation with javaObject.
But there are still static methods and fields.
 
Semmingly matlab does not allow static methods
invoked in java style like java.lang.System.load(...)
but i am not sure...

What about set and get fields?
is java.lang.Double.MAX_VALUE possible?
Maybe for set/get only?

Ernst Reissner <ernstreissner>
Sun 31 Jul 2016 08:36:46 PM UTC, comment #7: 

The only way I have to check the usefulness of supporting java.XXX only vs a proper fix is via Octave Forge.


$ grep -horP 'javaObject\s*\(['"'"'"]\s*[a-zA-Z\.]*['"'"'"]' octave-forge-super/ | grep -oP '(?<=['"'"'"]).*(?=['"'"'"])' | sort | uniq -c | sort -rn
     23 com.sun.star.uno.Type
      7 org.odftoolkit.odfdom.doc.table.OdfTableCell
      6 java.io.File
      4 org.odftoolkit.odfdom.doc.table.OdfTableRow
      3 org.odftoolkit.odfdom.doc.text.OdfTextParagraph
      3 java.lang.Short
      3 java.io.FileOutputStream
      3 com.extentech.ExtenXLS.WorkBookHandle
      2 jxl.write.Label
      2 java.util.LinkedList
      2 java.io.FileInputStream
      2 com.sun.star.beans.PropertyValue
      1 org.odftoolkit.odfdom.doc.table.OdfTable
      1 org.apache.xml.serialize.XMLSerializer
      1 org.apache.xml.serialize.OutputFormat
      1 org.apache.xerces.parsers.DOMParser
      1 org.apache.xerces.dom.DocumentImpl
      1 org.apache.poi.xssf.usermodel.XSSFWorkbook
      1 org.apache.poi.hssf.usermodel.HSSFWorkbook
      1 net.sourceforge.ltfat.SpectFrame
      1 net.sourceforge.ltfat.ContFrame
      1 jxl.write.Number
      1 jxl.write.Formula
      1 jxl.write.Boolean
      1 jxl.write.Blank
      1 javax.swing.table.DefaultTableModel
      1 java.io.StringWriter
      1 java.io.BufferedOutputStream


Which would suggest that supporting java.XXX only is not that important (almost all of the hits are from the io package and some from ltfat). That kind of makes sense, most of the stuff in the java namespace is core stuff that Octave will already support natively. If it's being used, is for use in java classes outside the java namespace.

Carnë Draug <carandraug>
Group Member
Sun 31 Jul 2016 05:38:19 PM UTC, comment #6: 

I borrowed this for pytave because every Python class is prefixed with "py." in Matlab's interface.

I agree this could be used for the "java.xxx" namespace only which may be partially useful, but also misleading since it won't support any other namespaces.

Mike Miller <mtmiller>
Group Member
Sat 30 Jul 2016 01:57:09 PM UTC, comment #5: 

The hack I proposed on comment #4 is now being used in pytave to support py.foo syntax so maybe it's not that bad and we could consider to support java.XXX only.


One thing that I now realize for supporting this notation on other java namespaces. If we want to cover anything in the javaclasspath, we will have to start the JVM the first time someone calls an undefined function of mistypes a variable name. That's a performance hit we might not want to have.

Carnë Draug <carandraug>
Group Member
Thu 14 Jul 2016 03:11:52 PM UTC, comment #4: 

This can be easily worked around for java.X classes with the following class:


$ cat @java/java.m
function j_obj = java ()
  j_obj = class (struct (), "java");
endfunction
$ cat @java/subsref.m
function obj = subsref (j_obj, idx)
  dots = strcmpi ({idx.type}, ".");
  args_start = find (! dots, 1);
  classname = strjoin ({"java", idx(1:(args_start -1)).subs}, ".");
  obj = javaObject (classname, idx(args_start:end).subs{:});
endfunction
$ octave
octave-gui:1> jstr = java.lang.String ("Hello World");
octave-gui:2> jstr.toString
ans = Hello World
octave-gui:3> jint = java.lang.Integer (5);
octave-gui:4> jint.byteValue
ans =  5


Obviously this will not be merged but it might be useful if anyone needs to make some Matlab code work and they only use java.XXX classes.

Carnë Draug <carandraug>
Group Member
Thu 07 Jul 2016 04:45:14 PM UTC, comment #3: 

I'd like to leave this up as a stand-alone item.  The list on the task tracker is pretty long, and I want to see if we can get this one feature implemented for 4.2.

Rik <rik5>
Group administrator
Thu 07 Jul 2016 04:25:32 PM UTC, comment #2: 

There's task tracker entry (task #12601) where this is also mentioned in your original report.

Should we close this bug and refer to the task tracker?

Philip Nienhuis <philipnienhuis>
Group Member
Wed 06 Jul 2016 11:00:48 PM UTC, comment #1: 

More specific examples that work in Matlab, but not Octave


jint = java.lang.Integer (5);
jstr = java.lang.String ("Hello World");



Rik <rik5>
Group administrator
Wed 06 Jul 2016 10:59:39 PM UTC, original submission:  

Matlab supports java.XXX syntax to easily use Java in m-files.  Octave only supports the longer form using javaObject, the class name, and any function arguments.

It would be nice to add a little bit of syntactic sugar in the parser to possiby convert


java.XXX.YYY (ZZZ)


to


javaObject ("java.XXX.YYY", ZZZ)




Rik <rik5>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by ernstreissner (Posted a comment)
  • -email is unavailable- added by rik5 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-02-15 mtmiller Carbon-CopyRemoved 80942 -
    2018-07-19 rik5 StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code