bugGNU Octave - Bugs: bug #44882, java arrays are not automatically...

 
 

bug #44882: java arrays are not automatically converted to octave types

Submitter:  Carnë Draug <carandraug>
Submitted:  Mon 20 Apr 2015 10:02:11 AM UTC
   
 
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Fixed Assigned to:  carandraug
Originator Name:  Open/Closed:  * Closed
Release:  * 4.0.0 Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 02 Jun 2015 04:46:16 PM UTC, comment #19: 

Retagging release from 4.0.0-rc3 to 4.0.0.

John W. Eaton <jwe>
Group administrator
Fri 24 Apr 2015 11:44:08 AM UTC, comment #18: 

I reported bug #44917 with this issue.

Carnë Draug <carandraug>
Group Member
Fri 24 Apr 2015 10:00:58 AM UTC, comment #17: 

I found the problem. I believe this should actually be a bug on liboctave

is_real_scalar only returns true to floating points but is_real_type returns true for integers as well. I was expecting is_real_scalar to be the same as "v.numel () == 1 && v.is_real_type ()"

Carnë Draug <carandraug>
Group Member
Fri 24 Apr 2015 09:39:57 AM UTC, comment #16: 

With a fresh clone incl. the new Java patches I see 8 FAILS with "make check":

>>>>> processing /home/philip/devel/octdev/oct40/libinterp/octave-value/ov-java.cc-tst
***** assert (javaMethod ("binarySearch", "java.util.Arrays", uint8  ([90 100 255]), uint8  (255)) < 0)
!!!!! test failed
[java] java.lang.ClassCastException: org.octave.OctaveReference cannot be cast to java.lang.Byte
***** assert (javaMethod ("binarySearch", "java.util.Arrays", uint8  ([90 100 128]), uint8  (128)) < 0)
!!!!! test failed
[java] java.lang.ClassCastException: org.octave.OctaveReference cannot be cast to java.lang.Byte
***** assert (javaMethod ("binarySearch", "java.util.Arrays", uint8  ([90 100 127]), uint8  (127)), 2)
!!!!! test failed
[java] java.lang.ClassCastException: org.octave.OctaveReference cannot be cast to java.lang.Byte
***** assert (javaMethod ("binarySearch", "java.util.Arrays", uint16 ([90 100 128]), uint16 (128)), 2)
!!!!! test failed
[java] java.lang.ClassCastException: org.octave.OctaveReference cannot be cast to java.lang.Short
***** assert (class (javaObject ("java.lang.Byte", uint8 (1))), "java.lang.Byte");
!!!!! test failed
[java] java.lang.NoSuchMethodException: java.lang.Byte
***** assert (class (javaObject ("java.lang.Byte", int8 (1))), "java.lang.Byte");
!!!!! test failed
[java] java.lang.NoSuchMethodException: java.lang.Byte
***** assert (class (javaObject ("java.lang.Short", uint16 (1))), "java.lang.Short");
!!!!! test failed
[java] java.lang.NoSuchMethodException: java.lang.Short
***** assert (class (javaObject ("java.lang.Short", int16 (1))), "java.lang.Short");
!!!!! test failed
[java] java.lang.NoSuchMethodException: java.lang.Short


All the Java-based io package spreadsheet I/O code works, however.

As to the last FAIL:

[java] java.lang.NoSuchMethodException: java.lang.Short


it looks like the constructor, doesn't accept any other input than doubles:

>> t = javaObject ("java.lang.Short", int16 (1))
error: [java] java.lang.NoSuchMethodException: java.lang.Short
>> t = javaObject ("java.lang.Short", 1)
t =

<Java object: java.lang.Short>

>> class (t)
ans = java.lang.Short
>> t = javaObject ("java.lang.Short", int8 (1))
error: [java] java.lang.NoSuchMethodException: java.lang.Short
>> t = javaObject ("java.lang.Short", int32 (1))
error: [java] java.lang.NoSuchMethodException: java.lang.Short
>> t = javaObject ("java.lang.Short", double (1))
t =

<Java object: java.lang.Short>


Is there something in the Javadocs outlining what argument types the constructor of java.lang.Short is supposed to accept? Or is this all related to conversion code in ov-java.cc / ov-java.h ?
(as you see, while I have written some Java-based code, I'm still a Java n00b when it comes to the nitty-gritty :-) )

Philip Nienhuis <philipnienhuis>
Group Member
Thu 23 Apr 2015 09:21:44 PM UTC, comment #15: 
Carnë Draug <carandraug>
Group Member
Thu 23 Apr 2015 09:11:23 PM UTC, comment #14: 

With the last fix all Java-based functions in the io package work.
I tested, after invoking chk_spreadsheet_support() to load the required Java class libs, using
- "io_ods_testscript ('uno')"  ## just checks UNO interface; OK
- test_spsh  ## tests all available interfaces; all worked fine.

I'd say just push you patch.

In the mean time I've changed some fields in the bug report, hope you don't mind :-)

Philip Nienhuis <philipnienhuis>
Group Member
Thu 23 Apr 2015 04:49:25 PM UTC, comment #13: 

I'll try the new patch.

That the code snippet now passes (if I understand correctly) is no guarantee that the whole of the UNO interface works - but AFAIR that PropertyValues code was one of the toughest pieces I had to tackle when writing the UNO interface stuff (apart from the queryInterface BS). LO's/OOo's Java-UNO bridge is extremely convoluted code IMO.

Building now, reporting back tonight or tomorrow.

Thanks for looking into this, Carnë

Philip Nienhuis <philipnienhuis>
Group Member
Thu 23 Apr 2015 08:58:15 AM UTC, comment #12: 

Ok. I believe that it is fixed now. Could you try with the new series of patches attached?

Or alternatively, change line 1312 of libinterp/octave-value/ov-java.cc from

  else if (val.is_real_type () || val.is_bool_scalar ())

into

  else if (val.is_real_scalar () || val.is_bool_scalar ())

Also, do you know of any java method that takes void or an empty array, that we can use for a test in this? Something in java "core" and not dependent on a jar file available locally.

(file #33744)

Carnë Draug <carandraug>
Group Member
Wed 22 Apr 2015 08:20:55 PM UTC, comment #11: 

To setup the javaclasspath I use io package's "chk_spreadsheet_support ('', 3, '/full/path/to/libreoffice/install/dir')".
On my Linux box arg#3 is /opt/libreoffice4.3, on Windows the javaclasspath looks like

:
C:\Program Files (X86)\LibreOffice 4\program
C:\Program Files (X86)\LibreOffice 4\program\classes\unoil.jar
C:\Program Files (X86)\LibreOffice 4\URE\java\jurt.jar
C:\Program Files (X86)\LibreOffice 4\URE\java\juh.jar
C:\Program Files (X86)\LibreOffice 4\URE\java\unoloader.jar
C:\Program Files (X86)\LibreOffice 4\URE\java\ridl.jar


FYI, here's a link to the relevant javadoc:
http://www.openoffice.org/api/docs/common/ref/com/sun/star/beans/Property.html

and I got my code templates from e.g.,
https://wiki.openoffice.org/wiki/Documentation/DevGuide/FirstSteps/Example:_Working_with_a_Spreadsheet_Document

I think once the LO program directory and .jar files are in the javaclasspath you can experiment with the line in comment #7 w/o requiring the rest of the OOo/LO code (it is fairly stand-alone code)

Philip Nienhuis <philipnienhuis>
Group Member
Wed 22 Apr 2015 05:55:59 PM UTC, comment #10: 

Can you give me a snippet where I can test this? The jar files I need to use and the line to use, and what was expected?

Carnë Draug <carandraug>
Group Member
Wed 22 Apr 2015 05:51:45 PM UTC, comment #9: 

With the patch, the UNO interface (invoking LibreOffice) doesn't work anymore.

After backing out the patch and rebuilding, UNO works again.

It is this statement in L.83 of io package's private/__UNO_spsh_open__.m where it breaks:

lProp = javaObject ...
("com.sun.star.beans.PropertyValue", "FilterName", 0, filtnam, []);


where "filtnam" = "MS Excel 97". (Could be "calc8" as well if one is to experiment on the command line)

AFAICS it is the empty array [] that is the offending argument ("error: invalid conversion from real matrix to real scalar") but I haven't uncovered what type is expected.
The only thing I know is that before the patch it somehow worked, after the patch is doesn't.

That said, the LibreOffice Java interface is very fragile and picky. Yet it is the only purely OSS-based interface that offers a wide variety of spreadsheet formats, so I'm hesitating to let it go.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 22 Apr 2015 06:12:12 AM UTC, comment #8: 

Upon request by Carnë I've tried his latest patch with the io package.

The Java-bases "interfaces" (~ dependency SW collections) still work, except for the UNO interface (LibreOffice/OpenOffice.org). That breaks at the very first UNO-specific Java call.

But that could have other causes as well. E.g.,:
Octave built using a 64-bit Java JDK will only work with 64-bit LO/OOo, same for 32-bot JDK and 32-bit LO/OOo.

But the fact that I never could get OpenOffice's/LibreOffice's Java-UNO bridge to run with Matlab is ominous....
I'm definitely sure it had to do with their Java support. OTOH with ML it broke further down in the UNO-specific code.

I have to experiment a bit more and try again w/o the patch to compare (hopefully tonight).

(changed release to 4.0.0-rc3)

Philip Nienhuis <philipnienhuis>
Group Member
Tue 21 Apr 2015 02:23:58 PM UTC, comment #7: 

Bug #38821 had the same cause as my comment #5 (everything was internally being casted into int32).  This patches also fix that.

Carnë Draug <carandraug>
Group Member
Tue 21 Apr 2015 12:13:53 AM UTC, comment #6: 

You might also want to check out previously reported bugs related to (long-standing known issues with) type conversion:

bug #38821 is about inability to construct a Byte object from an Octave uint8 type.

bug #39073 is about Octave converting a return value into a double when it should have stayed an Object of some type.

I am also not very familiar with JNI, but I'll take a look at your patch if I get a chance. Might want to make sure you get testing and feedback from Philip or other package maintainers who depend on the Java interface to make sure this won't break anything, if you are planning to push this for 4.0.

Mike Miller <mtmiller>
Group Member
Mon 20 Apr 2015 11:38:49 PM UTC, comment #5: 

While testing my patch, I found another problem with the way that even scalars are handled.  The problem is that all scalar integers are converted to Integer (int32) even when this is not possible (seems like it croaks when casting Byte into Integer).


octave> javaMethod ("binarySearch", "java.util.Arrays", int8 ([90 95 100]), int8 (95));
error: [java] java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Byte


Previously this was also not working, but after this patch, the vector gets casted into a Byte[] so now fails on the second conversion.  The problem is that all integer scalars are getting converted into an Integer:

http://hg.savannah.gnu.org/hgweb/octave/file/ddc5c7663329/libinterp/octave-value/ov-java.cc#l1292

which then fails when calling the actual method:

http://hg.savannah.gnu.org/hgweb/octave/file/ddc5c7663329/libinterp/octave-value/ov-java.cc#l1822

I checked with Matlab and it seems to cast down uint8 into int8, even if requires losing data.

The new attached patch, fixes all this in 4 cset and also adds tests which got run in Matlab 2010b



(file #33729)

Carnë Draug <carandraug>
Group Member
Mon 20 Apr 2015 06:48:21 PM UTC, comment #4: 

No, it does not. My patch only changes primitive arrays, which seem to not even be possible to create in Matlab, i.e., it converts "short[]" into "int16NDArray (dim_vector (len, 1))" but does nothing to java.lang.Short.

At least that is what I understood from the sparse JNI documentation and the people in ##java.

Carnë Draug <carandraug>
Group Member
Mon 20 Apr 2015 06:35:42 PM UTC, comment #3: 

I haven't try your patch, but does your change will auto-convert the result of "javaArray" function?

Matlab documentation indicates that, because Matlab passes arguments by value, changes made by a Java method to an auto-converted array will be lost. To avoid that, it suggest to use java arrays instead. but if javaArray's result is automatically converted, this is not possible anymore.

Michael Goffioul <goffioul>
Mon 20 Apr 2015 05:48:08 PM UTC, comment #2: 

I have now done the same but on the opposite direction, i.e., automatically convert octave vectors into an array of the closest java primitive. This is according to

http://uk.mathworks.com/help/matlab/matlab_external/passing-data-to-a-java-method.html#bqgs38j

Again, would be nice if anyone that knows some java would review this, but at least for the code I have here, it's now retuning the same as Matlab.

(file #33728)

Carnë Draug <carandraug>
Group Member
Mon 20 Apr 2015 02:06:05 PM UTC, comment #1: 

The attached cset seems to fix the issue for me. I'm not super familiar with JNI or the java codebase so it would be appreciated if anyone could take a look at it.

This seems to fix it to automatically convert output of java methods into octave types. I still need to figure out how to do it the other way around.

Also, Matlab documentation is not clear on what happens with Java multi-dimensional arrays (ragged arrays so they can't easily be converted). I'm guessing that this conversion will only happen with 1D arrays then which is what the patch does.

(file #33726)

Carnë Draug <carandraug>
Group Member
Mon 20 Apr 2015 10:02:11 AM UTC, original submission:  

Methods from java objects that return java base types are automatically converted into Octave types. However, this fails when returning arrays of such types:


octave> a = javaObject ("java.lang.String", "hello");
octave> b = a.getBytes()
b =

<Java object: byte[]>

octave> class (b)
ans = byte[]


while in Matlab R2010b:


>> b = a.getBytes()

b =

   72
  101
  108
  108
  111

>> class (b)

ans =

int8


Carnë Draug <carandraug>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #33744:  convert-java-primitive-arrays.cset added by carandraug (14KiB - application/octet-stream)
file #33729:  convert-java-primitive-arrays.cset added by carandraug (14KiB - application/octet-stream)
file #33728:  convert-java-primitive-arrays.cset added by carandraug (8KiB - application/octet-stream)
file #33726:  convert-java-primitive-arrays.cset added by carandraug (2KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by carandraug (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-06-02 jwe Release4.0.0-rc3 4.0.0
    2015-04-23 carandraug StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
    2015-04-23 philipnienhuis CategoryNone Libraries
        StatusNone Patch Submitted
        Assigned toNone carandraug
        Operating SystemGNU/Linux Any
    2015-04-23 carandraug Attached File- Added convert-java-primitive-arrays.cset, #33744
    2015-04-22 philipnienhuis Release4.0.0-rc2 4.0.0-rc3
    2015-04-20 carandraug Attached File- Added convert-java-primitive-arrays.cset, #33729
    2015-04-20 carandraug Attached File- Added convert-java-primitive-arrays.cset, #33728
    2015-04-20 carandraug Attached File- Added convert-java-primitive-arrays.cset, #33726

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code