bugGNU Octave - Bugs: bug #54170, java.lang.String.toCharArray...

 
 

bug #54170: java.lang.String.toCharArray result incorrect conversion to char matrix

Submitted by:  Liang Tang <lt1234>
Submitted on:  Sat 23 Jun 2018 04:28:38 PM UTC  
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Thu 19 Jul 2018 04:35:13 AM UTC, comment #16:

I'm having second thoughts about my patch here. It avoids the crash, but it produces bogus data and/or muddies the semantics of Octave `char`.

I think the right thing to do for the time being might be to raise an error when trying to convert Java char[] to Octave char, and have it tell the user to instead convert the Java char[] to a Java String and convert that, or convert the Java char[] to Octave uint16. And provide a uint16(Java char[]) conversion to do so.

Andrew Janke <apjanke>
Sun 01 Jul 2018 05:07:23 AM UTC, comment #15:

Attaching a patch that fixes the segfault without changing any Octave char type semantics: fix-java-char-boxing-segfault.patch

I had to pull the char conversion out of the macro in ov-java.cc, because the macro assumes that the sizes for the underlying primitive types are the same for Java and Octave, and char breaks this assumption.

With this in place, you can see some of the difficulty in the situation. It creates an entry point for creating Octave char values that contain UTF-16 instead of UTF-8 data, where most Octave functions working with char assume that the contents are UTF-8 (or the local character set?).

You can combine the results with chars produced using Octave '...' char literals, and they produce results without errors, but the contents are technically junk, because they contain a mix of UTF-8 and UTF-16 data, and are thus not valid Unicode strings.

Example:

I haven't played around with this much, but I bet at least some Octave functions (maybe in plotting?) will break on input like this, or even on valid UTF-16 data in char arrays.

For example:

The displayed title is "Hello, w". I bet this is because some C-style API is confused by the 0-valued bytes embedded in the string.

Andrew Janke <apjanke>
Mon 25 Jun 2018 10:30:05 AM UTC, comment #14:

Here's a concrete example of what I'm thinking of with respect to "I/O buffers":

In Matlab, it is impossible to use the java.io.Reader API from M-files.

This is because java.io.Reader takes a pass-by-reference char[] as an output parameter. See the java.io.Reader Javadoc and scroll down to the "read(char[] cbuf, ...)" methods.

In Matlab, because of the auto-conversion between Matlab char and Java char[], you can't get at changes that BufferedReader.read(...) makes to the `cbuf` that you pass in, because Matlab treats it as a pass-by-value Matlab char object, and object identity is lost. To use that char[] as a buffer, it would have to be treated as a mutable object, with just the object reference passed around.

I'm not saying that this is a reason to do something different with your API: if Matlab compatibility is a priority for you, this is probably something you have to give up. (Or introduce a custom non-autoconverting `java_primitive_array_reference` type or something.) Just trying to make it clear what I'm talking about.

Andrew Janke <apjanke>
Mon 25 Jun 2018 10:11:25 AM UTC, comment #13:

> What is the result for class(cs2{1}) in your example in comment #10?


It is 'char'. In Matlab, Java `char[]` arrays always automatically convert to Matlab `char`.

> We can still think about what calling "char" on a Java char array should do...


Yeah, it's a good question.

If you want full Matlab compatibility, I think there's only one thing to do: convert to a UTF-16-compatible native Octave type. In Matlab, there is no way to even call "char(...)" on a Java char[] array, because Java char[] arrays are always implicitly autoconverted to Matlab `char` arrays, so Java char[] objects are never exposed at the M-code level. (IMHO, this might be another Matlab design flaw, because it breaks compatibility with Java APIs that accept pass-by-reference Java char[] arrays to be used as buffers to fill and use as output parameters, e.g. in I/O libraries, but what's done is done.)

> It might lead to even more confusion if we introduced a special char class just for UTF-16 (and maybe another one for UTF-32?).


Hmm. Might not, depending on where it was used, because most users might not ever see it, if it were just for Java compatibility. Java APIs don't use `char[]` except for low level I/O buffer passing, and for storing passwords. In Java, if you are working directly with `char[]`, it is probably a design mistake and you should be working with java.lang.String instead.

Then again, it would probably need to exist somewhere for Matlab compatibility, too. And yeah, that would probably lead to confusion. (Though I'd argue that Octave's current use of `char` as bytes in a Unicode world is itself probably a source of confusion.)

I wonder why the original poster in this discussion was calling toCharArray() in the first place. @liangtang, can you chime in? Do you have an actual use case for this, or were you just exploring what these types do?

Andrew Janke <apjanke>
Mon 25 Jun 2018 09:52:31 AM UTC, comment #12:

You are raising some valid points. In the light of this, your options 1 or 2 in comment #6 seem to be the way to go.
What is the result for class(cs2{1}) in your example in comment #10? Maybe we should try and do what Matlab is doing.

It might lead to even more confusion if we introduced a special char class just for UTF-16 (and maybe another one for UTF-32?).

We can still think about what calling "char" on a Java char array should do: Should we convert to UTF-8? As far as I understand, the gnulib conversion functions don't do any normalization by default. So it might still be possible to get consistent results after a round trip Octave char -> Java String -> Octave char (i.e. UTF-8 -> UTF-16 -> UTF-8).

Markus Mützel <mmuetzel>
Project Member
Mon 25 Jun 2018 06:41:12 AM UTC, comment #11:

One side note: if you auto-convert java.lang.String values to Octave char values, it will break compatibility with some Java APIs, because you lose Java object identity in the conversion process. Some Java code compares Strings using `==` object identity comparisons, particularly in conjunction with String.intern().

Matlab does not auto-convert java.lang.String values to Matlab char. Maybe for this reason.

Andrew Janke <apjanke>
Mon 25 Jun 2018 06:35:58 AM UTC, comment #10:

Here's a concrete example that might help discussion.

Consider the case of `cs = {'foo', 'föö', 'foobar'}'`, and how it looks in 2-D char form.

From Matlab R2018a:

In Matlab, this can be round-tripped through java.lang.String and produce the same results.

If the individual char[] results of that operation were transcoded to UTF-8 before being returned as Octave values, you might end up with different values for cs/cs2, or not be able to produce that 2-D array.

Andrew Janke <apjanke>
Mon 25 Jun 2018 06:11:34 AM UTC, comment #9:

> But ideally the conversion UTF-8<->UTF-16 should be done automagically whenever a java.lang.String is constructed or toCharArray is called on it.


Doing this for java.lang.String might be desirable. But doing this for toCharArray results/`char[]` would deviate from Matlab compatibility. The problem is that Matlab's `char` type and Java's `char[]` represent arrays of UTF-16 code units, not strings per se. (IMHO, this is a design flaw in Matlab, but whatcha gonna do? At least they're addressing it now with the new `string` type.) There might also be edge cases related to Unicode normalization during the transcoding process that would make it hard to use Java APIs that use `char[]` as arguments.

This will run in to problems particularly with 2-D or higher `char` arrays. If you treat them as lists of strings and transcode them, the contents of the rows might end up being different lengths after transcoding, so they cannot be packed back in to an N-D char array. This could be an issue for Matlab compatibility because some Matlab code (particular older code, I think) uses blank-padded 2-D char arrays as a way of compactly representing lists of strings. Its string processing functions accept this as an input form.

IMHO, the abstraction levels for (Matlab) `char` and strings are different, and if you make the `char` autoconversion too smart, you might back yourself into a compatibility corner.

Andrew Janke <apjanke>
Mon 25 Jun 2018 06:01:13 AM UTC, comment #8:

I didn't have a look at how the Java interface is implemented. But ideally the conversion UTF-8<->UTF-16 should be done automagically whenever a java.lang.String is constructed or toCharArray is called on it.
The necessary conversion functions should be available in Octave dev.

Markus Mützel <mmuetzel>
Project Member
Mon 25 Jun 2018 03:19:27 AM UTC, comment #7:

(For 3., "wchar" might be a bad name for these UTF-16 Java-compatible characters, since it wouldn't map to the C++ `wchar_t` type. Need to pick a different name. Maybe `unichar`? `utf16char`?)

Andrew Janke <apjanke>
Sun 24 Jun 2018 10:23:30 PM UTC, comment #6:

I see a few workable alternatives here, in increasing order of difficulty:

1. Convert Java `char` to Octave `uint16` instead of Octave `char`

2. Do not auto-convert Java `char[]`; leave them as Java objects. (This leaves the question of what to do with scalar Java `char`s.)

3. Add a `wchar` type to Octave, and convert Java `char` to that.

4. Change Octave's `char` to be UTF-16, to match Matlab. (See http://lists.gnu.org/archive/html/octave-maintainers/2018-05/msg00137.html)

Thoughts on which is preferable in the short term? I may be able to do a patch for 1 or 2.

Andrew Janke <apjanke>
Sun 24 Jun 2018 07:54:34 PM UTC, comment #5:

Similar issue on 4.4.0 / macOS. It produces what to me is an unexpected string, and crashes.

That crash is reproducible; happens every time I do this.

I also get a malloc error sometimes, but this one is not as reproducible. It happens semi-consistently if I have used the up-arrow to retrieve the "str=" and "str1=" commands from my command history.

That double-spacing with only half the characters recovered does look like an issue converting 2-byte Java UTF-16 chars to 1-byte Octave chars. Note that those blanks are zeros, not spaces.

This probably works fine in Matlab because Matlab chars are 2-byte UTF-16 chars.

What are the expected semantics for converting Java `char`s to Octave `char`s?

Andrew Janke <apjanke>
Sun 24 Jun 2018 05:01:33 PM UTC, comment #4:

The easiest and most useful way to convert a Java string to an Octave char array is with the 'char' function.

But yes, there is undoubtedly a problem with the automatic conversion from a two-byte Java char array type into an Octave one-byte char array type.

Mike Miller <mtmiller>
Project Administrator
Sat 23 Jun 2018 06:56:07 PM UTC, comment #3:

Also reproduced on dev. The error message is somewhat different:

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 23 Jun 2018 05:24:44 PM UTC, comment #2:

I wonder if this is ASCII vs UTF issue.
(i>e. java uses 2-byte chars and octave uses 1-byte):

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 23 Jun 2018 05:16:40 PM UTC, comment #1:

I can confirm on 4.4.0 / linux as well. Also:

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 23 Jun 2018 04:28:38 PM UTC, original submission:

I cannot recover the string definition with java.lang.String within Octave 4.2.1 on win 10 pro 64x.

str=javaObject('java.lang.String', 'i am ............. here');

str1=str.toCharArray
str1 =

i

a
m

.
.
.
.
.
.
.

In matlab, the same command works.

Liang Tang <lt1234>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44471:  fix-java-char-boxing-segfault.patch added by apjanke (2KiB - application/octet-stream - fixes segfault without changing char semantics)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by lt1234 (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-07-01 apjanke Attached File- => Added fix-java-char-boxing-segfault.patch, #44471
    2018-06-24 mtmiller Summaryjava.lang.String.toCharArray result truncated => java.lang.String.toCharArray result incorrect conversion to char matrix
    2018-06-24 mtmiller StatusNone => Confirmed
        Release4.2.1 => dev
        Summaryjava.lang.String missing characters => java.lang.String.toCharArray result truncated

    Back to the top


    Powered by Savane 3.3