patchGNU Octave - Patches: patch #9980, JSON encoder and decoder,...

 
 

patch #9980: JSON encoder and decoder, alternative to object2json

Submitted by:  None
Submitted on:  Thu 15 Oct 2020 06:04:42 PM UTC  
 
Category:  Forge : new function Priority:  3 - Low
Status:  Done Privacy:  Public
Assigned to:  Philip Nienhuis <philipnienhuis> Originator Email:  -email is unavailable-
Open/Closed:  Closed

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sat 07 Nov 2020 09:17:27 PM UTC, comment #41: 

I guess this is for io-2.6.4 then (with 2.6.3 just released).

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sat 07 Nov 2020 05:22:14 PM UTC, comment #40: 

Usually Octave displays errors with their stack. That should make clear that regexprep was called from `fromJSON` in this case.

Markus Mützel <mmuetzel>
Project Member
Sat 07 Nov 2020 01:17:54 PM UTC, comment #39: 

comment #38:

> Sorry, I didn't read the changeset earlier. But this snippet might give misleading error messages:
>
> It might be better to remove the try-catch-block and let regexprep display an error message directly.


You are correct that the cause of the error could be misleading and it bothered me too.  But I'm not sure that resorting to 'regexprep' error is a better solutions.  

The problem I wanted to avoid is if 'fromJSON' was buried in another script/function, an error from 'regexprep' (rather than 'fromJSON') would be confusing or misleading too.  I know it is always possible to trace the problem to 'fromJSON' but not easily for novices.

Initially I tried to post-process the error from 'regexprep', but I could not replicate the error and verify that the solution was adequate.  As such, I opt with this.

While I too am unsure about what else might cause 'regexprep' error, since 'str' is already screen for classtype, unless 'regexprep' changes methods, I think it could be safe to say that a runtime error would only be from 'str' content.  Maybe not UTF-8 specifically but content nonetheless.

At best, it might be "safer" to just return a generic, 'Invalid JSON string' and drop the reference to UTF-8.  But I'm on the fence as to whether that is any better.

Or if someone share with me a sample file or method to replicate the problem, I can fix this better.

I await suggsetions....

Anonymous
Sat 07 Nov 2020 08:56:38 AM UTC, comment #38: 

Sorry, I didn't read the changeset earlier. But this snippet might give misleading error messages:

+  try
+    str = regexprep(str,'^[\s,]*','');
+  catch
+    error("Invalid UTF-8 encoding of JSON string");
+  end_try_catch

I am not certain if the only reason for which regexprep can possibly fail is invalid UTF-8. If it fails for a different reason, the error message that is shown might lead in the wrong direction.

It might be better to remove the try-catch-block and let regexprep display an error message directly.

Markus Mützel <mmuetzel>
Project Member
Mon 02 Nov 2020 06:33:49 PM UTC, comment #37: 

Thanks

Anonymous
Sun 01 Nov 2020 06:22:12 PM UTC, comment #36: 

I pushed the functions here:
http://hg.code.sf.net/p/octave/io/rev/6bb1f2a0e622
after a miniscule edit (add a newline at the end of one of the functions).

(Hmm I should add them to the INDEX file as well, will do.)

Thanks very much Ketan for your contributions here and the prompt replies.

Closing report as "Done".

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sun 01 Nov 2020 05:49:13 PM UTC, comment #35: 

The texinfo changes are good.  I tweaked a little on the wordage.

Likewise on pre-pending warning msg.

I understand the comment on the caution, it occurred to me too, but since the ND array outputs might appear okay, I worried end-user (current Matlab or future Octave 7) will not notice the reshaping.  Nevertheless, I'm OK with dropping it.

On whitespace, I added some more, at least on func def.  On func calls, I prioritized keeping code width under 80chars (and uniform) over more whitespace.

On single/double quotes, I sought to use single-quote on all symbols (single or multi-char) and double-quote on all sentence frags (i.e. with whitespaces).  In some places, however, single- and double- quote are not interchangeable, due to need of \ or " literals.

(file #50183, file #50184)

Anonymous
Sun 01 Nov 2020 02:06:39 PM UTC, comment #34: 

OK.
Note that your email is still in the functions themselves.

I've uploaded them here with the following changes:

  • Fixed & reorganized texinfo so that info is somewhat more organized per subject / input arg. I hope you like it.  Fixing Texinfo issues often boils down to adding "@" before offending characters ("@" is a Texinfo control char) and I use the on-line Texinfo docs:

https://www.gnu.org/software/texinfo/manual/texinfo/texinfo.html

  • Dropped caution at end of texinfo. I think we shouldn't refer to functions in future Octave releases (jsonencode/jsondecode are in 7.x which will be released maybe only later next year) or the proprietary competition in any way.
  • Updated copyright strings
  • Prepended warning- / error messages with function name.

and apart from minor typos and Octave's editor removing whitespace after each line, that's it.
For convenience I've also uploaded "diff -u" diffs to see what got changed.

There are still smaller style things (spaces between function arguments/after comma's, between function name and opening parenthesis, double rather than single quotes where possible) but I'm not going to chase that all down.
As I wrote in my first comment here: on the whole I think you've done a very good job.

If you want to make some last edits, here's your chance. If (when) you are satisfied I'll push them, just let me know.

(file #50178, file #50179, file #50180, file #50181)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sat 31 Oct 2020 11:02:21 PM UTC, comment #33: 

Okay on name, but not on email.  I'm okay with it in file but not on searchable website.  You can thank Google for that.

Anonymous
Sat 31 Oct 2020 05:05:59 PM UTC, comment #32: 

No need to apologize :-)

If you think the latest function versions are OK I'll test them briefly, push, and then polish Texinfo adapt style.

Would you mind if I push them with your name and you're credited in the commit message? If so can I use your name and email? (your email isn't mentioned in the on-line log, only when browsing he actual commits)
I'm asking because there are contributors who explicitly want to remain anonymous.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sat 31 Oct 2020 03:41:13 PM UTC, comment #31: 

very very sorry.

uploaded the wrong file!!!  do not use "fromJSON_v2_3.m"

use "fromJSON_v2_4.m" 

(file #50177)

Anonymous
Sat 31 Oct 2020 03:36:21 PM UTC, comment #30: 

sorry, I didn't do it 'native2...' correctly. 

now its an error for improperly encoded string. please use "fromJSON_v2_3.m"

(file #50176)

Anonymous
Sat 31 Oct 2020 03:35:22 PM UTC, comment #29: 

Sorry. I probably wasn't clear. `fromJSON` should not call `unicode2native`. Instead, the user (in the case of comment #21: Philip) must make sure that they call the function with valid strings.
Imho, it is ok if an error is thrown if the function is called with invalid strings.

Markus Mützel <mmuetzel>
Project Member
Sat 31 Oct 2020 03:28:00 PM UTC, comment #28: 

comment #26:

> The "invalid UTF-8" error is probably because `regexp` requires UTF-8 encoded input in Octave 6 or newer.
> I guess the input file has a different encoding. In Octave 6 or newer you can specify the encoding when opening a file with `fopen`. Otherwise you could use `native2unicode` to convert from whatever input encoding to UTF-8.


I've taken this advice and fixed the input with 'native2unicode' and a warning.  I can only verify that code did not break, I'm still at a loss to generate and test against a wrongly encoded string.   In Octave 7, I've tried writing out and reading back in a JSON string with non-UTF-8 encoding (including bizzare characters).  I was unable to replicate the `regexp` error (on Oct 5 or 7), so I can't tell if this fix works.

On the issue of multidim arrays, I think "to/fromJSON" and "jsonencode/decode" will have to go separate ways. I've updated the HELP doc to reflect this.  IMHO, Matlab's choice is at best inconsistent and at worst just wrong. Maybe "to/fromJSON" is wrong too, but I think at least its more consistent.   so...

Files are uploaded ("to/fromJSON_v2_2.m").  I have included some of the big BIST tests from "jsonencode" and "jsondecode".

(file #50174, file #50175)

Anonymous
Fri 30 Oct 2020 04:26:50 PM UTC, comment #27: 

>This brings me to a question.  At one time, I had the following BIST:
>    %! r = round(1000*rand(10,10,10,10,10));
>    %! assert(r, fromJSON( toJSON(r)));
>but later took it out to avoid referring to an external function.  >Can I put it back in?

If with "external function" you mean "round" and/or "rand", that seem OK to me. In core Octave many many BIST tests rely on behavior of other core functions.
Same for calling "toJSON" in a BIST for "fromJSON". We want to test  functions in a real environment where they are all available, isn't it?
Or do I misunderstand your question?

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 29 Oct 2020 04:38:51 PM UTC, comment #26: 

The "invalid UTF-8" error is probably because `regexp` requires UTF-8 encoded input in Octave 6 or newer.
I guess the input file has a different encoding. In Octave 6 or newer you can specify the encoding when opening a file with `fopen`. Otherwise you could use `native2unicode` to convert from whatever input encoding to UTF-8.
AFAICT, this is not a problem of `fromJSON` (which accepts strings). The problem is that an invalid string is passed to that function. The file has to be read with the correct encoding or the read byte stream has to be converted from that encoding to a valid string.

Markus Mützel <mmuetzel>
Project Member
Thu 29 Oct 2020 04:34:25 PM UTC, comment #25: 

I've added the "warning('off'...".  Turns out some of those warnings were inadvertent errors the BIST.

On the "Invalid UTF-8" issue, I'm having trouble finding/generating an offending JSON string.  Can you show rest of code on how it was read into Octave?  And/or upload a sample file? 

Anonymous
Thu 29 Oct 2020 03:19:22 PM UTC, comment #24: 

On comment #21, my code was wrong to allow the '\"' pass thru as is, I fixed that.   Turn's out my BIST for that check was wrong too.   I will upload soon.

On the changing of "remote-settings/monitor_changes" to "remote-settings_monitor_changes" (along with other fixings), I agree with you and not that design choice.  Provided it doesn't cause "fromJSON" to exit in error, I'm okay with "garbage in-garbage out" and leave it to the end user to catch and fix as he/she sees fit.  The "invalid UTF-8" error, on the other hand, bothers me.  Let me see what I can do.

On comment #19/20 vis a vis the ND array:

the difference is not a bug but difference in interpretation.  I  look (with my own eyeballs) at the JSON string and see multiple pages of 2x2 matricies, and "fromJSON" reflects that interpretation.  But it would seem jsondecode sees it as pages of 2x4 matricies.

You can see the difference more clearly in a simpler example (hypothetically-generated via "fromJSON_BIST.tst", you should check that manually):

jsonA = '[[1, 2], [3, 4]]';
mat1A = jsondecode(jsonA);
mat2A = fromJSON(jsonA);

jsonB = '[[[1, 2], [3, 4]], [[5, 6], [7, 8]]]';
mat1B = jsondecode(jsonB);
mat2B = fromJSON(jsonB);

returns the following.  One of these things is not like the others.

mat1A(:,:,1) => [1 2;3 4]    ## <-- inferred from "fromJSON_BIST.tst"
mat2A(:,:,1) => [1 2;3 4]    ## <-- actual

mat1B(:,:,1) => [1 3;5 7]    ## <-- inferred from "fromJSON_BIST.tst"
mat2B(:,:,1) => [1 2;3 4]    ## <-- actual

Which is correct: 'mat1B' or 'mat2B'? Actually, I don't know if this is really a difference in interpretation or ND arrays hurts the brain of the original "jsondecoder" developer too.  Either way, I prefer the output of "fromJSON", a more intuitive flow from "jsonA" to "jsonB".    BTW, "object2json" sees it the same as "jsonencode/jsondecode" and avoiding this was deliberate effort I made.

This brings me to a question.  At one time, I had the following BIST:

%! r = round(1000*rand(10,10,10,10,10));
%! assert(r, fromJSON( toJSON(r)));

but later took it out to avoid referring to an external function.  Can I put it back in?

Again, I feel a JSON parser should try to work to mate Javascript to Octave/Matlab, and Javascript only knows c-style, nested, 1-D arrays not FORTRAN-style multidim arrays.    In this regards, "fromJSON" and "toJSON" do not respect Octave column vectors: they treat any vector as a 1-D Javascript row vector...ALWAYS.

Anonymous
Wed 28 Oct 2020 05:21:39 PM UTC, comment #23: 

NaN vs. [] is probably a matter of Matlab-compatibility, which is of more importance for core Octave than for O-F packages.
In this case all the more so as your functions are named differently than in Matlab to start with.
So, with that out of the way I agree with you on this detail.

As to complexity of ND-arrays, I'm with you too :-)

I think it's worthwile to check out the tests I uploaded in comment #20 and check if there are no alarming differences or bugs.
AFAICS most of the tests work in toJSON and fromJSON.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Wed 28 Oct 2020 03:53:50 PM UTC, comment #22: 

On the quoted quoted-string, that's could be a bit tricky.  I will look into it.

On the difference in '[]' and 'NaN' for "null"...originally "fromJSON" conditionally return '[]' for "null": 'NaN' if in numerical array, '[]' if in mixed-class array.  However, given that Octave is a numerical computing environment, I made a deliberate choice to instead always make "null" into 'NaN'.  Aside form the symmetry to "toJSON" reflexively turns all 'NaN' to "null", "fromJSON" does recursive processing, and I worried about some edge-cases causing confusion.

I attribute this comprimise to what I regard as gross (teeth-grinding!) negligence in the JSON spec, which fails to accommodate for 'NaN' and 'Inf' despite both being valid IEEE754 values and valid Javascript values.  Besides, an empty-array is possible in JSON, so converting "null" to '[]' is a similarly dubious comprise.

In the end, with regards to NaN, [], UTF encoding, etc., this comes down to what is being transported via JSON: numerical dataset or information.  I was setting out to solve the problem of transporting numerical data (i.e. numerical ins and outs of Octave functions as backend to Javascript), where as 'jsonecode/decode' might have settled on transporting information (i.e. processing HTTP requests, etc).

On the ND array, I will look at that.  ND arrays hurts my brain, so I still have to wrap my head around what you show.

Anonymous
Wed 28 Oct 2020 08:42:37 AM UTC, comment #21: 

Just curious and for some more evaluation, in Octave-7.0.0 I tried fromJSON and jsonencode on some random json file in my Firefox and Seamonkey profiles (lots of json files there),
For example:

>> file = 'broadcast-listeners.json'
file = broadcast-listeners.json
>> fid = fopen (file);
txt = fread (fid, Inf, "*char")';
fclose (fid);
>> txt
txt = {"version":1,"listeners":{"remote-settings/monitor_changes":{"version":"\"1603569466778\"","sourceInfo":{"moduleURI":"resource://services-settings/remote-settings.js","symbolName":"remoteSettingsBroadcastHandler"}}}}

>> fromJSON (txt)
ans =
  scalar structure containing the fields:
    version = 1
    listeners =
      scalar structure containing the fields:
        remote-settings/monitor_changes =
          scalar structure containing the fields:
            version: 1x17 sq_string
            sourceInfo: 1x1 scalar struct
>> ans.listeners.('remote-settings/monitor_changes').version
ans = \"1603569466778\"

>> jsondecode (txt)
ans =
  scalar structure containing the fields:
    version = 1
    listeners =
      scalar structure containing the fields:
        remote_settings_monitor_changes =
          scalar structure containing the fields:
            version: 1x15 sq_string
            sourceInfo: 1x1 scalar struct
>> ans.listeners.remote_settings_monitor_changes.version
ans = "1603569466778"

Clearly Octave-7.0.0's jsondecode adapts structfield names and maybe struct field contents, I suppose for easier handling and maybe Matlab compatibility, while fromJSON delivers them 1:1. Yet fromJSON's output string with the escaped backslashes looks suspicious.
Again I have no idea if these results are right or wrong; opinions, anyone?

I'm inclined to think that it's a good thing that users can (also) get "raw" data from JSON files (provided they're correct). But that's me, being used to digging around in data files I tend to prefer to stay as close as possible to raw data before processing them my own way.

In another json file in my seamonkey profile I got

>> file = 'sessionstore-4.json';
:
>> jsondecode (txt);
## no errors
>> fromJSON (txt);
error: regexprep: the input string is invalid UTF-8
error: called from
    fromJSON at line 61 column 7

... but I suppose that's because the URL's inside can have varying encodings (peeking in the file I saw a lot of gunk).
Octave-7.0.0's jsondecode seems more resilient.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Tue 27 Oct 2020 10:36:06 PM UTC, comment #20: 

(attachments gone after "Preview", sigh)
Here they are:

(file #50144, file #50145)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Tue 27 Oct 2020 10:34:35 PM UTC, comment #19: 

Thanks Ketan for the prompt updates.

With Octave-7.0.0:

>> test fromJSON
warning: Unclosed block: "[1,2,3"
warning: Unclosed block: "[[1,2,3],[3"
warning: Unclosed block: "[3"
warning: Unclosed block: "{a:3,b"
warning: Malformed struct frag: "{a:3,b ...}"
warning: Unclosed block: "{a:[[[1,3],[2,4]],[["
warning: Unclosed block: "{re:2,im:4"
warning: Invalid frag: "im mi{}ing quotes"
warning: Invalid frag: "garbage"
warning: Invalid frag: "garbage"
warning: Invalid frag: "garbage"
PASSES 68 out of 68 tests

>> test toJSON
PASSES 53 out of 53 tests

That noise can (and IMO should) be shut off; I did that by adding
%! warning ("off", "all", "local");
statements before the fromJSON calls provoking these warnings.

I've reworked the functional BIST tests for Octave-7.0.0's jsonencode and jsondecode and after some editing attached them here as toJSON_BIST.tst and fromJSON_BIST.tst.
I randomly chose the third "%% try higher dimensions for the array" test block in fromJSON_BIST.tst,


%! json = ['[[[[1,-1], [2,-2]],[[3,-3],[4,-4]]],[[[5,-5],[6,-6]],[[7,-7],', ...
%!         '[8,-8]]],[[[9,-9], [10,-10]],[[11,-11],[12,-12]]],', ...
%!         '[[[13,-13],[14,-14]],[[15,-15],[16,-16]]]]'];
%! var1 = cat (3, [1, 3; 5, 7; 9, 11; 13, 15], [2, 4; 6, 8; 10, 12; 14, 16]);
%! var2 = cat (3, [-1, -3; -5, -7; -9, -11; -13, -15], ...
%!             [-2, -4; -6, -8; -10, -12; -14, -16]);
%! exp  = cat (4, var1, var2);
%! obs  = fromjSON (json);
%! assert (isequal (obs, exp));

and saw some different dimensions wih your fromJSON.
Similarly for some toJSON_BIST.tst tests.
In other (earlier) tests I saw that where jsondecode returns [] for 'null', fromJSON returns NaN.
I hope you are willing to have a look at the tests in these files  and try them with the toJSON and fromJSON functions.

As I know nothing about JSON (well, I know it exists); can you inform me if the different test results are something to worry about?
Background:
I do not expect full Matlab compatibility. It's just that as package maintainer I'd like functions to produce correct output, that is, output sufficiently correct to be used out in the wild w/o provoking undue bug reports. After all, we hope these new functions will attract new users rather than put them off :-)
So do you think fromJSON.m and toJSON.m as they stand are "safe" enough to include in the io package, or is more work needed?

They can always be included with the word "EXPERIMENTAL" (in capitals) in the NEWS file.
That word is FWIW anyway :-)

On another note: once finally accepted,

  • Before pushing to the repo I'll do some style edits (notably single quotes => double quotes). I don't mind, but you may also do it;
  • I'll have a look at the Texinfo, no worries. I'm no Texinfo guru but I think I can improve it a bit here and there.

Lastly:
Apologies if you expected your contributions to go rapidly and w/o much edits into some Octave add-on package :-)  But correctness of functions is a Big Thing for me and the io package is a "community package", a category for which we think this counts a bit more than for external packages.
These functions will only get better this way.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Tue 27 Oct 2020 03:03:12 PM UTC, comment #18: 

Yes, the change was intentional and was made to improve compatibility with Matlab, which allows simple function handles to be created that refer to functions that are not currently defined in the search path.

John W. Eaton <jwe>
Project Administrator
Tue 27 Oct 2020 01:30:13 PM UTC, comment #17: 

comment #15:

> What do you mean by
>

> v7 "break" of 'func2str'

>


By "break" I meant air-quoted "break". 

Its just that in v5.2, calling 'str2func("@nofunc")' where "nofunc" is non-existent returned an error, while in v7, it no longer is the case.  The resulting function will fail, of course, but 'str2func' will not.

I assume it is deliberate, but the change affected the behavior of my function.

Anonymous
Tue 27 Oct 2020 01:25:15 PM UTC, comment #16: 

Update 'toJSON' and 'fromJSON' have been uploaded. 

There were couple of other minor fixes needed for Octave 7 that produced no change in behavior, except one...

Octave 7 has changed its behavior regarding 'str2func'.  It is not as strict as in v5.2.  As a result, in v5.2, "fromJSON('@nofunc')" will return a string or function '@nofunc' depending on existence of 'nofunc' as a function; while in v7, "fromJSON" will always return a function '@nofunc', whether it is valid or not.

This is an edge-case scenario, in that transmitting functions over JSON is unusual in itself.  As such, to resolve this issue, I've simply removed the BIST test specifically for non-existing functions. 

Also, please note, the HELP doc remains broken with regards to layout.  I'm relying on some help on this issue.

(file #50136, file #50137)

Anonymous
Tue 27 Oct 2020 01:13:08 PM UTC, comment #15: 

What do you mean by

v7 "break" of 'func2str'

John W. Eaton <jwe>
Project Administrator
Tue 27 Oct 2020 12:41:58 PM UTC, comment #14: 

I got Octave 7 going.

Regarding empty index, there was no issue with replacing '{}' with '{:}'.

Regarding issue with anonymous function, yes, I discovered that too.  Fix implemented with solution I had tried to originally avoid by using nested subfunction.

Will upload soon, as soon as I deal with v7 "break" of 'func2str'.  In this case, v7 has become LESS strict that v5.2.

Anonymous
Tue 27 Oct 2020 11:57:02 AM UTC, comment #13: 

The empty indexing error is Octave-7.0.0 AFAIK.
Same probably for the end indexing.

But

error: '_num2str_' undefined near line 170, column 170"

occurred in octave-6.0.92 as well; that'll be the new release shortly, so also (better) try that one first.
What I saw during a brief debug session is that anonymous functions in the callers are NOT in scope in subfunctions (callees), nested or not. From what I read in the docs their scoping mimics that of plain variables. So maybe you'd need to add them as subfunctions; or add them in the input args for called functions.

All of which makes me wonder how these functions could work at all in your case. Maybe Octave-5.2.0 was less strict?

I'm no fan of nested subfunctions anyway, usually one sooner or later gets lost in all kinds of scoping issues, like here. But this is all your pick :-)

Octave prerelease source tarballs and Windows prerelease installers are on ftp://alpha.gnu.org/pub/octave

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Tue 27 Oct 2020 02:30:06 AM UTC, comment #12: 

Most information you will find in the wiki:

https://wiki.octave.org/Building

If you have trouble building Octave, please address a wider audience at https://octave.discourse.group/

Kai Torben Ohlhus <siko1056>
Project Member
Mon 26 Oct 2020 11:41:07 PM UTC, comment #11: 

BTW, where can I get Octave 7.0 source code?

Anonymous
Mon 26 Oct 2020 11:34:05 PM UTC, comment #10: 

Thanks.  I will look into ASAP.  I will try getting Octave 7.0 running first.

Anonymous
Mon 26 Oct 2020 08:07:21 AM UTC, comment #9: 

Thanks for the new patches.

Where in comment #2 I wrote that in Octave-7.0.0 these functions will become obsolete, if they are in the io package they should still run w/o issues for a few years until usage of then older Octave Out There is low enough. I think that is not unreasonable.

I checked and tested a few thing and I see the following in Octave-7.0.0:

>> test toJSON
***** test  ##  PREC input arg test
  assert(toJSON(pi,  0),'3')
  assert(toJSON(pi,  5),'3.1416')
  assert(toJSON(pi, -5),'3.1416')
  assert(toJSON(pi, 25),'3.141592653589793115997963')
  assert(toJSON(pi,  :),'3.14159265358979')
  assert(toJSON(pi, []),'3.14159265358979')
  assert(toJSON(pi, false),'3.14159265358979');  ## this is provoking stmt
  assert(toJSON(pi, {}),'3.14159265358979');          ## issue warning here
!!!!! test failed
invalid empty index expression

due to this statement on L.72:

    [PREC,COMPACT] = {[],PREC}{};

which (as we have discussed) is difficult to disentangle.
As of Octave-7.0.0 empty indexing e.g., var{}, is no longer allowed, see https://savannah.gnu.org/bugs/?56167.
Replacing by

%    [PREC,COMPACT] = {[],PREC}{};
    PREC = [];
    COMPACT = PREC(:);

makes it work but I don't know if that's the right fix.
The next test, 'toJSON(false)' fails elsewhere (L.234) and toJSON(reshape(1:8,2,2,2)) fails in L.217. Trying to fix those (just entering ":" between curly brackets) gave

"error: '_num2str_' undefined near line 170, column 170"

(I know it's defined on L.86) and then I gave up.
Please fix these issues.

Then,

>> test fromJSON
***** test  ## number
assert(fromJSON('4'),4)
!!!!! test failed
invalid use of 'end': may only be used to index existing value
>> fromJSON('4')
error: invalid use of 'end': may only be used to index existing value
error: called from
    fromJSON>_fromjson_ at line 70 column 25
    fromJSON at line 62 column 9

(perhaps also an Octave-7.0.0 issue)
variable 'obj' needs to be initialized (empty) before it can be indexed with (end+1).

Also in fromJSON I saw several empty var{} indexing which will error out.

Can you build Octave-7.0.0 to try yourself?
If you're on Linux it shouldn't be too hard, if on Windows I could cross-build a recent tip for you and upload it somewhere.

I'm on the verge of making a (bug fix) io-2.6.3 release, I really hope you can delivr toJSON and fromJSON also running in Octave-7+ so that I can include them in the io package. In fact that's all the release is waiting for.
I hope you can fix all of these issues

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sat 24 Oct 2020 03:54:58 PM UTC, comment #8: 

I've uploaded a new version.  Aside from adding suggested testing improvements, it they worker harder at mating native JSON/Javascript objects with native Octave objects, namely ND arrays and struct arrays.

fromJSON
========

Per comments, some more input checking and BIST has been added.  Pointing to exact character position could be VERY helpful, but 'fromJSON' works recursively, start with big blocks of string and working down to smaller fragments.  As such, error position is a "nice to have".  Instead, 'fromJSON' cites relevant fragments when malformed JSON is found. 

toJSON
======

Per comments, more input checking and BIST have been added/fixed.  On exotics like java or COM, the norm for stringifying non-(number,bool,string) is a placeholder with a quoted-string of its class name (e.g. '"[SomeClass]"'), which 'toJSON' observes. Provided that 'class()' works on any Octave object, there are no "invalid" inputs, just unfullfilling outputs.  I've added a BIST test with java object for verification.

On other input tests, i.e. PREC and COMPACT (formerly JSON), done.

The input checking statement

isbool(PREC) && ([PREC,JSON] = {[],PREC}{});

is my preferred contraction for:

if isbool(PREC);
    JSON = PREC;
    PREC = [];
endif

The resulting bool value in irrelavent.  I understand comment about being complicated.

Regarding compatibility, 'toJSON' and 'fromJSON' is made to work with Javscript's primative objects and its built-in parser ('JSON.parse' and 'JSON.stringify').  With all due respect, that is the standard to which any JSON parser should rise.  If 'jsondecode' plays nicely with Javascript, then 'toJSON' should work with 'jsondecode'.

(file #50087, file #50088)

Anonymous
Mon 19 Oct 2020 12:00:48 PM UTC, comment #7: 

ok. 

had epiphany when discussing pro/cons against 'jsonlab', 'object2json' and 'jsoneconde'.

will address cited issues, and upload soon

Anonymous
Sat 17 Oct 2020 09:04:58 PM UTC, comment #6: 

@Ketan:

Input validation:

fromJSON:
=========

  1. checking that there is input at all (nargin >= 1)
  2. checking that input is a character string
  3. ditto that is has sufficient minimum length (how'd you parse ""? or is that accepted and if yes, what class output would I get, a double? char?)

If parsing fails, can fromJSON indicate at what character position it lost track? (is this useful? is it a nice to have? - I'm unfamiliar with JSON so I'm not afraid to ask stupid questions)

BIST for input validation look like this:
%!error fromJSON (true) <here your error message between angular brackets>
I core Octave you can find many .m-file functions with examples.

toJSON:
=======

  1. does it accept classdef objects? Java objects? COM objects? empty objects?
  2. what if PREC is a struct?
  3. same for JSON?

What does the code at L.63 actually do?

isbool(PREC) && ([PREC,JSON] = {[],PREC}{})

Assigment happens to [PREC,JSON] but the rest? IMO this is unneededly convoluted code. I tried several values and classes for PREC and JSON and consistently got 0 (obviously) that is assigned to nothing.

At the bottom of both functions there's "scraps" that don't belong there. Should either be before the tests or deleted.

Version tracking:
"help version"
"help compare_versions"

Does toJSoN produce "sufficiently compatible" output for "regular" input, that can be read by e.g., jsondecode? (or any other JSON parser Out There? Matlab? we aim at Matlab compatbility)
I've read and understand your musings in comment #3 so please a short answer :-)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sat 17 Oct 2020 12:45:25 PM UTC, comment #5: 

Also, I looked into 'jsonlab' a little more. 

- it looks like its designed to a specific data format 'JData',  developed by the same author. https://github.com/OpenJData/jdata/blob/master/JData_specification.md

- it's designed with a specific, matching parser in mind; one for python, none for Javascript. Frankly, Octave-to-python seems moot because of NumPy.  Javascript on the other hand... https://github.com/fangq/jsonlab#sharing-jsonlab-created-data-files-in-python

> In short, to conveniently read/write data files created by
> JSONLab into Python, whether they are JSON based or binary
> JData/UBJSON based, one just need to download the below two
> light-weight python modules


- like 'object2json', 'jsonlab' can be slow.  speed, for some reason, is affected by ND array shape, not just size.  for example, on my desktop, "savejson(rand(1000000,1))" takes 1sec but takes 500sec for either "savejson(rand(1,1000000))"  or "savejson(rand(100,100,100))".  In contrast, "toJSON" takes 1sec for all 3 shapes and maintains the respective shape within the JSON file for straightforward parsing in Javascript and python (and others).  

- "loadjson" while appearing to work with non-JData style JSON files, mangles ND arrays similar to 'object2json'.  It assumes a strict FORTRAN style indexing not used in Javascript or python (plain), which use the c-style array indexing.  To be fair, "loadjson" is much faster than "fromJSON", presumably due to it's use of metadata.

Anonymous
Sat 17 Oct 2020 02:49:29 AM UTC, comment #4: 

comment #3:

> When I make the requested change, do I upload it here again?  how do I name or version number the new files?


Don't be too worried about this, just name it "toJSONv2.m" or alike.  It is not big deal to remove the "v2" suffix before testing it 😉

Kai Torben Ohlhus <siko1056>
Project Member
Sat 17 Oct 2020 01:28:31 AM UTC, comment #3: 

On input validation, Ok.  But "toJSON.m" is intended to take any and ALL objects and convert to JSON string, so I'm not sure what you have in minded there.  For "fromJSON.m", it works only on string, so that is no prob!   As for malformed JSON string, not sure how to valid other than just parsing the string.

Also, I don't know what you mean "yet input validation BIST tests are lacking"...as compared to what is already there.  The testing includes all the different Octave object classes I know off. More elaboration would be instructive.

On Octave version support, I don't know how to test for that, other than loading all versions on my computer.  The first iteration was develop on 4 but refactored on 5.  In refactoring, the only new feature added was default function args, so I'm 99.9999% sure it will still work on 4.  It uses strcut2cell, str2func, is_function_handle, cell2mat, regexp, regexprep and subsref/subsasgn, among other more mundane functions.

On JSON expertise, I don't know any.  I usually consult a web-browser's developer console (see JSON.parse).

On 'jsonlab', as well as the 'jsonencode' and 'jsondecode', I think 'toJSON' and 'fromJSON' still has some advantages.   I have very limited (<2hrs) experience with these so I cannot say expertly, but:

jsonlab seem okay, but it doesn't seem to be made to work well with other languages.   It seems to be made primarly for data storage, not data transmission.  Besides the fact that it's loaded with filesaving and compression (requiring external java library), it relies a lot on metadata.  For example, with 2D matrix, 'savejson.m' converts to a conventional JSON string, readily parsed to identical form in Javescript, python, etc., but not for ND arrays. For ND array, 'savejson' flattens the data into 1D array and includes metadata on array shape, variable name and class.  While still JSON compliant, it is not a conventional format for arrays, and require additional parsing on receiving end (i.e. non-trivial for Javascript, the JS in json).  

Also, setting aside the fact that I was unaware of the forthcoming 'jsonencode/decode', I also want to solve the problem of output data size and transmitting Inf and NaN, which I think neither 'jsonlab' nor 'jsonencode' really address properly.

First, 'toJSON' allows floating numbers to be truncated (precision), for more compact data transmission for large datasets.  For complex structured data, this is non-trivial either before or after encoding.

Second, I did not like their ('jsonlab', 'jsonencode', Firefox, Chrome, python, etc.) handling of Inf or NaN.   Either they turns both into 'null' (making them indistinguishable), or replace it with quoted-strings '"Infinity"' or '"NaN"'.  This require additional (computationally expensive for Javascript) parsing on receiving end.

The problem stems from the fact that JSON spec pretends to be IEEE754 complaint but doesn't actually make allowance for Infinity or NaN.  So, those numbers get trashed.  Instead 'toJSON' turns Inf into '1e999'.  While not a valid IEEE754 number, it automatically turns itself into Infinity when parsed by Javascript, Python, Ruby, Octave, Matlab, etc.  No fuss. No muss.  Nothing can be done about NaN, but with Infinity automatically parse, dealing with NaN is easier.

Likewise, 'fromJSON' cheats a little with regards to receiving Inf, it treats any floating number >= 1e308 as Inf.  Natively, Octave, JS, python, Ruby, etc. treat >= 2e308 as Inf, but by nudging this threshold down a tad, there's less work for transmitting end (JavaScript) to encode Inf, and it allows 'fromJSON' to very easily differentiate Inf from NaN.

Finally, 'toJSON' and 'fromJSON' handles complex number, which is slated to be unsupported in 'jsonencode' and 'jsondecode' (per MATLAB).  'jsonlab' uses more metadata.  Complex number are not native in Javascript nor supported by JSON, so structured object are the norm.  'jsonlab' does not follow this norm.  'toJSON', on the hand, handles complex number in a manner consistent with that norm. 'object2json' does similarly support complex numbers, however, where as it makes an array of structure object, 'toJSON' makes (for speed and output compactness) a structure object of arrays.  It's a design choice that super easy to change (add 2 lines, thanks to judicious use of recursion).  Additionally, 'object2json' is significantly (very!) slower, wrecks ND arrays, and there is no 'json2object'.  In contrast, 'fromJSON' reads in that structure object and converts it back into a Octave complex number. It works with either struct of array (small,fast) or array of struct (big,slow...for JSON).

When I make the requested change, do I upload it here again?  how do I name or version number the new files?

Anonymous
Fri 16 Oct 2020 07:28:26 PM UTC, comment #2: 

To reiterate what I wrote in the help-octave ML (typos fixed):

> In dev Octave we now have jsonencode and jsondecode functions which AFAICS are superior to object2json. I'll keep object2json in the io package for a while, to cater for users who have older Octave versions.
> In the mean time if you have JSON encode/decode functions that are better than what is now in io I'm interested.
> * Don't use the names jsonencode and jsondecode.
> * Be sure they're not depending on external libraries, just native Octave code.
> * At some time in the future the JSON functions in the io package will be deprecated. But that could take a few years ...
> * I'm not going to maintain them, that's up to you as author and maybe other volunteers. As package maintainer I'm just the first to make sure the package works, its functions have adequate documentation and no severe bugs.

So yes I'm not opposed to inclusion in the io package.
Other than Kai I'm not convinced that with Octave-7.0.0 next year your contributions would be obsolete; we regularly get bugs and question about ancient Octave versions that keep lingering on. Only when the io package no longer supports Octave < 7.x they'd be dropped.

Compared to JSONLAB, what do your functions offer? JSONLAB looks feature-complete, but I suppose there's a place for simple JSON I/O as well for upcoming Octave 6.1 and less.
What's the oldest Octave version your contributions will need? io currently supports Octave-4.2 so they should at least work there.

I can be a bit picky about contributions to packages I maintain (io and mapping), my quality demands are as follows, from more to less importance:

  1. Functions should have accurate output (and of course no obvious let alone severe bugs)
  2. Functions should be robust to wrong input (i.e., I like rigorous input validation)
  3. Functions should have adequate BIST tests if at all possible and practical, not only for their main function but also for input checks
  4. Octave coding style is strongly desired
  5. There should be a good Texinfo help text explaining all inputs and outputs of the functions.

Usually I'll polish the last two points if at least some attention was paid to them. Texinfo can be hard for novices so I don't mind helping out there.

Now, looking into toJSON.m and fromJSON.m:
1. (not checked yet, I don't use JSON myself). Can anyone help out here?
2. no input validation at all
3. excellent! yet input validation BIST tests are lacking ...
4. also excellent!
5. most of the required things are present, but needs more structure.

So if you add input validation and BIST tests, and someone verifies proper operation, I suppose they'll make it into the io package. Maybe not upcoming 2.6.3 but the next one after that should be possible.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Fri 16 Oct 2020 07:55:14 AM UTC, comment #1: 

Thanks for the contribution.  As described in the help mailing-list

https://lists.gnu.org/archive/html/help-octave/2020-10/msg00017.html

and the a must read (bug #53100) before putting too much effort in this project.

1. Octave 7 will have fast Matlab compatible json(en/de)code functions.
2. There is already a well-tested comprehensive JSON library as pure M-File implementation. https://github.com/fangq/jsonlab
3. Your function might fill the gap before the Octave 7 release next year by
3a. Inclusion in the io package
3b. Creation of an own lightweight package https://github.com/gnu-octave/pkg-example

Assigning this bug to Philip to decide about inclusion into the OF io package (3a.).  For core Octave your contribution will not be regarded (1.).  At your option (3b.) is always possible.

Kai Torben Ohlhus <siko1056>
Project Member
Thu 15 Oct 2020 06:04:42 PM UTC, original submission:  

Here (toJSON.m) is an alternative to object2json.m, along with a matching json parser (fromJSON.m).  It's faster, isn't constrained by numeric class, and conditions Inf, NaN, and NA for error-free parsing on receiving end.    It handles ND arrays better as well.

On documentation, I tried but am having trouble with formatting the layout.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50183:  fromJSON_v2_5.m added by None (21KiB - text/x-objcsrc - tweaked texinfo)
file #50184:  toJSON_v2_5.m added by None (18KiB - text/x-objcsrc - tweaked texinfo)
file #50178:  fromJSON.m added by philipnienhuis (21KiB - application/octet-stream)
file #50179:  toJSON.m added by philipnienhuis (18KiB - application/octet-stream)
file #50180:  fromJSON.diff added by philipnienhuis (14KiB - application/octet-stream)
file #50181:  toJSON.diff added by philipnienhuis (12KiB - application/octet-stream)
file #50177:  fromJSON_v2_4.m added by None (21KiB - text/x-objcsrc - fixed stupidity (fromJSON_v_2_3 is wrong file) and the UTF-8 error fix (in fromJSON_v_2_2.m))
file #50176:  fromJSON_v2_3.m added by None (10KiB - text/x-objcsrc - fixed the UTF-8 error fix)
file #50174:  toJSON_v2_2.m added by None (18KiB - text/x-objcsrc - fix UTF-8 encoding issue; added more BIST and help)
file #50175:  fromJSON_v2_2.m added by None (21KiB - text/x-objcsrc - fix UTF-8 encoding issue; added more BIST and help)
file #50144:  fromJSON_BIST.tst added by philipnienhuis (18KiB - application/octet-stream)
file #50145:  toJSON_BIST.tst added by philipnienhuis (23KiB - application/octet-stream)
file #50136:  fromJSON_v2_1.m added by None (17KiB - text/x-objcsrc - updated to comply with Octave 7)
file #50137:  toJSON_v2_1.m added by None (14KiB - text/x-objcsrc - updated to comply with Octave 7)
file #50087:  fromJSON_v2.m added by None (17KiB - text/x-objcsrc - update files)
file #50088:  toJSON_v2.m added by None (14KiB - text/x-objcsrc - update files)
file #49987:  toJSON.m added by None (9KiB - text/x-objcsrc)
file #49988:  fromJSON.m added by None (11KiB - text/x-objcsrc)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 24 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-11-01 philipnienhuis StatusIn Progress => Done
        Open/ClosedOpen => Closed
    2020-11-01 None Attached File- => Added fromJSON_v2_5.m, #50183
        Attached File- => Added toJSON_v2_5.m, #50184
    2020-11-01 philipnienhuis Attached File- => Added fromJSON.m, #50178
        Attached File- => Added toJSON.m, #50179
        Attached File- => Added fromJSON.diff, #50180
        Attached File- => Added toJSON.diff, #50181
    2020-10-31 None Attached File- => Added fromJSON_v2_4.m, #50177
    2020-10-31 None Attached File- => Added fromJSON_v2_3.m, #50176
    2020-10-31 None Attached File- => Added toJSON_v2_2.m, #50174
        Attached File- => Added fromJSON_v2_2.m, #50175
    2020-10-28 philipnienhuis StatusNeed Info => In Progress
    2020-10-27 philipnienhuis Attached File- => Added fromJSON_BIST.tst, #50144
        Attached File- => Added toJSON_BIST.tst, #50145
    2020-10-27 None Attached File- => Added fromJSON_v2_1.m, #50136
        Attached File- => Added toJSON_v2_1.m, #50137
    2020-10-24 None Attached File- => Added fromJSON_v2.m, #50087
        Attached File- => Added toJSON_v2.m, #50088
    2020-10-16 siko1056 Priority5 - Normal => 3 - Low
        StatusNone => Need Info
        Assigned toNone => philipnienhuis
    2020-10-15 None Attached File- => Added toJSON.m, #49987
        Attached File- => Added fromJSON.m, #49988

    Back to the top


    Powered by Savane 3.6