bugGNU Octave - Bugs: bug #53100, implement jsondecode, jsonencode...

 
 

bug #53100: implement jsondecode, jsonencode functions

Submitted by:  CH <atcl>
Submitted on:  Thu 08 Feb 2018 10:22:09 PM UTC  
 
Category:  Octave Forge Package Severity:  1 - Wish
Priority:  5 - Normal Item Group:  Feature Request
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Tue 04 Jun 2019 07:11:37 PM UTC, comment #16: 

@Andrew. Totally agreed. To match the feature set as in matlab's jsonencode/decode, I agree it is important for jsonstuff to first provide the basic data encoding/decoding capabilities.

JSONLab will likely serve for the JData encoder/decoder purposes and let jsonencode/decode to take care of the stream-level encoding/decoding, given their speed advantage.

Eventually, MATLAB will likely have to address the issue of serializing complex data structures (such as complex-valued array, sparse array, graphs etc), this will likely take many years. You can naturally add those in the future if Mathworks decides to update jsonencode/decode (and allow any specification of such serialization to solidify).

I don't have experience with rapidjson or jsoncpp, as I mostly code in C. I used cjson (https://github.com/DaveGamble/cJSON) which is super-lightweight and fast enough for most of my codes.

Binary JSON has smaller audiences but in my opinion is more advantageous in storing scientific data. JSONlab supports UBJSON (http://ubjson.org) and recently MessagePack (http://msgpack.org). Between the two, I like UBJSON a lot - it has the two top benefits of JSON - simplicity and readability (yes, despite it is a binary format, it is surprisingly human-readable). Might be beneficial to also implement UBJSON native support for Octave. The below github project provides a simple C encoder/decoder for UBJSON

https://github.com/Steve132/ubj

and I added a N-D array header in my fork.

https://github.com/fangq/ubj

Qianqian Fang <fangq>
Mon 03 Jun 2019 11:25:00 PM UTC, comment #15: 

Hi Qianqian! Thanks for chiming in.

I think that JSONlab and my jsonstuff package could happily co-exist. The goal of jsonstuff is simply to create Matlab-compatible implementations of jsonencode/jsondecode for Octave. My impression is that jsonencode/jsondecode is to allow Matlab to access external APIs that provide data in "native" JSON format. It doesn't cover more general or flexible JSON handling, or persisting of arbitrary Matlab data structures to JSON, like your JSONlab does. My goal is to eventually get jsonstuff merged into core Octave and cease to have it be an independent package.

I have no intention of expanding jsonstuff beyond the base jsonencode/jsondecode, or (probably) even supporting the Matlab Production Server JSON format, since that's not available in base Matlab, and as far as I know, MPS interoperability isn't an Octave design goal. jsonstuff's goals are: 1) Matlab-compatible jsonencode/jsondecode, 2) decent performance, and 3) nothing else.

I'll take a look at your JData stuff when I have some spare time; I'm pretty busy with the day job and a conference right now. Offhand, I think support for extended JSON functionality like that should probably stay independent from Octave's jsonencode/jsondecode stuff: since that's not provided by base Matlab either, it seems suited for a package that would work in both Matlab and Octave.

Andrew Janke <apjanke>
Sat 01 Jun 2019 06:09:10 AM UTC, comment #14: 

I just want to add that I made octave-rapidjson 2 years ago which I use regularly in my project. It uses the rapidjson SAX parser so big json files doesn't need to be held twice in memory:

https://github.com/Andy1978/octave-rapidjson

Andreas Weber <andy1978>
Project Member
Fri 31 May 2019 10:05:52 PM UTC, comment #13: 

Just found this thread. Glad that someone is working on a builtin JSON encoder/decoder for Octave. My JSONLab toolbox (http://github.com/fangq/jsonlab), as mentioned by the OP, does work for Octave, but the speed to encode/decode large sized JSON data is horrific on Octave.

I just want to mention a JSON-base complex data encoding scheme - JData specification - that I have been working on. Most of these were derived from the encoding method I already used in my JSONLab toolbox, but generalized significantly. Happy to hear if you have any feedback/suggestions/thoughts

http://github.com/fangq/jdata

more background info can be found in this post

https://mailarchive.ietf.org/arch/msg/json/z7cUcdABZSS_sgvhVExWUW5y1Yc

I also want to point out jsonencode/jsondecode in MATLAB do not support saving/loading complex/sparse arrays, as well as some other complex data structures. I noticed that starting from MATLAB R2018a, matlab production server (mps) includes a more complex JSON encoder/decoder

https://www.mathworks.com/help/mps/restfuljson/json-representation-of-matlab-data-types.html

it also uses annotated struct mwtype/mwsize/mwdata to encode complex data, almost exactly like what I did in JSONLab and JData.

anyways, happy to hear if you have any thoughts on this project.

Qianqian Fang <fangq>
Tue 05 Mar 2019 03:00:22 AM UTC, comment #12: 

Gotcha.

I looked at a few different JSON libraries before picking JsonCpp. I liked its interface and documentation, and its "consolidated source" option made it really easy to include in a self-contained Octave package for a non-C++-expert like myself.

RapidJson wasn't one of the libraries I considered, though. Don't know why I didn't find it in my original search. And its header-only nature makes it as easy as the "consolidated source" of JsonCpp. So when I find some time I'll have a closer look at it and consider switching over before the jsonstuff implementation gets much bigger. Added it as a TODO at https://github.com/apjanke/octave-jsonstuff/issues/4.

Andrew Janke <apjanke>
Tue 26 Feb 2019 08:12:19 PM UTC, comment #11: 

@apjanke - I wasn't suggesting that the code should support multiple backends at all, rather suggesting that you might look at the various options available before locking in on one. If possible, maybe design the code with some separation to make it feasible to port the functions to a different parser if the need arises someday.

Mike Miller <mtmiller>
Project Administrator
Sat 02 Feb 2019 08:50:07 AM UTC, comment #10: 

I agree: this should end up in Octave core, since these are core (non-toolbox) Matlab functions. Just, uhh, gimme a moment to shake out the bugs.

Supporting multiple parser library back-ends might make this a lot more complicated – JSON is a super simple format, and as such the parser libraries have really simple interfaces, but they're not at all the same interface. Writing a compatibility layer to switch between different parser library back-ends at run- or compile-time might be more complicated than the whole rest of this library. IMHO.

I looked at rapidjson - that looks really cool. I think I'd be down to switch it out for JsonCpp here, if folks have a preference. But I dunno about supporting both/either: that compatibility/flexibility would mean a lot of code.

If people actually care, maybe I could code up an alternate version using rapidjson and we could discuss which one we prefer, and then pick one?

Andrew Janke <apjanke>
Wed 30 Jan 2019 06:18:50 PM UTC, comment #9: 

You can continue to develop and stabilize the jsonstuff any way you want to, but ultimately I would hope to see them integrated into Octave's core library. It is up to you if you want to integrate into a package first and then into Octave, but eventually they should be pulled into Octave one way or another.

Linking with the libjsoncpp library is no problem, the license is clearly GPL compatible.

There seem to be no shortage of json parsing library options in C and C++ with different tradeoffs. If possible I would try to keep the options open to allow porting the functions to different parsing backends as development progresses. For example rapidjson sounds like a very appealing alternative.

Mike Miller <mtmiller>
Project Administrator
Wed 30 Jan 2019 05:08:49 PM UTC, comment #8: 

> But getting it installed and available on Windows is a big pain. And I'd like to see all 3 big OSes supported.


Three source files should be easy to build during package installation on all 3 "big" OS-es.

Clipper (lib) in geometry is just two source files (cpp + h plus a mex file) so looks comparable.

"Public domain" sounds a bit fishy but MIT should be good. I'll gladly let the lawyer types around here (if any) shed more light on it. Or ask on the maintainers list.

If incorporated in io, make sure you include configure.in and Makefile.in stanzas + NEWS + INDEX changes.
If required, using configure you might avoid building JSON libs already present on a system. Remember, configure should check capabilities not versions.

As to "stable", IMO all Forge stuff is beta stuff at best :-)  There's ample experimental code in OF packages.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 29 Jan 2019 10:38:18 PM UTC, comment #7: 

Hi Philip!

> Are we sure the io package is a good home for it? If yes, good, if no, maybe miscellaneous could be better, I really don't know. Opinions?


To me, io seems like the best fit among the existing supported Octave Forge packages. io is for "Input/output in external formats", and JSON seems to qualify as an external format. io's already got support for Office formats, which seems somewhat analogous. miscellaneous seems to be mostly numeric stuff, and doesn't currently have any file format support that I can see.

> OTOH if the JSON library is not too big, comprises a limited nr. of files, and has a GPL compatible license, it can be included in, and shipped with, the io package. JuanPi and I did s/th similar for the clipper polygon clipping library in the geometry package.


This is the approach I'm taking with jsonstuff now, using the JsonCpp library. It's 400KB including both the tarball and decompressed source, and is only three source files when using the "amalgamated source" approach.

"Vendoring" the library like this is by far my preferred approach: installing the lib using Linux or Mac package managers is really easy. But getting it installed and available on Windows is a big pain. And I'd like to see all 3 big OSes supported.

JsonCpp's license is "Public Domain in jurisdictions where that is recognized, and MIT License elsewhere". I don't know if that counts as GPL-compatible. If it's not, switching to a different GPL-compatible library might be the best choice.

I think for now I'd like to continue developing jsonstuff as its own package where I have commit rights so I can iterate on it quickly. And then once it's feature-complete and stable maybe it could be moved to io or core Octave.

Andrew Janke <apjanke>
Tue 29 Jan 2019 09:58:40 PM UTC, comment #6: 

(io pkg maintainer here)

Yes, I'd happily accept compiled JSON stuff, but I'm afraid I can't maintain it due to lack of knowledge about JSON - so maintenance is left to contributor(s).
Are we sure the io package is a good home for it? If yes, good, if no, maybe miscellaneous could be better, I really don't know. Opinions?

If it introduces a new dependency I'd prefer to make it a "soft" dependency - if the io package is installed (built) and the underlying JSON lib is absent, the pkg should be installed w/o the JSON functions depending on that library, please with a suitable message.
As an example, Carnë created a particularly luxurious version for gdalread in the mapping package and the GDAL library. IMO it need not be that slick but something along those lines would be nice, the bottom line is users do need to know and be able to find out if their io package installation supports the JSON stuff or not.

OTOH if the JSON library is not too big, comprises a limited nr. of files, and has a GPL compatible license, it can be included in, and shipped with, the io package. JuanPi and I did s/th similar for the clipper polygon clipping library in the geometry package.

Maybe then the existing JSON .m-file function could be replaced by faster compiled ones?

Philip Nienhuis <philipnienhuis>
Project Member
Tue 29 Jan 2019 03:02:20 PM UTC, comment #5: 

atcl: Here's a new package with jsonencode/jsondecode for you: https://github.com/apjanke/octave-jsonstuff.

Andrew Janke <apjanke>
Fri 18 Jan 2019 08:26:41 PM UTC, comment #4: 

Would the io package maintainers be willing to take a dependency on jsoncpp or one of the other mentioned C/C++ JSON libraries? Performance for a native M-code JSON parser is probably going to be grody.

Andrew Janke <apjanke>
Fri 09 Feb 2018 04:09:50 PM UTC, comment #3: 

I would be happy to follow Philip's suggestion and introduce these two functions in to the io package, and then migrate them to core after they have proved stable and useful.

Matlab has a habit of adding functions that persist for only a few releases before being renamed or even removed.  These two have the feel of functions that are going to eventually change.

Rik <rik5>
Project Administrator
Fri 09 Feb 2018 09:42:30 AM UTC, comment #2: 

Also have a look at JSONio (C MEX wrapper around jsmn):
  https://www.artefact.tk/software/matlab/jsonio/
and octave-rapidjson (C++ Octave wrapper around rapidjson):
  https://github.com/Andy1978/octave-rapidjson

Guillaume <gyom>
Fri 09 Feb 2018 07:51:47 AM UTC, comment #1: 

I'd rather call it's a feature request.

The io package contains a (contributed) json2object function.
What if people who want to write jsonencode and jsondecode contribute it to the io package (I'll happily accept them) to mature there? Once stable enough they can be moved to core Octave, if the core devs agree of course.
The io package is more regularly updated than Octave releases.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 08 Feb 2018 10:22:09 PM UTC, original submission:  

Matlab introduced the function jsondecode(string) which parses a json formatted string to a cell array of predefined output types based on the json input type. Similarly, jsonencode converts data into a json formatted string. Further details are in the documentation.

An open-source json parser for Octave and MATLAB is here: https://github.com/fangq/jsonlab which may be of help.

CH <atcl>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by andy1978 (Posted a comment)
  • -email is unavailable- added by fangq (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by siko1056 (interested)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by gyom (Posted a comment)
  • -email is unavailable- added by atcl (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-02-26 mtmiller Severity3 - Normal => 1 - Wish
    2018-02-12 siko1056 Carbon-Copy- => Added siko1056
    2018-02-09 rik5 CategoryNone => Octave Forge Package
        Item GroupMatlab Compatibility => Feature Request
        StatusNone => Confirmed
        Summaryjsondecode (and jsonencode) => implement jsondecode, jsonencode functions

    Back to the top


    Powered by Savane 3.5