bugGNU Octave - Bugs: bug #53100, implement jsondecode, jsonencode...

 
 

bug #53100: implement jsondecode, jsonencode functions

Submitter:  CH <atcl>
Submitted:  Thu 08 Feb 2018 10:22:09 PM UTC
   
 
Category:  Octave Function Severity:  1 - Wish
Priority:  5 - Normal Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 16 Sep 2020 01:13:48 AM UTC, comment #21: 

Yes, this can be closed.  Abdallah did a great job this summer

https://abdallahshamy.wordpress.com/

Bugs / improvements of jsondecode, jsonencode can be handled in different bug reports.

Many thanks for introducing all your JSON projects here.  This discussion was very helpful 🙂

Closing as fixed.

Kai Torben Ohlhus <siko1056>
Group Member
Tue 15 Sep 2020 05:52:04 PM UTC, comment #20: 

This year's GSoC student Abdallah Elshamy added these functions to core Octave. And they have been pushed to the default branch (Octave 7) in the meantime.

Can this report be closed?

Markus Mützel <mmuetzel>
Group administrator
Tue 18 Feb 2020 12:28:20 PM UTC, comment #19: 

I'm still working on JSONStuff - https://github.com/apjanke/octave-jsonstuff - and I think it's about feature-complete, but it's not Fast (because some stuff is still in the M-code layer, that should be down at the C++ oct-file layer). I'm working on adding more unit tests to it, and once I'm comfortable with the test coverage, will work on optimizing it with C++.

I'd like to get it on to Octave Forge at this point, if you're still taking Forge submissions. I think it's good enough for end users to work with. And while there are other Octave JSON packages out there, users might be interested in this one in particular since it is Matlab-compatible, so there's no need to modify M-code being ported from Matlab.

Andrew Janke <apjanke>
Sun 15 Dec 2019 03:22:26 AM UTC, comment #18: 

Is there anything happening on this front?

Looking at my mails I was working on adding JSONLab to Octave back in 2014 (when it was on svn). It seems I was pretty happy with it.

What needs to be done?

Juan Pablo Carbajal <juanpi>
Group Member
Thu 28 Nov 2019 12:57:28 PM UTC, comment #17: 

Hi all,

I think Mike suggestion is the way to go.
In other words, what is needed in json-stuff to separate the user interface with the low level library?

My first observation would be that the parsing of options should be done at the m-file level (so we do not need to re-do it if we change the low-level library)

For jsonencode we need to parse the 'ConvertInfAndNaN' option

I do not see any other option for jsondecode.

So what is stopping us from using json-stuff as the first working implementation of the functionality?

If the separation is well done, the decision of what is a good low level library (or even checking for different libraries, like https://github.com/nlohmann/json (MIT)) can be postponed.

Juan Pablo Carbajal <juanpi>
Group Member
Tue 04 Jun 2019 07:11:37 PM UTC, comment #16: 

@Andrew. Totally agreed. To match the feature set as in matlab's jsonencode/decode, I agree it is important for jsonstuff to first provide the basic data encoding/decoding capabilities.

JSONLab will likely serve for the JData encoder/decoder purposes and let jsonencode/decode to take care of the stream-level encoding/decoding, given their speed advantage.

Eventually, MATLAB will likely have to address the issue of serializing complex data structures (such as complex-valued array, sparse array, graphs etc), this will likely take many years. You can naturally add those in the future if Mathworks decides to update jsonencode/decode (and allow any specification of such serialization to solidify).

I don't have experience with rapidjson or jsoncpp, as I mostly code in C. I used cjson (https://github.com/DaveGamble/cJSON) which is super-lightweight and fast enough for most of my codes.

Binary JSON has smaller audiences but in my opinion is more advantageous in storing scientific data. JSONlab supports UBJSON (http://ubjson.org) and recently MessagePack (http://msgpack.org). Between the two, I like UBJSON a lot - it has the two top benefits of JSON - simplicity and readability (yes, despite it is a binary format, it is surprisingly human-readable). Might be beneficial to also implement UBJSON native support for Octave. The below github project provides a simple C encoder/decoder for UBJSON

https://github.com/Steve132/ubj

and I added a N-D array header in my fork.

https://github.com/fangq/ubj

Qianqian Fang <fangq>
Mon 03 Jun 2019 11:25:00 PM UTC, comment #15: 

Hi Qianqian! Thanks for chiming in.

I think that JSONlab and my jsonstuff package could happily co-exist. The goal of jsonstuff is simply to create Matlab-compatible implementations of jsonencode/jsondecode for Octave. My impression is that jsonencode/jsondecode is to allow Matlab to access external APIs that provide data in "native" JSON format. It doesn't cover more general or flexible JSON handling, or persisting of arbitrary Matlab data structures to JSON, like your JSONlab does. My goal is to eventually get jsonstuff merged into core Octave and cease to have it be an independent package.

I have no intention of expanding jsonstuff beyond the base jsonencode/jsondecode, or (probably) even supporting the Matlab Production Server JSON format, since that's not available in base Matlab, and as far as I know, MPS interoperability isn't an Octave design goal. jsonstuff's goals are: 1) Matlab-compatible jsonencode/jsondecode, 2) decent performance, and 3) nothing else.

I'll take a look at your JData stuff when I have some spare time; I'm pretty busy with the day job and a conference right now. Offhand, I think support for extended JSON functionality like that should probably stay independent from Octave's jsonencode/jsondecode stuff: since that's not provided by base Matlab either, it seems suited for a package that would work in both Matlab and Octave.

Andrew Janke <apjanke>
Sat 01 Jun 2019 06:09:10 AM UTC, comment #14: 

I just want to add that I made octave-rapidjson 2 years ago which I use regularly in my project. It uses the rapidjson SAX parser so big json files doesn't need to be held twice in memory:

https://github.com/Andy1978/octave-rapidjson

Andreas Weber <andy1978>
Group Member
Fri 31 May 2019 10:05:52 PM UTC, comment #13: 

Just found this thread. Glad that someone is working on a builtin JSON encoder/decoder for Octave. My JSONLab toolbox (http://github.com/fangq/jsonlab), as mentioned by the OP, does work for Octave, but the speed to encode/decode large sized JSON data is horrific on Octave.


I just want to mention a JSON-base complex data encoding scheme - JData specification - that I have been working on. Most of these were derived from the encoding method I already used in my JSONLab toolbox, but generalized significantly. Happy to hear if you have any feedback/suggestions/thoughts

http://github.com/fangq/jdata

more background info can be found in this post

https://mailarchive.ietf.org/arch/msg/json/z7cUcdABZSS_sgvhVExWUW5y1Yc


I also want to point out jsonencode/jsondecode in MATLAB do not support saving/loading complex/sparse arrays, as well as some other complex data structures. I noticed that starting from MATLAB R2018a, matlab production server (mps) includes a more complex JSON encoder/decoder

https://www.mathworks.com/help/mps/restfuljson/json-representation-of-matlab-data-types.html

it also uses annotated struct mwtype/mwsize/mwdata to encode complex data, almost exactly like what I did in JSONLab and JData.

anyways, happy to hear if you have any thoughts on this project.

Qianqian Fang <fangq>
Tue 05 Mar 2019 03:00:22 AM UTC, comment #12: 

Gotcha.

I looked at a few different JSON libraries before picking JsonCpp. I liked its interface and documentation, and its "consolidated source" option made it really easy to include in a self-contained Octave package for a non-C++-expert like myself.

RapidJson wasn't one of the libraries I considered, though. Don't know why I didn't find it in my original search. And its header-only nature makes it as easy as the "consolidated source" of JsonCpp. So when I find some time I'll have a closer look at it and consider switching over before the jsonstuff implementation gets much bigger. Added it as a TODO at https://github.com/apjanke/octave-jsonstuff/issues/4.

Andrew Janke <apjanke>
Tue 26 Feb 2019 08:12:19 PM UTC, comment #11: 

@apjanke - I wasn't suggesting that the code should support multiple backends at all, rather suggesting that you might look at the various options available before locking in on one. If possible, maybe design the code with some separation to make it feasible to port the functions to a different parser if the need arises someday.

Mike Miller <mtmiller>
Group Member
Sat 02 Feb 2019 08:50:07 AM UTC, comment #10: 

I agree: this should end up in Octave core, since these are core (non-toolbox) Matlab functions. Just, uhh, gimme a moment to shake out the bugs.

Supporting multiple parser library back-ends might make this a lot more complicated – JSON is a super simple format, and as such the parser libraries have really simple interfaces, but they're not at all the same interface. Writing a compatibility layer to switch between different parser library back-ends at run- or compile-time might be more complicated than the whole rest of this library. IMHO.

I looked at rapidjson - that looks really cool. I think I'd be down to switch it out for JsonCpp here, if folks have a preference. But I dunno about supporting both/either: that compatibility/flexibility would mean a lot of code.

If people actually care, maybe I could code up an alternate version using rapidjson and we could discuss which one we prefer, and then pick one?

Andrew Janke <apjanke>
Wed 30 Jan 2019 06:18:50 PM UTC, comment #9: 

You can continue to develop and stabilize the jsonstuff any way you want to, but ultimately I would hope to see them integrated into Octave's core library. It is up to you if you want to integrate into a package first and then into Octave, but eventually they should be pulled into Octave one way or another.

Linking with the libjsoncpp library is no problem, the license is clearly GPL compatible.

There seem to be no shortage of json parsing library options in C and C++ with different tradeoffs. If possible I would try to keep the options open to allow porting the functions to different parsing backends as development progresses. For example rapidjson sounds like a very appealing alternative.

Mike Miller <mtmiller>
Group Member
Wed 30 Jan 2019 05:08:49 PM UTC, comment #8: 


> But getting it installed and available on Windows is a big pain. And I'd like to see all 3 big OSes supported.


Three source files should be easy to build during package installation on all 3 "big" OS-es.

Clipper (lib) in geometry is just two source files (cpp + h plus a mex file) so looks comparable.

"Public domain" sounds a bit fishy but MIT should be good. I'll gladly let the lawyer types around here (if any) shed more light on it. Or ask on the maintainers list.

If incorporated in io, make sure you include configure.in and Makefile.in stanzas + NEWS + INDEX changes.
If required, using configure you might avoid building JSON libs already present on a system. Remember, configure should check capabilities not versions.

As to "stable", IMO all Forge stuff is beta stuff at best :-)  There's ample experimental code in OF packages.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 29 Jan 2019 10:38:18 PM UTC, comment #7: 

Hi Philip!

> Are we sure the io package is a good home for it? If yes, good, if no, maybe miscellaneous could be better, I really don't know. Opinions?


To me, io seems like the best fit among the existing supported Octave Forge packages. io is for "Input/output in external formats", and JSON seems to qualify as an external format. io's already got support for Office formats, which seems somewhat analogous. miscellaneous seems to be mostly numeric stuff, and doesn't currently have any file format support that I can see.

> OTOH if the JSON library is not too big, comprises a limited nr. of files, and has a GPL compatible license, it can be included in, and shipped with, the io package. JuanPi and I did s/th similar for the clipper polygon clipping library in the geometry package.


This is the approach I'm taking with jsonstuff now, using the JsonCpp library. It's 400KB including both the tarball and decompressed source, and is only three source files when using the "amalgamated source" approach.

"Vendoring" the library like this is by far my preferred approach: installing the lib using Linux or Mac package managers is really easy. But getting it installed and available on Windows is a big pain. And I'd like to see all 3 big OSes supported.

JsonCpp's license is "Public Domain in jurisdictions where that is recognized, and MIT License elsewhere". I don't know if that counts as GPL-compatible. If it's not, switching to a different GPL-compatible library might be the best choice.

I think for now I'd like to continue developing jsonstuff as its own package where I have commit rights so I can iterate on it quickly. And then once it's feature-complete and stable maybe it could be moved to io or core Octave.

Andrew Janke <apjanke>
Tue 29 Jan 2019 09:58:40 PM UTC, comment #6: 

(io pkg maintainer here)

Yes, I'd happily accept compiled JSON stuff, but I'm afraid I can't maintain it due to lack of knowledge about JSON - so maintenance is left to contributor(s).
Are we sure the io package is a good home for it? If yes, good, if no, maybe miscellaneous could be better, I really don't know. Opinions?

If it introduces a new dependency I'd prefer to make it a "soft" dependency - if the io package is installed (built) and the underlying JSON lib is absent, the pkg should be installed w/o the JSON functions depending on that library, please with a suitable message.
As an example, Carnë created a particularly luxurious version for gdalread in the mapping package and the GDAL library. IMO it need not be that slick but something along those lines would be nice, the bottom line is users do need to know and be able to find out if their io package installation supports the JSON stuff or not.

OTOH if the JSON library is not too big, comprises a limited nr. of files, and has a GPL compatible license, it can be included in, and shipped with, the io package. JuanPi and I did s/th similar for the clipper polygon clipping library in the geometry package.

Maybe then the existing JSON .m-file function could be replaced by faster compiled ones?

Philip Nienhuis <philipnienhuis>
Group Member
Tue 29 Jan 2019 03:02:20 PM UTC, comment #5: 

atcl: Here's a new package with jsonencode/jsondecode for you: https://github.com/apjanke/octave-jsonstuff.

Andrew Janke <apjanke>
Fri 18 Jan 2019 08:26:41 PM UTC, comment #4: 

Would the io package maintainers be willing to take a dependency on jsoncpp or one of the other mentioned C/C++ JSON libraries? Performance for a native M-code JSON parser is probably going to be grody.

Andrew Janke <apjanke>
Fri 09 Feb 2018 04:09:50 PM UTC, comment #3: 

I would be happy to follow Philip's suggestion and introduce these two functions in to the io package, and then migrate them to core after they have proved stable and useful.

Matlab has a habit of adding functions that persist for only a few releases before being renamed or even removed.  These two have the feel of functions that are going to eventually change.

Rik <rik5>
Group administrator
Fri 09 Feb 2018 09:42:30 AM UTC, comment #2: 

Also have a look at JSONio (C MEX wrapper around jsmn):
  https://www.artefact.tk/software/matlab/jsonio/
and octave-rapidjson (C++ Octave wrapper around rapidjson):
  https://github.com/Andy1978/octave-rapidjson

Guillaume <gyom>
Fri 09 Feb 2018 07:51:47 AM UTC, comment #1: 

I'd rather call it's a feature request.

The io package contains a (contributed) json2object function.
What if people who want to write jsonencode and jsondecode contribute it to the io package (I'll happily accept them) to mature there? Once stable enough they can be moved to core Octave, if the core devs agree of course.
The io package is more regularly updated than Octave releases.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 08 Feb 2018 10:22:09 PM UTC, original submission:  

Matlab introduced the function jsondecode(string) which parses a json formatted string to a cell array of predefined output types based on the json input type. Similarly, jsonencode converts data into a json formatted string. Further details are in the documentation.

An open-source json parser for Octave and MATLAB is here: https://github.com/fangq/jsonlab which may be of help.

CH <atcl>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by juanpi (Posted a comment)
  • -email is unavailable- added by andy1978 (Posted a comment)
  • -email is unavailable- added by fangq (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by siko1056 (interested)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by gyom (Posted a comment)
  • -email is unavailable- added by atcl (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-09-16 siko1056 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2020-09-15 mmuetzel CategoryOctave Package Octave Function
    2020-04-27 mtmiller Dependencies- bugs #58217 is dependent
    2019-02-26 mtmiller Severity3 - Normal 1 - Wish
    2018-02-12 siko1056 Carbon-Copy- Added siko1056
    2018-02-09 rik5 CategoryNone Octave Package
        Item GroupMatlab Compatibility Feature Request
        StatusNone Confirmed
        Summaryjsondecode (and jsonencode) implement jsondecode, jsonencode functions

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code