bugGNU Octave - Bugs: bug #45951, Add "xmlread" to core...

 
 

bug #45951: Add "xmlread" to core Octave or replace the one in io package

Submitter:  Pantxo Diribarne <pantxo>
Submitted:  Sun 13 Sep 2015 08:39:55 AM UTC
   
 
Category:  Octave Package Severity:  1 - Wish
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  philipnienhuis
Originator Name:  Open/Closed:  * Closed
Release:  * other Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 07 Jan 2016 07:57:15 AM UTC, comment #29: 

Great, thanks for your work Philip.

Pantxo Diribarne <pantxo>
Group Member
Wed 06 Jan 2016 05:50:09 PM UTC, comment #28: 

@Panxto:
I took the liberty of uploading io-2.4.0 several weeks ago.
AFAICS Carnë silently released it, it's up on octave.sf.net.packages.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 06 Jan 2016 04:02:20 PM UTC, comment #27: 

@Philip: I used the functions intensively and didn't see any obvious problem. I think they ready for live tests.

Pantxo Diribarne <pantxo>
Group Member
Fri 20 Nov 2015 03:32:30 PM UTC, comment #26: 

Well I am not sure you should halt the release process on those tests which BTW run fine at the moment. Furthermore I will probably get off-line soon (waiting for my first child in 2 weeks :-)) and not be very responsive for a few month (how many???).

Pantxo Diribarne <pantxo>
Group Member
Thu 19 Nov 2015 05:54:16 PM UTC, comment #25: 

Oh, then I wait until after those 3 weeks before making a stable new io pkg release?

What I can do is make an unstable 2.3.0 io release and send it to you and the rest of my (currently 1-person) test panel. Yes?

Philip Nienhuis <philipnienhuis>
Group Member
Wed 18 Nov 2015 08:59:02 PM UTC, comment #24: 

Many thanks Philip. I didn't find any useful BIST either.

I am currently (and for three weeks) using those functions intensively for refreshing an svg drawing, the synopsis of an experiment, with data retrieved from Octave. This will be a kind of test :-)

Pantxo Diribarne <pantxo>
Group Member
Wed 18 Nov 2015 06:12:20 PM UTC, comment #23: 

I've updated xmlread.m / xmlwrite.m in the io package repo.

I found no solution for tests, for the same reason many spreadsheet functions have no test: it is simply not possible to properly cope with absence/presence of various levels of dependencies presence/absence in one and the same test.
Furthermore, the most reliable tests for proper files is to read files made by verified XML writing external SW, likewise for writing.

I'll make a 2.4.0 io release soon.

Closing report.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 12 Nov 2015 10:18:15 PM UTC, comment #22: 

Thanks again for the review.

as to xmlwrite:

  • Yep I'll swap the first two stanzas, thx for noticing.


  • About the return arg: sure you are right but the "some elements in list of return values are undefined" message is useless as well. If a return arg is defined it should have been assigned some value at successful return.


I'll think about some tests (contributions welcome!) and prepare for io-2.4.0 with this new xml functionality.

BTW I had another look at xmltree (see comment #2) and I'm thinking of maybe absorbing (some of) it into io as well.
The Matlab way of XML processing (using xerces) wouldn't be affected as none of xmltree's public functions shadow xmlread/xmlwrite.
Currently just an idea, maybe not a very good thought-out one, but I'm always in favor of supplying some choice to users.
It doesn't look like a big maintenance burden AFAICS

Philip Nienhuis <philipnienhuis>
Group Member
Thu 12 Nov 2015 09:43:46 PM UTC, comment #21: 

@Philip:

>> The numeric argument isn't undocumented, see e.g., "help xmlread". It could be more clear, though.


Sorry I hadn't carefully read the help string :-)

>> Indeed, I could change behavior into persistently checking ... Would that be better? That'd be an easy fix - see attached.


Sure that is exactly what I meant. A user shouldn't have to use debug arguments to force a function to check the path again.

>> Why check for "nargout > 0" in xmlwrite.m when returning the xml string?


You are right, checking the output stream is a StringWriter is enough.
Now about returning an int on success when writing to file, I'd say it is useless as the function will error out otherwise. I'd leave the warning but that is no big deal.

I notice that you have moved base Java support checks a little below the first java calls in xmlwrite :-). Could you move them at the top again? Or I can do it if you want.




Pantxo Diribarne <pantxo>
Group Member
Wed 11 Nov 2015 06:37:11 PM UTC, comment #20: 

@Panxto:
Thanks for testing.

The numeric argument isn't undocumented, see e.g., "help xmlread". It could be more clear, though.

Indeed, I could change behavior into persistently checking during each call for class libs until those requirements have been fulfilled; and echoing version if called with no arguments.
Would that be better?  That'd be an easy fix - see attached.

Experience with the Java-based spreadsheet stuff learned that just echoing some error like "xmlread: couldn't load Xerces parser object" doesn't help users tracking down the cause; I fear we'd only get help requests in the help-octave ML or stackoverflow.  Checking for completeness and proper version is simply required IMO.  All the more as those same class libs are also required for the ODF Toolkit spreadsheet support interface (OTK).

"demo xmlread" &  "demo xmlwrite" work fine for me.

Another thing:
Why check for "nargout > 0" in xmlwrite.m when returning the xml string? I'd say an xml string could safely be returned in "ans". I'd say it's up to the user to decide whether to supply an output arg.
I added a str initialize stmt. at L. 49 + an else clause at the bottom to set str when writing to file, to suppress "warning: xmlwrite: some elements in list of return values are undefined".  I'm unsure as to whether it should be 0 or 1 upon success - what's the convention?

<New xmlread.m/xmlwrite.m attached>

(file #35435, file #35436)

Philip Nienhuis <philipnienhuis>
Group Member
Wed 11 Nov 2015 10:15:29 AM UTC, comment #19: 

@Philip:
I tried to run the demos without adding .jar files to the path and got the expected error message "xmlread: no xercesImpl.jar and/or xml-apis.jar > v2.11.0 in javaclasspath".
Then I added the required .jar files to the javapath and tried again. This time I get "xmlread: required Java support (-files) lacking" because the function doesn't check the path again.

Do we really need this construct with a numeric undocumented argument to reset the "java_ok" variable? What about simply leaving "java_ok" empty and checking again?

Pantxo Diribarne <pantxo>
Group Member
Tue 10 Nov 2015 12:23:56 PM UTC, comment #18: 

Thanks Philip, I'll look into your changes ASAP and report back here.

Pantxo Diribarne <pantxo>
Group Member
Mon 09 Nov 2015 06:43:11 PM UTC, comment #17: 

I've replaced the binary xml stuff from the io package by Panxto's xmlread.m and xmlwrite.m and added a private function for checking xerces jars for those functions.

@Panxto: (and othe interested parties)
you can check out a tarball of upcoming io-2.2.12 (or maybe it will be io-2.4.0; votes?) and see if you like what I did.

I'll leave this bug report open until I have had some feedback.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 26 Oct 2015 09:49:33 PM UTC, comment #16: 

@Panxto:
Sure I'm (very) happy to host them in the io package.

Discussion of the transfer of the spreadsheet stuff to core has stalled a bit, JWE must be too busy with other stuff. No problem, in fact I like it better to be able to steadily clean up the io package in the mean time (I'm busy eliminating a lot of duplicate code). Kicking out the old xml stuff fits well in that respect :-)

Philip Nienhuis <philipnienhuis>
Group Member
Mon 26 Oct 2015 07:17:49 AM UTC, comment #15: 

@Philip: I'd be happy if xmlread/write functions could go first in the IO package. If I understand correctly they would be fitted with the same library checks that are currently used also in spreadsheet functions.
The next step would then be wait for integration of spreadsheet functions to core and let xmlread/write follow the same path.

Pantxo Diribarne <pantxo>
Group Member
Sun 25 Oct 2015 10:29:06 PM UTC, comment #14: 

@Panxto,
sorry I didn't actively follow this bug report - I'll add myself to the cc-list.
In the mean time the io pkg is already 2 minor versions further 8-Z

What I forgot to mention is that (1) we might have to test for xerces & xml-apis being in the javaclasspath, (2) in addition we might need to include them in Windows binary installers.
(1) is implemented in the io package as private function chk_jar_entries.m, I'll see if I can morph it into some useful form for these new function scripts. Or if you prefer to have xmlread.m and xmlwrite.m live in the io package for a while it is even easier.

Normally I have a persistent variable remembering the outcome of all checks, for performance reasons; I'll try to add this to xmlread/-write.m as well.

As to Java exceptions, I know just enough Java to invoke some methods so I can't be of any help.

Ah, and I just found that with xml-apis.jar in the javaclasspath as well (next to xercesImpl.jar), ODF Toolkit works fine (to some extent) with newer xerces releases than 2.9.1 - one less obstacle for the io package.

So far your xmlwrite/-read.m work fine here.

Status => in progress


Philip Nienhuis <philipnienhuis>
Group Member
Fri 09 Oct 2015 11:57:19 PM UTC, comment #13: 

I attached two modified versions where I added a few try/catch around java.

I don't really like the the error messages in case of Java exception. ATM the functions just display the raw "lasterr ()" before throwing an error.
Are there ways to retrieve Java exception messages in a more friendly format than "lasterr ()"?


(file #35130, file #35131)

Pantxo Diribarne <pantxo>
Group Member
Mon 05 Oct 2015 10:48:50 PM UTC, comment #12: 

I attached xmlread.m and xmlwrite.m. There is a demo in each if you want to test and give me feedback.

None of the matlab documented examples work out of the box but it is not a show stopper:

  • xmlread example[1]: dom elements don't have a getMethods method so Octave fails to list their methods using "methods" function. A try/catch can be added and the example works fine.
  • xmlwrite example [2]: ML uses its own implementation of some classe to create a dom document. The workaround is a two liner described in the demo.


[1] http://fr.mathworks.com/help/matlab/ref/xmlread.html
[2] http://fr.mathworks.com/help/matlab/ref/xmlwrite.html

(file #35078, file #35079)

Pantxo Diribarne <pantxo>
Group Member
Tue 15 Sep 2015 09:41:19 PM UTC, comment #11: 

@Panxto:
Is it meant for core Octave or for the io package? If it is for core I'll immediately deprecate io's xmlread/-write/-tree in soon-to-come io-2.2.10. If for io it can replace its old xmlread right away.
I have no preference here. Your pick?

One thing about the object.method syntax you described that seems to bug me is that using xerces no "native" Octave objects are obtained (e.g., structs) but Java objects.
Not such a big deal, but Java objects do have some drawbacks. E.g., they cannot be saved to file (neither can classdef objects). But I suppose xml doc objects are destined to be volatile anyway.
On the ML site there is a code example to convert it to structs but obviously we cannot use that.

Your XML read code might also need a stanza along the lines of:

if (! octave_config_info.features.JAVA)
  error ("xmlread: Octave was built without Java support, exiting");
elseif (! usejava ("jvm"))
  error ("xmlread: no Java JRE or JDK detected, exiting");
endif


+ the try-catch Mike suggested to catch obvious Java problems with some sort of meaningful error message.

The stanza above could be adapted for any future Java-calling function in Octave I guess.


In the mean time I also found this:
https://github.com/sbigaret/Euclide/tree/master/utils/octave/xmltoolbox
After this weekend I'll have some bits of time to explore this one. It is 5 years old but also uses xerces (the C++ version I suppose). Hmmm no license there...

Philip Nienhuis <philipnienhuis>
Group Member
Tue 15 Sep 2015 01:15:23 PM UTC, comment #10: 

I am fine with leaving users the task of adding the relevant .jar files to their Java path.

@Philip: I didn't mention xmlwrite which is also essential I agree: at least I need it for my project (read/modify/write svg files). I will look into all this in the coming weeks.

Pantxo Diribarne <pantxo>
Group Member
Mon 14 Sep 2015 10:26:03 PM UTC, comment #9: 

@Mike:
It would surely be preferrable to just provide the function. I wouldn't be surprised if several Linux distros provide xerces and other Java libs.
Matlab ships a plethora of Java libs, but the associated catch is that they are all somehow interdependent. Replacing one (better/newer/more functionality/...) often breaks things elsewhere (like "dll hell").  Let's not copy this in Octave.

@Panxto:
Your reasoning happens to be the same I have for invoking Java libs for the spreadsheet I/O. With one exception: regular expressions usually outperform XML parsers when data lists (arrays) are to be extracted from (not too complicated) XML. Having a parsed XML-tree in RAM (or rather, the JVM) makes for a large overhead compared to just processing a long string.

Would this xmlread be put into core right away or would you first want to have it live in the io package? I'd love to see xmlwrite as well, in core or io, could be as easy as xmlread.
FYI, io-2.2.10 is very near.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 14 Sep 2015 07:45:23 PM UTC, comment #8: 

That looks good to me. Maybe just add a try-catch for the ClassNotFoundException with a better error message for the user: "xmlread: unable to load Xerces XML parser" or something like that and call it done.

To what extent do you think Octave should be able to provide/detect/find/install the Xerces library? At the bare minimum we just provide this function and it's on the user to set up the Java runtime path to make sure the library can be found, right?

Mike Miller <mtmiller>
Group Member
Mon 14 Sep 2015 07:23:15 PM UTC, comment #7: 


>> Why is classdef required, or so advatageous, for reading XML?


I don't know of another way to have "object.method (args)" syntax (as in ML, see e.g. [1]) in the interpreter ... well except for Java objects.

I am also completely ignorant about xml parsers and Java. Simply saying that wrapping all (very many) methods from libxml2 to obtain an interface that we could have using a few lines of code is a waste of time:


function doc = xmlread (fname)
  parser = javaObject ('org.apache.xerces.parsers.DOMParser');
  parser.parse(filename);
  doc = parser.getDocument;
endfunction


Once we have the path to relevant Xerces jar files, as explained in the wiki, the above is enough to run Matlab examples.

The only problem I had to run examples in [2] is that the returned object has no "getMethods" method so e.g. "methods (doc)" doesn't work in Octave out of the box (I used try/catch).


[1] http://fr.mathworks.com/help/matlab/import_export/importing-xml-documents.html#bsmj8la
[2] http://fr.mathworks.com/help/matlab/ref/xmlread.html

Pantxo Diribarne <pantxo>
Group Member
Mon 14 Sep 2015 05:37:04 PM UTC, comment #6: 

Why is classdef required, or so advatageous, for reading XML?
I'd suppose an XML parser simply spits out some (nested) struct array, or maybe cell array. Once there it is easy to process the data.

For libxml2 I think we need some .oct file as an interface, don't we. That would have to be written largely from scratch. But maybe it isn't that hard, I suppose basic bits and pieces of code can be found on the Web.
The io package would be a good home for a replacement xmlread/xmlwrite until it is mature enough to go to core.

One thing though as regards xerces: in the io package there's an "OTK" (ODF Toolkit) .ods spreadsheet interface that requires xerces-2.9.1 and only exactly that version. That might interfere with newer xerces versions (or rather, vice versa).

Philip Nienhuis <philipnienhuis>
Group Member
Mon 14 Sep 2015 05:16:32 PM UTC, comment #5: 

For my part, I'm speaking from ignorance about the strengths and weaknesses of different xml parsers. Just saying from a packaging and dependency perspective, libxml would be a nice option.

If Xerces does a lot of work for you, by all means go with that solution. We do have Java support in core, I have no problem with functions in core depending on Java libraries if it helps.

Mike Miller <mtmiller>
Group Member
Mon 14 Sep 2015 04:42:49 PM UTC, comment #4: 

@Philip:

  • I had tried XMLTree before noticing the comment in the wiki. It works indeed (eventhough I couldn't have it use compiled versions of some functions).
  • D. Bateman's comment doesn't help much in deciding whether the xmlread.m function in IO is still useful for anyone.


Yes libxml and XMLTree seam to be mature XML parsers. Unfortunately if we wanted to have a ML compatible version of xmlread we would have to reimplement Xerces interface using e.g. classdef syntax in order to have "object.method ()" syntax ... this is clearly a waste of time considering how easy it is to use directly Xerces and have any Matlab code work out of the box (well not completely I admit).

If Java libraries are not welcome in core, what about IO package?

Pantxo Diribarne <pantxo>
Group Member
Mon 14 Sep 2015 02:16:13 PM UTC, comment #3: 

Is libxml2 not good enough? It's a basic but well-used xml parsing C library that is present on pretty much every Linux system these days, part of the LSB, and is already built in the mxe-octave project.

Mike Miller <mtmiller>
Group Member
Sun 13 Sep 2015 03:39:13 PM UTC, comment #2: 

BTW, a GPL-ed and recent (June 2015) XML toolbox is here:

http://www.artefact.tk/software/matlab/xml/

I played a few minutes with it, the demos work fine except for the graphics. There's a mex file included, in the source I read "#if !defined (HAVE_OCTAVE)" and there's more mention of GNU Octave there so that gives some hope .... that:

Maybe this toolbox is a good start.

As to current xmlread, here's a comment by David Bateman:
http://octave.1599824.n4.nabble.com/xmlread-tt4640427.html#a4640428
that explains its -let's say- limited scope.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 13 Sep 2015 02:54:20 PM UTC, comment #1: 

Yes, the io pkg xmlread / xmlwrite / xmltree is completely undocumented. But I tend to keep it around until either
- A good replacemnt comes up, or
- It no longer builds with (then) current Octave.
Maybe some svn archeology or google searches might turn up hints how to get it working but I haven't bothered yet.

Several Octave users seem to have gripes against Java so invoking Java class libs like xerces for core functions may not meet much enthusiasm. (I have no opinion here)
But in the io package things are different :-)

Depending on what you want to achieve, there are some alternatives:

- For small parts of XML files there's getxmlnode.m and getxmlattv.m in the io package. I wrote those to be able to get some special localized data from spreadsheet files (often zipped archives) in small XML support files when speed isn't critical.
Looking at the example you referred to I think a few lines of code invoking them will do as well.

- Reading series of data can be done very fast using regular expressions. Markus Bergholz showed how to do it with .xlsx Excel files, later on I've used it for gnumeric and several xml based data logger files. In io pkg's private directory there are several examples for the OCT interface.
XML gurus may frown here but tests at work show that regexps work 2 to 3 times as fast as when using real XML parsers.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 13 Sep 2015 08:39:55 AM UTC, original submission:  

Hi,

There is currently an "xmlread" function in the IO package that is not really documented and that doesn't seam to work as ML counterpart (actually I don't understand what it is supposed to do). I couldn't find any use of the function in the IO package itself and I doubt that many users will miss the current version.
 
On the other hand according to a comment in the wiki [1], it would be fairly easy to have a ML compliant version : it is just a mater of adding a dependency (in IO or in core) on Xerces java library.

FWIW, I tested a few examples from ML doc, and replacing "xmlread" by the 3 lines mentioned in [1] actually does the job.

[1] http://wiki.octave.org/Cookbook#Load_XML_files

Pantxo Diribarne <pantxo>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #35435:  xmlwrite.m added by philipnienhuis (5KiB - text/x-matlab)
file #35436:  xmlread.m added by philipnienhuis (3KiB - text/x-matlab)
file #35130:  xmlread.m added by pantxo (2KiB - application/vnd.wolfram.mathematica.package)
file #35131:  xmlwrite.m added by pantxo (4KiB - application/vnd.wolfram.mathematica.package)
file #35078:  xmlread.m added by pantxo (2KiB - application/vnd.wolfram.mathematica.package)
file #35079:  xmlwrite.m added by pantxo (3KiB - application/vnd.wolfram.mathematica.package)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by pantxo (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 15 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-11-18 philipnienhuis CategoryOctave Function Octave Package
        StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2015-11-12 philipnienhuis StatusReady For Test In Progress
    2015-11-11 philipnienhuis Attached File- Added xmlwrite.m, #35435
        Attached File- Added xmlread.m, #35436
    2015-11-09 philipnienhuis StatusIn Progress Ready For Test
        Assigned toNone philipnienhuis
        Release4.0.0 other
    2015-10-25 philipnienhuis StatusNone In Progress
        Carbon-Copy- Added philipnienhuis
    2015-10-09 pantxo Attached File- Added xmlread.m, #35130
        Attached File- Added xmlwrite.m, #35131
    2015-10-05 pantxo Attached File- Added xmlread.m, #35078
        Attached File- Added xmlwrite.m, #35079

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code