bugGNU Octave - Bugs: bug #59245, Implementation of an XML interface...

 
 

bug #59245: Implementation of an XML interface without Java dependency

Submitted by:  Markus Mützel <mmuetzel>
Submitted on:  Fri 09 Oct 2020 07:22:10 PM UTC  
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Mon 12 Oct 2020 04:50:07 PM UTC, comment #12: 

Sorry if I'm stating the obvious but the point I was trying to make is that xmlread/xmlwrite can be used to implement readstruct/writestruct but not the other way round. OOXML files probably require the level of detail that xmlread/xmlwrite but I would guess these functions would be too slow to use when manipulating the output object.
I used yxml (https://dev.yorhel.nl/yxml) to parse XML and I liked it - it's not as widespread as RapidXML or libxml2 but it's easy to use and I can see it being used in Fuchsia to parse SVG files so there are credentials.

Guillaume <gyom>
Mon 12 Oct 2020 09:44:02 AM UTC, comment #11: 

About error messages:
Excel may show file names and line numbers [*], but that is of little use in a long XML string of several tens-of-thousands or more characters - it'll be a long search.
LibreOffice is so nice to also tell you at what character position it encountered a problem.

[*] Excel 2013 and 2016 only tell me something along the lines of "We found a problem in <filename>. Delete content"?
Furthermore, are line numbers really useful? OOXML files usually consist of just two lines: Line 1 is the XML version header, line 2 contains all the actual XML content.

Philip Nienhuis <philipnienhuis>
Project Member
Mon 12 Oct 2020 09:33:32 AM UTC, comment #10: 

RapidXML's I/O? => that was mentioned in the decision tree you referred to in comment #5. Go to that site, scroll down to "Maximum XML Parsing Performance" and then scroll a little more down under the RapidXL header to read this;

> One other downside with RapidXML is that it is painful for writing XML. It requires you to do a lot of explicit memory allocation of string names in order to build its DOM. It does provide a kind of string buffer, but that still requires a lot of explicit work on your end. It's certainly functional, but it's a pain to use.


But if you've already sorted it out this is obviously moot.

Philip Nienhuis <philipnienhuis>
Project Member
Mon 12 Oct 2020 07:22:43 AM UTC, comment #9: 

@Philip: Thank you very much for your feedback.
Could you please elaborate on what you think is bad about how RapidXML implements IO?
It takes a pointer to an array for input and writes to any string iterator. Imho, it is a plus point that it doesn't try to hide file IO in its API because that is bound to cause issues on some platforms (Windows, I am looking at you!). We have several wrappers in Octave already that handle the platform specific peculiarities...

Excel's error messages (including file names and line numbers) proved to be quite useful to identify the issue...

AFAICT, there is no XML standard that dictates or suggests order of sibling nodes. That must be OOXML specific.

Markus Mützel <mmuetzel>
Project Member
Sat 10 Oct 2020 09:38:45 PM UTC, comment #8: 

@Markus
I welcome your attempt very much.
All I wanted to convey in comment #3 is that IMO a project comprising implementing XML I/O with an underlying aim of providing spreadsheet I/O needs very careful thinking over.

RapidXML seems a pain for I/O; for me that would be enough to just skip it at first sight. But maybe some clever extension or wrapper can alleviate the writing complications.

A hint:
Excel may complain about invalid OOXML but only offers to wipe the offending content w/o further detail.
To get the exact error location you need LibreOffice. That tells you exactly at what character position in which XML file it lost track.

It is reassuring to read that Excel is picky about ordering of tags. Reassuring because such order is the major weakness of parsing XML with regular expressions.
Do XML standards dictate or suggest a default tag ordering, e.g., alphabetical or so? AFAICS OOXML has a fairly alphabetical tag ordering (but not consistently, as nothing from Microsoft is consistent).

Philip Nienhuis <philipnienhuis>
Project Member
Sat 10 Oct 2020 03:45:21 PM UTC, comment #7: 

I tried to read the xml of an .xlsx file with `readstruct` and write it again (without changing anything) with `writestruct`. There were issues writing the header (which are easy enough to work around).
I then zipped everything pack to an .xlsx file.
If I try to open the resulting file, Excel complains about invalid syntax.
The main issue seems to be that the order of tags seems to be important for OOXML. E.g. it looks like v-tags must not appear before f-tags. If I manually edit the xml to correct the order of these tags, Excel seems to be able to open the resulting file.

It looks like `readstruct` successively adds fields to the struct arrays while iterating through the DOM nodes. That means that the order of fields can differ from the one in the file if some tags don't exist in earlier nodes (see the attached screenshot).

This is probably what you were hinting at in comment #6.

I'm still debating how to proceed.
Maybe I shouldn't give up on xmlread/xmlwrite so quickly.
One possible strategy might be to rename them to oxmlread/oxmlwrite (to avoid having them shadowed by the functions in the io package which will always be more Matlab compatible). That way we could leave them deliberately incompatible where we don't need full compatibility.
We could also make them internal functions for the (not yet existing) readcell/writecell. But I guess they could come handy for end users, too.

Also, implementing readstruct/writestruct might still be worthwhile even if it won't be possible to use those functions for reading and writing OOXML...

I'll have to think about that. Input very welcome.


Markus Mützel <mmuetzel>
Project Member
Sat 10 Oct 2020 01:49:32 PM UTC, comment #6: 

readstruct/writestruct are targetting very specific types of XML files, e.g. it starts to get ugly with this:

<a e="f"><b>c</b>d<Text>g</Text><b>3</b></a>

>> readstruct('test.xml')
ans =
  struct with fields:
    eAttribute: "f"
             b: ["c"    "3"]
          Text: "d"
        Text_1: "g"

xmlread/xmlwrite always felt more Java than Matlab but you can access appropriately any content of an XML file. In the example above, you can see that the data order is lost with "3" appearing before "g" in the struct, so you wouldn't be able to use readstruct/writestruct if you wanted to parse an XHTML file. I guess it would be a problem with OOXML too.
I can see why they added the new functions though as that's what many people want (it is equivalent to the convert() method of the xmltree class I wrote a long long time ago https://www.artefact.tk/software/matlab/xml/).

Guillaume <gyom>
Sat 10 Oct 2020 11:49:37 AM UTC, comment #5: 

@Guillaume: I wasn't aware of these functions. Their interface looks much cleaner and "more native" than xmlread/xmlwrite.
For a test, I unzipped a random .xlsx file I had lying around and tried to read one of the worksheet .xml files in Matlab R2020b. It was successfully read into a (nested) structure that resembles the DOM tree of that file.
I wonder if working with these structures would cost performance as opposed to accessing the XML DOM tree more directly. But afaict, it might be possible to use these functions as intermediate layers for reading/writing .xlsx files.
Given that, I'll drop trying to implement xmlread/xmlwrite in favor of readstruct/writestruct.
I just wish I knew about these functions earlier...

@Philip: There are also readcell/writecell which work with cell arrays instead of tables. We might be able to implement those already. That could be done either in Octave language using readstruct/writestruct (if the performance penalty isn't too bad) or by accessing the XML DOM tree directly in C++. (Or maybe first implement it in Octave language to get the hierarchy sorted out and then translate that to C++ if necessary.)

AFAICT, the only XML "library" that core Octave is using already is QtXml. But Qt is an optional dependency (for the GUI and graphics toolkit). Additionally, QtXml isn't even mentioned as a contender in comparisons I found...
There might be different reasons why other software choses to use different XML libraries. But RapidXML seems to outperform libxml in comparisons (for XML it is able to read at least):
https://pugixml.org/benchmark.html

A nice decision tree with different XML parsers is e.g. here:
https://stackoverflow.com/questions/9387610/what-xml-parser-should-i-use-in-c

Also RapidXML seems to be stable and equally available on many distros.

I'd prefer to implement these features in incremental steps. Imho, we don't need to support .xlsx, .ods and .xls right from the start. Just one of them (.xlsx) would be better than none. If users want to read/write other formats, they will still be able to use the functions in the io package...

Being able to read "plain" XML would be a good initial step imho.

Markus Mützel <mmuetzel>
Project Member
Sat 10 Oct 2020 10:09:42 AM UTC, comment #4: 

@Philip: you are right, I was distracted by the title of this report. If the aim was to improve XML support in Octave, I thought it might be worth focussing on readstruct/writestruct but this will not really help if the aim is to build the tools in order to implement xlsread/xlswrite. Given the rather generic naming of readstruct/writestruct, I wonder if they will go beyond XML in future releases.

Guillaume <gyom>
Fri 09 Oct 2020 11:07:57 PM UTC, comment #3: 

Some remarks, in no particular order:

In the mxe-octave installers there's libxml2 (and libxml++). Can that also be used? I suppose it's also in many Linux distros so if yes it may be a better library to use.

xmlread and xmlwrite are Java based and as such highly Matlab compatible.  I'd suppose many use cases and Matlab user code Out There are also Java based; could that compatibility be partly or wholly lost when switching to non-Java based parsers and other XML functions?

I'd support getting some form of xlsread and xlswrite into core Octave. Years ago JWE wrote that he also wanted it.
If this is done IMO it should read .xlsx, .ods and if possible, .xls (BIFF8 and/or BIFF5) as many embedded systems spew out .xls. Gnumeric is a very easy format so could also be included.

But writing .xlsx is a daunting task. The ECMA specs for the OOXML specs comprise thick tomes and to further put you off I'd suggest to read e.g., this:
http://ooxmlisdefectivebydesign.blogspot.com/2007/08/microsoft-office-xml-formats-defective.html   (hmm yet I found many useful hints in there :-) )
That said, .ods is even more complicated to handle with many implicit rules and no address in spreadsheet cell nodes.
In the io package OOXML and ODS formats have already been sorted out (many corner cases excluded) so that could serve for a start.

Matlab these days has readtable and writetable that can read from and write to spreadsheets. Their interface / calling form looks much more practical than the xlsread/xlswrite requirements of fixed but inconsistent argument order.
To implement readtable and writetable Octave needs the Table class. Andrew Janke has made Tablicious and on the task tracker there's a task for Table I/O based on the io package (task #15419)

@gyom:
readstruct doesn't look like it can read a spreadsheet file, can it?

Philip Nienhuis <philipnienhuis>
Project Member
Fri 09 Oct 2020 08:50:10 PM UTC, comment #2: 

Have you noticed that there are two new functions in Matlab R2020b, readstruct and writestruct:
https://www.mathworks.com/help/matlab/ref/readstruct.html
https://www.mathworks.com/help/matlab/ref/writestruct.html
They are probably more convenient than xmlread/xmlread for most use cases.

Guillaume <gyom>
Fri 09 Oct 2020 07:25:28 PM UTC, comment #1: 

Attached the patch.

I only tested it with toy examples. A better test would be to try if they work for reading and writing .xlsx files.
Is that possible at all or are there missing features?
How do the functions perform?

(file #49955)

Markus Mützel <mmuetzel>
Project Member
Fri 09 Oct 2020 07:22:10 PM UTC, original submission:  

The io package provides the functions `xmlread` and `xmlwrite` that are basically wrappers for the Xerces Java jar files.
I'm not sure if this (Java) is the reason why they aren't moved to core Octave. But Philip hinted at this in the past.

My "real" goal here is to get `xlsread` and `xlswrite` into core Octave.
Those function are also provided by the io package. They are implemented using several different interfaces. But those are either slow (the OCT interface), rely on proprietary software (the COM interface) or need Java (the remainder?).
IIUC, the most likely to ever be implemented in core Octave is the OCT interface. Unfortunately, it is quite slow for reasonably large .xlsx files. This is probably because it uses regular expressions that are getting more and more complex to match the XML of .xlsx files.
My hopes are that using an XML parser will be easier and faster than matching the XML with regular expressions.

There are several free C/C++ XML parsers. To be able to read and write the XML of .xlsx files, we won't need to have full XML compliance.
Others [1] are using RapidXML [2] for that task which is supposedly very fast and has a simple but adequate interface.

So, I gave it a shot and wrote a simple interface from Octave to RapidXML which is more or less compatible to Matlab.
I'll upload a work-in-progress patch as soon as I have a bug number.

CC'ing Philip as the maintainer of the io package.

[1]: https://readxl.tidyverse.org/
[2]: http://rapidxml.sourceforge.net/manual.html

Markus Mützel <mmuetzel>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49955:  bug59245_xmlread_xmlwrite.patch added by mmuetzel (55KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gyom (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  • -email is unavailable- added by mmuetzel
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-10-10 mmuetzel Attached File- => Added readstruct_fieldorder.PNG, #49958
    2020-10-10 mmuetzel SummaryImplementation of xmlread/xmlwrite without Java dependency => Implementation of an XML interface without Java dependency
    2020-10-09 mmuetzel Attached File- => Added bug59245_xmlread_xmlwrite.patch, #49955
    2020-10-09 mmuetzel Carbon-Copy- => Added philipnienhuis

    Back to the top


    Powered by Savane 3.5