bugGNU Octave - Bugs: bug #66236, [octave forge](io) Regex in...

 
 

bug #66236: [octave forge](io) Regex in __OCT_xlsx2oct__.m fails on 'Style' tag

Submitter:  None
Submitted:  Fri 20 Sep 2024 10:08:46 AM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  In Progress Assigned to:  philipnienhuis
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * other
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 13 Oct 2024 07:10:35 PM UTC, comment #12: 

My email address used for Octave development is in the copyright string on top of most of the io package files. Sourceforge automatically puts it through to my real (hobby) email address.

Thank you, today or tomorrow I'll push the fix under your name.

I can't vouch that the additional regression testing is completely reliable but it'll suffice. For now.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 13 Oct 2024 01:12:00 PM UTC, comment #11: 

@philipnienhuis
Seems a non-public reply on 2024-10-02 failed to reach you. :)

In case you seem it fit to attribute credits for the changes:
Marc Becker <marc.becker@astos.de>


Thanks (again) for the swift replies and additional regression testing.

Marc Becker <becm>
Sat 12 Oct 2024 07:49:56 PM UTC, comment #10: 

@becm:
Ping!

I've tried your regexp on a fair number of bug report spreadsheets and some of my own work collection and I see no regressions so I'm ready to push the fix.
But:
Would you like to be credited? in that case only your name and email address is required

Philip Nienhuis <philipnienhuis>
Group Member
Tue 01 Oct 2024 05:30:05 PM UTC, comment #9: 

@becm:
Thanks very much, your amended regexp works, at least on the files you uploaded.
I collected quite a few of .xlsx files from various bug reports and other test cases over the years, I'll try some of those as well and if I see no issues I'm ready to push the fix.

Would you like to be credited? just your name + email address are needed for that, they'll appear in the logs in the io repo.

Your remarks about stray 't' chars make me wary that other regexps in the OCT interface code may use some attention as well. But until now the code (written originally in -I believe- 2009 or 2010) has been working fine. I lack time to hunt it all down all potential weaknesses, other priorities if no real issues pop up.
If anything must be overhauled for spreadsheet I/O, I think we'd better switch to a reliable XML parser. But one of the core devs gave it a try a few years ago with libxml2 and IIUC he gave up as it got too complicated and took too much time.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 30 Sep 2024 08:31:30 PM UTC, comment #8: 

If the non-greedy expression is not too much of a performance hit
it may be reasonable to make further changes (see `oct_regex.m`).

This will still not catch all the possible silliness, but at least:
- totally different attributes names just ending in 't'
- any stray 't' anywhere before the actual attribute.

Problem: Attribute values have less strict rules than XML data:
https://www.w3.org/TR/xml/#NT-AttValue

Only reliably excluded character seems to boil down to '<':
- still accepted as reference indicator: &
- inconclusive/alternate start and end: ' "

(file #56468)

Marc Becker <becm>
Mon 30 Sep 2024 10:10:08 AM UTC, comment #7: 

Current patch suggestion would be

@@ -96,7 +96,7 @@

   ## Get cell addresses and contents of t=".." tags, "<f>", "<v>" and "<is>"
   ## nodes, optionally just placeholders
-  val = regexp (rawdata, '<c r="(\w+)"(?:[^t>]*(>)|(?:[^t>]*t="(\w+)")>)(?:<f.+?(?:</f>|/>))?(?:<v(?:[^>]*)>([^<]*)</v>|<is><t>([^<]*)</t></is>|<v/>([^<]*))', "tokens");
+  val = regexp (rawdata, '<c r="(\w+)"(?:[^t>]*(>)|(?:[^t>]*t="(\w+)")[^>]*>)(?:<f.+?(?:</f>|/>))?(?:<v(?:[^>]*)>([^<]*)</v>|<is><t>([^<]*)</t></is>|<v/>([^<]*))', "tokens");
   ## <inlineStr>: <is><t>(.*?)</t></is>  ## formula: <f>..</f>
   if (any (cellfun (@length, val) != 3))
     warning ("xls2oct: error reading data from sheet %d", wsh);


(i.e., shifted 1 pos to the right to outside the 't=' attribute :-) ) - with that change it works.
AFAICS (it's been a long time since I last touched this part of the code) catching optional attributes incl. 's=' between 'r=' and 't=' attributes was already catched (see below in my comment #4 for what I meant). So I guess the above patch (BTW similar to the existing preceding clauses) would fix it.

I'll wait a few days and if no more comment come in I'll push the fix.


Philip Nienhuis <philipnienhuis>
Group Member
Sun 29 Sep 2024 05:11:11 PM UTC, comment #6: 

Thanks becm for your feedback.

You're right with the '[^>]*' part, have you tried yourself?

AFAIK whitespace shouldn't matter unless between "". IIRC all regexps should be able to ignore themwhitespace between attributes. That said I've never met them in the wild. The OCT interface has been existing for some 15 years now and never had such issues.

'>' (and 4 other) characters are illegal anyway in XML.

As to malformed XML in general, I think trusting all SW out there writing spreadsheet XML to behave well, also has advantages: catering for and gracefully catching XML errors only takes expensive time, and reading spreadsheets this way is slow enough already.

If you care you can upload a better patch here ;-)

Philip Nienhuis <philipnienhuis>
Group Member
Sat 28 Sep 2024 08:43:09 PM UTC, comment #5: 

This works at least for well-behaving input,
but going the secure way, I'd still rather opt for '[^>]*'
instead of the much broader (and overly-optional)  '.*?'

No reason to start accepting malformed XML now/yet and match
'<c r="A1" t="n" s="0">><v>5</v></c>' :)


There may still be the whole issue with whitespace between tags.
Or likely show strange effects when attribute values have '>'.
'<c r="A1" t="n" s="0>"> <v>5</v> </c>'

But this already breaks and the suggested change should be
sufficient to tackle most regular trailing attributes.


--
original submitter, created account for proper interaction

Marc Becker <becm>
Fri 27 Sep 2024 10:35:53 AM UTC, comment #4: 

Try with attached __OCT_xlsx2oct__.m and report back, please.

To replace it, find where io package lives:
(in Octave)
cd (strrep (which ("oct2xls"), "oct2xls.m", "private"))

I just applied the following patch:

@@ -96,7 +96,7 @@

   ## Get cell addresses and contents of t=".." tags, "<f>", "<v>" and "<is>"
   ## nodes, optionally just placeholders
-  val = regexp (rawdata, '<c r="(\w+)"(?:[^t>]*(>)|(?:[^t>]*t="(\w+)")>)(?:<f.+?(?:</f>|/>))?(?:<v(?:[^>]*)>([^<]*)</v>|<is><t>([^<]*)</t></is>|<v/>([^<]*))', "tokens");
+  val = regexp (rawdata, '<c r="(\w+)"(?:[^t>]*(>)|(?:[^t>]*t="(\w+)".*?)>)(?:<f.+?(?:</f>|/>))?(?:<v(?:[^>]*)>([^<]*)</v>|<is><t>([^<]*)</t></is>|<v/>([^<]*))', "tokens");
   ## <inlineStr>: <is><t>(.*?)</t></is>  ## formula: <f>..</f>
   if (any (cellfun (@length, val) != 3))
     warning ("xls2oct: error reading data from sheet %d", wsh);

(i.e., just added '.*?' w/o quotes to the regexp)
but I'm not sure if this is the right thing to do.
The issue is that first of all, regexps aren't quite made for reading XML; up till now the OCT interface scripts relied on (nowhere documented but seemingly generally accepted) alphabetical order of XML node tags. The Apache POI folks apparently chose to break this habit with adding this "s=" tag; anytime they may "fix" that.
Maybe an additional "s=" clause has to inserted before the "t=" (type) tag, who knows.

So maybe in due time I'll ask again to try with another __OCT_xlsx2oct__.m version :-)

(file #56459)

Philip Nienhuis <philipnienhuis>
Group Member
Tue 24 Sep 2024 06:40:44 PM UTC, comment #3: 

Thanks for uploading some files, I'll have a look.

BTW a comment with message that you did upload them would have been handy, I wouldn't have noticed were it not that I perused the bug tracker archive mailing list.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 23 Sep 2024 02:37:14 PM UTC, comment #2: 

(io maintainer here)

Yeah Apache Poi makes for many problems these days. I stopped updating the spreadsheet I/O functions for POI since a few years as with every new POI release new methods were invented and other required methods shifted around over several .jar files. It is a big PITA to track which .jars are needed for every new POI release.

Anyway, I second Andy's request:
Please put up a test case here. Reading your report, a sample .xlsx file created by Apache POI may suffice.

("Release" and "Status" fields updated)

Philip Nienhuis <philipnienhuis>
Group Member
Sat 21 Sep 2024 07:02:51 AM UTC, comment #1: 

Thank you for your report. Can you please add a testcase which shows the problem? For example create a simple xlsx and an .m script which tries to read it and fails. (And then attach both to the bugreport)

Thank you, Andy

Andreas Weber <andy1978>
Group Member
Fri 20 Sep 2024 10:08:46 AM UTC, original submission:  

Apache POI started to add a Style tag to cells: '<c ... s="...">'

As a result, cell content is not identified as such (empty sheet).

This now constitutes a "in the wild" issue as mentioned in
respective comment for regex call (for other tags).

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #56468:  oct_regex.m added by becm (860B - text/x-objcsrc)
file #56445:  POI.xlsx added by None (3KiB - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet - Test cases for reading POI Excel file and MS-modified version)
file #56446:  POI.m added by None (50B - application/octet-stream - Test cases for reading POI Excel file and MS-modified version)
file #56447:  POI_ms.xlsx added by None (8KiB - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet - Test cases for reading POI Excel file and MS-modified version)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by becm (Posted a comment)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by andy1978 (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-09-30 becm Attached File- Added oct_regex.m, #56468
    2024-09-27 philipnienhuis Attached File- Added _OCT_xlsx2oct_.m, #56459
        StatusNeed Info In Progress
    2024-09-24 philipnienhuis Carbon-Copy- Added philipnienhuis
    2024-09-24 philipnienhuis Assigned toNone philipnienhuis
    2024-09-23 None Attached File- Added POI.xlsx, #56445
        Attached File- Added POI.m, #56446
        Attached File- Added POI_ms.xlsx, #56447
    2024-09-23 philipnienhuis StatusNone Need Info
        Release9.2.0 other

    Back to the top

    Powered by Savane 3.13-5884.
    Corresponding source code