bugGNU Octave - Bugs: bug #59273, [octave forge] (io) can't reliably...

 
 

bug #59273: [octave forge] (io) can't reliably load XLSX files in release 2.6.2

Submitted by:  Sébastien Villemot <svillemot>
Submitted on:  Thu 15 Oct 2020 09:37:52 AM UTC  
 
Category:  Octave Forge Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  In Progress Assigned to:  Philip Nienhuis <philipnienhuis>
Originator Name:  Open/Closed:  Open
Release:  other Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sat 17 Oct 2020 08:13:46 PM UTC, comment #10: 

Sebastien,

Attached are two new versions of xlsopen.m and private/getinterface.m, could you please try these?
xlsopen.m echoes some debug info to the screen, notably about inferred file type and the supported interfaces; I'd be interested in that.

I can't reproduce nor figure out why your test code would work in io-2.6.1 but not in 2.6.2. I tracked back the logs of the io package but AFAICS both files haven't seen relevant changes between 2.6.1 and 2.6.2. The only difference I can think of is that for io-2.6.1, program flow in xlsopen.m fell trough to using OCT interface behind your back, but even that is mere speculation.
A candidate patch could be:
http://hg.code.sf.net/p/octave/io/rev/91353b9092d8
before which the OCT interface was always active even if the user didn't select it.

Also enigmatic is why .xlsx didn't work with any interface but .xls did with the patched xlsopen.m you tried before.
One can try interfaces by hand using the io_testscript function:
io_testscript ("<interface>", "io-test.xlsx")

There was a bug when xlsopen.m was called for the first time, requested interfaces were wiped, which might explain the erratic behavior I saw. But your testsuite doesn't specify any interface argument so this looks irrelevant.

(file #50003, file #50004)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Fri 16 Oct 2020 09:13:55 PM UTC, comment #9: 

ATM most of this bug has been sorted out. Over in bug #59277 a related issue was reported.

There are actually 2 interfering issues:

  1. User selection of an interface is ignored the first time a spreadsheet is opened after loading the io package;
  2. Detection of POI OOXML support can be incomplete;

and a related, more subdued issue:

  • Fiddling with interface support SW by a user (adding/removing jars to the javaclasspath, unloading windows package) while the io package is loaded might not always be adequately detected, let alone catched.

One may wonder if the latter should be detected & catched in the first place (isn't that user responsibility?) but I did implement it to cut down on bug reports (and bad Octave reputation). But it's usually hard to make such code sufficiently robust, let alone avoid performance degradation.

All of these three issues have to be considered for a fix.

Hopefully tomorrow I'll come up with replacement functions that I hope you can try for me.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 15 Oct 2020 06:22:23 PM UTC, comment #8: 

Thanks, so the Debian testsuite does about the same as test-spsh.m - (for those unfamiliar with io package innards:) the latter finds supported interfaces and tests them one by one.

Anyway, see also bug #59277.

(To be continued tomorrow night)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 15 Oct 2020 05:28:45 PM UTC, comment #7: 

In Debian we have a testsuite that checks all interfaces (by enforcing them at test time), and all interfaces used to work well in 2.6.1, see:
https://ci.debian.net/data/autopkgtest/testing/amd64/o/octave-io/7420457/log.gz

This testsuite is run with the Java libraries installed for JXL and POI+OOXML.
The source of the testsuite can be seen at:
https://sources.debian.org/src/octave-io/2.6.1-3/debian/tests/

Those very same tests that used to work with 2.6.1 now fail with 2.6.2 (without your patch). All tested XLS and XLSX interfaces fail (XLS/JXL, XLS/POI, XLSX/OCT, XLSX/POI). The environment has not changed, in particular the same Java libraries are present.

I’ve applied your patches and they fix the testsuite for XLS loading (i.e. XLS/JXL and XLS/POI now work). However it still fails for XLSX/OCT and XLSX/POI.

So you clearly fixed an issue, but there is still something going on.

Sébastien Villemot <svillemot>
Thu 15 Oct 2020 05:26:44 PM UTC, comment #6: 

(Our posts crossed)

Thanks for the javaclasspath echo, that looks good enough. I'm at POI-4.0.0; FTR it turns out that some POI classes of versions 4.0 and 3.15 can be mixed w/o loss of functionality of the io package:

   DYNAMIC JAVA PATH

      C:\Users\philip\Java\poi-ooxml-schemas-4.0.0.jar
      C:\Users\philip\Java\jOpenDocument-1.4rc2.jar
      C:\Users\philip\Java\xml-apis.jar
      C:\Users\philip\Java\xercesImpl-2.11.0.jar
      C:\Users\philip\Java\gwt-servlet-deps.jar
      C:\Users\philip\Java\jxl.jar
      C:\Users\philip\Java\backup\commons-compress-1.18.jar
      C:\Users\philip\Java\commons-collections4-4.2.jar
      C:\Users\philip\Java\dom4j-1.6.1.jar
      C:\Users\philip\Java\xmlbeans-3.0.1.jar
      C:\Users\philip\Java\backup\poi-ooxml-3.15.jar
      C:\Users\philip\Java\backup\poi-3.15.jar

I'd be curious how your original example works in io-2.6.2 with the patched xlsopen/getinterfaces functions of comment #5 when chk_spreadsheet_support.m's output equals 74.

No doubt you know io package's innards :-)
Thing is, I cannot reliably reproduce your issues here, that's why I'm asking you for several trials that perhaps look quite useless for you.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 15 Oct 2020 05:10:50 PM UTC, comment #5: 

Looks like you've hit a bug that has been looming for quite some time.

It boils down to always trying to invoke POI for .xls and .xlsx even if only partial POI support (just for .xls) was present. Turns out I fixed this already in 2017 but after last year's interfaces reshuffling for io-2.6.x that fix got lost.

The other, subordinate issue is L.131 in xlopen.m,
      reqinterface = [];
which needs unconditionally changing to
      reqinterface = "";

... and maybe it shouldn't be there in that form at all.

As a first step, would you please try again with attached xlsopen.m and getinterfaces.m ? (the latter to be put in private/ subdir)

Next up = finding out what else you hit when you did have full POI support. If still needed.

Taking a step backward for some overview it would be better to rethink the "verbose" option. You say that it worked in io-2.6.1 (and before?) but you -unknowingly- may have actually invoked OCT rather than POI&OOXML. xlsopen.m as it stands doesn't give you any clue.

(file #49985, file #49986)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 15 Oct 2020 04:04:42 PM UTC, comment #4: 

commentaire #3 :

> chk_spreadsheet_support.m's return value of 78 is what you'd get with JXL (8), full POI (2 + 4) and JOD (64) support (+ OCT) and 74 is the same with basic POI support(just .xls).
> 4 is the added value for POI-OOXML; see "help chk_spreadsheet_support".


Note that I’m well familiar with all that stuff (I’ve been packaging io for Debian since 2012).

> Again, what does:
>
> javaclasspath
>
> return?


Here it is:

octave:2> chk_spreadsheet_support("/usr/share/java", [])
ans =  78
octave:3> javaclasspath
   STATIC JAVA PATH

      - empty -

   DYNAMIC JAVA PATH

      /usr/share/java/poi-4.0.1.jar
      /usr/share/java/poi-ooxml-4.0.1.jar
      /usr/share/java/xmlbeans-3.0.2.jar
      /usr/share/java/poi-ooxml-schemas-4.0.1.jar
      /usr/share/java/dom4j-2.1.1.jar
      /usr/share/java/commons-collections4.jar
      /usr/share/java/commons-compress.jar
      /usr/share/java/jxl-2.6.12.jar
      /usr/share/java/xercesImpl-2.12.0.jar
      /usr/share/java/xml-apis-1.3.04.jar
      /usr/share/java/jOpenDocument-1.3-1.3repack.jar

Sébastien Villemot <svillemot>
Thu 15 Oct 2020 03:58:32 PM UTC, comment #3: 

chk_spreadsheet_support.m's return value of 78 is what you'd get with JXL (8), full POI (2 + 4) and JOD (64) support (+ OCT) and 74 is the same with basic POI support(just .xls).
4 is the added value for POI-OOXML; see "help chk_spreadsheet_support".

Again, what does:

javaclasspath

return?

(just to get that bit out of the way. For me as maintainer and potential bug-solver that does matter. That's not meant to challenge your POV; probably like you, I think the actual bug here lies elsewhere.)

I'm currently investigating why xlsopen didn't fall back to OCT as you didn't select an explicit interface and from the messages in your comment #0, OCT should have been the only active interface that supports .xlsx.

Thanks.

<title adapted>

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 15 Oct 2020 12:36:34 PM UTC, comment #2: 

commentaire #1 :

> In your post I do not see "POI (& OOXML)" in xlsopen.m's output but just "POI". That implies that only .xls support for POI is active, not .xlsx. What does your javaclasspath look like after the call to chk_spreadheet_support? BTW the ouput of chk_spreadsheet_support is a bit mask indicating which interfaces have been successfully checked. 74 as in your post hints at incomplete POI initialization.


For my Debian packaging work, I use several containers, which don’t necessarily have the same set of Java libraries installed. Hence the discrepancy between the body of my post (bitmask 74) and the log file I attached (bitmask 78). The problem occurs in both cases.

> On my box I see Java complaints during POI initialization about a "reflective error" or so (bug #58266) and the threat to disable that in future Java implementations. Do you have newer Java where they actually did so?


I don’t see that error message. All my tests were performed using Debian unstable, which defaults to OpenJDK 11.

> In the entries for OCT there's a "subscripts must be either integers 1 to (2^63)-1 or logicals" error. That leads me to the following:
> Can you try replacing L.131 in xlopen.m,
>       reqinterface = [];
> by
>       reqinterface = "";
> try again and and report back?


I applied this changes and the problem persists (I attach the log, the summary table at the end is the same as the previous one, but the error messages slightly differ, as you can see with a text diff tool).

(file #49982)

Sébastien Villemot <svillemot>
Thu 15 Oct 2020 12:16:23 PM UTC, comment #1: 

Thanks for the report, we already had some limited private discussion.

Afer installing io using -forge, I could reproduce the bug on Windows, but erratically.
On my Linux system (Mageia 7) I can not, it all works fine there.

You've attached the output of "test_spsh.m", one of the dedicated spreadsheet I/O test scripts, thanks.
That script also shows that operation is erratic. JXL and POI don't work, OCT partly and JOD works fine.

In your post I do not see "POI (& OOXML)" in xlsopen.m's output but just "POI". That implies that only .xls support for POI is active, not .xlsx. What does your javaclasspath look like after the call to chk_spreadheet_support? BTW the ouput of chk_spreadsheet_support is a bit mask indicating which interfaces have been successfully checked. 74 as in your post hints at incomplete POI initialization.

On my box I see Java complaints during POI initialization about a "reflective error" or so (bug #58266) and the threat to disable that in future Java implementations. Do you have newer Java where they actually did so?

In the entries for OCT there's a "subscripts must be either integers 1 to (2^63)-1 or logicals" error. That leads me to the following:
Can you try replacing L.131 in xlopen.m,
      reqinterface = [];
by
      reqinterface = "";
try again and and report back?

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 15 Oct 2020 09:37:52 AM UTC, original submission:  

Hi,

After a call to chk_spreadsheet_support, loading XLSX files no longer work in 2.6.2. Here is an example session:

octave:1> pkg load io
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "org.octave.OctClassLoader"). To use archiv
ed non-system classes, this property must be not be set
octave:2> chk_spreadsheet_support("/usr/share/java", [])
ans =  74
octave:3> cd ~/debian/octave-io/debian/tests/
octave:4> xlsfinfo('test.xlsx', [], true)
Detected interfaces: POI; JXL; JOD;
None.
warning: xlsopen: no'.xlsx' spreadsheet I/O support with available interfaces.
octave:5>

Note that the problem did not occur with 2.6.1.

Also note that XLSX loading works fine if chk_spreadsheet_support is not called.

I also attach a diary file showing the output of test_spsh.

Sébastien Villemot <svillemot>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50003:  xlsopen.m added by philipnienhuis (17KiB - application/octet-stream)
file #50004:  getinterfaces.m added by philipnienhuis (8KiB - application/octet-stream)
file #49985:  xlsopen.m added by philipnienhuis (17KiB - application/octet-stream)
file #49986:  getinterfaces.m added by philipnienhuis (8KiB - application/octet-stream)
file #49981:  io.log added by svillemot (5KiB - text/x-log)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by svillemot (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-10-17 philipnienhuis Attached File- => Added xlsopen.m, #50003
        Attached File- => Added getinterfaces.m, #50004
    2020-10-16 philipnienhuis StatusConfirmed => In Progress
    2020-10-15 philipnienhuis Attached File- => Added xlsopen.m, #49985
        Attached File- => Added getinterfaces.m, #49986
    2020-10-15 philipnienhuis Summary[octave forge] (io) can't load XLSX files after chk_spreadsheet_support in 2.6.2 => [octave forge] (io) can't reliably load XLSX files in release 2.6.2
    2020-10-15 svillemot Attached File- => Added io-xlsopen-patched-L131.log, #49982
    2020-10-15 philipnienhuis StatusNone => Confirmed
        Assigned toNone => philipnienhuis
        Operating SystemGNU/Linux => Any
    2020-10-15 svillemot Attached File- => Added io.log, #49981

    Back to the top


    Powered by Savane 3.5