bugGNU Octave - Bugs: bug #42044, octave 3.8.1: huge memory use on...

 
 

bug #42044: octave 3.8.1: huge memory use on pkg load io

Submitter:  None
Submitted:  Fri 04 Apr 2014 07:18:40 AM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Fixed Assigned to:  None
Originator Name:  A.R. Burgers Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 3.8.1
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 17 Apr 2014 07:25:17 PM UTC, comment #9: 

I've uploaded a new io pkg version (2.2.0) to the release tracker; automatic loading of Java class libs for spreadsheet I/O on Linux has been disabled in that version.

FYI, investigation on my Linux boxes showed it is the search for Java class libs from LibreOffice installation that apparently makes for a long delay (~5 seconds on my core i5 laptop).

Strikingly it doesn't take that much time on Windows (a mere 0.8 seconds for all class libs incl. LibreOffice).

I'll investigate later (no time now) to see if there's a better solution for Linux. So this automatic loading of Java class libs may return in a future io pkg release.

OTOH, as io-2.2.0 now has full read/write support for OOXML and ODS w/o external support SW (albeit a bit slow for ODS) and other file formats will hopefully follow, the need for Java class libs will become less and less stringent anyway.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 05 Apr 2014 06:54:50 PM UTC, comment #8: 

Thanks (at least one other user where that mechanism is confirmed to work).

Any suggestion on how to proceed?
I'm not sure if much automation is the right thing for the average Linux user, and perhaps this is too much behind-ones-back automation.

E.g., the operation of that environment variable could be reversed on Linux (more like "opt in" rather than "opt out").

Philip Nienhuis <philipnienhuis>
Group Member
Sat 05 Apr 2014 04:12:49 PM UTC, comment #7: 

Setting OCTAVE_IO_JAVALIBS to `no` indeed fixes this:


octave-cli-3.8.1:1> tic; pkg load io; toc
Elapsed time is 0.0373571 seconds.


maybe I'm just spoiled for never having to deal much with Java...

Carnë Draug <carandraug>
Group Member
Sat 05 Apr 2014 05:32:42 AM UTC, comment #6: 

Oops, sorry, make that 'comment #3'

Philip Nienhuis <philipnienhuis>
Group Member
Sat 05 Apr 2014 05:31:34 AM UTC, comment #5: 

Pls also try setting the env var to "no" as I outlined in comment #1. Can you report back then,  please?

Maybe for Linux systems the search should be skipped; for the on average more helpless Windows users :-) it could be retained

Philip Nienhuis <philipnienhuis>
Group Member
Fri 04 Apr 2014 04:29:20 PM UTC, comment #4: 

For me it takes a lot of time as well


octave-cli-3.8.1:1> tic; pkg load io; toc
Elapsed time is 8.15525 seconds.


I don't have any of java libraries required by the IO package though.

Carnë Draug <carandraug>
Group Member
Fri 04 Apr 2014 03:43:52 PM UTC, comment #3: 

This may be due to the way the io pkg tries to automatically find Java spreadsheet class libs.

PKG_ADD (invoked when you load io) starts a function that searches in a few predefined places + subdirs (max 2 subdir levels deep), see below. So if you have many subdirs and files there, searches may take some time.

Hint: if you do not need spreadsheet I/O, try to set an environment flag:

export OCTAVE_IO_JAVALIBS=no

before starting Octave, then PKG_ADD will skip searching.


FYI, the places where PKG_ADD searches on Linux systems are (for "regular" spreadsheet I/O java class libs like Apache POI):
/usr/share/java/
~/java/
+  a subdir conveyed using environment var "OCTAVE_IO_JAVALIBS" (I suppose you didn't set that)

... and for OpenOffice /LibreOffice:
/usr/lib/<bnam>
/opt/<bnam>
where <bnam> is "OpenOffice.org*", "LibreOffice*" or "ooo", in that order.

So if you have exceptionally many files in one or more of those places PKG_ADD's search can take a while.

On all Linux systems I peeked in, /usr/share/java usually didn't have more than some tens of files so a search there should go fairly rapidly. Same holds for the other searched subdirs mentioned above.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 04 Apr 2014 12:31:53 PM UTC, comment #2: 

I don't doubt that it is taking a long time to load for you, but that is not the way to measure CPU time.  The cputime function returns the number of CPU seconds since starting Octave.  So you need to subtract the value returned from the first call:


t0 = cputime ()
pkg load io
cputime () - t0


John W. Eaton <jwe>
Group administrator
Fri 04 Apr 2014 11:55:18 AM UTC, comment #1: 

Seems to be related to java.
With java.opts memory usage can be controlled.

Stil io package takes long to load:

octave:2> cputime;pkg load io;cputime
ans =  34.632

When I recompile with --disable-java io package loads quickly.

Anonymous
Fri 04 Apr 2014 07:18:40 AM UTC, original submission:  

if I launch the command-line version of octave 3.8.1, and then enter "pkg load io", this takes considerable time and, watching the process list with top, I see that the memory footprint (column VIRT in top) immediately goes to 33.6Gb. Is this normal?

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by jwe (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-04-17 philipnienhuis StatusNeed Info Fixed
        Open/ClosedOpen Closed
    2014-04-13 mtmiller CategoryInterpreter Octave Package
        StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code