bugGNU Octave - Bugs: bug #47508, package installation very slow due...

 
 

bug #47508: package installation very slow due to makeinfo

Submitter:  Carnë Draug <carandraug>
Submitted:  Wed 23 Mar 2016 04:01:22 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 30 Mar 2016 04:18:48 PM UTC, comment #7: 

Does this function fit the bill? I think this could drop into scripts/help, any time lookfor is updated it can be updated, and pkg can rely on it to validate any doc-cache files that are shipped in the pkg source tarball.

(file #36799)

Mike Miller <mtmiller>
Group Member
Wed 30 Mar 2016 03:31:48 PM UTC, comment #6: 

We could introduce a new function, say doc_cache_valid(), that would do what lookfor does, and simply return a bool if it's a doc-cache file format that lookfor knows how to handle.

Mike Miller <mtmiller>
Group Member
Wed 30 Mar 2016 03:24:26 PM UTC, comment #5: 

I can say that the structure hasn't changed in a very long while.  The last substantive change was in 2009 when the file name changed to doc-cache from DOC, and even that wasn't a format change.  I think you would be fine coding to the existing format.

Rik <rik5>
Group administrator
Wed 30 Mar 2016 12:17:15 PM UTC, comment #4: 


> As you wrote, this will be more about changing the pkg.m script than altering other parts of Octave.


Yes, this is what I was proposing too. But the main issue is what will happen if we ever change doc-cache structure in core? To move this forward we need to set some sort of stable spec for doc-cache (or at very least, a doc-cache spec version number).

Carnë Draug <carandraug>
Group Member
Thu 24 Mar 2016 02:53:06 PM UTC, comment #3: 

I was in a hurry and only read the upper part of lookfor.m; Certainly more of the infrastructure for this is in place than I thought.  As you wrote, this will be more about changing the pkg.m script than altering other parts of Octave.

Rik <rik5>
Group administrator
Thu 24 Mar 2016 02:36:36 PM UTC, comment #2: 

Rik, I think the only thing needed is an agreement that Forge package maintainers who want to use this feature do something to build the doc-cache file into the package source distribution. And some small code in pkg.m (and subfunctions) to check for its existence and not generate the cache if it already exists in the source package.

The lookfor function already has code to look for files named "doc-cache" in the load path and use them, pkg already builds doc-cache files from the m-files and oct-files at install time, we just need a way to bypass that and an easy way to build the doc-cache at package release time (if the maintainer so chooses).

Mike Miller <mtmiller>
Group Member
Wed 23 Mar 2016 05:23:38 PM UTC, comment #1: 

Seems like a good idea to me.  Core Octave without a doc-cache is painfully slow because of Makeinfo and the distributed doc-cache is the only thing that makes it workable.

An implementation is almost there.  The function doc_cache_create will create a doc-cache file.  You might want to inquire on the maintainers list about the best approach.  It seems like we might want to expand doc_cache_file.m to return a cell array of doc-cache files.  The function lookfor would also need to be expanded to loop over all results returned from doc_cache_file.

Rik <rik5>
Group administrator
Wed 23 Mar 2016 04:01:22 PM UTC, original submission:  

Since version 5.0 of texinfo, when it was rewritten in perl, installing packages can take a long time.  The actual numbers are package dependent but generation of doc-cache is usually the performance bottleneck.

I have reports from Marco Azteri that in windows this is worse and installing ltfat takes more than 10 minutes with hundreds of perl process forked.

We could add support for doc-cache in the package like it currently happens for Octave core releases.  This could be optional so that the absence of doc-cache would still trigger generation of doc-cache.

The advantages of these are:

  • faster installation of packages;
  • reduce importance of makeinfo as runtime dependency;
  • in theory, everyone installing the package is doing the same work to generate doc-cache so stops wasting time and energy (environment friendly).


A possible caveat is:

  • if the doc-cache model ever changes in core, there may be issues.


At the moment doc-cache is a 3xN cell array.  An extra doc-cache version variable could be added to it so generation of doc-cache would only be triggered is necessary.  Also, how often has this changed in core?

The presence of doc-cache could also be completely optional.  A good place to start making use of it is with octave binary packages which are often already version dependent.

Carnë Draug <carandraug>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #36799:  doc_cache_valid.m added by mtmiller (1KiB - d2l/unknowntype)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by matzeri
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by carandraug (Submitted the item)
  • -email is unavailable- added by carandraug (Marco Atzeri - downstream cygwin packager)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-03-30 mtmiller Attached File- Added doc_cache_valid.m, #36799
    2016-03-24 mtmiller StatusNone Confirmed
    2016-03-23 matzeri Carbon-Copy- Added matzeri
    2016-03-23 carandraug Carbon-Copy- Added marco.atzeri@gmail.com>

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code