bugGNU Octave - Bugs: bug #39479, 'pkg -forge list' takes 56.5...

 
 

bug #39479: 'pkg -forge list' takes 56.5 seconds

Submitter:  Rik <rik5>
Submitted:  Sat 13 Jul 2013 05:40:33 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 26 Mar 2019 08:23:30 AM UTC, comment #28: 

Nice! Runs in a couple seconds for me too in default Octave now.

Andrew Janke <apjanke>
Tue 26 Mar 2019 05:23:26 AM UTC, comment #27: 

I implemented a complicated scheme to compare the list of published packages to the list of all packages ever published with their version numbers.  See https://hg.savannah.gnu.org/hgweb/octave/rev/222f5ac853c8.  I think this will do for now (30X speedup to 1.8 seconds on my computer) until something better comes along to replace the pkg command.

Rik <rik5>
Group administrator
Mon 25 Mar 2019 06:26:57 PM UTC, comment #26: 

No, I have no idea how the SourceForge web stuff works.

Mike Miller <mtmiller>
Group Member
Sat 23 Mar 2019 09:23:54 AM UTC, comment #25: 

I just incorporated some ideas to the wiki: https://wiki.octave.org/Summer_of_Code_Project_Ideas#Octave_Package_management

@Rik comment #18:  This would be a great first step inserting the fraction of packajoozle that speeds up things, if the long term goal of replacements takes longer than Octave 6.0.

Regarding comment #21: Those changes have to be done at  https://sourceforge.net/p/octave/project-web/ci/default/tree/.  Mike are you able to push changes there and familiar with that project Oliver created last year?

I would love to help out here.

Kai Torben Ohlhus <siko1056>
Group Member
Sat 23 Mar 2019 12:08:15 AM UTC, comment #24: 


> Does Octave even have a linter?


Nope, no shortage of tasks :)

Mike Miller <mtmiller>
Group Member
Sat 23 Mar 2019 12:02:56 AM UTC, comment #23: 

All right! Those all sound good to me. With that, I think there's plenty for a GSoc project to work on, even if it's taking Packajoozle as a base instead of writing a new package manager.

> ...doing package linting, ...


Does Octave even have a linter?

Andrew Janke <apjanke>
Fri 22 Mar 2019 11:59:54 PM UTC, comment #22: 

Andrew - I agree on all of your points for improvements, and have even more, and could probably be broken up into several distinct projects that could be worked on independently and incrementally.

Package Installation - This seems like the topic that Packajoozle is mainly addressing. This covers the `pkg` or `pkj` function itself, the options, the downloading of packages, the handling of metadata, building, handling dependencies, etc. I'd include the serve-side package repository details here too. I think you've got that well covered.

Package Quality - This covers what you touched on in comment #20, fixing packages and making sure they pass their tests, but could be expanded a lot. Do packages build, install, have tests, do the tests pass, do they have demos, do they have help strings, do the help strings have examples, etc. I can see a lot of room for improvement in automating these kinds of checks, doing package linting, auditing, and so on. This is on my mind since we have a large backlog of unreleased Forge packages right now, and I recently proposed that we try to do peer review to help distribute the package QC workload, and I think this could be automated and streamlined a lot. I do intend to do some work on this aspect.

Package Building / Maintenance - There was a thread recently about the downsides of the makefile system Octave Forge packages currently use for routine maintenance tasks. Someone mentioned the rust cargo command, and having something like that for Octave packages might be an interesting project. A set of commands for turning a directory, git repo, or hg repo into a package, probably available as a package itself. This would take over all the tasks that the top-level maintainer's makefile does right now, including running tests from the developer side, running CI tests, running doctest, running QA checks, making the tarball the right way, building the HTML docs, running a package against multiple versions of Octave if they are available, stuff like that.

Mike Miller <mtmiller>
Group Member
Fri 22 Mar 2019 11:31:21 PM UTC, comment #21: 

Rik:

> Why doesn't the original URL which is a php-generated page have the version number in it? Could we do that?


Yeah, that sounds like the right way to do it. Might need to be a new separate php page to maintain back-compatibility for existing old versions of Octave's pkg parsing code. My way is just a clumsy hack to make it work without changes on the Forge side.

Andrew Janke <apjanke>
Fri 22 Mar 2019 11:30:12 PM UTC, comment #20: 

A couple more GSoC packaging ideas:

  • Improving the Forge metadata publishing situation, so that e.g. this bug's original issue can be fixed the Right Way, and package metadata can be discovered without having to download all the distribution tarballs
  • Fixing issues with all the individual Forge packages so that they install on all three OSes (Win/Lin/Mac) and pass their tests (maybe including doctest)
  • Older Octave version support: so that in Octave 4.4.x or 5.x.x, you can say `pkj install foo` and it will find the latest version of package foo that is supported for your version of Octave (and maybe your OS version)
  • Other `pkj` features: I wonder what other things Dildar sees in nix that he would like Octave to support? The student could draw them up as requirements specs and implement them
Andrew Janke <apjanke>
Fri 22 Mar 2019 11:03:53 PM UTC, comment #19: 


> When do we replace pkg by octave-packajoozle? ;-) Looks great, I just tested it!


Aw, thank you!

> Unfortunately you are no student for GSoC, right? ^^ Recently, "Dildar Sk" decided to work on improving pkg.


Nope, I'm a 40-year old grizzled veteran hacker. :)

> I think another competing from scratch approach will not be beneficial. Maybe he can help documenting and merging your code into Octave core for 6.0?


I think so too. If this is going to become part of core Octave, it will need some real help, because the pkg code is so central and widely used; it has to be robust. And if I understand it correctly, this would be the first major Octave feature to make use of Octave OOP, so it would need both testing and documentation of both the classes and OOP techniques it uses. I think that could be a good GSoC project. It could involve:

  • Testing, both interactive and setting up test scripts and harnesses
  • Documentation: getting the Packajoozle documentation into a shape where a GSoC student could understand it and effectively work with the code would be a good demonstration of its documentation and design.
  • Dependency resolution code. The current dependency resolution logic is rather naive, and dependency resolution is complicated. Developing real, robust dependency resolution could be a good part of a summer project.
  • Working with Octave to fix issues with the classdef/OOP support (like bug #55976 and bug #55810). There seem to be some basic shortcomings in object support; I'm guessing this is because people don't use it very much?
  • Developing a public API for the Packajoozle objects. Right now, they're all in +internal, but I'd like to present a public OOP API for developers that want to script or extend package management.
  • Shepherding Packajoozle through the process of becoming an Octave Forge package, so people can use and test it, on its way to getting into core Octave.
  • Making sure it works on Linux and Windows
  • Getting rid of weird errors and warnings, like complaints about missing paths after a package has been installed
  • Integration with Testify or the core BIST code, so `pkj test <package>` works well, and we can run scripts that test all the Octave Forge packages
  • Support for additional, temporary package installation directories besides the regular "global" and "user" ones, so that package testing can be done in a clean installation space and not mess up the user's existing package installations.


Though I'm having enough fun with this myself, who knows how much of that I'll get done by the time GSoC starts.

Andrew Janke <apjanke>
Fri 22 Mar 2019 06:20:24 PM UTC, comment #18: 

I modified the code so that it might be inserted in the current pkg code.  One issue is that it was returning the entire list of all packages and all package versions which is over 600 entries.  I chopped off the version numbers with another regexp and then used unique to get a list of true packages.


## Get the list of all packages
[html, r] = urlread ('https://sourceforge.net/projects/octave/files/Octave%20Forge%20Packages/Individual%20Package%20Releases');

if (! r)
  error ("pkg: failed to fetch list of packages from sourceforge.net");
endif

## Scrape the HTML itself
ptn = '<tr\s+title="(.*?gz)"\s+class="file';
[r, tok] = regexp (html, ptn, "start", "tokens");
if (isempty (r))
  error ("pkg: failed to parse list of packages from sourceforge.net");
endif

## Remove version numbers and produce unique list
files = cellstr (tok);
files = cellstr (regexp (files, '^.*?(?=-\d)', "match"));
files = unique (files);


However, the output is still not the same as 'pkg -forge list' which contains 69 entries, whereas this code produces 90 packages.  The difference is possibly in deprecated or removed packages.  For example, "pt_br-1.0.9.tar.gz" from 2010.

I don't think we should list packages like that.

It gets more complicated, but we could get the list of active packages from the original URL 'https://octave.sourceforge.io/list_packages.php'.  And then get the list of all packages with version numbers from the URL Andrew provided.  And then use the names from the first to search in the second and get the version number.  This seems pretty complicated though.

Why doesn't the original URL which is a php-generated page have the version number in it?  Could we do that?


(file #46621)

Rik <rik5>
Group administrator
Fri 22 Mar 2019 07:37:08 AM UTC, comment #17: 

When do we replace pkg by octave-packajoozle? ;-)  Looks great, I just tested it!

`pkj list -forge`  5.36779 seconds
`pkg list -forge` 85.1127  seconds

And the pkg version selection is great.


Off topic:

Unfortunately you are no student for GSoC, right? ^^ Recently, "Dildar Sk" decided to work on improving pkg.

https://lists.gnu.org/archive/html/help-octave/2019-03/msg00172.html

I think another competing from scratch approach will not be beneficial.  Maybe he can help documenting and merging your code into Octave core for 6.0?

Kai Torben Ohlhus <siko1056>
Group Member
Thu 21 Mar 2019 09:32:56 PM UTC, comment #16: 

Found a workaround you can do now!

If you download the Octave Forge file list from https://sourceforge.net/projects/octave/files/Octave%20Forge%20Packages/Individual%20Package%20Releases, it has a complete file list that you can parse with a simple regex. (And there's even an embedded JSON list with additional metadata support, for if & when Octave gets JSON support.)

I added it to my little pkg-replacement toy. https://github.com/apjanke/octave-packajoozle/blob/master/inst/%2Bpackajoozle/%2Binternal/OctaveForgeClient.m#L149-L165

It's a biggish file to parse, so it still takes a bit, but it brings the `pkg -forge list` time down to under 1 second, with about ten lines of code.

Andrew Janke <apjanke>
Sun 20 Jan 2019 04:55:16 AM UTC, comment #15: 

Updated patch with user documentation for the pkg('-forge', 'list') signature: file #46033:  pkg-forge-csv-package-list-v2.patch

Andrew Janke <apjanke>
Sun 20 Jan 2019 04:04:55 AM UTC, comment #14: 

How about a good ol' CSV file with a column header line? That will be easy to parse, and forward-compatible for extensibility by adding additional columns.

file #46032 is an example CSV package list file.

file #46030 is a patch to modify Octave to use it. It includes a `setappdata(0, 'forge.base_url' 'file://...')` back door to use for local testing. Doing this also removes the need to scrape the HTML for individual package pages to get their current version.

file #46031:  gen_forge_package_list.m will generate that CSV file using the current Octave pkg code.

Andrew Janke <apjanke>
Fri 18 Jan 2019 10:31:16 PM UTC, comment #13: 

The file can be updated automatically from the metadata, but that is a server side script.  Separate that part of the problem, and assume that there is such a script and it generates a file (your choice as to format).  I was imagining starting with the client side and writing the code to parse and display this file.  When that is working, then we can go back and work on the server side script to produce it.

The file ~/.octave_packages is generated by the pkg scripts.  Try


pkg install -forge symbolic


which will install the symbolic package and also create ~/.octave_packages.
 

Rik <rik5>
Group administrator
Fri 18 Jan 2019 09:17:35 PM UTC, comment #12: 

Hmm. Shouldn't that file be dynamically generated from the package metadata so it doesn't add one more thing to manually update when a new package version is posted?

What repo is that ~/.octave_packages file in? I don't see it in octave-code or octave-project-web.

Andrew Janke <apjanke>
Fri 18 Jan 2019 09:14:11 PM UTC, comment #11: 

For testing, I don't think you need to clone Octave Forge.  The suggestion was that there be a unique URL that points to a file (format still undetermined) which can be fetched with urlread.  This file is then parsed locally and displayed to the user.

For prototyping, you can just skip the urlread and have a local file that you load() or otherwise read.  The code that needs to be written would then format and display this information.  The suggestion was made either to use a struct variable saved in Octave's native text format, or possibly in the format in ~/.octave_packages which seems to be a cell array.

Rik <rik5>
Group administrator
Fri 18 Jan 2019 06:47:02 PM UTC, comment #10: 

I'd like to take a stab at fixing this.

Is there any documentation on how to set up a local instance of Octave-Forge for testing this?

Andrew Janke <apjanke>
Thu 24 Nov 2016 12:35:48 AM UTC, comment #9: 

The problem is still there in Octave 4.3.0+


tic; pkg list -forge; toc
Elapsed time is 40.4616 seconds.


Juan Pablo Carbajal <juanpi>
Group Member
Thu 16 Jun 2016 01:22:21 PM UTC, comment #8: 

@Mike (comment #5)

The php scripts that you're looking for are in the project-web repository, in htdocs:

https://sourceforge.net/p/octave/project-web/ci/default/tree/htdocs/

Julien Bect <jbect>
Thu 16 Jun 2016 01:19:51 PM UTC, comment #7: 

I meant "still takes"

Julien Bect <jbect>
Thu 16 Jun 2016 01:18:55 PM UTC, comment #6: 

tic; pkg list -forge;  toc

stills taks approximately 30 seconds on Octave 4.0.2

Julien Bect <jbect>
Sun 28 Jul 2013 05:10:30 PM UTC, comment #5: 

I could not find any *.php scripts in the octave-forge SVN.

So the process will be you or someone else runs a script to update a hosted static file every time a new Forge package release is made?

Should the format of this file be the same as the struct saved in the local packages cache file (~/.octave_packages) so we can reuse the code that understands that structure?

Mike Miller <mtmiller>
Group Member
Wed 17 Jul 2013 09:14:47 PM UTC, comment #4: 

I think there's a copy of the scripts on the SVN repository.

Anyway, that has been discussed before on the mailing. It was agreed that the best solution was to have an octave struct saved in a text file and hosted in octave.org. That would have the advantage of keep working if octave forge ever moves to another server, and that be very easy for pkg to read.

Carnë Draug <carandraug>
Group Member
Wed 17 Jul 2013 06:53:24 PM UTC, comment #3: 

A workaround is to use


s = pkg ("-forge", "list");


which completes much faster, but only lists the names of available packages without their versions.

A real solution to this will probably involve an improved server-side script that returns all the information needed in some structured format like XML, JSON, or even CSV. The pkg function can then get everything it needs from one URL fetch instead of tens of URLs.

Can anyone comment where are the Octave Forge server-side scripts maintained, for example list_packages.php and packages.php?

Mike Miller <mtmiller>
Group Member
Tue 16 Jul 2013 06:19:01 PM UTC, comment #2: 

In addition, the listing of packages doesn't bother to respect the pager - it just scrolls on and on ....

Philip Nienhuis <philipnienhuis>
Group Member
Sat 13 Jul 2013 09:18:59 PM UTC, comment #1: 

Using a tic (); pkg -forge list; toc (), says


Elapsed time is 25.1274 seconds.


Which is faster than Rik's result, but still too slow (imo).

Ben Abbott <bpabbott>
Group Member
Sat 13 Jul 2013 05:40:33 PM UTC, original submission:  

This is overwhelmingly slow and discourages using the built-in interface.  It is faster to actually go to the octave-forge website to browse around.

The problem seems to be that get_forge_pkg is called on each package name and each call takes ~0.5 seconds.  It might be easiest just to maintain a list of the Octave-Forge packages and their version numbers somewhere "magic" on the Octave-Forge site that the pkg command could fetch with urlread.


Rik <rik5>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #46621:  pkglist.m added by rik5 (622B - text/x-matlab)
file #46033:  pkg-forge-csv-package-list-v2.patch added by apjanke (8KiB - application/octet-stream - update patch with user documentation)
file #46030:  pkg-forge-csv-package-list.patch added by apjanke (7KiB - application/octet-stream)
file #46031:  gen_forge_package_list.m added by apjanke (476B - application/octet-stream)
file #46032:  package_list.csv added by apjanke (1003B - text/csv)

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by juanpi (Posted a comment)
  • -email is unavailable- added by jbect (Posted a comment)
  • -email is unavailable- added by bpabbott (Posted a comment)
  • -email is unavailable- added by rik5 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-04-11 mtmiller Carbon-CopyRemoved 80942 -
    2019-03-26 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2019-03-22 rik5 Attached File- Added pkglist.m, #46621
    2019-01-20 apjanke Attached File- Added pkg-forge-csv-package-list-v2.patch, #46033
    2019-01-20 apjanke Attached File- Added pkg-forge-csv-package-list.patch, #46030
        Attached File- Added gen_forge_package_list.m, #46031
        Attached File- Added package_list.csv, #46032
    2016-02-09 rik5 Dependencies- bugs #47102 is dependent
    2013-07-13 bpabbott StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code