Nice! Runs in a couple seconds for me too in default Octave now.

No, I have no idea how the SourceForge web stuff works.

> Does Octave even have a linter?

Nope, no shortage of tasks :)

Andrew - I agree on all of your points for improvements, and have even more, and could probably be broken up into several distinct projects that could be worked on independently and incrementally.

Package Installation - This seems like the topic that Packajoozle is mainly addressing. This covers the `pkg` or `pkj` function itself, the options, the downloading of packages, the handling of metadata, building, handling dependencies, etc. I'd include the serve-side package repository details here too. I think you've got that well covered.

Package Quality - This covers what you touched on in comment #20, fixing packages and making sure they pass their tests, but could be expanded a lot. Do packages build, install, have tests, do the tests pass, do they have demos, do they have help strings, do the help strings have examples, etc. I can see a lot of room for improvement in automating these kinds of checks, doing package linting, auditing, and so on. This is on my mind since we have a large backlog of unreleased Forge packages right now, and I recently proposed that we try to do peer review to help distribute the package QC workload, and I think this could be automated and streamlined a lot. I do intend to do some work on this aspect.

Package Building / Maintenance - There was a thread recently about the downsides of the makefile system Octave Forge packages currently use for routine maintenance tasks. Someone mentioned the rust cargo command, and having something like that for Octave packages might be an interesting project. A set of commands for turning a directory, git repo, or hg repo into a package, probably available as a package itself. This would take over all the tasks that the top-level maintainer's makefile does right now, including running tests from the developer side, running CI tests, running doctest, running QA checks, making the tarball the right way, building the HTML docs, running a package against multiple versions of Octave if they are available, stuff like that.

A couple more GSoC packaging ideas:

• Improving the Forge metadata publishing situation, so that e.g. this bug's original issue can be fixed the Right Way, and package metadata can be discovered without having to download all the distribution tarballs
• Fixing issues with all the individual Forge packages so that they install on all three OSes (Win/Lin/Mac) and pass their tests (maybe including doctest)
• Older Octave version support: so that in Octave 4.4.x or 5.x.x, you can say `pkj install foo` and it will find the latest version of package foo that is supported for your version of Octave (and maybe your OS version)
• Other `pkj` features: I wonder what other things Dildar sees in nix that he would like Octave to support? The student could draw them up as requirements specs and implement them

I modified the code so that it might be inserted in the current pkg code.  One issue is that it was returning the entire list of all packages and all package versions which is over 600 entries.  I chopped off the version numbers with another regexp and then used unique to get a list of true packages.

## Get the list of all packages
[html, r] = urlread ('https://sourceforge.net/projects/octave/files/Octave%20Forge%20Packages/Individual%20Package%20Releases');

if (! r)
  error ("pkg: failed to fetch list of packages from sourceforge.net");
endif

## Scrape the HTML itself
ptn = '<tr\s+title="(.*?gz)"\s+class="file';
[r, tok] = regexp (html, ptn, "start", "tokens");
if (isempty (r))
  error ("pkg: failed to parse list of packages from sourceforge.net");
endif

## Remove version numbers and produce unique list
files = cellstr (tok);
files = cellstr (regexp (files, '^.*?(?=-\d)', "match"));
files = unique (files);

However, the output is still not the same as 'pkg -forge list' which contains 69 entries, whereas this code produces 90 packages.  The difference is possibly in deprecated or removed packages.  For example, "pt_br-1.0.9.tar.gz" from 2010.

I don't think we should list packages like that.

It gets more complicated, but we could get the list of active packages from the original URL 'https://octave.sourceforge.io/list_packages.php'.  And then get the list of all packages with version numbers from the URL Andrew provided.  And then use the names from the first to search in the second and get the version number.  This seems pretty complicated though.

Why doesn't the original URL which is a php-generated page have the version number in it?  Could we do that?


(file #46621)

Found a workaround you can do now!

If you download the Octave Forge file list from https://sourceforge.net/projects/octave/files/Octave%20Forge%20Packages/Individual%20Package%20Releases, it has a complete file list that you can parse with a simple regex. (And there's even an embedded JSON list with additional metadata support, for if & when Octave gets JSON support.)

I added it to my little pkg-replacement toy. https://github.com/apjanke/octave-packajoozle/blob/master/inst/%2Bpackajoozle/%2Binternal/OctaveForgeClient.m#L149-L165

It's a biggish file to parse, so it still takes a bit, but it brings the `pkg -forge list` time down to under 1 second, with about ten lines of code.

How about a good ol' CSV file with a column header line? That will be easy to parse, and forward-compatible for extensibility by adding additional columns.

file #46032 is an example CSV package list file.

file #46030 is a patch to modify Octave to use it. It includes a `setappdata(0, 'forge.base_url' 'file://...')` back door to use for local testing. Doing this also removes the need to scrape the HTML for individual package pages to get their current version.

file #46031:  gen_forge_package_list.m will generate that CSV file using the current Octave pkg code.

Hmm. Shouldn't that file be dynamically generated from the package metadata so it doesn't add one more thing to manually update when a new package version is posted?

What repo is that ~/.octave_packages file in? I don't see it in octave-code or octave-project-web.

I'd like to take a stab at fixing this.

Is there any documentation on how to set up a local instance of Octave-Forge for testing this?

@Mike (comment #5)

The php scripts that you're looking for are in the project-web repository, in htdocs:

https://sourceforge.net/p/octave/project-web/ci/default/tree/htdocs/

tic; pkg list -forge;  toc

stills taks approximately 30 seconds on Octave 4.0.2

I think there's a copy of the scripts on the SVN repository.

Anyway, that has been discussed before on the mailing. It was agreed that the best solution was to have an octave struct saved in a text file and hosted in octave.org. That would have the advantage of keep working if octave forge ever moves to another server, and that be very easy for pkg to read.

In addition, the listing of packages doesn't bother to respect the pager - it just scrolls on and on ....

This is overwhelmingly slow and discourages using the built-in interface.  It is faster to actually go to the octave-forge website to browse around.

The problem seems to be that get_forge_pkg is called on each package name and each call takes ~0.5 seconds.  It might be easiest just to maintain a list of the Octave-Forge packages and their version numbers somewhere "magic" on the Octave-Forge site that the pkg command could fetch with urlread.