bugGNU Octave - Bugs: bug #58299, Local packages for different...

 
 

bug #58299: Local packages for different Octave versions cannot be installed in parallel

Submitted by:  Markus Mützel <mmuetzel>
Submitted on:  Mon 04 May 2020 10:35:40 AM UTC  
 
Category:  Interpreter Severity:  1 - Wish
Priority:  3 - Low Item Group:  Feature Request
Status:  Postponed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Thu 05 Nov 2020 07:20:07 PM UTC, comment #9: 

I've put the three lines of comment #5 in my site-wide octaverc (/usr/local/share/octave/site/m/startup/octaverc) and it works beautifully with octave-6.0.93 and octave-7.0.0 n my Mageia system.

Philip Nienhuis <philipnienhuis>
Project Member
Mon 26 Oct 2020 10:08:36 PM UTC, comment #8: 

Maybe we are so focused on fixing problems by patching function code in Octave that we don't see the obvious...

As also suggested in comment #5, why can't we simply add code along the lines of the suggestion in comment #5 in the site-wide .octaverc? or alternatively in the version-dependent global .octaverc?

For newer Octave versions the problem will be solved automatically that way.
IMO a big benefit would be that no changes for pkg.m are needed then as all required functionality has already been implemented there; all we need to do is invoke it.
If we're quick we can even do it for upcoming 6.1.

We'd need to agree on a good path, e.g.,
~/octave/<full_release>/
-or-
~/octave/<major_version>/  ## ~ cf. Mike comment #3
-or-
~/octave/local_packages/<major_version>/  ## a variant
-or-
~/<whatever>

We can enter any code in that site-wide or global .octaverc, e.g. the code in comment #5, or adapted paths, or even:

if (ispc)
  <set local package store somewhere in %APPDATA%/Roaming/...>
else
  <set local store in dedicated Linux or Mac way>
endif

Philip Nienhuis <philipnienhuis>
Project Member
Mon 26 Oct 2020 07:47:21 AM UTC, comment #7: 

A workaround exists (comment #5, that I use too).  Fixing this issue requires major refactoring of the existing "pkg" tool, and some more thoughts.  Nothing for an ready to release Octave 6.

Mark as "dev".

Kai Torben Ohlhus <siko1056>
Project Member
Fri 18 Sep 2020 08:46:24 AM UTC, comment #6: 

In some cases, package installations may not be compatible across minor or even patch version upgrades of Octave. At least on macOS, oct-files have linkage dependancy on the exact installation location of the Octave they were built under. And some installation mechanisms, such as Homebrew or Octave.app, have the full version name of Octave in the path to the installation (to allow side-by-side installations of multiple versions). So for this to work generally, you may need to make the package repositories completely version-specific.

This is what Octave.app does. https://github.com/octave-app/octave-app-bundler/blob/f41254a6e35fb50923564d62e7e3e0a3ea6899f7/bundle_octave#L431-L437

Andrew Janke <apjanke>
Sun 24 May 2020 04:37:35 PM UTC, comment #5: 

In bug #58225  I offered a workaround, repeated here:

Put the following in e.g., your local .octaverc:

vo = ver ("octave").Version(1);
pkg ("prefix", [ "~/octave" vo ], [ "~/octave" vo ]);
pkg ("local_list", [ "~/octave" vo "/.octave" vo "_packages" ]);

which leads to different local package stores for different versions of Octave.
It could even be incorporated in the global (site-wide) .octaverc

Philip Nienhuis <philipnienhuis>
Project Member
Tue 05 May 2020 08:39:44 PM UTC, comment #4: 

@Markus: see bug #36477, bug #40444, bug #57463 and bug #55822

To be precise in bug #58225 I referred to .m files in OF packages, and it's not those .m-files that differ between Octave versions but rather many OF packages have .m files that explicitly depend on some range of Octave versions. (sorry for nitpicking)
FWIW my experience is that .m-files usually have a much longer lifetime across Octave releases than .oct files. Other than that I think Mike's suggestion is a good one.

But maybe the mxe-octave installer should be adapted to wipe local package directories relating to its Octave version (after user consent, of course).

This bug report is about locally installed packages but for global packages I see a lot of similarities.
On Windows all global stores are installed separately for mxe-octave cross-builds so little problems there. But AIU on *nix the global package stores (both in share/ and lib/) are shared between Octave versions. Will e.g., updating or reinstalling a global package running a newer Octave version (with different api no.) lead to removal of the existing subdir with .oct files for the older api? (I guess so, looking at the logic in pkg.m)

Philip Nienhuis <philipnienhuis>
Project Member
Mon 04 May 2020 09:42:52 PM UTC, comment #3: 

Ok, confirmed. I've found that by deleting the package cache file, Octave doesn't know about the other versions of the package oct files and won't delete them, but then fails when it tries to extract doc strings from the incompatible oct files. Added that fact to bug #57826.

So yes, I think we're on the same page on all operating systems, and I agree it would be a good enhancement to install packages by default in a per-Octave-version directory in the user's home / profile directory. I think it's an enhancement because most users don't regularly use multiple versions of Octave at the same time, and normally need to reinstall all packages when updating to a new version anyway.

IMHO, that directory should be based on the major version of Octave, so packages installed in version 7.1.0 are still found and work in version 7.3.0.

Mike Miller <mtmiller>
Project Administrator
Mon 04 May 2020 08:46:19 PM UTC, comment #2: 

I tried by installing the "windows" package locally with the following command in Octave 5.2:

pkg install -local -forge windows

After this, the package is installed in C:\Users\Markus\octave\windows-1.4.0:

>> cd C:\Users\Markus\octave\windows-1.4.0
>> dir
.                           @octave_com_object          iscom.m                     packinfo                    x86_64-w64-mingw32-api-v53
..                          doc-cache                   mat2xls.m                   test

Then I closed Octave 5.2 and tried to install the package again using the same command in a self-compiled Octave 6.0.1.
The installation succeeds. But all files from the previous installation are gone. In particular the "x86_64-w64-mingw32-api-v53" with the binaries for Octave 5.2 has disappeared:

>> cd C:\Users\Markus\octave\windows-1.4.0
>> dir

.                           ..                          @octave_com_object          doc-cache                   iscom.m                     mat2xls.m                   packinfo                    test                        x86_64-w64-mingw32-api-v54

This might lead to problems if a user wants to change between different Octave versions frequently and has local packages.

Markus Mützel <mmuetzel>
Project Member
Mon 04 May 2020 08:03:01 PM UTC, comment #1: 

Putting aside the issue of modifying the m-files dynamically, can you please test whether this works or doesn't work on Windows? It would be best if we can start from a known baseline of what does work and what would be nice to enhance.

On GNU/Linux, with the default configuration, the same set of local packages can be installed and used with multiple Octave versions. M-files are installed on top of each other in the same directory, so again, assume that's correct for now. Oct files are installed in a directory that is named for the CPU, OS, and Octave API version, so multiple sets of oct files can coexist. The local package store is shared between versions, but that is not a bug, again assuming the same set of packages.

There is definitely room for improvement, but please confirm first that you can get the same current behavior on Windows so we characterize this correctly.

I've offered this before elsewhere, but on my system I install packages in $XDG_DATA_HOME/octave/$OCTAVE_VERSION_MAJOR/packages (e.g. ~/.local/share/octave/6/packages/...).

Mike Miller <mtmiller>
Project Administrator
Mon 04 May 2020 10:35:40 AM UTC, original submission:  

The local package store for Windows is in %USERPROFILE%\octave atm.
It is possible to install several different versions of Octave (very easy on Windows). All of these versions share the same local package store.
But .oct files are usually not binary compatible between versions. In another bug report [1], Philip wrote that also some .m files might be different depending on the Octave version.
So updating a local package in one version of Octave can break it for the other version.

This issue will get more prominent because we plan on installing packages to the local store by default on Windows starting Octave 6.

Maybe the local package store should also include the Octave (major) version or API version in the path.

I believe there was a bug report about moving the local package store to %LOCALAPPDATA% or something. But I can no longer find that bug. So opening a new bug here.

I'd like to propose to store the local packages at %LOCALAPPDATA%/octave/{version} where {version} is the string returned by "version" (e.g. "6.1.0").

While this bug is focused mainly on the situation on Windows, it might make sense to make a similar change on all platforms. After all, after updating to a newer Octave version (or a downgrade fwiw), it is necessary to recompile all installed packages. So it would make sense to not list them from their old location when they probably won't work anyway.

Any thoughts?

[1]: https://savannah.gnu.org/bugs/index.php?58225#comment10

Markus Mützel <mmuetzel>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-10-26 siko1056 StatusConfirmed => Postponed
        Release6.0.90 => dev
    2020-05-04 mtmiller Severity3 - Normal => 1 - Wish
        Priority5 - Normal => 3 - Low
        Item GroupUnexpected Error or Warning => Feature Request
        StatusNone => Confirmed

    Back to the top


    Powered by Savane 3.5