bugGNU Octave - Bugs: bug #58299, Local packages for different...

 
 

bug #58299: Local packages for different Octave versions cannot be installed in parallel

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Mon 04 May 2020 10:35:40 AM UTC
   
 
Category:  Interpreter Severity:  1 - Wish
Priority:  3 - Low Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 18 Nov 2021 04:41:21 PM UTC, comment #16: 

This should be working now.

Closing as fixed.

If this change should cause any trouble, we can re-open or open new reports.

Markus Mützel <mmuetzel>
Group administrator
Thu 15 Jul 2021 08:18:17 AM UTC, comment #15: 

Forgot to update the documentation as part of the previous change. Made up for here:
https://hg.savannah.gnu.org/hgweb/octave/rev/8859d8d1d275

Markus Mützel <mmuetzel>
Group administrator
Wed 14 Jul 2021 09:49:01 AM UTC, comment #14: 

Like we've seen with Octave 6.3, we are trying to keep the API stable through minor versions. But we cannot guarantee that this will always be the case.

To avoid the issue that motivated this change re-appearing when the API should change between minor versions in the future, I pushed a change here that uses the "api_version" string instead of the "major_version" as an identifier in the path for the default local package store:
https://hg.savannah.gnu.org/hgweb/octave/rev/2cc14e275a51

Using something like "MAJOR_VERSION.MINOR_VERSION" would require to re-install packages even if it would be possible to use the same installation.

Markus Mützel <mmuetzel>
Group administrator
Wed 24 Mar 2021 08:09:52 PM UTC, comment #13: 

The last patch didn't work correctly.
I fixed the issues, adapted the manual and pushed everything together here:
https://hg.savannah.gnu.org/hgweb/octave/rev/280defaf2023

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Mon 22 Mar 2021 10:00:00 AM UTC, comment #12: 

Thanks for the feedback.
If we can also guarantee that the API will only change for major versions, using the major version in the path sounds reasonable to me.

Attached is an updated patch with that change. As of now untested.

(file #51108)

Markus Mützel <mmuetzel>
Group administrator
Mon 22 Mar 2021 12:02:57 AM UTC, comment #11: 

We might as well use the major version number since the api string has changed with every new major version and the major version number is probably easier for users to understand and for us to explain.

John W. Eaton <jwe>
Group administrator
Sun 21 Mar 2021 07:26:37 PM UTC, comment #10: 

The attached patch moves the local package installation folder to the platform specific user data path and adds the Octave API version string, e.g. `/home/osboxes/.local/share/octave/api-v55+/packages`.
Similarly, it moves the local package configuration file to the platform specific user config path and adds the Octave API version string, too, e.g. `/home/osboxes/.config/octave/api-v55+/.octave_packages`.

I opted for the Octave API version string because that string will probably differ for incompatible binaries.

Until now, I only did some minor testing on Ubuntu 20.10. But I hope this will work on all platforms.


(file #51102)

Markus Mützel <mmuetzel>
Group administrator
Thu 05 Nov 2020 07:20:07 PM UTC, comment #9: 

I've put the three lines of comment #5 in my site-wide octaverc (/usr/local/share/octave/site/m/startup/octaverc) and it works beautifully with octave-6.0.93 and octave-7.0.0 n my Mageia system.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 26 Oct 2020 10:08:36 PM UTC, comment #8: 

Maybe we are so focused on fixing problems by patching function code in Octave that we don't see the obvious...

As also suggested in comment #5, why can't we simply add code along the lines of the suggestion in comment #5 in the site-wide .octaverc? or alternatively in the version-dependent global .octaverc?

For newer Octave versions the problem will be solved automatically that way.
IMO a big benefit would be that no changes for pkg.m are needed then as all required functionality has already been implemented there; all we need to do is invoke it.
If we're quick we can even do it for upcoming 6.1.

We'd need to agree on a good path, e.g.,
~/octave/<full_release>/
-or-
~/octave/<major_version>/  ## ~ cf. Mike comment #3
-or-
~/octave/local_packages/<major_version>/  ## a variant
-or-
~/<whatever>

We can enter any code in that site-wide or global .octaverc, e.g. the code in comment #5, or adapted paths, or even:

if (ispc)
  <set local package store somewhere in %APPDATA%/Roaming/...>
else
  <set local store in dedicated Linux or Mac way>
endif


Philip Nienhuis <philipnienhuis>
Group Member
Mon 26 Oct 2020 07:47:21 AM UTC, comment #7: 

A workaround exists (comment #5, that I use too).  Fixing this issue requires major refactoring of the existing "pkg" tool, and some more thoughts.  Nothing for an ready to release Octave 6.

Mark as "dev".

Kai Torben Ohlhus <siko1056>
Group Member
Fri 18 Sep 2020 08:46:24 AM UTC, comment #6: 

In some cases, package installations may not be compatible across minor or even patch version upgrades of Octave. At least on macOS, oct-files have linkage dependancy on the exact installation location of the Octave they were built under. And some installation mechanisms, such as Homebrew or Octave.app, have the full version name of Octave in the path to the installation (to allow side-by-side installations of multiple versions). So for this to work generally, you may need to make the package repositories completely version-specific.

This is what Octave.app does. https://github.com/octave-app/octave-app-bundler/blob/f41254a6e35fb50923564d62e7e3e0a3ea6899f7/bundle_octave#L431-L437

Andrew Janke <apjanke>
Sun 24 May 2020 04:37:35 PM UTC, comment #5: 

In bug #58225  I offered a workaround, repeated here:

Put the following in e.g., your local .octaverc:

vo = ver ("octave").Version(1);
pkg ("prefix", [ "~/octave" vo ], [ "~/octave" vo ]);
pkg ("local_list", [ "~/octave" vo "/.octave" vo "_packages" ]);


which leads to different local package stores for different versions of Octave.
It could even be incorporated in the global (site-wide) .octaverc

Philip Nienhuis <philipnienhuis>
Group Member
Tue 05 May 2020 08:39:44 PM UTC, comment #4: 

@Markus: see bug #36477, bug #40444, bug #57463 and bug #55822

To be precise in bug #58225 I referred to .m files in OF packages, and it's not those .m-files that differ between Octave versions but rather many OF packages have .m files that explicitly depend on some range of Octave versions. (sorry for nitpicking)
FWIW my experience is that .m-files usually have a much longer lifetime across Octave releases than .oct files. Other than that I think Mike's suggestion is a good one.

But maybe the mxe-octave installer should be adapted to wipe local package directories relating to its Octave version (after user consent, of course).

This bug report is about locally installed packages but for global packages I see a lot of similarities.
On Windows all global stores are installed separately for mxe-octave cross-builds so little problems there. But AIU on *nix the global package stores (both in share/ and lib/) are shared between Octave versions. Will e.g., updating or reinstalling a global package running a newer Octave version (with different api no.) lead to removal of the existing subdir with .oct files for the older api? (I guess so, looking at the logic in pkg.m)

Philip Nienhuis <philipnienhuis>
Group Member
Mon 04 May 2020 09:42:52 PM UTC, comment #3: 

Ok, confirmed. I've found that by deleting the package cache file, Octave doesn't know about the other versions of the package oct files and won't delete them, but then fails when it tries to extract doc strings from the incompatible oct files. Added that fact to bug #57826.

So yes, I think we're on the same page on all operating systems, and I agree it would be a good enhancement to install packages by default in a per-Octave-version directory in the user's home / profile directory. I think it's an enhancement because most users don't regularly use multiple versions of Octave at the same time, and normally need to reinstall all packages when updating to a new version anyway.

IMHO, that directory should be based on the major version of Octave, so packages installed in version 7.1.0 are still found and work in version 7.3.0.

Mike Miller <mtmiller>
Group Member
Mon 04 May 2020 08:46:19 PM UTC, comment #2: 

I tried by installing the "windows" package locally with the following command in Octave 5.2:

pkg install -local -forge windows


After this, the package is installed in C:\Users\Markus\octave\windows-1.4.0:

>> cd C:\Users\Markus\octave\windows-1.4.0
>> dir
.                           @octave_com_object          iscom.m                     packinfo                    x86_64-w64-mingw32-api-v53
..                          doc-cache                   mat2xls.m                   test


Then I closed Octave 5.2 and tried to install the package again using the same command in a self-compiled Octave 6.0.1.
The installation succeeds. But all files from the previous installation are gone. In particular the "x86_64-w64-mingw32-api-v53" with the binaries for Octave 5.2 has disappeared:

>> cd C:\Users\Markus\octave\windows-1.4.0
>> dir

.                           ..                          @octave_com_object          doc-cache                   iscom.m                     mat2xls.m                   packinfo                    test                        x86_64-w64-mingw32-api-v54


This might lead to problems if a user wants to change between different Octave versions frequently and has local packages.


Markus Mützel <mmuetzel>
Group administrator
Mon 04 May 2020 08:03:01 PM UTC, comment #1: 

Putting aside the issue of modifying the m-files dynamically, can you please test whether this works or doesn't work on Windows? It would be best if we can start from a known baseline of what does work and what would be nice to enhance.

On GNU/Linux, with the default configuration, the same set of local packages can be installed and used with multiple Octave versions. M-files are installed on top of each other in the same directory, so again, assume that's correct for now. Oct files are installed in a directory that is named for the CPU, OS, and Octave API version, so multiple sets of oct files can coexist. The local package store is shared between versions, but that is not a bug, again assuming the same set of packages.

There is definitely room for improvement, but please confirm first that you can get the same current behavior on Windows so we characterize this correctly.

I've offered this before elsewhere, but on my system I install packages in $XDG_DATA_HOME/octave/$OCTAVE_VERSION_MAJOR/packages (e.g. ~/.local/share/octave/6/packages/...).

Mike Miller <mtmiller>
Group Member
Mon 04 May 2020 10:35:40 AM UTC, original submission:  

The local package store for Windows is in %USERPROFILE%\octave atm.
It is possible to install several different versions of Octave (very easy on Windows). All of these versions share the same local package store.
But .oct files are usually not binary compatible between versions. In another bug report [1], Philip wrote that also some .m files might be different depending on the Octave version.
So updating a local package in one version of Octave can break it for the other version.

This issue will get more prominent because we plan on installing packages to the local store by default on Windows starting Octave 6.

Maybe the local package store should also include the Octave (major) version or API version in the path.

I believe there was a bug report about moving the local package store to %LOCALAPPDATA% or something. But I can no longer find that bug. So opening a new bug here.

I'd like to propose to store the local packages at %LOCALAPPDATA%/octave/{version} where {version} is the string returned by "version" (e.g. "6.1.0").

While this bug is focused mainly on the situation on Windows, it might make sense to make a similar change on all platforms. After all, after updating to a newer Octave version (or a downgrade fwiw), it is necessary to recompile all installed packages. So it would make sense to not list them from their old location when they probably won't work anyway.

Any thoughts?

[1]: https://savannah.gnu.org/bugs/index.php?58225#comment10

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-11-18 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2021-03-24 mmuetzel StatusPatch Submitted Ready For Test
    2021-03-22 mmuetzel Attached File- Added bug58299_pkg_local_prefix_v2.patch, #51108
        StatusPostponed Patch Submitted
    2021-03-21 mmuetzel Attached File- Added bug58299_pkg_local_prefix.patch, #51102
    2020-10-26 siko1056 StatusConfirmed Postponed
        Release6.0.90 dev
    2020-05-04 mtmiller Severity3 - Normal 1 - Wish
        Priority5 - Normal 3 - Low
        Item GroupUnexpected Error or Warning Feature Request
        StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code