bugGNU Octave - Bugs: bug #51632, make global pkg paths always...

 
 

bug #51632: make global pkg paths always relative to OCTAVE_HOME, no need to pkg rebuild -global when path changes

Submitted by:  Tasos Papastylianou <tpapastylianou>
Submitted on:  Mon 31 Jul 2017 06:57:31 PM UTC  
 
Category: Octave FunctionSeverity: 1 - Wish
Priority: 5 - NormalItem Group: Feature Request
Status: ConfirmedAssigned to: None
Originator Name: Tasos Papastylianou <poacheR>Open/Closed: Open
Release: 4.2.1Operating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 09 Aug 2017 12:59:42 PM UTC, comment #32:

I would be in support of that. Since OCTAVE_HOME is already set on launch it would make this a non issue.

Nicholas Jankowski <nrjank>
Tue 08 Aug 2017 09:34:03 PM UTC, comment #31:

Since bug #51684 is the new meta-bug about all path portability issues, can we focus this bug on the specific issue that when OCTAVE_HOME changes, the user needs to run "pkg rebuild -global"?

I found an old thread where it was discussed to make the global pkg prefix no longer configurable and have it always derive from OCTAVE_HOME, avoiding the need to rebuild the cache when that path changes: https://lists.gnu.org/archive/html/octave-maintainers/2014-11/msg00018.html

I would propose that we revive that idea to resolve this bug, and make it so that the global cache only needs to be built once, or when a new pkg is installed, and changing OCTAVE_HOME when Octave moves to a different location is all that is needed.

Mike Miller <mtmiller>
Project Administrator
Tue 08 Aug 2017 01:59:36 AM UTC, comment #30:

FYI, separate wishlist bug report created at bug #51684 for a truly portable" version.

Nicholas Jankowski <nrjank>
Thu 03 Aug 2017 08:29:27 PM UTC, comment #29:

I agree that 'going truly portable' is probably worth a separate discussion elsewhere.

the problem is either (1) actually let people know what to do, (2) remove the need for them to do anything.

(1) gets into the fact that the wiki is poorly linked from the main Octave and download pages. I see a link at the bottom of the main page, and a couple scattered Wiki links to the 'how to use' pages. Nothing to installation notes pages.

(2) Since the zip version's kind of a hack anyway, would the comment #17 proposal make this all a non-discussion? portability, Octave documentation/linking/wiki etc could be another issue for another day.

Alternatively, could mxe-octave add a line of text somewhere within the Octave welcome message (no idea what file that is) saying:
"You are using the Window zip package. On first use and anytime you change the Octave location you will need to run 'pkg rebuild' to access installed packages"

this will guarantee it's there for people to see.

Nicholas Jankowski <nrjank>
Thu 03 Aug 2017 06:34:58 PM UTC, comment #28:

Doug makes an interesting point; the need for a pkg rebuild is less a 'at first launch' issue, but more of a 'whenever you change computer by treating the zip version as a portable octave' issue. Therefore it might make less sense to rely on the presence or absence of an `octave_packages` file, and more sense to simply check (via octaverc) whether `pkg list` is displaying appropriate output at each launch, and attempt a pkg rebuild if it is not, under the assumption that the paths have changed.

On the other hand, it would be reasonable to expect that a user that treats the octave zip-version "portably" in this way, is likely aware of the need to do a pkg rebuild, once they've done it once, and that this is a trivial thing to do so ... therefore to me this is still a documentation-first issue. It's an extremely simple issue to address, once you know what's going on. The biggest obstacle is not knowing what's going on in the first place, and not being aware of the relevant wiki post (as per the original stackoverflow post that triggered this discussion).

I feel the solution is more along the lines of having a simple line saying "you may need to rebuild / upgrade the packages", or a link to the wiki, next to the download link to the zipped version; this could also clarify what Mike is saying at the point of download, e.g. "link to alternative zip version: use this if you're looking for a standalone version that doesn't require admin rights, or to use as a 'portable' distribution (but see point on wiki regarding portability)". I think the fact that the download link in the main website simply redirects to a dry webpage of ftp links instead, devoid of any helpful instructions, is more relevant to the problem than an octaverc fix, however nice a touch the latter would be.

Tasos Papastylianou <tpapastylianou>
Thu 03 Aug 2017 06:11:23 PM UTC, comment #27:

Yes, Portable is a separate issue and I guess we are getting sidetracked. Doug's comment #26 describes a Portable distribution as it's usually intended. Octave is close to being so, but yes, currently Mike's comment #25 description of the zip usage is probably more correct.

What options are there for a 'run once' set of commands that would re-run after a 'reinstall' or 'upgrade' of the zip package? There is a

C:\Octave\octave-4.2.1\share\octave\site\m\once_only

folder. What are the checks/conditions to execute things in that folder?

I know that on a new machine I get the 'this is your first time running Octave' dialogue since (i assume) it doesn't see anything in my User folder, but it only ever happens once on the machine. If I delete Octave and restore from the zip, or put a new version on, it doesn't create that dialogue again.

Nicholas Jankowski <nrjank>
Thu 03 Aug 2017 04:50:53 PM UTC, comment #26:

Before I retired I had octave installed on a memory stick, and I carried it with me to different rooms. I then inserted the memory stick in whatever computer was in the room. I found that I had to do a rebuild so that the drive letter would work in that room. Each computer seemed to assign a different drive letter to the usb slot that I plugged the memory stick into. I don't know if this is the same issue but it might be related, and I don't know how many people would still be doing it this way.

Doug Stewart <dastew>
Thu 03 Aug 2017 03:57:01 PM UTC, comment #25:

To my mind, the zip distribution of Octave is an archive, that's all. Calling it a portable distribution is a bit disingenuous.

I haven't heard much call from users over the years for a truly portable version of Octave. If that's something that is now desired, maybe we should tackle the sub-issues in separate bug reports. For now, Octave definitely does need to know its own root directory (which can be overridden by the OCTAVE_HOME environment variable for many functions), and the pkg installation prefixes and cache file locations (which are not corrected to OCTAVE_HOME because they can be separately configured to be anywhere on the filesystem).

I thought the zip version was intended to be a workaround for not having admin privileges on a Windows system. Unpack the archive in a specific directory that the user does have privileges on and then run post-installation commands that a proper packaging format would normally run.

Mike Miller <mtmiller>
Project Administrator
Thu 03 Aug 2017 03:49:38 PM UTC, comment #24:

Why would relative vs absolute paths even be an issue here?
All that would need to be done is to modify octaverc in the prepackaged windows zip version specifically such that it runs a simple pkg rebuild command at first launch (as per the one-liner below).
Are we getting sidetracked here, or is there a legitimate reason this would not be the preferred approach?

Also, regardless, it may still be worth treating this issue partly as a documentation issue, and simply make it more visible to a user at first launch; while any programmatic fix might address the `rebuild` issue, the user might still want to be informed that it may be worth to `update` their pkg database too, as per the wiki. And you certainly wouldn't want to automatically trigger an update programmatically at first launch.

Tasos Papastylianou <tpapastylianou>
Thu 03 Aug 2017 01:39:09 PM UTC, comment #23:

re: comment #20: that was my original thought, as I was messing with in comment #7 and comment #9. I had hoped that pkg would process the strings and expand out OCTAVE_HOME(), so that the only thing to do was store all the bundled packages with either relative path names or names relative to OCTAVE_HOME. then no pkg rebuild would be required for either zip or exe. However, it appears I was mislead by how pkg displays the path and actually stores and uses the path, and haven't looked at it further.

re: comment #21: many people consider the zip versions of programs to be 'portable' versions. run from flash drive, etc. Octave zip is not actually a portable version since it relies on absolute paths and makes use of the user folder. Having everything able to work relative to OCTAVE_HOME, which the zip version successfully finds on startup, would be step 1 to making it actually portable in addition to solving the pkg rebuild issue. I guess 'relative to octave_home' is still an absolute path in a sense, depending on where the full path expansion occurs.

getting sidetracked into how to 'auto rebuild' as per comments #10-18 I stopped short of diving into pkg to see where it gets locked into absolute paths. First thought would be to see if there are places you could parse and 'eval' the string, but I know some people dislike eval in general.

jwe: (re: comment #19 re: comment #6) hah... brilliant ... let's just pretend I never made that comment.

Nicholas Jankowski <nrjank>
Thu 03 Aug 2017 12:44:17 PM UTC, comment #22:

for the building of octave_packages file,

cd $(HOST_PREFIX)
OCTAVE_HOME=. octave-cli --eval 'pkg rebuild -global'

appears to create the octave_packages file, with paths now as ./share/octave/packages/commuications-1.2.1 and ./lib/octave/packages/communications-1.2.1, so if the package database worked with names relative to OCTAVE_HOME, I think that would work.

John Donoghue <lostbard>
Project Member
Thu 03 Aug 2017 12:10:40 PM UTC, comment #21:

Or, if there are actually legitimate uses for relative directory names (are there?) then we could extend the syntax of the file to somehow tag names that are relative to OCTAVE_HOME. However, I'm not sure that we want to use $(OCTAVE_HOME) or similar since no other parts of Octave expand variables in path or directory names in this way.

John W. Eaton <jwe>
Project Administrator
Thu 03 Aug 2017 12:08:22 PM UTC, comment #20:

It seems to me that we should just make the package database work with relative directory names so that any directory name that is stored in the file and that is not absolute is assumed to be relative to OCTAVE_HOME.

John W. Eaton <jwe>
Project Administrator
Thu 03 Aug 2017 12:05:46 PM UTC, comment #19:

Nicholas: Regarding your statement in comment #6 that

works, it only "works" because % is a comment character, so this expression is equivalent to calling "cd" without any arguments, which, in Octave, changes to the home directory.

John W. Eaton <jwe>
Project Administrator
Wed 02 Aug 2017 09:10:32 PM UTC, comment #18:

I guess if there's a desire to remove those two lines from the octaverc after it's done could add:

after the zip line. (i'm sure there is a 'delete two lines' syntax for sed, but that's the limit of my knowledge)

Nicholas Jankowski <nrjank>
Wed 02 Aug 2017 09:00:43 PM UTC, comment #17:

maintainers list discussion [1] advises trying to just make the change to the windows zip-distribution if possible.

I think this could be done changing the following in binary-dist-rules.mk in the 'define make-zip-dist' section (near line 194):

can anyone who actually knows how the makefile/building process works comment on whether that's right?

[1] http://octave.1599824.n4.nabble.com/any-impacts-of-adding-this-line-to-default-octaverc-tp4684287.html

Nicholas Jankowski <nrjank>
Wed 02 Aug 2017 06:17:10 PM UTC, comment #16:

comment on the maintainers list mentioned that the code would throw an error every time Octave started if it was on a system where the user didn't have write access to the normal global_list location. Suggested using a try/catch to hide the error.

or the ultracompact 1 line version:

looks like this could just be added to the octaverc sitting in mxe-octave. But that would force this change on everyone to fix a problem only occurring in the Windows zip-distribution. Maybe a line could be added to the binary-dist-rules.mk file to append that line(s) to the octaverc file prior to zipping? Or maybe there's a better way to accomplish that?

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 08:31:20 PM UTC, comment #15:

a couple options that would work:

I'm sure there are some others. the latter would cover it if that file designation ever changes.

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 08:21:37 PM UTC, comment #14:

is there even a time running a pkg rebuild, when global octave_packages is missing, would break something?

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 08:19:46 PM UTC, comment #13:

even better, you could check if the file referenced by pkg global_list exists; if not rebuild.

Tasos Papastylianou <tpapastylianou>
Tue 01 Aug 2017 08:13:21 PM UTC, comment #12:

sorry, missed your comment #8, John. I was intermittently changing both dir and archprefix, mainly because I wasn't sure what archprefix did.

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 07:58:34 PM UTC, comment #11:

I had thought adding a simple find/replace on octave_packages before zipping would have been the simplest fix. Looks like not.

'on first launch' is still a bit of the problem. When 4.3.0 comes out, I'm pretty sure it won't see it as first launch on my machine since I've run Octave before and my profile has the right files in place. Same if I delete and 'reinstall'. It'll only run 'first launch' the first time any version of Octave is run on the machine. we want something that detects when this copy hasn't been run yet.

octave_packages is missing on first run. would putting a

or similar in .octaverc break anything?

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 07:18:06 PM UTC, comment #10:

A simpler approach might be to just include the pkg rebuild command in the octaverc that gets executed on first launch. (I have checked this and it works, and it would be a simple bugfix). If there's a way to detect this is the first launch via some sort of flag, and only launch this once, even better; I've noticed there's a separate "octave-fistlaunch.vbs" script in the root folder: perhaps an environment variable can be set that is checked for within octaverc?

Whether there should be some form of documentation to inform of the potential need for a pkg update as well is a slightly separate issue.

Tasos Papastylianou <tpapastylianou>
Tue 01 Aug 2017 05:32:37 PM UTC, comment #9:

ok. must be more to this.

but put within the octave_packages, it produces:

trying the long version of fullfile:

but using that in octave_packages:

it doesn't check for a valid path:

C:\Octave\OCTAVE~1.1\share\octave\packages123\signal-1.3.2

but if I make it something arbitrarily short:

C:\Octave\OCTAVE~1.1\share\signal-1.3.2

so... haven't yet figured out how to be sure it's actually storing/reading the correct folder, and whether the truncation is just a display thing or if it's actually stored incorrectly.

so, not quite sure it'll work, and something about how octave_packages is read seems to 'sometimes' truncate things. eg:

despite the location being the same in the text file.

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 05:18:28 PM UTC, comment #8:

I think you changed the wrong path - archprefix, instead of dir.

There is a comment #length that provides info on the length of the value, which has changed as you added the fullpath in.

John Donoghue <lostbard>
Project Member
Tue 01 Aug 2017 05:03:35 PM UTC, comment #7:

interesting find: octave_packages DOES work if you put a fullfile() function in for path.

replacing

C:\Octave\OCTAVE~1.1\lib\octave\packages\signal-1.3.2

with

fullfile(OCTAVE_HOME (), "\share\octave\packages\signal-1.3.2")

allows pkg list to read octave_packages. however, it doesn't quite work perfectly.

not sure why it's truncating the -1.3.2 from the path.

fixing that, this should be a straightforward way to have the portable one find the packages right away taking advantage of the environment variables set on octave startup to avoid needing to hardcode package links.

honestly, this would be an interesting way to store the global package paths anyway if we wanted to make a truly portable octave version.

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 02:20:11 PM UTC, comment #6:

well, it's the syntax that normally works at the Octave interpreter in windows (cd $OCTAVE_HOME gives an error, but cd %OCTAVE_HOME% works). Was hopeful it would be transparent to pkg, but no luck. Would have made this a trivial fix. maybe add it to the ever longer 'pkg improvement wishlist' absent any other workaround...

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 02:07:31 PM UTC, comment #5:

That's DOS environment syntax - $OCTAVE_HOME would be better, however pkg does not expand env variables

John Donoghue <lostbard>
Project Member
Tue 01 Aug 2017 01:47:36 PM UTC, comment #4:

just to answer my own question: no, currently octave_packages cannot handle variables in paths.

Took my octave_packages file, replaced the absolute paths for the signal package from

C:\Octave\OCTAVE~1.1\lib\octave\packages\signal-1.3.2

to

%OCT_HOME%lib\octave\packages\signal-1.3.2

It renders the entire file unreadable to a 'pkg list'. i.e.:

pkg list:

manually change paths in signal section of octave_packages to:

and get:

change it back to "C:\Octave\Octave~1.1..." and 'pkg list' returns to normal.

so something about that syntax prevents pkg from parsing the entire file. (also tested by changing the path to signal and going a 'pkg load control' and 'pkg load video', each listed before and after signal in the file, respectively. pkg returned a 'package control not installed' until I removed the %OCT_HOME% from signal.)

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 01:17:05 PM UTC, comment #3:

Yes, but I guess what I meant was dp0 is used to grab the program location on startup and store it in an environment variable:

set OCT_HOME=%~dp0

Could OCT_HOME be used within the installed octave packages file so that the rebuild command would not need to be run?

Nicholas Jankowski <nrjank>
Tue 01 Aug 2017 12:38:01 PM UTC, comment #2:

~dp0 is used by the .bat file, which gets the real path before it is ever passed to octave, so currently octave doesn't know what ~dp0 is.

John Donoghue <lostbard>
Project Member
Mon 31 Jul 2017 08:59:13 PM UTC, comment #1:

a good question: do the paths 'need' to be hardcoded and absolute? the zip version already uses ~dp0 to find the local path at startup. could the zip packgage have a manually created package list with entries using that nomenclature? All subsequent pkg installs would get absolute paths, and so would a pkg rebuild. Would it work?

Nicholas Jankowski <nrjank>
Mon 31 Jul 2017 06:57:31 PM UTC, original submission:

As per this StackOverflow post: https://stackoverflow.com/q/45399494/4183191

The need to run pkg rebuild (plus pkg update) in order for the preinstalled packages to become accessible in the zipped version, while documented somewhere on the wiki, it is highly unlikely that a new user would know this in advance, and is likely to be a source of frustration and "first papercut" to the very first experience of installing octave on windows in that way.

While there are probably good technical reasons to not provide a generic octave_packages file in the portable version (presumably the paths need to be hardcoded and absolute), this information would probably need to be documented in the README file, and / or in the information window that comes up at first launch, informing the new user of this problem, and / or redirecting to the wiki for further instructions at the very least.

Tasos Papastylianou <tpapastylianou>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -unavailable- added by dastew (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by lostbard (Posted a comment)
  • -unavailable- added by nrjank (Posted a comment)
  • -unavailable- added by nrjank
  • -unavailable- added by tpapastylianou (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 15 Aug 2017 04:56:42 PM UTCmtmillerCategoryConfiguration and Build System=>Octave Function
      Severity3 - Normal=>1 - Wish
      Item GroupDocumentation=>Feature Request
      StatusNone=>Confirmed
      SummaryMS Windows portable (i.e. zip) version cannot find pre-installed packages, needs pkg rebuild, not mentioned at first run=>make global pkg paths always relative to OCTAVE_HOME, no need to pkg rebuild -global when path changes
    Tue 08 Aug 2017 08:44:45 PM UTCmtmillerDependencies-=>bugs #51684 is dependent
    Mon 31 Jul 2017 08:59:13 PM UTCnrjankCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup1