@Rik: Thank you for overhauling the "is_same_file" function.
I noticed that you removed the condition (! is_pc ()) from the test:

%!assert (is_same_file ("~", tilde_expand ("~")))

This test fails on Windows. I think this is because "~" is not an alias for the home directory on Windows. It is perfectly fine to create a file or directory with that name.

@Markus: It would be great if Octave could deal with spaces and any other odd characters throughout the interpreter.  However, there have been problems in the past with installing Octave in to a directory with spaces and also with accessing external commands, like ghostscript, when the path to the external command contains spaces.  You can use the Savannah Bug Search interface to search for "path spaces" and come up with a list of reports.

At least this one, bug #53124, is still open.  The others have been closed because we have worked around the issues on Windows by requiring that Octave be installed in a path without spaces.  Relaxing that requirement would be nice, but is not essential.

@Markus: Thank you for the new is_same_file function.  I made a few changes I think improved the implementation in this cset https://hg.savannah.gnu.org/hgweb/octave/rev/24618a510749.

New patch attached.

Tested on Linux, Windows and with mxe-octave installer (that will create the global octave-packages file during installation). Listing, uninstalling, installing packages & rebuilding package lists work OK now.

I've put if "(ispc)" blocks around the relevant statements (calls to fixup_paths.m) in pkg.m, install.m and uninstall.m as pkg.m wasn't affected on Linux. W/o them it still works on Linux, so maybe the if's can be dropped.

@Markus, comment #15:
AFAIK there are still some issue when Octave is installed in a path containing spaces in the name, but IIRC that related to invoking command line functions through "system ()" or eval/feval calls.
The mxe-octave NSIS-installer as it stands won't let you install in a path containing spaces.

(file #47751)

I got my idea to work beautifully now for an existing Window installation.
I just added this private function:

## <texinfo>
function pkg_list = fixup_paths (pkg_list)

  for ii=1:numel (pkg_list)
    pkg_list{ii}.dir = canonicalize_file_name (pkg_list{ii}.dir);
    pkg_list{ii}.archprefix = canonicalize_file_name (pkg_list{ii}.archprefix);
  endfor

endfunction

and added calls to it in pkg.m (rebuild clause in switch), install.m and uninstall.m.

Next:
- testing on linux
- testing installation of a crossbuild. That doesn't need all that build time, I know a few tricks :-) (Besides I can run the mxe "installation" in mxe-octave/dist/ on Linux directly from Windows.

I looked at dir_in_loadpath but that isn't going to work for installing / uninstalling packages (the 2nd part of this extended bug report).

Thx Markus, turning it into an easy reloctable .oct file is a Good Idea.

BTW I'm entering relatively long posts here to keep the discussion going in order to arrive at an optimal solution. I hope no one is offended or irritated.

is_same_file() won't easily fix the original problem ("pkg list" showing no loaded packages) as the pertinent if conditions are not about comparing files but rather about finding path name strings in a long text constructed by path().

Of course I can do s/th like (here for installed_packages.m):

:
tmppath = strsplit (path (); pathsep);
## Find which packages are loaded and thus appear in the path
for i = 1:numel (installed_pkgs_lst)
  if (any (cellfun (@(x) is_same_file (x, installed_pkgs_lst{i}.dir), tmppath, "uni", 0)))
    ....

(yet untested, but ripe for vectorizing) however similar constructs need to be added in several other places in pkg.m and private/ functions.
I'm hoping that by just choosing one path name style (i.e., LFN) in the octave_packages files the existing code can be kept as-is (with the exception of the strrep()s for replacing fileseps) and more intensive surgery can be avoided.
IOW presently I'm believe more in Rik's philosophy :-)

But we'll see what the best approach is.

AFAICS is_same_file() could be a very useful addition to the user functions. I wonder if it works with symlinks on *nix and (quite relevant but often overlooked) NTFS "joins" and SUBST etc. on Windows.

@Markus,

_It sounds like you already identified all points in pkg that need changing_

If only that were true ...

pkg.m and its functions are a beast. It's all every complicated and big and the path names pop up in various places.
I tried to identify all places where a package directory is defined. private/install.m is an obvious candidate but als for "pkg rebuild" in pkg.m itself some fixes are due.

The underlying issue is that mxe-octave builds are configured with 8.3 style path names for OCTAVE_HOME and various prefixes. The reason is avoiding problems when Octave is installed in a path containing spaces in subdir names.
OTOH, AFAICS pkg.m works fine with LFN path names.

My current idea is to do this:
1. grep for 'save (global_list' and 'save (local_list' in share octave/packages/pkg/
2. before each occurrence, add in a call to a new private function
3. create that function which does nothing more than invoke canonicalize_file_name on all path names in in each item of the package struct and replace those path names.

That solution won't interfere with why the 8.3 style path names were configured in the first place, and users can see "real" paths when typing "pkg list". In addition fileseps won't need to be adapted before comparison at various places in pkg.m and friends.

@Markus:
is_same_file could be a solution.
Does it also work for paths rather than files? (I guess so).

I'll first see if I can go along the lines sketched above. I agree with Rik that we'd better use one type of file/path name and I interpret it as hiding 8.3 filenames for user functions.
An mxe crossbuild, needed to get is-same_file working, takes > 2 hours here so is_same_file will have to wait a little anyway.

Later on we can decide what the best option would be.

@Rik:
I think the cset for bug #57267 you mentioned in comment #3 has had a few more consequences on Windows than just "pkg list". It turns out "pkg unload" won't work either for packages (global or local) installed in paths with 8.3 style filenames in pkg prefix.

As to "pkg unload" and "pkg list", a crossbuild from Oct. 8 works fine, one from Oct. 17 fails. In between Markus' patches for bug #57267 were pushed.

Some investigation shows that the patch for this bug report has nothing to do with "pkg unload" failing; the patch itself is just fine and harmless, it doesn't change the return arguments of private/installed_packages.m either. But it's just a cosmetic fix.
"pkg unload" and "pkg list" failing are mere manifestations of a deeper lying bug here.

At the bottom of this is that pkg.m on Windows stores 8.3 style filenames in the octave_packages files and for any action compares these 8.3 style path names to the LFN path names that are now in the path. So I tend to think that a more profound fix for pkg.m is required.

AFAICS Octave happily loads .m, .oct and .mex files from paths containing spaces in subdir names. A fix might boil down to simply writing LFN file names (by invoking canonicalize-file_name) to octave_packages files as that's where pkg.m relies on for all its actions.
I experimented with manually adapting all 8.3 tyle path names in octave_packages to LFN ad that seems to solve all problems with pkg.m due to the fix for bug #57267.

I'll try to prepare a better cset.

It may be more of a project than anticipated, but I'm sure that the software design should only use unique paths in comparisons.  Until that is done we are going to keep having problems where a choice of representation leads to a miscompare.

Patch attached. Tested on Linux and Windows.

As regards Mike's comment, the last hunk changes 8.3 pathnames into full (LFN) path names in pkg.m's output as well (see verbatim blocks in comment #6). That is not what Mike meant but it may make the bug he mentioned more prominent, esp. on Windows.
Personally I find the LFN paths much more meaningful than the 8.3 style paths. The latter only fullfil internal Octave tasks (i.e., make Octave stuff work when it is installed in a path containing spaces in the name) but are meaningless for many Windows users.

If only the actual bug fix is desired that last hunk can be dropped.

(file #47741)

right, I already got it working on Windows.
pkg list now shows (w. fixed Octave):

>> pkg list
Package Name         | Version | Installation directory
---------------------+---------+-----------------------
     communications  |   1.2.1 | ...\mingw64\share\octave\packages\communications-1.2.1
            control *|   3.2.0 | ...\Octave-6.0.0_20191017L\mingw64\share\octave\packages\control-3.2.0
     data-smoothing  |   1.3.0 | ...\mingw64\share\octave\packages\data-smoothing-1.3.0
:

rather than (w. Octave from before bug):

>> pkg list
Package Name         | Version | Installation directory
---------------------+---------+-----------------------
     communications  |   1.2.1 | ...\OCTAVE~2.0_2\mingw64\share\octave\packages\communications-1.2.1
            control *|   3.2.0 | ...\Octave\OCTAVE~2.0_2\mingw64\share\octave\packages\control-3.2.0
     data-smoothing  |   1.3.0 | ...\OCTAVE~2.0_2\mingw64\share\octave\packages\data-smoothing-1.3.0
:

I need to check the fix on Linux first, somewhere next weekend I'll have a cset.

While there, I find the listing of package installation subdirs not very enlightening. Maybe pkg should first mention Octave's install directory (OCTAVE_HOME) before mentioning any package and/or show global package dirs relative to OCTAVE_HOME or so.
But that's for another bug report.

Your hunch is a good one.

For e.g., geometry I get in the stanza you mention:

tmppath = ....;C:/Programs/Octave/Octave-6.0.0_20191017L/mingw64/share/octave/packages/geometry-4.0.0;...

while the string it is compared with looks like:

debug> strrep (installed_pkgs_lst{i}.dir, '\', '/')
ans = C:/Programs/Octave/OCTAVE~4.0_2/mingw64/share/octave/packages/geometry-4.0.0

but:

debug> strrep (canonicalize_file_name (installed_pkgs_lst{i}.dir), '\', '/')
ans = C:/Programs/Octave/Octave-6.0.0_20191017L/mingw64/share/octave/packages/geometry-4.0.0

gets me the right string.

Looking at it from a little distance I get the impression that the strrep() statements to morph the Windows fileseps to forward slashes serves no purpose anymore here. But the short pathnames (exclusively for Windows, the ones with "...\OCTAVE~4.0_2\..." rather than "...\Octave-6.0.0_20191017L\...") occur elsewhere in the path() as well and appear to clobber up.

Perhaps the way to go here is to use canonicalize_file_name() to reconstruct the (on Windows) native long file names (LFN) everywhere in the private/installed_packages.m function, keep the fileseps as-is and only then compare.
IOW something along the lines of

strfind (path, canonicalize_file_name (installed_pkgs_lst{i}.dir))

(Of course I already see a few gotchas but those can be tackled.)
AFAIU "pkg list" isn't a time-critical function so a little more processing time, if any, won't hurt.

IMO another benefit is that "pkg list" would then return installed package pathnames that are much more understandable to users then the short path names that are returned currently (compare the second verbatim block with the first or third).

What do you think?
If you agree I'll try to come up with a cset along these lines

Makes no difference.
I just made a new build, after I manually wiped all Octave build files from the "install" dir (mingw64/) in the mxe-octave build tree and tried again.

I have no mxe builds between 8 and 17 Oct so it is a bit hard to "bisect".
The installed versions (8, 18 and 23 Oct) all use the same user .octaverc and the one from 8 Oct works fine with it so it must be in Octave itself.

Since a few days, when listing the OF packages with "pkg list", loaded global packages aren't shown with an "*" asterisk.
The last installed version where it worked fine was a tip from Oct. 8.

That some globally installed packages are loaded is easily verified using "which <some_loaded_package_function>"

:
              nurbs  |  1.3.13 | ...\mingw64\share\octave\packages\nurbs-1.3.13
                ocs  |   0.1.5 | ...\mingw64\share\octave\packages\ocs-0.1.5
            octproj *|   1.1.5 | C:\Users\philip\octave\octproj-1.1.5
             odepkg  |   0.8.5 | ...\mingw64\share\octave\packages\odepkg-0.8.5
             optics  |   0.1.4 | ...\mingw64\share\octave\packages\optics-0.1.4
              optim  |   1.6.0 | ...\mingw64\share\octave\packages\optim-1.6.0
:
:
              video  |   1.2.4 | ...\mingw64\share\octave\packages\video-1.2.4
            windows  |   1.3.1 | ...\mingw64\share\octave\packages\windows-1.3.1
             zeromq  |   1.5.0 | ...\mingw64\share\octave\packages\zeromq-1.5.0
>> which actxserver
'actxserver' is a function from the file C:\Programs\Octave\Octave-6.0.0_20191018L\mingw64\lib\octave\packages\windows-1.3.1\x86_64-w64-mingw32-api-v53+\__COM__.oct
>>

Formally I think it's a regression but this bug is merely cosmetic.