bugGNU Octave - Bugs: bug #62847, addpath incorrectly processes...

 
 

bug #62847: addpath incorrectly processes symlinks on mapped network drives

Submitter:  Philip Nienhuis <philipnienhuis>
Submitted:  Mon 01 Aug 2022 10:14:15 AM UTC
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  Philip Nienhuis Open/Closed:  * Closed
Release:  * 7.2.0 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       

 

( Jump to the original submission )

Fri 02 Sep 2022 05:45:12 AM UTC, comment #62: 

Looks like we are there (finally). ­čÄë

Regarding the output of `ls`: `\` and `/` are both valid file separators on Windows afaict. I'm not sure if using `/` can be a problem somewhere. Anyway, if it is, we should handle it in a different bug report.

Closing this one as fixed (finally after 62 comments). Thanks everyone for testing this in multiple different environments.

Markus M├╝tzel <mmuetzel>
Project Member
Thu 01 Sep 2022 08:25:31 PM UTC, comment #61: 

ok, much improved!

>> cd R:
>> [c,s,m] = canonicalize_file_name ('current_backup_20210803_1550.pst')
c = \\full.server.name\PSTStorage\PST_Storage_nicholas.jankowski\current_backup_20210803_1550.pst
s = 0
m =

>> ls(c)
//full.server.name/PSTStorage/PST_Storage_nicholas.jankowski/current_backup_20210803_1550.pst

>> [c,s,m] = canonicalize_file_name ('R:\current_backup_20210803_1550.pst')
c = R:\current_backup_20210803_1550.pst
s = 0
m =

>> ls(c)
R:/current_backup_20210803_1550.pst

>> cd temp

>> [c,s,m] = canonicalize_file_name ('current_backup_20210803_1550.pst')
c = \\full.server.name\PSTStorage\PST_Storage_nicholas.jankowski\temp\current_backup_20210803_1550.pst
s = 0
m =

>> ls(c)
//full.server.name/PSTStorage/PST_Storage_nicholas.jankowski/temp/current_backup_20210803_1550.pst

>> [c,s,m] = canonicalize_file_name ('R:\temp\current_backup_20210803_1550.pst')
c = R:\temp\current_backup_20210803_1550.pst
s = 0
m =

>> ls(c)
R:/temp/current_backup_20210803_1550.pst

I'd call that fixed for me.  I don't know if the change from \ to / in the ls output is any concern.

Nicholas Jankowski <nrjank>
Project Member
Thu 01 Sep 2022 03:58:04 PM UTC, comment #60: 

simplest test on my end seems to simply be:

- cd to root of mapped network drive, e.g., R:

c = canonicalize_file_name ('some_file')
ls(c)

c = canonicalize_file_name ('R:\some_file')
ls(c)

- cd to any subfolder and repeat:

c = canonicalize_file_name ('some_file')
ls(c)

c = canonicalize_file_name ('R:\folder\some_file')
ls(c)

the c's will either be correct or not, and if the ls command didn't return a file not found error, most likely it was correct.

i'm downloading the sep 1 build now (hg id: 5b021ecc8bfe) that should include Markus's comment #56 change and will report back

Nicholas Jankowski <nrjank>
Project Member
Thu 01 Sep 2022 06:48:02 AM UTC, comment #59: 

AFAICS (and I tried a few things) the patch still solves my original issue.

I didn't pursue things like Nick mentioned, simply due to lack of time at work, sorry. My work with Octave involves a lot of file I/O on various network drives and apart from the path issues in comment #0, to this day I didn't hit other issues (yet?).

Philip Nienhuis <philipnienhuis>
Project Member
Wed 31 Aug 2022 06:15:24 AM UTC, comment #58: 

@Philip: More tests definitely wouldn't hurt. As we have seen, there are many subtleties that can make the difference between success and failure here...

Markus M├╝tzel <mmuetzel>
Project Member
Tue 30 Aug 2022 08:50:12 PM UTC, comment #57: 

Should I check if it still works for me at work? Hopefully I can get to it this week.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 30 Aug 2022 04:28:23 PM UTC, comment #56: 

Thanks for testing.

The UNC path corresponding to the mapped network drive doesn't end with a file separator for Philip or me (see comment #14 and comment #15). But it looks like it does for you (see comment #51).

I pushed a change here that removes any trailing file separators from the UNC path that corresponds to the mapped network drive:
https://hg.savannah.gnu.org/hgweb/octave/rev/f111d1caa73d

I hope that way we don't need to change any of the subsequent logic.

Could you please check with a nightly build from tomorrow or later if this fixes the remaining issue for you?

Markus M├╝tzel <mmuetzel>
Project Member
Tue 30 Aug 2022 02:36:52 PM UTC, comment #55: 

using the Aug 30 build, improved but still not quite there. drops the \ after the drive letter.

GNU Octave Version: 7.2.1 (hg id: 9ad55d2e1bbf)

>> cd R:
>> [c,s,m] = canonicalize_file_name ('current_backup_20210803_1550.pst')
c = \\full.server.name\PSTStorage\PST_Storage_nicholas.jankowski\current_backup_20210803_1550.pst
s = 0
m =

>> ls(c)
//full.server.name/PSTStorage/PST_Storage_nicholas.jankowski/current_backup_20210803_1550.pst

>> [c,s,m] = canonicalize_file_name ('R:\current_backup_20210803_1550.pst')
c = R:current_backup_20210803_1550.pst
s = 0
m =

>> ls(c)
ls: cannot access 'R:current_backup_20210803_1550.pst': No such file or directory

>> cd temp

>> [c,s,m] = canonicalize_file_name ('current_backup_20210803_1550.pst')
c = \\full.server.name\PSTStorage\PST_Storage_nicholas.jankowski\temp\current_backup_20210803_1550.pst
s = 0
m =

>> ls(c)
//full.server.name/PSTStorage/PST_Storage_nicholas.jankowski/temp/current_backup_20210803_1550.pst

>> [c,s,m] = canonicalize_file_name ('R:\temp\current_backup_20210803_1550.pst')
c = R:temp\current_backup_20210803_1550.pst
s = 0
m =

>> ls(c)
ls: cannot access 'R:temp/current_backup_20210803_1550.pst': No such file or directory

Nicholas Jankowski <nrjank>
Project Member
Tue 30 Aug 2022 09:58:43 AM UTC, comment #54: 

@Nik: Did the recent change fix it for you? Could you please test with a nightly build from Kai's buildbots?

Markus M├╝tzel <mmuetzel>
Project Member
Tue 23 Aug 2022 05:48:02 PM UTC, comment #53: 

Forgot that 'R:' corresponds to the current folder of that drive (not necessarily the root of that drive). I pushed yet another change here that makes sure to get the canonicalized path of the root of a potential mapped network drive:
https://hg.savannah.gnu.org/hgweb/octave/rev/4c38cf0ce06c

I hope we are slowly getting closer to a stable solution for this...

Markus M├╝tzel <mmuetzel>
Project Member
Tue 23 Aug 2022 02:49:29 PM UTC, comment #52: 

bumping, as last test occurred during savannah email outage.  testing with latest patch didn't solve issues.

Nicholas Jankowski <nrjank>
Project Member
Mon 15 Aug 2022 06:45:39 PM UTC, comment #51: 

latest build that just finished today (hg id: 820d2c802247) I believe has the comment #49 mentioned change, but still causes path problems, especially with subfolders:

>> cd R:
>> canonicalize_file_name('.')
ans = \\full.server.name\PSTStorage\PST_Storage_nicholas.jankowski\
>> canonicalize_file_name('R:')
ans = R:
>> canonicalize_file_name('R:\')
ans = R:
>> canonicalize_file_name('R:\current_backup_20210803_1550.pst')
ans = R:current_backup_20210803_1550.pst
>> cd temp
>> canonicalize_file_name('.')
ans = \\full.server.name\PSTStorage\PST_Storage_nicholas.jankowski\temp
>> canonicalize_file_name('R:\temp')
ans = R:
>> canonicalize_file_name('R:\temp\')
ans = R:\
>> canonicalize_file_name('current_backup_20210803_1550.pst')
ans = \\full.server.name\PSTStorage\PST_Storage_nicholas.jankowski\temp\current_backup_20210803_1550.pst
>> canonicalize_file_name('R:\temp\current_backup_20210803_1550.pst')
ans = R:\current_backup_20210803_1550.pst

Nicholas Jankowski <nrjank>
Project Member
Fri 12 Aug 2022 08:51:09 PM UTC, comment #50: 

> ... only network shares can be mapped to a drive letter

There's also the SUBST command. It works differently than 'net use ...' but in the end it produces some path (any path) mapped to a drive letter. I haven't tried if 'subst' w/o an argument lists network drives, maybe it does, too.

> ... doubt if treating mapped drives differently in 'canonicalize_file_name` was a good idea in the first place


Maybe don't overhaul the subject too hastily. It is a bit of a minefield.
My issues have been solved (thanks again), hopefully Nicholas's issues as well.

Philip Nienhuis <philipnienhuis>
Project Member
Fri 12 Aug 2022 02:47:28 PM UTC, comment #49: 

I pushed a change to the stable branch that should allow getting the correct canonicalized path for files on the mapped network drives that point to a directory on a UNC network share:
https://hg.savannah.gnu.org/hgweb/octave/rev/84df79fb637d

That actually allowed to simplify the function a bit.

It should be part of the next nightly build for Windows.

Marking as ready for test again.

Markus M├╝tzel <mmuetzel>
Project Member
Fri 12 Aug 2022 06:51:52 AM UTC, comment #48: 

I was (erroneously) assuming that only network shares can be mapped to a drive letter. But it looks like any directory on a share can be mapped to a drive letter (e.g., `net use X: \\server\share\some\folder\on\the\share` will map that folder as the entry point to the drive letter `X:`).

I'll try to have a look at it again with that in mind.

Given all those complications, I'm starting to doubt if treating mapped drives differently in `canonicalize_file_name` was a good idea in the first place.
See: https://hg.savannah.gnu.org/hgweb/octave/rev/b47bf773c508

Maybe, we should just revert that change (and the follow-up changes) and leave it at the UNC path that the Windows API functions return?

Markus M├╝tzel <mmuetzel>
Project Member
Thu 11 Aug 2022 06:22:06 PM UTC, comment #47: 

testing with Octave 7.2.1 (hg id: 51311efef5dd), I still get the same behavior as in comment #30.

>> [c, s, m] = canonicalize_file_name ('current_backup_20210803_1550.pst')
c = \\full.server.name\PSTStorage\PST_Storage_nicholas.jankowski\current_backup_20210803_1550.pst
s = 0
m =
>> [c, s, m] = canonicalize_file_name ('R:\current_backup_20210803_1550.pst')
c = R:\PST_Storage_nicholas.jankowski\current_backup_20210803_1550.pst
s = 0
m =

where the first call above returns a valid path, but the second one does not due to the insertion of the aliased folder name.

Nicholas Jankowski <nrjank>
Project Member
Sun 07 Aug 2022 06:13:44 PM UTC, comment #46: 

> (It's a pity that we only noticed this shortly after 7.2.0 was released. But better late than never...)

Well, I should feel a bit guilty because I noted this bug already several weeks ago.
Because I usually use dev Octave with some extra mods I usually assume issues are my own fault - not in this case BTW. 2nd after that, our IT facilities are somewhat in turmoil and sometimes cause sneaky problems. Therefore, bugs in Octave usually only come 3rd on my "suspicion" list :-) unless they're obvious, and that wasn't quite the case here.
And due to other priorities, sorting it out and promoting this issue to #1 on the list took some time :-)

Philip Nienhuis <philipnienhuis>
Project Member
Sun 07 Aug 2022 08:58:41 AM UTC, comment #45: 

There was a flaw with the patch from comment #31.
I pushed a slightly different patch which (hopefully) avoids that flaw to the stable branch here:
https://hg.savannah.gnu.org/hgweb/octave/rev/51311efef5dd

Marking as ready for test.

Markus M├╝tzel <mmuetzel>
Project Member
Sun 07 Aug 2022 08:44:17 AM UTC, comment #44: 

@jwe: I copied your comment to bug #62865 where we could discuss potential changes in the behavior of the load path functions.
In this report, I'd like to concentrate on the actual error in Octave 7.2.0 when adding a path from a mapped network share that contains a symbolic link to a different share.

(It's a pity that we only noticed this shortly after 7.2.0 was released. But better late than never...)

Markus M├╝tzel <mmuetzel>
Project Member
Sat 06 Aug 2022 03:33:19 PM UTC, comment #43: 

My point wasn't specifically about Unix-style symbolic links, but that if there can be more than one name for a directory (symbolic link, second mount point, whatever) it made sense to me to only store the One True Directory Name as an index for load path searching.  Maybe we should also store the name initially used (the one passed to addpath or other functions) and use that as the one to display to the user?

John W. Eaton <jwe>
Project Administrator
Sat 06 Aug 2022 10:10:58 AM UTC, comment #42: 

The mapped drive itself isn't treated as a symbolic link.
But there is a symbolic link in the path that you feed to `addpath` (which in turn is passed to `canonicalize_file_name`).
Afaict, that symbolic is `Q:\full`. (I.e., what you described as "a shortcut - apparently to '\\<server>\company.local\full$'".)
That symbolic link is resolved by `canonicalize_file_name`. Not the mapped network drive letter itself.

Markus M├╝tzel <mmuetzel>
Project Member
Sat 06 Aug 2022 09:02:45 AM UTC, comment #41: 

> ... it seemed to me that it is only reasonable to have a single name for any directory in the load path and that should be the actual name of the directory, not the name of a symbolic link.


AFAIU a drive mapping (a Windows idiosyncrasy, admittedly) is different than a symbolic link; it's more like a mount. Yet I can understand that it can be perceived as some symbolic link as well.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 06 Aug 2022 07:38:12 AM UTC, comment #40: 

> will you push the cset?


I'd like to do a few more tests before pushing this (maybe today or tomorrow).

Markus M├╝tzel <mmuetzel>
Project Member
Sat 06 Aug 2022 04:56:41 AM UTC, comment #39: 

"IIUC, the paths that are stored in the load path are canonicalized. I don't recall the exact reason for that."

If I remember correctly, the reason was that with symbolic links, a single real directory could have multiple names but it seemed to me that it is only reasonable to have a single name for any directory in the load path and that should be the actual name of the directory, not the name of a symbolic link.

John W. Eaton <jwe>
Project Administrator
Fri 05 Aug 2022 06:53:25 PM UTC, comment #38: 

@Markus: will you push the cset? In that case I wouldn't have to back it out from my local tree.

Philip Nienhuis <philipnienhuis>
Project Member
Fri 05 Aug 2022 06:52:03 PM UTC, comment #37: 

> IIUC, the paths that are stored in the load path are canonicalized. I don't recall the exact reason for that.

IMHO that's something that simply happened along the way. I never found it handy, yet not worth a bug report / feature request. But now that we're here ...

>`canonicalize_file_name` ... should return a UNC path in your case. The behavior in previous versions of Octave was arguably incorrect.

Sure, but this bug started with 'addpath', not 'canonicalize_file_name'. I'd expect the result of 'addpath' (i.e., the search path) to return something that resembles as much as possible what I have fed it :-)

> Does something no longer work because of those UNC paths in the load path?

No no, nothing broke (yet?), but there are several motives, see bug #62865 that I've just entered.

Philip Nienhuis <philipnienhuis>
Project Member
Fri 05 Aug 2022 05:46:52 PM UTC, comment #36: 

clarification to my comment #30:

the v7.2.0 canonicalized filename given in the first examples (just giving the local filename rather than the "R:/..." and returning UNC paths starting with \\) are the correct fully resolved UNC paths to the files.  The only problems were the ones returned that starting with "R:\PST_Storage_nicholas.jankowski...". (haven't been able to try the patch yet)

the UNC paths returned do work for navigation, etc., within octave and windows explorer. the only place they don't work is in the Windows Command Line, which does work (out of the box) with UNC paths ("CMD does not support UNC paths as current directories")[1]. It seems that it works from within Octave CLI, e.g., calling a cd or ls command. I only see the error if passing something to system().  I do not know of any places within octave that might encounter this specific problem (pkg?) mixing system commands with UNC paths, but that is the only concern I can see with UNC path usage.

[1] https://devblogs.microsoft.com/oldnewthing/20070215-05/?p=28003

Nicholas Jankowski <nrjank>
Project Member
Fri 05 Aug 2022 05:09:00 PM UTC, comment #35: 

Thanks for checking!

If I understand the POSIX documentation of `canonicalize_file_name` correctly, a path containing a symbolic link (including to a UNC share) should be returned with that symbolic link resolved. That means that the behavior of `canonicalize_file_name` in Octave 7 is correct (bar the issue addressed in this patch). It should return a UNC path in your case. The behavior in previous versions of Octave was arguably incorrect.

IIUC, the paths that are stored in the load path are canonicalized. I don't recall the exact reason for that.
But these two facts are probably why you are seeing those UNC paths in the load path now.

Does something no longer work because of those UNC paths in the load path? Or is it just for cosmetic reasons that you prefer the non-canonical path with the mapped drive letter over the UNC path?
But you are right: This is probably a better topic for a new bug report. (A first step would be to check why we are canonicalizing paths in the load path currently. Maybe that is not necessary?)

Markus M├╝tzel <mmuetzel>
Project Member
Fri 05 Aug 2022 10:54:04 AM UTC, comment #34: 

Patch works.
Thanks Markus.

But I have a remark: in the output of the 'path' command I see the UNC path (that's also the case with 7.1.0). But in 6.4.0 and before the drive letter path was mentioned, and I find that much more practical and reliable than working with UNC paths.
Another motive is that Matlab (at least r2014a, we still use that for its 'old style' graphics; and IIRC r2018b as well) also echoes drive letter paths, not UNC paths.
After all we're working with drive letters all the time, not UNC paths (that can change unpredictably when ICT staff seems to be in the mood for it). We regularly copy full pathnames using the 'path' command output.

Can this behavior be restored? I'm willing to enter a separate feature request for this.

Philip Nienhuis <philipnienhuis>
Project Member
Fri 05 Aug 2022 06:08:17 AM UTC, comment #33: 

That patch is meant to go on a current head of the repository (without other changeset being backed out).

Markus M├╝tzel <mmuetzel>
Project Member
Thu 04 Aug 2022 06:44:58 PM UTC, comment #32: 

@Markus:
At the stock default Octave tree, or with the the cset mentioned in comment #8 backed out? (I suppose the former).

I'll happily do that, I might report back later tomorrow.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 04 Aug 2022 04:38:19 PM UTC, comment #31: 

Could you please check with the attached patch if this fixes the behavior of `canonicalize_file_name` for you?
It does for the configuration I tried to describe in comment #26. But it's still not clear to me if that is the same issue that you guys are seeing.

(file #53504)

Markus M├╝tzel <mmuetzel>
Project Member
Thu 04 Aug 2022 03:51:46 PM UTC, comment #30: 

FYI on my corporate setup I can confirm similar(?) issues.  We have a archive network store set up as drive R: where I have a few old outlook PST files:

>> cd R:
>> ls
 Jankowski_upto20190101.pst
 current_backup_20210803_1550.pst

i made a subfolder and copied one file into there to test.

>> cd temp
>> ls
current_backup_20210803_1550.pst

here are the results on different octave releases:

in Octave 6.4.0:

>> [c, s, m] = canonicalize_file_name ('current_backup_20210803_1550.pst')
c = R:\current_backup_20210803_1550.pst
s = 0
m =
>> [c, s, m] = canonicalize_file_name ('R:\current_backup_20210803_1550.pst')
c = R:\current_backup_20210803_1550.pst
s = 0
m =
>>cd temp
>> [c, s, m] = canonicalize_file_name ('current_backup_20210803_1550.pst')
c = R:\temp\current_backup_20210803_1550.pst
s = 0
m =
>> [c, s, m] = canonicalize_file_name ('R:\temp\current_backup_20210803_1550.pst')
c = R:\temp\current_backup_20210803_1550.pst
s = 0
m =

Octave 7.2.0:

>> [c, s, m] = canonicalize_file_name ('current_backup_20210803_1550.pst')
c = \\alcnxfs.nae.ds.army.mil\PSTStorage\PST_Storage_nicholas.jankowski\current_backup_20210803_1550.pst
s = 0
m =
>> [c, s, m] = canonicalize_file_name ('R:\current_backup_20210803_1550.pst')
c = R:\PST_Storage_nicholas.jankowski\current_backup_20210803_1550.pst
s = 0
m =
>> cd temp
>> ls
current_backup_20210803_1550.pst
>> [c, s, m] = canonicalize_file_name ('current_backup_20210803_1550.pst')
c = \\top.level.server.name\PSTStorage\PST_Storage_nicholas.jankowski\temp\current_backup_20210803_1550.pst
s = 0
m =
>> [c, s, m] = canonicalize_file_name ('R:\temp\current_backup_20210803_1550.pst')
c = R:\PST_Storage_nicholas.jankowski\temp\current_backup_20210803_1550.pst
s = 0
m =

note that none of the forms returned by octave 7.2.0 are valid.

>> ls(c)
ls: cannot access 'R:/PST_Storage_nicholas.jankowski/temp/current_backup_20210803_1550.pst': No such file or directory

Nicholas Jankowski <nrjank>
Project Member
Thu 04 Aug 2022 03:03:40 PM UTC, comment #29: 

Reading comment #26, I'm unsure if we see the same thing. But maybe they are different manifestations of the same deeper lying issue.

FWIW, the drive mapping script that I run about every year does something along the lines of mapping e.g., some drive letter 'X:' to
'\\company.local\SomeDir$'
and in the end, in Explorer I see an entry along the lines of
 > SomeDir (X:)
where the '$' character has disappeared. In the drive properties' DFS tab the original UNC path is shown but with a server name prepended to it.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 04 Aug 2022 02:44:39 PM UTC, comment #28: 

Forgot to mention, this is with 7.2.0

Philip Nienhuis <philipnienhuis>
Project Member
Thu 04 Aug 2022 02:43:49 PM UTC, comment #27: 

(couldn't resist to try, though I'm a bit swamped in work)

AFAICS, if the network subdir where opt_lbm.m lives is the active directory on network drive letter Q, but not yet in the path:
when inside that subdir:

  • canonicalize_file_name returns a UNC path if I feed it just the file name 'opt_lvm.m') as input
  • if its input is 'Q:opt_lvm' (drive letter filename), it returns a path with a drive letter (no UNC path)
  • same if I feed it the full (non-UNC) path

__wglob__ and 'dir' always return a non-UNC path

When that network subdir IS in the path, from that network subdir or elsewhere:

  • which always returns a UNC path
  • canonicalize_file_name and __wglob__ return what I feed them, UNC path or drive letter-path

It makes no difference whether there's a UNC or 'drive letter' path in the subdir box in the file browser and main menu.

I hope I have described it adequately. There are myriads of options: in the path or not, UNC or drive letter path in browser pane, in or outside pertinent network subdir, network subdir active directory on the network drive or not, ... you name it. Plus I'm getting a bit hazy after a long day behind the screen.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 04 Aug 2022 12:57:44 PM UTC, comment #26: 

I think I'm able to reproduce the issue with this information now.
On the server side within a shared folder:

mklink /D test_C \\Markus-PC\C$

Additionally, I had to execute this on the client side or the symlink couldn't be followed:

fsutil behavior set symlinkevaluation R2R:1

When I map the share on the client to Z:, the following happens with `canonicalize_file_name` in Octave:

canonicalize_file_name ('Z:\test_C\Octave')
ans = Z:\Octave

The first part of the path got "eaten". Maybe that is what is causing the issues for you...

I'd guess the correct behavior according to the POSIX documentation would be for this to return `\\Markus-PC\C$\Octave`.

Markus M├╝tzel <mmuetzel>
Project Member
Thu 04 Aug 2022 12:47:18 PM UTC, comment #25: 

Afaict, Octave's `canonicalize_file_name` should behave the same as the POSIX function of the same name:
https://man7.org/linux/man-pages/man3/canonicalize_file_name.3.html

What might matter here is: "In the returned string, symbolic links are resolved".

The "shortcut" in the root of the mapped drive might be a symbolic link that is resolved by `canonicalize_file_name`.

I'd guess `glob` or `__wglob__` shouldn't resolve symbolic links. But I don't know if they actually aren't doing that.

I don't know by heart which functions are used when the content of the load path is scanned. But it might be `canonicalize_file_name`, `glob`, `__wglob__`, ... or a combination of those.

Could you please check what each of those functions returns for a file on the mapped network drive?

Markus M├╝tzel <mmuetzel>
Project Member
Thu 04 Aug 2022 09:40:15 AM UTC, comment #24: 

> If you browse that drive with the Windows Explorer, do the folders also end with `$`?

  • If I enter '\\<server>\company.local' in Windows Explorer's address bar there is nothing shown. But that server address exists as there are no errors.
  • If I enter '\\<server>\company.local\full$' I see the contents of directory 'full$' (including 'path' (from 'full\path\to').
  • If I enter '\\<server>\company.local\full' (w/o trailing '$') I get an error "Windows cannot access ...".
  • Now, if I enter 'Q:\' I see that root subdir 'full' (w/o trailing '$') is actually a shortcut - apparently to '\\<server>\company.local\full$'. I cannot verify the latter, but I suppose it has to be. And I have no idea where the "physical" location of that shortcut is. Could be anywhere on the network. Maybe it's even some sort of virtual thing.

So, there are also shortcuts in the mix. Do addpath / canonicalize_file_name stumble over shortcuts? That would fit in nicely in the behavior I see.

BTW in a CMD window UNC paths aren't accepted on my virtual PC. The 'cd' command there even says so: "CMD does not support UNC paths as current directories"

Philip Nienhuis <philipnienhuis>
Project Member
Thu 04 Aug 2022 06:33:07 AM UTC, comment #23: 

> Makes me wonder when Octave started to morph drive letter paths into UNC paths


The backbone of `__wglob__` changed from an implementation from gnulib to using the Windows API directly.
See also what I wrote in the other report:

> IIUC, Windows is returning `\\server\share\file` for files on network shares for `FindFirstFileW` and `FindNextFileW`. `__wglob__` (the underlying work horse for the `dir` function on Windows) is using these functions to be able to handle non-ASCII characters in paths. That'll probably stay that way.


The gnulib implementation of that function can't cope with paths containing non-ASCII characters on Windows.

Markus M├╝tzel <mmuetzel>
Project Member
Wed 03 Aug 2022 10:53:18 PM UTC, comment #22: 

Yep it's pretty mysterious.

Reading up on bug #62576 + the Stackoverflow posting mentioned there I get more and more confused about the UNC paths versus drive letter issue.
I can follow your rationale for making the chance that I backed out locally.
Makes me wonder when Octave started to morph drive letter paths into UNC paths, and most of all, why it is specifically 'addpath' that doesn't work anymore with drive letters. 'which' and 'cd' seem to have no issue with it.

Anyway, as I wrote I can work around it for the time being.
I suggest to lower priority (but leave Item Group "regression").

I hope (but maybe in vain) that after the summer I'll have more free time to spend on Octave.  Even then I see no way I can debug this, if only for the sheer size of the unstripped Octave "installer" or "dist archive". There's no way I can transfer files that large to my virtual work PC, not even in chunks.

Philip Nienhuis <philipnienhuis>
Project Member
Wed 03 Aug 2022 10:14:02 PM UTC, comment #21: 

(Hmmm again a posting got lost. Maybe due to the Seamonkey browser, works less and less well these days. I get logged out of Savannah on each restart as well)

Correction:
7.1.90 DOES work fine.
Apologies Markus, I mixed up the by now tens of shortcuts on my desktop. (Time to clean up there.)

So that is consistent with your initial thought cf. comment #8.

As to shared folders with a trailing '$' in their names on remote servers, I'd have to try, no idea if that works on our company network. Later tomorrow I'll hopefully have time for that.

Philip Nienhuis <philipnienhuis>
Project Member
Wed 03 Aug 2022 05:52:54 AM UTC, comment #20: 

Maybe one more thing: If you browse that drive with the Windows Explorer, do the folders also end with `$`?

Markus M├╝tzel <mmuetzel>
Project Member
Wed 03 Aug 2022 05:48:00 AM UTC, comment #19: 

> 7.1.90 didn't "work" (w.r.t. this bug report)


This is getting more and more mysterious. The changeset that you backed out wasn't part of 7.1.90 afaict. I wonder why backing it out now on the default branch helps...

I guess for the time being we probably need to accept that something is going on "behind the scenes" that we currently don't understand. Maybe we can try to come back to this when we figured out what is happening. Ideally, I would be able to reproduce and try with different Windows API functions how to best tackle with this.

Markus M├╝tzel <mmuetzel>
Project Member
Tue 02 Aug 2022 08:45:29 PM UTC, comment #18: 

I build dev Octave from default more or less regularly, that's where I hit the issue.

7.1.90 didn't "work" (w.r.t. this bug report)

6.3.0 because the OP in bug #62576 mentioned his scripts were working with 6.4.0. But reading more closely I see that for him it changed adversely in 7.1.0 which works fine for me. Right, saves me some time :-)

>> There are two things I currently don't understand: Why does the name of the share (`Total`) disappear in the output of `which`? Where does the `$` sign in that output come from?

Yeah, good questions :-) That's really beyond me. The company (actually 'authority', it's semi-gov) is slowly shifting its IT resources to the cloud (Sharepoint, Azure) so probably there's quite a bit of reshuffling going on behind the scenes.
OTOH I have reminiscence of this ('Total' being replaced / dropped) having been the case for many many years, because I spotted that too when I had to renew my drive mappings each and every time our desktop PCs, and later on virtual PCs, got 'rolled out' anew (which now morphed into a yearly tradition in the busy days before Xmas. sigh).
Until recently Octave coped well with it.

Anyway I have little time for experiments. I'll help but maybe at a lower pace - after all I have workarounds.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 02 Aug 2022 08:00:49 PM UTC, comment #17: 

We still don't have enough information to tell if this is related to bug #62576. Please, see my comment #15.

Why 6.3.0? It is no longer supported.
Please test with 7.1.90.

Markus M├╝tzel <mmuetzel>
Project Member
Tue 02 Aug 2022 07:53:32 PM UTC, comment #16: 

Looks to me that this bug report should be linked to bug #62576.

Reading up there I also fail to understand what is happening at my work.
IIRC I still have 6.3.0 installed there so maybe I'll try that as well (as s/th related changed in 7.1.0), but TTBOMK 6.3.0 has always worked fine while I used it. I'll run a check tomorrow anyway.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 02 Aug 2022 07:50:25 PM UTC, comment #15: 

The default branch has other changes with respect to the stable branch that could be relevant here.
Does a build from the default branch without that changeset backed out work?
Did the release candidate 7.1.90 still work?

There are two things I currently don't understand: Why does the name of the share (`Total`) disappear in the output of `which`? Where does the `$` sign in that output come from?

Markus M├╝tzel <mmuetzel>
Project Member
Tue 02 Aug 2022 07:34:50 PM UTC, comment #14: 

Hmmm, I thought I typed in two comments but they don't show up.

Another try:

H:\>>net use

Status       Lokal     Remote                    Netzwerk
-------------------------------------------------------------------------------
:
             Q:        \\company.local\Total     Microsoft Windows Network
:

so there seems to be some stripping / substitution going on (of directory 'Total'). We work in a domain so perhaps that's where all the wizardry comes from.

Anyway, an 8.0.0 build with the cset mentioned in comment #8 backed out seems to work fine w.r.t. this bug report.

I don't know how we should proceed further. I'll buy it blindly if it is suggested that the situation at my work side is "special" (for other reasons as well, don't get me started ;-) ).
For the time being my workaround is Good Enough for me.
Or I can leave the pertinent cset backed out for some time (maybe easiest for me).

Philip Nienhuis <philipnienhuis>
Project Member
Tue 02 Aug 2022 04:04:26 PM UTC, comment #13: 

I'm not sure how that is setup on your end. I'm not able to map a drive letter to a server. But that seems to be the case for you...

From a command shell:

C:\Users\Markus>net use
Neue Verbindungen werden gespeichert.


Status       Lokal     Remote                    Netzwerk

-------------------------------------------------------------------------------
OK           Y:        \\Markus-PC\D$            Microsoft Windows Network
OK           Z:        \\Markus-PC\share         Microsoft Windows Network
Der Befehl wurde erfolgreich ausgef├╝hrt.


C:\Users\Markus>net use X: \\Markus-PC
Systemfehler 67 aufgetreten.

Der Netzwerkname wurde nicht gefunden.


C:\Users\Markus>net use X: \\Markus-PC\C$
Der Befehl wurde erfolgreich ausgef├╝hrt.


C:\Users\Markus>net use
Neue Verbindungen werden gespeichert.


Status       Lokal     Remote                    Netzwerk

-------------------------------------------------------------------------------
OK           X:        \\Markus-PC\C$            Microsoft Windows Network
OK           Y:        \\Markus-PC\D$            Microsoft Windows Network
OK           Z:        \\Markus-PC\share         Microsoft Windows Network
Der Befehl wurde erfolgreich ausgef├╝hrt.


C:\Users\Markus>net use X: /delete
X: wurde erfolgreich gel├Âscht.

(German system messages. But I hope it is still clear what is going on.)

I'm able to map to shares (hidden or visible) on a server. I'm not able to map directly to a server. Maybe there is something different because you seem to be a domain member???

Could you please run `net use` from a command shell and show the result?

Markus M├╝tzel <mmuetzel>
Project Member
Tue 02 Aug 2022 03:44:18 PM UTC, comment #12: 

I've tried a hopefully more systematic approach (found some time in between assignments). It looks inconsistent with parts that I wrote in comment #7, notably behavior of the subdir boxes in the file browser and main menu. That seems to always work, also with UNC paths.

##========================================================================
## with pertinent network subdir IN the path:

 >> cd ('Q:\full\path\to')
## Works fine. Note: no '$' after 'full' entered. With it it won't work

## from within pertinent subdir on network where opt_lvm.m lives
 >> which opt_lvm
'opt_lvm' is a function from the file '\\<server>.company.local\full$\path\to\opt_lvm.m'
## Note '$' appearing after first subdir on server (or is it already a share)

 >> canonicalize_file_name ('opt_lvm')
ans = '\\<server>.company.local\full$\path\to\opt_lvm.m'
## Note '$' appearing after first subdir on server (or is it a share)

## Elsewhere on that network drive, or on local drive
 >> which opt_lvm
'opt_lvm' is a function from the file '\\<server>.company.local\full$\path\to\opt_lvm.m'
## Note '$' appearing after first subdir on server (or is it a share)

 >> canonicalize_file_name ('opt_lvm')
ans =
## (empty)

## Copying Q:\full\path\to into box in main menu or browser pane + <Enter>
## ==> Works fine, is accepted

##==============================================================================
## With pertinent subdir NOT in path
 >> cd ('Q:\full\path\to')
## Works fine

## from within pertinent subdir
 >> which opt_lvm
'opt_lvm' is a function from the file '\\<server>.company.local\full$\path\to\opt_lvm.m'
## Note '$' appearing after first subdir on server (or is it already a share)

 >> canonicalize_file_name ('opt_lvm')
ans = '\\<server>.company.local\full$\path\to\opt_lvm.m'
## Note '$' appearing after first subdir on server (or is it a share)

## But now look here:
 >> canonicalize_file_name ('Q:opt_lvm')
ans = 'Q:\full\path\to\opt_lvm.m'
## Note drive letter rather than UNC

## Elsewhere on that network drive, or on local drive
 >> which opt_lvm
## empty

 >> canonicalize_file_name ('opt_lvm')
ans =
## (empty)

## Copying Q:\full\path\to into box in main menu or browser pane + <Enter>
## ==> Works fine, is accepted

## ==========================================================

Funnily, starting in the network subdir holding 'opt_lvm.m', then going to E: (entering 'E:\' in the box), then:
 >> canonicalize_file_name ('T:opt_lvm.m')
 ans = Q:\full\path\to\opt_lvm.m

Then, in file browser back to the network subdir, and pressing the 'one subdir level up' button, I get
 >> canonicalize_file_name ('T:opt_lvm.m')
 ans =
## empty

The '$' character that shows up in UNC paths may be a red herring, I think it's merely the literal "low-level" name of the share.
Again, simply strrep'ing 'Q:' by '\\<server>.company.local' in 'Q:\full\path\to' makes it all work.

So, as far is I can see it's just addpath() that doesn't work with the network drive letters on my work system.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 02 Aug 2022 02:51:44 PM UTC, comment #11: 

Oops. Typo. I meant 7.1.90.

> From which location, local drive (= outside path) or network subdir (in the path)? and what do you mean exactly with 'full path' and which part might have been 'eaten'?


The current directory shouldn't matter.
Any part might be "eaten" if the logic is wrong that converts an UNC path (as returned by the Windows API) back to the mapped drive.
Especially, the first part of the path might be affected.

Markus M├╝tzel <mmuetzel>
Project Member
Tue 02 Aug 2022 12:30:35 PM UTC, comment #10: 

Last comment for now, I really have to get work done ... :-)

7.1.0 works OK, so I suppose 6.1.90 works as well. If you insist I'll give it a try later on.

>> Did you `cd` to the folder on the mapped drive Q: that contains `opt_lvm.m`, and `which` still returned the UNC path?

Yep, see comment #7

>> Was that folder also in the load path (maybe using the UNC path)?

AFAICS, no. It was all done w/o the 'workaround' mentioned in comment #2, even w/o the script that adds the pertinent network subdirs to the path.

>> Does `canonicalize_file_name ('Q:\full\path\to\opt_lvm.m')` return the full path (without "eating" part of it)?

From which location, local drive (= outside path) or network subdir (in the path)? and what do you mean exactly with 'full path' and which part might have been 'eaten'?

Thx for your insistence :-)

Philip Nienhuis <philipnienhuis>
Project Member
Tue 02 Aug 2022 12:21:46 PM UTC, comment #9: 

And again: Does `full\path\to\opt_lvm.m` really start immediately after the server name (`\\<server>.company.local`)?

Markus M├╝tzel <mmuetzel>
Project Member
Tue 02 Aug 2022 12:20:17 PM UTC, comment #8: 

Did the release candidate (6.1.90) still work?
If it did, this change might be a candidate (though I don't understand why it would break your use case...):
https://hg.savannah.gnu.org/hgweb/octave/rev/b47bf773c508

Did you `cd` to the folder on the mapped drive Q: that contains `opt_lvm.m`, and `which` still returned the UNC path?
Was that folder also in the load path (maybe using the UNC path)?

I understand you redacted some of the names. Does `canonicalize_file_name ('Q:\full\path\to\opt_lvm.m')` return the full path (without "eating" part of it)?

Markus M├╝tzel <mmuetzel>
Project Member
Tue 02 Aug 2022 12:08:46 PM UTC, comment #7: 

Thanks for the suggestions Markus, I was already busy trying most of them out.

Below obfuscated paths are shown for my company's IT security, sorry for that. I tried with the real ones, of course.
And all in the GUI (I rarely if ever use the CLI).

cd ('Q:\full\path\to\network\subdir')
works fine. As do 'dir' and 'ls'.
In the browser pane and main menu 'Current Directory' box I see a drive letter and full path.

## From the network subdir where opt_lvm.m lives:
 >> which opt_lvm  (## lives in a network subdir)
'opt_lvm' is a function from the file \\<server>.company.local\full\path\to\opt_lvm.m
==> works

 >> canonicalize_file_name ('opt_lvm.m')
ans = \\<server>.company.local\full\path\to\pt_lvm.m
==> works

## From a different, "local" drive (FYI it's all on virtualized desktops) &
## after restarting Octave, just to be sure
 >> canonicalize_file_name ('Q:\full\path\to\opt_lvm.m')
ans = Q:\full\path\to\opt_lvm.m
==> hey this works as well ... same for just the subdir.

canonicalize_file_name and make_absolute_filename yield the same results.

These results makes me wonder where the expanded UNC paths come from under what circumstances. Doesn't seem consistent across various local vs. network locations.

In the GUI I cannot enter the path into the file browser pane nor in the "Current subdirectory" box in the main menu. That is, I can paste it but hitting <Enter> does nothing.
However, when using the drop-down menus there for previous subdirs, when I click on the relevant network subdir it does work and gets me to that location.
(Those entries came into that list thru the workaround I described in comment #2, temporarily disabled for the tests above.)

BTW the actual full path has various intermediate directories with spaces in the name.

Would you have any other suggestions?
Today I have a busy work day, maybe tonight or tomorrow night I could do more tests and/or peruse the log at savannah to search for a possible candidate cset that broke the "proper" behavior (or would you know a candidate cset?). If found I can back it out (hopefully) and try a build with that tree.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 02 Aug 2022 08:23:37 AM UTC, comment #6: 

> Could be the other way round, sure. Is that really relevant?


When canonicalizing the path, patterns are used to detect if the input was a mapped network share. If it is possible to map a server (as opposed to a share), the patterns might need to be adapted.

Other things that you could try could be testing if you can `cd` to that mapped drive. What does `canonicalize_file_name` return for a file on that mapped drive? Does `which mfile` work for an .m file in the current directory on a mapped drive?

Markus M├╝tzel <mmuetzel>
Project Member
Tue 02 Aug 2022 08:14:43 AM UTC, comment #5: 

> Are you sure that the drive letter maps to the server ("\\server")? I'm not able to do that. I can only map shares to a drive letter ("\\server\share").

Could be the other way round, sure. Is that really relevant?

What seems to have happened is that between Octave-7.1.0 and 7.2.0 there has been a chance that broke the way that network drive letters on some large company networks can be added to the path. My Octave-7.1.0 and 7.2.0 installations at work are based on the ftp.gnu.org downloads.

I just reported my experiences. If it can be fixed, nice. But maybe it can't - I've already indicated some of the magic I see when delving down into the file systems at work.
I'll try a few more things today; maybe look at file encodings, or saving files to the network, etc.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 02 Aug 2022 07:09:09 AM UTC, comment #4: 

Still unable to reproduce. It also works for me with a hidden network share (trailing '$').

Are you sure that the drive letter maps to the server ("\\server")? I'm not able to do that. I can only map shares to a drive letter ("\\server\share").

Markus M├╝tzel <mmuetzel>
Project Member
Mon 01 Aug 2022 07:27:59 PM UTC, comment #3: 

Maybe related: in Octave's path, after adding a network drive cf. my method 3 of comment #0, the path entry looks like

'\\PS315.company.local\rest$\of\full\path'

Note the '$' character after 'rest': that means that that subdir is hidden for normal browsing actions.

Philip Nienhuis <philipnienhuis>
Project Member
Mon 01 Aug 2022 07:24:36 PM UTC, comment #2: 

Thanks Markus for responding quickly :-)

A correction: I see the behavior also with 7.2.0, but not with 7.1.0.
(I think I erred there because the NSIS installer, when installing several Octave versions side by side, usually picks up the previous Octave installation and puts its path into the new desktop shortcuts. I forgot to fix that.)

Not an error but a warning (typed over from the remote desktop, I can't copy/paste over VMWare and Savannah is blocked by the company's firewall), along the lines of:

warning: addpath: Q:\full\path\to\some\network\subdir: no such file or directory
warning: called from
     < some local helper .m-file that adds paths > at line x column y
     E:\.octaverc at line z column t

(where E:\ is a local drive) and from which we can conclude that

  1. an error (not being able to add a requested path) is disguised as a warning. Looks inconsistent to me and something users wouldn't perceive as worth merely a warning
  2. ? addpath seems prepared to add files to the path? (OK that's mere cosmetics.)

Admittedly, at the company I work for the network mapping vagaries and intricacies are a bit untractable for mere mortals like me.
E.g., where I wrote that drive letter "Q:\" may resolve to '\\SP500.company.local', after adding the path cf. my example 3 the pertinent path entry looks like '\\BC312.company.local\rest\of\path'. So the base server name has been silently substituted as well. Furthermore in the network disk's properties under the DFS [*] tab, the start of the path just looks like '\\company.local\'.
A sysadmin once assured me this is all run-of-the-mill Windows network mapping management and processing. So I conclude this should all be transparent to programs like Octave. And it apparently is, up till 7.1.0.

BTW I've noted this behavior since maybe a few weeks.

I do have a kludge workaround: in the helper .m file I mentioned I've added a strrep to replace the drive letter with the pertinent '\\server.company.local' UNC start, but as that can chance unpredictably (for me) it's not a robust solution.
Furthermore, it's a bit untenable to include strrep into any addpath call at the command prompt (e.g., by aliasing addpath) as we have several drive letters each pointing to possibly separate servers.

[*] DFS stands for 'Distributed File System'

Philip Nienhuis <philipnienhuis>
Project Member
Mon 01 Aug 2022 12:58:20 PM UTC, comment #1: 

I tried to reproduce on Windows 11 with hg id 929c05cf2afa. But it seems to work for me. Functions in folders on a mapped network drive are found for me and correctly executed.

What do you mean by "doesn't work"? Which error messages (if any) do you see?

Markus M├╝tzel <mmuetzel>
Project Member
Mon 01 Aug 2022 10:14:15 AM UTC, original submission:  

At my work, addpath cannot process pathnames with drive letters that behind the scenes map onto some UNC path.
Unfortunately due to strict security I cannot literally mention those path names but I can come up with some lookalikes.

1. What doesn't work is a mapped drive letter, where the path to the user looks like e.g., 'Q:\full\path\to\subdir\on\network' and 'Q:\' points to some server like '\\SP500.company.local'.

2. What does work is

addpath ('\full\path\onto\some\local\disk')

3. What also works is

addpath ( ['\\servername.local_NTFS_joint_name' <rest of path]] )

addpath still works fine on Octave-7.2.0 (pfew) but not on dev Octave.

Philip Nienhuis <philipnienhuis>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by philipnienhuis (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only project members can vote.

     

     

     

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-10-18 mmuetzel Dependencies- bugs #63227 is dependent
    2022-09-02 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2022-08-12 mmuetzel StatusIn Progress Ready For Test
    2022-08-12 mmuetzel StatusReady For Test In Progress
    2022-08-07 mmuetzel StatusPatch Reviewed Ready For Test
        Summaryaddpath fails to process symlinks on mapped network drives addpath incorrectly processes symlinks on mapped network drives
    2022-08-07 mmuetzel Releasedev 7.2.0
    2022-08-05 mmuetzel StatusNeed Info Patch Reviewed
    2022-08-04 mmuetzel Attached File- Added bug62847-symlinks-mapped-drive.patch, #53504
        Summaryaddpath fails to process some mapped network drives addpath fails to process symlinks on mapped network drives
    2022-08-03 mmuetzel Summaryaddpath fails to process network pathnames addpath fails to process some mapped network drives
    2022-08-01 mmuetzel StatusNone Need Info

    Back to the top


    Powered by Savane 3.9