bugGNU Octave - Bugs: bug #57664, dir() function folder element is...

 
 

bug #57664: dir() function folder element is empty for Windows UNC network-based files

Submitted by:  Dan Sebald <sebald>
Submitted on:  Sat 25 Jan 2020 06:02:04 AM UTC  
 
Category:  Octave Function Severity:  3 - Normal
Priority:  3 - Low Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  dev Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 09 Feb 2020 11:45:28 AM UTC, comment #16: 

I opened bug #57776 for the issues with dir on share roots and ".." with UNC shares.

Marking this report as fixed.

Markus Mützel <mmuetzel>
Project Member
Mon 03 Feb 2020 09:49:09 PM UTC, comment #15: 

Marking as Ready For Test.

Rik <rik5>
Project Administrator
Sun 02 Feb 2020 04:20:09 PM UTC, comment #14: 

Oh yeah, PowerShell, Windows remote login.  I was also wondering about MXE-Octave and the like.  I think this is one we get little bits at a time as all the allowable formats come to light.

The shared folder and literal characters issues are related, but seem different: dir failing as opposed to missing the "folder" field.

Dan Sebald <sebald>
Sun 02 Feb 2020 01:20:10 PM UTC, comment #13: 

Just after posting my previous comment, I found that the PowerShell does allow forward slashes:

PS C:\WINDOWS\system32> cd Microsoft.PowerShell.Core\FileSystem:://SERVER/share/test
PS Microsoft.PowerShell.Core\FileSystem::\\SERVER\share\test>

Markus Mützel <mmuetzel>
Project Member
Sun 02 Feb 2020 01:17:59 PM UTC, comment #12: 

I couldn't find a place in Windows where starting UNC shares with "//" is valid syntax. So I ended up pushing Mike's patch here:
http://hg.savannah.gnu.org/hgweb/octave/rev/673fb7081ebe

If we want to allow forward slashes. This can still be done in a later patch.

There might be another one or two somewhat independent issues with dir and UNC paths:

  • Calling "dir" directly on a shared folder fails:

    >> dir('\\SERVER\share')
    warning: dir: nonexistent directory '\\SERVER\share'
    warning: called from
        dir at line 193 column 5

  • Paths starting with "\\" are literal on Windows. E.g., it would be possible to list the content of a folder with name "..". That leads to lstat failing:

    >> dir('\\SERVER\share\test')
    warning: dir: 'lstat (\\SERVER\share\test\..)' failed: No such file or directory
    warning: called from
        dir at line 132 column 9

    .                 Octave (GUI).lnk  test.txt          test_octave.bat   test_unc.m

  Maybe, we could skip lstat on ".."?

New bug report(s)?

Markus Mützel <mmuetzel>
Project Member
Sat 01 Feb 2020 07:50:18 PM UTC, comment #11: 

Looks good.

The filesep() substitution is a good idea.  There are the linux Windows emulators, and I'm not sure how they handle UNC.  Difficult to have BIST tests when there is the need for directory hierarchy within a network.  I'm sort of use to Qt at the moment where it handles both forward and back slashes, but Qt suggests the forward slash convention.  That is, I think in Qt one can use either, but if asking for info from QDir or something it only returns forward slash for file separator.

Dan Sebald <sebald>
Sat 01 Feb 2020 05:56:48 PM UTC, comment #10: 

I've revised your patch, attached. I think it's slightly better. But maybe we should also do a global replace of forward slashes into backslashes as a poor man's canonicalization.

(file #48327)

Mike Miller <mtmiller>
Project Administrator
Sat 01 Feb 2020 08:29:43 AM UTC, comment #9: 

How about the attached patch?  It's as good as anything, I suppose.

(file #48326)

Dan Sebald <sebald>
Sat 01 Feb 2020 03:46:27 AM UTC, comment #8: 

I agree Mike.

Actually, I think this is easier than we imagine, but I haven't had time all week to look at this.  The UNC is much more restrictive, and ostensibly is already in canonical format.  There has to be a full path beginning with // or \\.  There are no relative paths or present or parent directory symbols that you described.  Stuff like //SERVER/path/to/../to/server doesn't work.

So, I'm thinking all we need do is verify that some files return (which they do), and if canonicalize_file_name() is empty (and on Windows?), run the name through remove-whitespace routine, check for // or \\ at the start, if present then just set the folder to that white-space free name.

The silliness in this UNC stuff is Windows.  Three or four decades now after the advent of the PC, and Windows still has a 247 character limit on path names, not file names, but path names.  Do you know how easy it is to burn up 247 characters in a UNC name?  Terabyte servers aren't going to have all their files at the root path.  Uncompress something with deep directories, and it's "path too long".

Dan Sebald <sebald>
Fri 31 Jan 2020 07:39:41 PM UTC, comment #7: 

The 'canonicalize_file_name' function is great when you want to get the real absolute path of a file that actually exists as a real local file. If the file doesn't exist, the return value is empty. So this works for any absolute or relative file name specification, for example '.', '..', '/tmp/foo/bar', '../images', 'C:\Files\doc.txt', '\Programs\one\two', and so on.

As I understand it, UNC paths are really closer to URLs than to file names. That all Windows file I/O functions understand how to resolve and access UNC paths is great, makes it easy on programs. But it's the same as a path like 'host:/some/file' on a Unix-like system, it's a network resource that needs to be fetched from elsewhere, and 'canonicalize_file_name' won't work on these types of paths either.

So my opinion is that 'canonicalize_file_name' is working correctly, it should return an empty string when given a UNC path specifier, just like a URL or a host:file specifier.

If you want the 'dir' function to work correctly on UNC names, which I agree it should, then 'canonicalize_file_name' should be skipped, either always or only in the case of UNC names.

Mike Miller <mtmiller>
Project Administrator
Tue 28 Jan 2020 09:15:37 AM UTC, comment #6: 

I'm seeing pretty much the same as what Markus describes.  Also,

ls('\\SRVR\engineering\path\to\file')

works too, but in that case Octave is using a system call to "dir \\SRVR\engineering\path\to\file" (not the Octave dir() routine).

My guess would be that canonicalize_file_name() sort of originated as a Unix routine, then along the way gnulib extended it to Windows.  There doesn't seem to be a lot out there on canonicalize_file_name() and UNC paths.  Instead, in the Windows world realpath() is mentioned more often.

It would be nice if gnulib already handled UNC.  We could cobble something together using "dir" system command, but it feels like the best approach is to use some of the low level Windows routines.  In any case, processing of file paths generally in Octave could utilize the same "pre-process" code.

Dan Sebald <sebald>
Mon 27 Jan 2020 04:47:18 PM UTC, comment #5: 

Is there some reason that canonicalize_file_name can't be made to work for these directories?

John W. Eaton <jwe>
Project Administrator
Mon 27 Jan 2020 04:19:51 PM UTC, comment #4: 

Because canonicalize_file_name() doesn't work this is going to be quite difficult to fix.  Ensuring uniform support for UNC paths not just in dir(), but throughout Octave will be quite a project, if anyone wants to take it on.

This won't get fixed for 5.2.0 so I have changed the Release field to "dev".  I've also lowered the Priority to Low.  This could be fixed, but someone would need to be very motivated and there are probably workarounds to the use of UNC paths.

Rik <rik5>
Project Administrator
Mon 27 Jan 2020 03:02:49 PM UTC, comment #3: 

"canonicalize_file_name" does not work for UNC paths.
It returns an empty string on Octave 4.2.2, 4.4.1, 5.1.0, and 5.1.90. (I don't have other installed versions atm. But IIRC, it also doesn't work on dev.)
"cd"ing to the same UNC path does work.

Markus Mützel <mmuetzel>
Project Member
Mon 27 Jan 2020 02:08:37 AM UTC, comment #2: 

The function canonicalize_file_name we get from gnulib.  Can you check whether the UNC path actually exists at the time canonicalize_file_name is called?  You can do that without calling dir() by just invoking the function directly.

r = canonicalize_file_name ("//hostname/path/to/directory")

The function will return "" if it cannot find the path which could be caused by typos, network errors, etc.

Also, see bug #45816 from 5 years ago in which other problems with canonicalize_file_name were addressed.

Rik <rik5>
Project Administrator
Sat 25 Jan 2020 02:01:43 PM UTC, comment #1: 

FWIW, I had a small debug session using dev Octave with dir.m and found that the command

dir ('\\nsa310\photo')

(where 'photo' is a directory on our NAS) fails a lot earlier, on L. 101, where

_wglob_ (directory)

returns empty. Forward slashes or backslashes make no difference.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 25 Jan 2020 06:02:04 AM UTC, original submission:  

I used the latest beta version of Octave for Windows and ran across an issue in which the "folder" field of the structure returned from the dir() function is empty for something like the following UNC path:

rslt = dir("//hostname/path/to/directory");

However, the dir() routine and other file-access routines are able to handle UNC paths without a problem.

It comes down to the following hunk of code in dir():

        tmpdir = regexprep (fn, re, '$1');
        fn = regexprep (fn, re, '$2$3');
        info(i).name = fn;
        if (! strcmp (last_dir, tmpdir))
          ## Caching mechanism to speed up function
          last_dir = tmpdir;
          last_absdir = canonicalize_file_name (last_dir);
        endif
        info(i).folder = last_absdir;

In particular, the line

          last_absdir = canonicalize_file_name (last_dir);

is the only line that appears to not behave as expected.

The first question to ask is should "folder" report something?  It's not obvious it should because I'm aware that UNC paths are kind of odd from the DOS perspective.  There may even be some Windows utilities that need UNC paths mapped to a network drive in order to access the share volume.  But, as I pointed out, UNC paths seem fine otherwise in Octave.

The second question is why does canonicalize_file_name() fail?  Is it because it is a C-library routine and therefore works at the OS (DOS) level?  Would some other routine be needed if the beginnings of a UNC path is identified?

Some references:

https://savannah.gnu.org/bugs/?50504
https://savannah.gnu.org/bugs/?55448

Dan Sebald <sebald>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #48327:  dir.patch added by mtmiller (1KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by sebald (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-02-09 mmuetzel StatusReady For Test => Fixed
    2020-02-03 rik5 StatusNone => Ready For Test
        Open/ClosedOpen => Closed
    2020-02-02 mmuetzel StatusPatch Submitted => None
    2020-02-01 mtmiller Attached File- => Added dir.patch, #48327
        StatusNone => Patch Submitted
    2020-02-01 sebald Attached File- => Added octave-dir_unc_paths-djs_2020-01-31.patch, #48326
    2020-01-27 rik5 Priority5 - Normal => 3 - Low
        Release5.1.90 => dev

    Back to the top


    Powered by Savane 3.5