bugGNU Octave - Bugs: bug #36677, make_absolute_filename: extend to...

 
 

bug #36677: make_absolute_filename: extend to work with Windows drive letters

Submitter:  Philip Nienhuis <philipnienhuis>
Submitted:  Mon 18 Jun 2012 05:27:39 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  3 - Low Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Philip Nienhuis Open/Closed:  * Closed
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 04 Mar 2019 09:40:18 PM UTC, comment #25: 

I'd vote for closing. It can always be reopened if needed.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 04 Mar 2019 06:28:51 PM UTC, comment #24: 

I agree that handling file URIs is better be handled at a higher level function.
I backed out the respective changeset here:
http://hg.savannah.gnu.org/hgweb/octave/rev/5025f274e8f9

Can we consider this bug fixed? Or is there still something actionable remaining?

Markus Mützel <mmuetzel>
Group administrator
Sun 03 Mar 2019 09:31:48 PM UTC, comment #23: 

@Markus, comment #19:

A short practical answer:
I didn't hit problems related to this bug the last years. IIRC there was a somewhat related issue but it was either solved or didn't hurt anymore.

A longer, more philosophical and maybe/probably less useful answer is this:

Reading back upon this old bug I think one unsolved aspect, or peculiarity, is how make-absolute_filename would be supposed to work on Windows systems, where can have several distinct drive letters, and on each drive one has a different "working" directory.

So, when my current working directory is C:\Users\Philip\octave, I could do in a cmd window:

cd F:\some\other\directory

and then if I would type:

type 'dir F:' (w/o quotes and w/o filesep) the contents of F:\some\other\directory would be listed rather than the contents of F:\ (F:'s root dir).

Now, in Octave it is already impossible to do something like
dir F:
while one CAN do
dir F:/
and
dir C:
(hmm, another inconsistency)

IMI we'd better ignore these DOS-era (but still working) idosyncrasies.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 03 Mar 2019 09:14:39 PM UTC, comment #22: 

Yeah, I tend to agree, I don't think this function should be heuristically deciding that a file name looks like a URL or not. I think that 'make_absolute_filename("file:///asdf")' should probably return the same as 'fullfile(pwd, "file:///asdf")'.

Mike Miller <mtmiller>
Group Member
Sun 03 Mar 2019 09:04:11 PM UTC, comment #21: 

The handling of "file:" as a prefix should probably be limited to contexts where that is expected.  The low-level file functions that test for absolute file names are probably not the right place.  Those should be making tests that correspond to the rules for file names on the OS (and filesystem).  So it's OK to test for things like "C:" as a device name prefix, but only on Windows systems.

John W. Eaton <jwe>
Group administrator
Sun 03 Mar 2019 08:29:01 PM UTC, comment #20: 

If a file name string is "file:foo.txt", is the intention that that is

  • a relative file named "file:foo.txt"?
  • a file URI specifying a relative file named "foo.txt"?
  • an invalid file URI?


I see lots of conflicting information over whether at least one slash is required after "file:" for it to be interpreted as a URI scheme, and whether it is legal to represent relative file names with "file:" or not.

Mike Miller <mtmiller>
Group Member
Sun 03 Mar 2019 08:12:38 PM UTC, comment #19: 

Thanks, Mike for the input. With that I am inclined to say that the function is already working as expected now.

@Philip: Does the current behaviour of make_absolute_filename fit your use case? Or would you need a function that canonicalizes file names of non-existent files with absolute path?

Markus Mützel <mmuetzel>
Group administrator
Sun 03 Mar 2019 07:20:48 PM UTC, comment #18: 

@mmuetzel As long as 'make_absolute_filename' still fully supports nonexistent file and directory names, it doesn't matter to me whether it attempts to canonicalize names or not.

I guess the argument could be made that the only purpose of the function is to turn a relative file name into an absolute file name. If the input is already an absolute file name it should be returned unmodified.

Mike Miller <mtmiller>
Group Member
Sun 03 Mar 2019 07:06:22 PM UTC, comment #17: 

I pushed a patch here that supports the file URI scheme in make_absolute_filename:
http://hg.savannah.gnu.org/hgweb/octave/rev/79efd0653155

At the moment, make_absolute_filename "canonicalizes" the file name if the input is a relative path. But it does not if the input is an absolute path.
Imho, this is inconsistent. But the documentation is not clear on what is expected.

The attached patch adds preliminary support for "canonicalizing" absolute paths.
It isn't quite complete because it doesn't correctly treat something like "file://hostname/C:/test/../../../file".

But before I continue: Would we expect make_absolute_filename to "canonicalize" absolute paths?

(file #46404)

Markus Mützel <mmuetzel>
Group administrator
Sun 03 Mar 2019 04:39:14 PM UTC, comment #16: 

Do "file://path/file" constructs need attention?

In the io package, spreadsheet I/O, UNO interfce, these constructs are needed for file I/O using the Java-UNO bridge. Other than that I haven't seen them in the wild.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 03 Mar 2019 04:17:52 PM UTC, comment #15: 

make_absolute_filename and canonicalize_filename behave consistently on Linux and Windows for me.

I pushed a patch that should prevent traversing past the file letter on Windows here to fix the remaining issue:
http://hg.savannah.gnu.org/hgweb/octave/rev/670ebc29ec69

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Mon 25 May 2015 08:49:19 PM UTC, comment #14: 

Rik,
As my C++ proficiency is a bit lacking I think I'm not the one to fix make_absolute_filename().
When I have time (and think of it) I'll check what Matlab makes of one of the original issues (accessing files on different drive letters on Windows).

Philip Nienhuis <philipnienhuis>
Group Member
Fri 22 May 2015 03:34:02 PM UTC, comment #13: 

@Philip: I removed the 'postponed' status for this report.  If you want to make changes on the 4.2 branch to make_absolute_filename that would be fine.  Otherwise, it seems you've already worked around it so I made the priority low.

Rik <rik5>
Group administrator
Sun 20 Jan 2013 08:11:31 PM UTC, comment #12: 

With some workarounds (if-else constructs) the io pkg scripts work OK.
Fixing this "bug" (rather: inconsistency) can wait as far as I'm concerned and as long no no one else reports to be affected.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 02 Sep 2012 08:36:56 PM UTC, comment #11: 

Thanks.

My C++ proficiency is simply too meager to be able to fix this.
But I can outline what the code might need to do:


- check if we're on Windows
  - if yes, check if "filename" contains leading drive letter
    (e.g., a pattern like [a-zA-Z]: )
    ## FIXME: what to do with 'file:///...' constructs? These
    ## are not supported but are fairly generally used
    - if yes, get/isolate that driveletter
      .compare with drive letter of cwd
      - if (not equal)
        opwd = pwd  # save current dir
        # get current dit on other drive. s/th like "system"?
        .[~, npwd] = system ("cd <driveletter>:")
        .cd (npwd)
        .make_absolute_filename (filename)
        # Ready. Restore
        .cd (opwd)
      endif
    endif
  endif
endif


The idea is of course to temporarily switch to the other drive to avoid do_make_absolute cluttering up.

Other than that on windows it doesn't work across drive letters (with files that exist or not), make_absolute_filename indeed fits the bill.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 02 Sep 2012 04:57:18 PM UTC, comment #10: 

make_absolute_filename() is defined in libinterp/interpfcn/utils.cc.  However, I had already traced back the series of internal function calls to the function which needs work.  That function is do_make_absolute at line 326 of liboctave/system/oct-env.cc. 

Separately, I just updated the source tree for liboctave to group related function into subdirectories.  You will need a version of the development sources that is greater than or equal to 15273:535bb601e7db.

Rik <rik5>
Group administrator
Sun 02 Sep 2012 02:38:03 PM UTC, comment #9: 

I (that is, "search") searched the entire octave source tree for "make_absolute_filename" but couldn't find it, except as a call in help.m.
In oct-env.cc I see do_make_absolute() and do_absolute_pathname().

Where is make_absolute_filename() defined?

Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Aug 2012 11:25:21 PM UTC, comment #8: 

canonicalize_file_name() is never going to be the right function for this situation.  It isn't particularly broken, it is just defined a particular way by GNU for their version of libc.  Lots of people rely on the current behavior so there isn't much hope of changing it.

make_absolute_pathname() isn't a GNU or gnulib function but an Octave one.  Therefore, we could potentially modify it to do whatever we want.  The function is pretty short.  If you are familiar with C++ you should take a look.  The file is liboctave/oct-env.cc.  The function is make_absolute() at line 326.  I see that there is already a FIXME comment that the behavior may not be correct on all systems.  Duh!  It seems like this must have been coded in a bit of a hurry.

Rik <rik5>
Group administrator
Thu 30 Aug 2012 10:49:53 PM UTC, comment #7: 

"...begins with a drive letter..." = probably same as ispc().
Anyway that wouldn't make much difference as the second & third examples in the 1st post show. It would only solve part of the issue.

The issue is that a general mechanism is needed that also returns the full path of target dirs for non-existent files. Say I'm in /home/philip and I want to create a file in /tmp. Entering "../../tmp/newfile.fil" should return /tmp/newfile.fil.

For the same token I could say that the GNU behavior of canonicalize_file_name is broken.

Anyway some behavior on Windows. With 3.6.2 MSVC2010:
(we're in X:\ = root of drive X:)


octave-3.6.2.exe:1> cd x:\
octave-3.6.2.exe:2> pwd
ans = x:\
octave-3.6.2.exe:3> filename = '../../../tmp/notfound.fil'
filename = ../../../tmp/notfound.fil
octave-3.6.2.exe:4> make_absolute_filename (filename)
ans = xtmp/notfound.fil        ## Faulty
octave-3.6.2.exe:5> canonicalize_file_name (filename)
ans = x:\tmp\notfound.fil      ## Correct
octave-3.6.2.exe:6>


so make_absolute_filename does return info (but faulty).
But now in subdir x:\octave\octave-3.7.x\bin:


octave-3.6.2.exe:7> pwd
ans = x:\octave\octave-3.7.x\bin
octave-3.6.2.exe:8> filename = '../../../tmp/notfound.fil'
filename = ../../../tmp/notfound.fil
octave-3.6.2.exe:9> canonicalize_file_name (filename)
ans = x:\tmp\notfound.fil      ## Correct
octave-3.6.2.exe:10> make_absolute_filename (filename)
ans = x:\tmp/notfound.fil      ## Correct
octave-3.6.2.exe:11>


...so there's a bug too in the sense that make_absolute_filename doesn't note that the ..\..\.. series descends below the "root level". Canonicalize_file_name seems robust (but one could argue that it incorrectly returns correct or rather expected info).

I'd better sort this out in more detail.....
Are there any tests for canonicalize_file_name and make_absolute_filename in the sources? Do they cover this?

And regarding gnulib:
what is the gnulib-defined behavior of make_absolute_filename?

Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Aug 2012 10:07:17 PM UTC, comment #6: 

There was a bit of a miscommunication.  Octave-Forge scripts shouldn't have to do anything with gnulib.  Octave core needs to implement canonicalize_file_name consistently and it should be doing this by pulling the gnulib module.  For some reason ./configure on your system didn't pull the module as it was supposed to.  This is really a problem with the build system problem for core Octave.

But, if we can get gnulib working then canonicalize_file_name will definitely return an empty string.  Thus, the original problem remains.

Maybe you can check the file name to see if it begins with a drive letter and if it doesn't you can use make_absolute_pathname.

Rik <rik5>
Group administrator
Thu 30 Aug 2012 09:48:13 PM UTC, comment #5: 

gnulib? AFAIK, until now gnulib is a bit alien to Octave-Forge.
Furthermore the spreadsheet functions are pure m-files. There's good reason to keep it that way.

The behavior you describe (GNU describes) simply kills the functionality that the spreadsheet functions need. Hopefully make_absolute_filename() will keep that functionality.

But I'll also investigate if full pathnames are really needed. If not this whole problem can simply be avoided.


Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Aug 2012 09:25:14 PM UTC, comment #4: 

I suppose you should work around this for now.  The working behavior of canonicalize_file_name is defined here (http://www.gnu.org/software/libc/manual/html_node/Symbolic-Links.html).  It is very clear that the function should return an empty string if the file name object does not exist.  We should probably me moving in the direction of getting the platforms to do the same thing.  In this case, I don't exactly know why you aren't pulling the gnulib module.

Rik <rik5>
Group administrator
Thu 30 Aug 2012 07:05:44 PM UTC, comment #3: 

Rik, any plans to get this fixed soon? (I suppose you lack time)

If not I'll add a workaround for this to various functions in the OF io package and change status of this bug to "postponed".

Philip Nienhuis <philipnienhuis>
Group Member
Fri 29 Jun 2012 09:19:38 PM UTC, comment #2: 

Thanks Rik.

Well, it works to some extent, but now I get other bugs (specific for Windows).
See below for a CLI session, sorry *nix users for the Windows-specific idiosyncrasies:


Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

  rem ========== go to Octave bin dir ===============

C:\Home\Philip>cd c:\programs\octave\octave-3.6.2\bin

  rem ========== set a working dir on drive F: ======

C:\Programs\Octave\Octave-3.6.2\bin>cd f:\svn\octave.sf.net\extra\java

C:\Programs\Octave\Octave-3.6.2\bin>dir f:
 Volume in drive F is BIGWIN1
 Volume Serial Number is 746C-E43B

 Directory of F:\svn\octave.sf.net\extra\java

28-Apr-12  19:50    <DIR>          .
28-Apr-12  19:50    <DIR>          ..
28-Apr-12  19:51    <DIR>          doc
28-Apr-12  19:52                75 configure
28-Apr-12  19:52               369 DESCRIPTION
28-Apr-12  19:52               110 PKG_DEL
28-Apr-12  19:52    <DIR>          src
28-Apr-12  19:52               372 INDEX
28-Apr-12  19:52    <DIR>          inst
28-Apr-12  19:52            17 983 COPYING
28-Apr-12  19:52             1 092 PKG_ADD
28-Apr-12  19:52             4 035 Makefile
29-Jun-12  19:05             3 226 preinstall.m
               8 File(s)         27 262 bytes
               5 Dir(s)     823 967 744 bytes free

  rem ========== OK, now starting Octave ===========

C:\Programs\Octave\Octave-3.6.2\bin>octave
GNU Octave, version 3.6.2
<snip>
Octave was configured for "i686-pc-mingw32".
<snip>

octave:1> pwd
ans = c:\home\philip\mydocs\octave

octave:2> filename = 'notfound.fil'
filename = notfound.fil
octave:3> make_absolute_filename (filename)
ans = c:\home\philip\mydocs\octave\notfound.fil
  # Correct

octave:4> filename = '..\notfound.fil'
filename = ..\notfound.fil
octave:5> make_absolute_filename (filename)
ans = c:\home\philip\mydocs\notfound.fil
  # Correct

octave:6> filename = 'f:PKG_ADD'
filename = f:PKG_ADD
octave:7> make_absolute_filename (filename)
ans = c:\home\philip\mydocs\octave\f:PKG_ADD
  # Wrong
octave:8> canonicalize_file_name (filename)
ans = F:\svn\octave.sf.net\extra\java\PKG_ADD
  # Correct


(Probably unneeded explanation for Windows haters:)
On windows you have a current directory on every drive (-letter). So while you're on drive C:, accessing a file on another drive depends on that other drive's current dir.
That's where make_absolute_filename() fails, but canonicalize_file_name() works correctly.

So yes, in principle if on *nix I could use make_absolute_filename() and on Windows canonicalize_file_name()

But, could canonicalize_file_name() be made to fall back on make_absolute_filename() on *nix, and vice versa on Windows?

That would be a "better" solution, not so much because I wouldn't have to fix the io pkg scripts then, but rather because the two functions would work consistently (cross-platform) in core Octave.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 29 Jun 2012 07:49:20 PM UTC, comment #1: 

Philip,

You have indeed found an inconsistency.  Check the source code for file-ops.cc in the liboctave directory.  The code has a number of #idefs.  We first try to use the gnulib module, then we try to use the RESOLVEPATH module, then we look to see whether were on a Win32 platform.

For your specific application, would make_absolute_filename() work?  It appears to work correctly for a non-existent file in the pwd, in a relative directory location '../', and in an absolute location '/tmp/'.  The testing was done on a Linux platform.

Rik <rik5>
Group administrator
Mon 18 Jun 2012 05:27:39 PM UTC, original submission:  

canonicalize_file_name behaves differently under Linux than under Windows (file 'notfound.fil' doesn't exist):

Windows (both MSVC & MinGW) Octave-3.6.2_MSVC

octave-3.6.2.exe:1> filename = 'notfound.fil'
filename = notfound.fil
octave-3.6.2.exe:2> canonicalize_file_name (filename)
ans = c:\home\philip\mydocs\octave\notfound.fil
octave-3.6.2.exe:3> filename = '..\notfound.fil'
filename = ..\notfound.fil
octave-3.6.2.exe:4> canonicalize_file_name (filename)
ans = c:\home\philip\mydocs\notfound.fil
octave-3.6.2.exe:5> pwd
ans = c:\home\philip\mydocs\octave


Linux: (Octave-3.6.1 (Mandriva)  &  tip d.d. June 17, 2012)

octave-3.6.1:1> filename = 'notfound.fil'
filename = notfound.fil
octave-3.6.1:2> ls filename
ls: cannot access filename: No such file or directory
error: ls: command exited abnormally with status 2
octave-3.6.1:2> canonicalize_file_name (filename)
ans =
octave-3.6.1:3> filename = './notfound.fil'
filename = ./notfound.fil
octave-3.6.1:4> canonicalize_file_name (filename)
ans =
octave-3.6.1:5>


Another example (on Linux:)

octave-3.6.1:1> filename = '/tmp/notfound.fil'
filename = /tmp/notfound.fil
octave-3.6.1:2> canonicalize_file_name (filename)
ans =
octave-3.6.1:3>


Some more background:

I wouldn't be able to tell for sure on which of the two platforms canonicalize_file_name behaves faulty (plus, I can't test on Mac OSX).
But the behavior is inconsistent across platforms.

As for me, I would prefer that on Linux, like on Windows, canonicalize_file_name assumes that a non-existing file would reside in the current directory.

This is a.o., needed in the UNO (OpenOffice.org) sections of the io package, where to be able to open or create a file, OOo needs the fully qualified file name.
On Windows the same seems to be needed for COM/ActiveX.

A clumsy workaround would be to first test for existence, and if nonexistent, prepend the filename with
['pwd' filesep]
but this would break for nonexistent files with fully qualified pathnames in another directory, as the third example shows.
Obviously, more involved workarounds can be imagined; but IMO that functionality should rather be contained in the canonicalize_file_name function.

Philip Nienhuis <philipnienhuis>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #46404:  bug36677_canonicalize_absolute.patch added by mmuetzel (3KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by philipnienhuis (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 18 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-03-04 mtmiller StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2019-03-04 mmuetzel StatusIn Progress Ready For Test
    2019-03-03 mmuetzel Attached File- Added bug36677_canonicalize_absolute.patch, #46404
        StatusReady For Test In Progress
        Open/ClosedClosed Open
    2019-03-03 mtmiller Carbon-CopyRemoved 80942 -
    2019-03-03 mmuetzel StatusNone Ready For Test
        Open/ClosedOpen Closed
    2015-05-22 rik5 Priority5 - Normal 3 - Low
        Item GroupIncorrect Result Feature Request
        StatusPostponed None
    2015-02-03 rik5 CategoryLibraries Octave Function
        Operating SystemAny Microsoft Windows
        Summarycanonicalize_file_name: inconsistent behavior for non-existing files make_absolute_filename: extend to work with Windows drive letters
    2014-01-19 mtmiller CategoryNone Libraries
    2013-01-20 philipnienhuis StatusNeed Info Postponed
    2012-06-29 rik5 StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code