bugmake - Bugs: bug #40344, Can't handle Windows path names...

 
 

bug #40344: Can't handle Windows path names longer than 259 characters

Submitter:  Mike Hommey <glandium>
Submitted:  Tue 22 Oct 2013 04:49:10 AM UTC
   
 
Severity:  1 - Wish Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.0 Operating System:  MS Windows
Fixed Release:  None Triage Status:  Major Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 11 Mar 2014 06:15:48 PM UTC, comment #10: 

Note that being able to create file names longer than 259 characters is a mixed blessing: once created, these files cannot be manipulated by any Windows utilities, except if those utilities themselves use the techniques mentioned in this discussion.  Having at some point gotten myself into that trap, I now very much advise people not to do that, if they can, because chances are you will have to write your own utilities to delete, move, and copy these files, as none of the programs available on Windows can handle them.

Caveat emptor!

Eli Zaretskii <eliz>
Group Member
Tue 11 Mar 2014 08:16:01 AM UTC, comment #9: 

Vladimir, when will your patch be ready?

Tim Lebedkov <nasf82309>
Fri 29 Nov 2013 09:17:34 AM UTC, comment #8: 

One other thing: such a patch will probably be quite substantial, so you will need legal paperwork to assign the copyright to the FSF, before the patch can be admitted.  If that's fine with you, I suggest to start the paperwork ASAP, as it can take a few weeks.

Thanks.

Eli Zaretskii <eliz>
Group Member
Fri 29 Nov 2013 08:34:03 AM UTC, comment #7: 

Before passing file names to wide (a.k.a. "Unicode") APIs, you will need to convert the names to wchar_t, of course.  But I'm not sure this is even required; some parts of MSDN documentation seem to indicate that just using the \\?\ form is enough even for ANSI APIs.  (Which makes sense, since the ANSI APIs are just thin wrappers around Unicode APIs.)  So I suggest to try that first, it might save you some work.  It will also avoid breaking the Windows port on Windows 9X, which don't have the Unicode APIs.  Actually, I wonder whether \\?\ names will even work on Windows 9X; so perhaps the conversion to \\?\ should be conditioned on the OS version, to be safe.

Yes, you will have to normalize all file names, and make sure they use backslashes (forward slashes are not supported by the \\?\ format).  The tricky part about that is that some APIs that compute an absolute file name fail if the file does not exist, while file creation via fopen, _creat, etc. must work for non-existing files as well.  Also, the normalization must remove all "." and ".." parts, I think, as the \\?\ form cannot have these.

Such a patch will be welcome, indeed, but please use names like w32_fopen, w32_stat, etc., and #define the standard names to those.

Note that you will also need to do similar surgery with the functions that start processes, in sub_proc.c etc.  And then there's an issue with the programs Make invokes, because if they get a too-long file name, they will fail more often than not, since most programs don't use the \\?\ format of file names and don't support it.  There's nothing Make can do about these.

TIA

Eli Zaretskii <eliz>
Group Member
Fri 29 Nov 2013 05:51:56 AM UTC, comment #6: 

A full fix could be to wrap all of the functions that directly take paths (e.g. fopen, stat, etc.) with versions that convert the path to a full \\?\-based path, and then pass that to the wchar_t-based versions of those functions in the Windows standard library.

The only tricky thing there is that \\?\-based paths can't be relative; so every path would need to be converted to an absolute path.  That shouldn't be that difficult, though.

Would such a patch be considered if I were to write it?  It would involve either #define'ing things like fopen, stat, etc. to, say, fopen_longpath, stat_longpath in makeint.h, and implementing those.

Vladimir Vukicevic <vukicevic>
Tue 22 Oct 2013 04:01:06 PM UTC, comment #5: 

For the record, the place where the long file name causes trouble is in name_mtime, where Make calls stat to get the file's time stamp.  It looks like Windows prepends the current directory to the file name, without paying attention to the fact that the leading string of ../../.. makes the absolute file name shorter.  The result of prepending the current directory is longer than 259 characters, so stat fails, and Make treats the file as non-existent.

One possible workaround is to use $(abspath), but I didn't try that.

Eli Zaretskii <eliz>
Group Member
Tue 22 Oct 2013 08:28:53 AM UTC, comment #4: 

Make does not necessarily prepend the current directory, but the library functions it calls might do that internally, as part of file-name normalization.  (So it's not the current directory that counts, but CURRENT_DIR/../../../../../../ in your case.)  When they do that, and the result exceeds 259 characters, all these functions fail due to the limitation.

As for SUBST, you can use it to work around this because then the name will be shorter, like X:/this-is-a-long-path-component-0/this-is-a-long-filename.txt.  Windows APIs never normalize file names beyond the drive letter, even if it is a SUBSTed drive letter.

Eli Zaretskii <eliz>
Group Member
Tue 22 Oct 2013 07:50:47 AM UTC, comment #3: 

It looks like it would be the case, as

success: ../../../../../../this-is-a-long-path-component-0/this-is-a-long-filename.txt

fails too, with the current path being the original long path. I don't think there's any SUBST or CD way out of this.

Mike Hommey <glandium>
Tue 22 Oct 2013 07:46:27 AM UTC, comment #2: 

Note that "../../../../../../this-is-a-long-path-component-0/this-is-a-long-path-component-1/this-is-a-long-path-component-2/this-is-a-long-path-component-3/this-is-a-long-path-component-4/this-is-a-long-path-component-5/this-is-a-long-filename.txt" is actually 238 characters long. Is make prepending the full current directory path too?

Mike Hommey <glandium>
Tue 22 Oct 2013 07:13:34 AM UTC, comment #1: 

I presume the result exceeded the 259-character limit of file names that Win32 APIs and CRT library functions can handle.  Make doesn't support longer names, as it relies on APIs which normalize file names, including (but not limited to) automatic conversion of forward slashes to backslashes.  Switch to using Unicode APIs is a major effort in any program which came from Unix and assumes that character strings can be processed with 'char ' pointers and library functions that accept 'char ' arguments.

Patches are welcome to lift this limitation.  Until then, use SUBST or CD to work around it.

Eli Zaretskii <eliz>
Group Member
Tue 22 Oct 2013 04:49:10 AM UTC, original submission:  

(Reported by Vladimir Vukicevic on the mozilla bug tracker)

make 4.0 can't handle long path names, which need some special magic under windows -- you need to use the unicode versions of the IO functions, and prepend paths with the \\?\ UNC prefix.  (Note that also turns off automatic / -> \ conversion.)  So, \\?\C:\foo\bar\baz instead of C:/foo/bar/baz or similar.

STR:

$ cd c:/tmp   # or wherever

$ mkdir -p this-is-a-long-path-component-0/this-is-a-long-path-component-1/this-is-a-long-path-component-2/this-is-a-long-path-component-3/this-is-a-long-path-component-4/this-is-a-long-path-component-5

$ cd !$

$ echo hello > this-is-a-long-filename.txt

$ cat > Makefile <<EOF

all: success

success: ../../../../../../this-is-a-long-path-component-0/this-is-a-long-path-component-1/this-is-a-long-path-component-2/this-is-a-long-path-component-3/this-is-a-long-path-component-4/this-is-a-long-path-component-5/this-is-a-long-filename.txt
        cp $< $@
EOF

$ make
make: * No rule to make target '../../../../../../this-is-a-long-path-component-0/this-is-a-long-path-component-1/this-is-a-long-path-component-2/this-is-a-long-path-component-3/this-is-a-long-path-component-4/this-is-a-long-path-component-5/this-is-a-long-filename.txt', needed by 'success'.  Stop.

Mike Hommey <glandium>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by nasf82309 (Posted a comment)
  • -email is unavailable- added by vukicevic (Posted a comment)
  • -email is unavailable- added by eliz (Posted a comment)
  • -email is unavailable- added by glandium (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-10-22 eliz Severity3 - Normal 1 - Wish
        Item GroupBug Enhancement
        Triage StatusNone Major Effort
        SummaryCan\'t handle Windows long path names Can't handle Windows path names longer than 259 characters

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code