bugGNU Wget - Bugs: bug #21714, File name too long

 
 

bug #21714: File name too long

Submitted by:  John Doe <johndoe32102002>
Submitted on:  Tue 04 Dec 2007 05:29:29 AM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 7 - HighStatus: Fixed
Privacy: PublicAssigned to: Giuseppe Scrivano <gscrivano>
Originator Name: JohnOpen/Closed: Closed
Release: 1.10.2Operating System: GNU/Linux
Reproducibility: Every TimeFixed Release: None
Planned Release: 1.12.xRegression: Yes
Work Required: 0 - HoursPatch Included: No

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 29 Sep 2012 02:21:24 PM UTC, comment #51:

Thanks. I've verified that the current git version fixes the problem.

Frank Heckenbach <frank>
Sat 29 Sep 2012 11:49:26 AM UTC, comment #50:

Applied as commit: 67e6027ea130d06aeff365adfbc83f34d019b968

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Fri 28 Sep 2012 04:35:09 PM UTC, comment #49:

I am going to take a look at the patch in the next few days.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Fri 28 Sep 2012 03:05:32 AM UTC, comment #48:

What exactly are we supposed to do to get this patch finally included?

We (i.e., I and other contributors) have repeatedly answered to the maintainer's questions, fixed issues they pointed out, ported the fix to several new versions of wget, etc.

Now that everything's resolved, the issue seems to be ignored.

It's been going on for years now.

So what should we do to move on???

Frank Heckenbach <frank>
Mon 09 Jul 2012 06:43:07 PM UTC, comment #47:

Hello,

just as a reminder. Maybe it just got forgoten. Could the patch be added?

Thank you.

Noël Köthe <nok>
Wed 30 May 2012 12:03:22 AM UTC, comment #46:

Looks good to me.

Frank Heckenbach <frank>
Tue 29 May 2012 11:23:49 AM UTC, comment #45:

Sorry for the delay - I had been off pc.

ret is long again and spaces for errno=0 added.

I changed the return type of get_max_length() to size_t, since the return value is directly compared to strlen().

Instead of returning INT_MAX - which was inconvenient anyway, should have been LONG_MAX - it returns 0 for 'no limit' / 'unknown limit'.

Any complaints ?

Tim Ruehsen <rockdaboot>
Project Member
Fri 18 May 2012 07:43:17 PM UTC, comment #44:

Two minor things about the new patch, both in get_max_length ():

- Why did you change the type of ret from long to int? pathconf () returns long, at least on GNU/Linux.

- Why did you remove the spaces in "errno = 0;"?

Frank Heckenbach <frank>
Fri 18 May 2012 03:11:50 PM UTC, comment #43:

Just accidentially named the path 1.14,.
I meant ;-) 1.13.4, fixed up for current git.

Why hasn't this feature been applied yet ?
It would also close Debian bug #672131.

Tim Ruehsen <rockdaboot>
Project Member
Thu 29 Mar 2012 06:18:12 PM UTC, comment #42:

Can this patch now finally be included, or is anything missing?

Frank Heckenbach <frank>
Sun 27 Nov 2011 11:29:55 AM UTC, comment #41:

I wrote:

> So perhaps you should use the current directory (or the target directory, resp.) instead of "/". Of course then the function couldn't cache the result since it might be different for different files. However, the cost of a pathconf() call is probably negligible compared to downloading a file through the net.


I did that now. A special case to note is when the target directory doesn't exist yet (it may be created in the current retrieval). In this case, I strip directories and check the path again until I find an existing one. Since the newly created directories will be on the same file system as their parent directories, this should be safe.

I also adopted the patch for 1.13. I hope it's finally ready for inclusion now.

(wget_filename_length.v3-1.13.patch)

(file #24473)

Frank Heckenbach <frank>
Sat 12 Nov 2011 04:40:54 AM UTC, comment #40:

> So perhaps you should use the current directory (or the target directory, resp.) instead of "/". Of course then the function couldn't cache the result since it might be different for different files. However, the cost of a pathconf() call is probably negligible compared to downloading a file through the net.


I believe Micah brought up the same point earlier (comment #28). I know I'm still assigned to this bug, but I won't have time to change anything now. It's been a couple of years (almost 3?!) since I submitted my patch and that is the only work I've done on wget. I suspect if that safety improvement is really needed (checking target directory params versus root directory), then I won't have the time to get it done. Maybe someone else will add that check?

Anyway, glad the patch is still more-or-less working for folks (I know it's still doing the job on our server). Hopefully the fix makes it into the next release (even without the safer file system check).

Joseph Rios <alotau>
Fri 11 Nov 2011 11:50:39 PM UTC, comment #39:

> Here it goes (sorry for italian message: "Nome del file troppo lungo" means "File name too long")


BTW, if you run it with "LC_ALL=C /opt/wget...", you get default (English) messages.

> >Are you sure you've applied the patch properly?
>
> I'm sorry, I messed up with patching!
>
> I do confirm it does work indeed on wget 1.12!
>
> Thank you for the good job!


Good to hear so.

Though when investigating the previous message, I saw a possible problem (but even then the patch in an improvement to the status quo):

get_name_max() does:

if ((ret = pathconf ("/", _PC_NAME_MAX)) == -1)
if ((ret = pathconf ("/", _PC_PATH_MAX)) == -1)

But the manpage says:

_PC_NAME_MAX
returns the maximum length of a filename in the directory path or fd that the process is allowed to create.
_PC_PATH_MAX
returns the maximum length of a relative pathname when path or fd is the current working directory.

So imagine the situation that the root file system allows a larger limit than the current directory. It would probably still fail (I can't test it easily).

So perhaps you should use the current directory (or the target directory, resp.) instead of "/". Of course then the function couldn't cache the result since it might be different for different files. However, the cost of a pathconf() call is probably negligible compared to downloading a file through the net.

Frank Heckenbach <frank>
Fri 11 Nov 2011 06:42:26 PM UTC, comment #38:

>Are you sure you've applied the patch properly?


I'm sorry, I messed up with patching!

I do confirm it does work indeed on wget 1.12!

Thank you for the good job!

Giovanni <timido>
Thu 10 Nov 2011 09:33:29 PM UTC, comment #37:

Are you sure you've applied the patch properly? There are some logprintf's that I had included that I don't see in your example output.

From the patch:

+ if (strlen (temp_fnres.base) > get_name_max () - CHOMP_BUFFER)
{
- append_char (FN_QUERY_SEP, &fnres);
- append_uri_pathel (u_query, u_query + strlen (u_query), true, &fnres);
+ logprintf (LOG_NOTQUIET, "The name is too long, %lu chars total.\n",
+ (unsigned long) strlen (temp_fnres.base));
+ logprintf (LOG_NOTQUIET, "Trying to shorten...\n");
+
+ /* Shorten the file name. */
+ temp_fnres.base[get_name_max () - CHOMP_BUFFER] = '\0';
+
+ logprintf (LOG_NOTQUIET, "New name is %s.\n", temp_fnres.base);
}

Or am I missing something else?

Joseph Rios <alotau>
Thu 10 Nov 2011 05:24:29 PM UTC, comment #36:

> What doesn't work, which version and system?


Linux lenny 2.6.26-2-686
ext3
wget 1.12 patched with yours

Here it goes (sorry for italian message: "Nome del file troppo lungo" means "File name too long")

$ /opt/wget-1.12-patch/bin/wget "google.de?q=`seq 200|tr -d 'n'`"
--2011-11-10 06:31:50-- http://google.de/?q=1%0A2%0A3%0A4%0A5%0A6%0A7%0A8%0A9%0A10%0A11%0A12%0A13%0A14%0A15%0A16%0A17%0A18%0A19%0A20%0A21%0A22%0A23%0A24%0A25%0A26%0A27%0A28%0A29%0A30%0A31%0A32%0A33%0A34%0A35%0A36%0A37%0A38%0A39%0A40%0A41%0A42%0A43%0A44%0A45%0A46%0A47%0A48%0A49%0A50%0A51%0A52%0A53%0A54%0A55%0A56%0A57%0A58%0A59%0A60%0A61%0A62%0A63%0A64%0A65%0A66%0A67%0A68%0A69%0A70%0A71%0A72%0A73%0A74%0A75%0A76%0A77%0A78%0A79%0A80%0A81%0A82%0A83%0A84%0A85%0A86%0A87%0A88%0A89%0A90%0A91%0A92%0A93%0A94%0A95%0A96%0A97%0A98%0A99%0A100%0A101%0A102%0A103%0A104%0A105%0A106%0A107%0A108%0A109%0A110%0A111%0A112%0A113%0A114%0A115%0A116%0A117%0A118%0A119%0A120%0A121%0A122%0A123%0A124%0A125%0A126%0A127%0A128%0A129%0A130%0A131%0A132%0A133%0A134%0A135%0A136%0A137%0A138%0A139%0A140%0A141%0A142%0A143%0A144%0A145%0A146%0A147%0A148%0A149%0A150%0A151%0A152%0A153%0A154%0A155%0A156%0A157%0A158%0A159%0A160%0A161%0A162%0A163%0A164%0A165%0A166%0A167%0A168%0A169%0A170%0A171%0A172%0A173%0A174%0A175%0A176%0A177%0A178%0A179%0A180%0A181%0A182%0A183%0A184%0A185%0A186%0A187%0A188%0A189%0A190%0A191%0A192%0A193%0A194%0A195%0A196%0A197%0A198%0A199%0A200
Risoluzione di google.de (google.de)... 74.125.39.103, 74.125.39.104, 74.125.39.105, ...
Connessione a google.de (google.de)|74.125.39.103|:80... connesso.
HTTP richiesta inviata, in attesa di risposta... 301 Moved Permanently
Posizione: http://www.google.de/?q=1%0A2%0A3%0A4%0A5%0A6%0A7%0A8%0A9%0A10%0A11%0A12%0A13%0A14%0A15%0A16%0A17%0A18%0A19%0A20%0A21%0A22%0A23%0A24%0A25%0A26%0A27%0A28%0A29%0A30%0A31%0A32%0A33%0A34%0A35%0A36%0A37%0A38%0A39%0A40%0A41%0A42%0A43%0A44%0A45%0A46%0A47%0A48%0A49%0A50%0A51%0A52%0A53%0A54%0A55%0A56%0A57%0A58%0A59%0A60%0A61%0A62%0A63%0A64%0A65%0A66%0A67%0A68%0A69%0A70%0A71%0A72%0A73%0A74%0A75%0A76%0A77%0A78%0A79%0A80%0A81%0A82%0A83%0A84%0A85%0A86%0A87%0A88%0A89%0A90%0A91%0A92%0A93%0A94%0A95%0A96%0A97%0A98%0A99%0A100%0A101%0A102%0A103%0A104%0A105%0A106%0A107%0A108%0A109%0A110%0A111%0A112%0A113%0A114%0A115%0A116%0A117%0A118%0A119%0A120%0A121%0A122%0A123%0A124%0A125%0A126%0A127%0A128%0A129%0A130%0A131%0A132%0A133%0A134%0A135%0A136%0A137%0A138%0A139%0A140%0A141%0A142%0A143%0A144%0A145%0A146%0A147%0A148%0A149%0A150%0A151%0A152%0A153%0A154%0A155%0A156%0A157%0A158%0A159%0A160%0A161%0A162%0A163%0A164%0A165%0A166%0A167%0A168%0A169%0A170%0A171%0A172%0A173%0A174%0A175%0A176%0A177%0A178%0A179%0A180%0A181%0A182%0A183%0A184%0A185%0A186%0A187%0A188%0A189%0A190%0A191%0A192%0A193%0A194%0A195%0A196%0A197%0A198%0A199%0A200 [segue]
--2011-11-10 06:31:51-- http://www.google.de/?q=1%0A2%0A3%0A4%0A5%0A6%0A7%0A8%0A9%0A10%0A11%0A12%0A13%0A14%0A15%0A16%0A17%0A18%0A19%0A20%0A21%0A22%0A23%0A24%0A25%0A26%0A27%0A28%0A29%0A30%0A31%0A32%0A33%0A34%0A35%0A36%0A37%0A38%0A39%0A40%0A41%0A42%0A43%0A44%0A45%0A46%0A47%0A48%0A49%0A50%0A51%0A52%0A53%0A54%0A55%0A56%0A57%0A58%0A59%0A60%0A61%0A62%0A63%0A64%0A65%0A66%0A67%0A68%0A69%0A70%0A71%0A72%0A73%0A74%0A75%0A76%0A77%0A78%0A79%0A80%0A81%0A82%0A83%0A84%0A85%0A86%0A87%0A88%0A89%0A90%0A91%0A92%0A93%0A94%0A95%0A96%0A97%0A98%0A99%0A100%0A101%0A102%0A103%0A104%0A105%0A106%0A107%0A108%0A109%0A110%0A111%0A112%0A113%0A114%0A115%0A116%0A117%0A118%0A119%0A120%0A121%0A122%0A123%0A124%0A125%0A126%0A127%0A128%0A129%0A130%0A131%0A132%0A133%0A134%0A135%0A136%0A137%0A138%0A139%0A140%0A141%0A142%0A143%0A144%0A145%0A146%0A147%0A148%0A149%0A150%0A151%0A152%0A153%0A154%0A155%0A156%0A157%0A158%0A159%0A160%0A161%0A162%0A163%0A164%0A165%0A166%0A167%0A168%0A169%0A170%0A171%0A172%0A173%0A174%0A175%0A176%0A177%0A178%0A179%0A180%0A181%0A182%0A183%0A184%0A185%0A186%0A187%0A188%0A189%0A190%0A191%0A192%0A193%0A194%0A195%0A196%0A197%0A198%0A199%0A200
Risoluzione di http://www.google.de (http://www.google.de)... 74.125.39.103, 74.125.39.104, 74.125.39.105, ...
Riutilizzo della connessione esistente a google.de:80.
HTTP richiesta inviata, in attesa di risposta... 200 OK
Lunghezza: non specificato [text/html]
index.html?q=1%0A2%0A3%0A4%0A5%0A6%0A7%0A8%0A9%0A10%0A11%0A12%0A13%0A14%0A15%0A16%0A17%0A18%0A19%0A20%0A21%0A22%0A23%0A24%0A25%0A26%0A27%0A28%0A29%0A30%0A31%0A32%0A33%0A34%0A35%0A36%0A37%0A38%0A39%0A40%0A41%0A42%0A43%0A44%0A45%0A46%0A47%0A48%0A49%0A50%0A51%0A52%0A53%0A54%0A55%0A56%0A57%0A58%0A59%0A60%0A61%0A62%0A63%0A64%0A65%0A66%0A67%0A68%0A69%0A70%0A71%0A72%0A73%0A74%0A75%0A76%0A77%0A78%0A79%0A80%0A81%0A82%0A83%0A84%0A85%0A86%0A87%0A88%0A89%0A90%0A91%0A92%0A93%0A94%0A95%0A96%0A97%0A98%0A99%0A100%0A101%0A102%0A103%0A104%0A105%0A106%0A107%0A108%0A109%0A110%0A111%0A112%0A113%0A114%0A115%0A116%0A117%0A118%0A119%0A120%0A121%0A122%0A123%0A124%0A125%0A126%0A127%0A128%0A129%0A130%0A131%0A132%0A133%0A134%0A135%0A136%0A137%0A138%0A139%0A140%0A141%0A142%0A143%0A144%0A145%0A146%0A147%0A148%0A149%0A150%0A151%0A152%0A153%0A154%0A155%0A156%0A157%0A158%0A159%0A160%0A161%0A162%0A163%0A164%0A165%0A166%0A167%0A168%0A169%0A170%0A171%0A172%0A173%0A174%0A175%0A176%0A177%0A178%0A179%0A180%0A181%0A182%0A183%0A184%0A185%0A186%0A187%0A188%0A189%0A190%0A191%0A192%0A193%0A194%0A195%0A196%0A197%0A198%0A199%0A200: Nome del file troppo lungo

Impossibile scrivere in "index.html?q=1%0A2%0A3%0A4%0A5%0A6%0A7%0A8%0A9%0A10%0A11%0A12%0A13%0A14%0A15%0A16%0A17%0A18%0A19%0A20%0A21%0A22%0A23%0A24%0A25%0A26%0A27%0A28%0A29%0A30%0A31%0A32%0A33%0A34%0A35%0A36%0A37%0A38%0A39%0A40%0A41%0A42%0A43%0A44%0A45%0A46%0A47%0A48%0A49%0A50%0A51%0A52%0A53%0A54%0A55%0A56%0A57%0A58%0A59%0A60%0A61%0A62%0A63%0A64%0A65%0A66%0A67%0A68%0A69%0A70%0A71%0A72%0A73%0A74%0A75%0A76%0A77%0A78%0A79%0A80%0A81%0A82%0A83%0A84%0A85%0A86%0A87%0A88%0A89%0A90%0A91%0A92%0A93%0A94%0A95%0A96%0A97%0A98%0A99%0A100%0A101%0A102%0A103%0A104%0A105%0A106%0A107%0A108%0A109%0A110%0A111%0A112%0A113%0A114%0A115%0A116%0A117%0A118%0A119%0A120%0A121%0A122%0A123%0A124%0A125%0A126%0A127%0A128%0A129%0A130%0A131%0A132%0A133%0A134%0A135%0A136%0A137%0A138%0A139%0A140%0A141%0A142%0A143%0A144%0A145%0A146%0A147%0A148%0A149%0A150%0A151%0A152%0A153%0A154%0A155%0A156%0A157%0A158%0A159%0A160%0A161%0A162%0A163%0A164%0A165%0A166%0A167%0A168%0A169%0A170%0A171%0A172%0A173%0A174%0A175%0A176%0A177%0A178%0A179%0A180%0A181%0A182%0A183%0A184%0A185%0A186%0A187%0A188%0A189%0A190%0A191%0A192%0A193%0A194%0A195%0A196%0A197%0A198%0A199%0A200" (Nome del file troppo lungo).

Giovanni <timido>
Thu 10 Nov 2011 01:12:15 AM UTC, comment #35:

Giovanni <timido> wrote:

> Doesnt seem to work even with patch applied!


What doesn't work, which version and system?

The example I gave still works for me (Linux 2.6.34, ext[2-4], wget 1.12):

wget "google.de?q=`seq 200|tr -d '\n'`"

> Are you sure Linux VFS allows wget to save a file whose name is longer than 255 bytes ?


No, and that's just the point. The patch makes wget shorten the
output file name.

> The issue is when wget creates temporary files from following HTTP redirections... which you cannot save with -O option indeed


Or with recursive retrieval if one of the linked files has a longer name.

Frank Heckenbach <frank>
Wed 09 Nov 2011 06:39:24 PM UTC, comment #34:

Doesnt seem to work even with patch applied!

Are you sure Linux VFS allows wget to save a file whose name is longer than 255 bytes ?

This problem still is in wget version 1.13.4 .

The issue is when wget creates temporary files from following HTTP redirections... which you cannot save with -O option indeed

Giovanni <timido>
Mon 13 Jun 2011 04:27:50 AM UTC, comment #33:

Rob Mangiafico wrote:

> Is there an updated patch for version 1.12? I tried massaging the 1.11 patch, but now filenames that are short have random invalid characters appended to the filenames.


I did the same and found the same problem. It was because temp_fnres wasn't 0-terminated. The bug is not actually related to the version, just happens to occur now (and perhaps even before, unnoticed).

I also fixed the problems Micah pointed out except "Why is temp_fnres a growable?" (I'm not familiar with those types and at least it seems to work as is, if maybe a little less efficient).

I'll upload my patch here. I hope it's ready for inclusion now. Should I do anything else to make that happen?

To test it:

Before:

% wget "google.de?q=`seq 200|tr -d '\n'`"
[...]
Cannot write to `index.html?q=[...]' (File name too long).
% echo $?
3

With the patch:

% wget "google.de?q=`seq 200|tr -d '\n'`"
The name is too long, 505 chars total.
Trying to shorten...
New name is index.html?q=[...].
[...]
% echo $?
0

(file #23520)

Frank Heckenbach <frank>
Fri 01 Apr 2011 02:15:26 AM UTC, comment #32:

Is there an updated patch for version 1.12? I tried massaging the 1.11 patch, but now filenames that are short have random invalid characters appended to the filenames.

Rob Mangiafico <robman>
Mon 02 Aug 2010 06:19:21 PM UTC, comment #31:

Hello,
I have this same problem, it's in wget version 1.12 as well.

How this patch does apply to ?
Is it a final version?

Thank you for support.

Giovanni <timido>
Thu 10 Sep 2009 06:19:32 AM UTC, comment #30:

> Why is temp_fnres a growable?


I wanted to use the append_uri_pathel() function which seemed to be doing smarter things than buffer/string tricks I'd play on my own.

Joseph Rios <alotau>
Fri 04 Sep 2009 03:17:03 AM UTC, comment #29:

Some additional comments on the latest patch.

It's not quite "GNU-style"; the indentation is good, but there's a number of places where the spacing around parentheses is off: "if( foo )" instead of "if (foo)".

Why is temp_fnres a growable? ... you never grow it, and I don't see any usage that wouldn't have been just as good with an ordinary buffer. Also, I don't understand why you're restoring the string length just before freeing it altogether. free() doesn't care where the NUL is; it doesn't even know it's a string.

printf("%ld", strlen(...)) is dangerous. Make sure the return size of the variable matches the specifier: size_t's size isn't guaranteed to match a long. And strlen will never return a signed value, so your best bet is something like printf("%lu", (unsigned long) strlen(...)).

BTW, unified diffs are easier for me to deal with than contextual diffs; easier for me to double-check that they were applied correctly.

Micah Cowan <micahcowan>
Project Administrator
Tue 01 Sep 2009 05:45:58 AM UTC, comment #28:

You're right that the fallback behavior may not be necessary; but I'm still not crazy about running a risk of prematurely truncating the filename before we're certain it's too big (especially since get_max_name isn't necessarily checking the same file system). The chances of checking the right filesystem could be improved if we invoke pathconf on the directory argument to -P (or the current working directory, if -P wasn't used), though it still wouldn't be fool-proof (the fail cases, however, are quite tolerably rare).

What about a boolean option, instead of a max-length parameter? When we get ENAMETOOBIG we could just toggle the flag, and call url_file_name just one more time. Basically, I'd just like to eliminate any potential for already-working behavior to change.

Micah Cowan <micahcowan>
Project Administrator
Tue 01 Sep 2009 04:50:49 AM UTC, comment #27:

Your suggestions make sense, but I wonder at the necessity. I think the patch fixes the bug as described by all who reported it. If NAME_MAX has issues on someone's system, then a working wget for long file names may work around a problem that should be addressed elsewhere. I guess I'm saying that adding the additional fail-safe you describe (for the reasons you describe) seems like we're fixing a problem that isn't wget's to worry about. I understand, though, that pro-action is good too.

I'm not trying to be lazy and am open to working on it some more (though I'm not sure when)... I'm just opening this for further discussion before I do so.

Joseph Rios <alotau>
Sun 30 Aug 2009 04:11:36 AM UTC, comment #26:

So, what I liked about the original patch was that it waited until it was sure that the name was going to be too long before it adjusted anything; what I like about the new patches is that it uses {NAME_MAX} intelligently. But it would also have been nice for it to fall back on the first patch's behavior, in the event that {NAME_MAX} is wrong or something.

What about something that does the following?

- Modify url_file_name to take a maximum-length parameter, with a potential value representing no limit.
- First time url_file_name gets called, give it the no-limit parameter
- If http.c gets an ENAMETOOBIG error, then use get_max_name for that parameter, and call url_file_name again.
- If we still get ENAMETOOBIG, use the first patch's behavior and keep chomping off the filename.

Micah Cowan <micahcowan>
Project Administrator
Sun 05 Jul 2009 09:50:35 PM UTC, comment #25:

Pursuing copyright assignment.

Micah Cowan <micahcowan>
Project Administrator
Fri 06 Feb 2009 07:20:18 PM UTC, comment #24:

I've clean the code to make it "GNU style". I'll consider this the final patch and that the bug is fixed unless I hear otherwise. I have placed TODO tags in the code where there should be a check for path length violations. I didn't do that check because I have no idea what to do if the path is too long. I don't see an obvious solution. In addition, that hasn't been the complaint I originally had or any users have posted.

Below is the result of the only test I ran on the code. On the first try with standard wget, the 'File Name Too Long' error occurs. I then use the patched version and the page is saved. I then run it again and another copy is saved, demonstrating that the *.1 suffix didn't break anything on the second download.

This is my first GNU bug fix, so if there is something else I am supposed to do, please let me know.

Thanks.

(file #17414)

Joseph Rios <alotau>
Fri 06 Feb 2009 04:37:55 AM UTC, comment #23:

I'm uploading a new patch. The major changes are in url_file_name() (of url.c) as discussed below. I also added a couple of functions to util.c: get_name_max() and get_path_max(). I think I wrote them in a way such that repeated calls won't generate additional system calls.

Instead of chomping 1/3's off of the file length, I actually just trim it to a preferred length. I calculate the preferred length based on the value of get_name_max() and a rough guess of how much junk might be appended to the file name later in the code (specifically in get_http() I think). Things like ".html" and ".orig" and various numerals might get appended after url_file_name().

I don't think the code is in the preferred style, but I can fix that if it seems that the patch does the right thing.

Thanks for the quick response earlier, David. I didn't think about sending junk params to increase the URL length.

If anyone can provide feedback, the sooner you can get it to me the better. I can't say how much more time I'll have to work on this after tomorrow.

Thanks.

(file #17409)

Joseph Rios <alotau>
Fri 06 Feb 2009 12:45:27 AM UTC, comment #22:

Hi Joseph,

Here's one of the longest URL that wget was able to save for me:

http://personal.optus.com.au/web/ocaportal.portal?_nfpb=true&_pageLabel=shoppingcart&FP=/personal/shoppingcart&LHP=/personal/customerhelp/producthelp/mobilehelp/howtoguidesmobile/usingsimbackup/nokia6280&site=personal

Maybe if you try to add some dummy parameters behind it would reach wget's limit.

Here's an example of what I get if I put dummy parameters at the end of the URL.

Note that the problem might be amplified by Windows' maximum pathname limitation.

David Han <kayhadrin>
Thu 05 Feb 2009 07:45:06 PM UTC, comment #21:

Hi Micah,

Can you (or anyone for that matter) provide a url upon which I can test my code? The site I used for testing previously doesn't seem to be offering long URLs anymore and I don't have any locally. Or, better yet, can you provide a wget command (or commands) complete with URLs that should DTRT when this bug is fixed?

I've been sitting on this for a while and have a little time to poke at it now.

Joseph Rios <alotau>
Wed 12 Nov 2008 11:15:28 PM UTC, comment #20:

Note that the current version of the patch doesn't DTRT when -E is given (that is, the final filename won't end in .html).

(Be that as it may, it still allowed me to download Savannah's 1.12 buglist, which the current Wget won't.)

Micah Cowan <micahcowan>
Project Administrator
Wed 29 Oct 2008 08:00:53 PM UTC, comment #19:

Yes, url_file_name is the perfect place for this.

We should probably determine the values for PATH_MAX and NAME_MAX just once, and then save the values away from that point on. The case where they will differ at different points in the hierarchy strikes me as so rare as not to be worth considering.

Then we could place fallback code around the actual point of opening the file. Although really, for a first go, PATH_MAX and NAME_MAX should be plenty for our needs.

Currently Wget finds a unique filename and then opens it a little later, which is a race condition (some other file can "pop into existence"). At some point we'll want to have the unique-name-finder actually attempt to open the file, and create it if it doesn't exist, so that we already have an open filehandle and know for certain that it's now ours. That would be an excellent point at which to introduce the fallback code, if you want to wait until then.

Micah Cowan <micahcowan>
Project Administrator
Wed 29 Oct 2008 07:17:40 PM UTC, comment #18:

It makes sense to do it as you describe. I was catching the error at the point the file failed to open, which is long after the path and filename had been determined. So any chomping at that point could dig into the path.

Figuring out the best place to perform the check for filename and path length will take some time since I'm not that familiar with the code yet. I started looking a bit more, though, so I can ask: does it make sense to perform the check in the url_file_name() function? It looks like that is where ultimate filename is created. Or maybe in unique_name() since that may be called after url_file_name() and could actually add to the ultimate length? Or do you have a sense for a better place?

If I don't hear from you, I'll use my best judgement. I probably won't get to this for a few days.

Joseph Rios <alotau>
Wed 29 Oct 2008 05:41:09 AM UTC, comment #17:

Ah. Well I never actually meant to chomp the path: I meant a third of the filename.

I'd strongly prefer the use of NAME_MAX (and possibly PATH_PAX) (obtained from pathconf, not macros), as that's supposed to be able to tell us exactly how long to make it: no guessing. The other's just for fallback.

I recommended chopping by thirds mainly because for long paths it's a lot quicker than shaving a few at a time (O(log_3 N) versus O(N)). If the max size is 1000 chars and you've got 9000, you'll get there in six steps, rather than 2,666 (with a step-down of 3).

Micah Cowan <micahcowan>
Project Administrator
Wed 29 Oct 2008 02:07:13 AM UTC, comment #16:

Hi Micah,

I think I have a fix that doesn't require explicit knowledge of the {NAME_MAX} and {PATH_MAX} macros, though I do query them diagnostically. Now instead of chomping of a set amount of characters and hoping for the best, I'm chomping off a few characters and trying again. I do that until either the filename is legal or fopen() fails for some other reason. I don't know if this is better or worse than cutting it by a third each time, but I think the spirit is the same. Chomping off a third makes me nervous because if the path is long, but the file name is actually short, we could be chomping into the path which would cause other problems. Right now I am the NAME_CHOMP_AMT is set to 3, but whatever seems reasonable to you is fine with me.

Thanks for the info on the C/POSIX stuff. I've never had to deal with that directly, so it was helpful looking up a little bit of it.

Let me know if you think the attached patch has any further issues to address in regards to this bug. Also let me know if you think it actually fixes the problem sufficiently.

Thanks for maintaining wget.

(file #16746)

Joseph Rios <alotau>
Tue 28 Oct 2008 06:04:30 PM UTC, comment #15:

Hi Joe, and thanks for the patches.

Some comments: actually, POSIX does have FILENAME_MAX, just like the C standard does. And {NAME_MAX} isn't necessarily the name of a macro: the most reliable way to get {NAME_MAX} is to use fpathconf or pathconf with the _PC_NAME_MAX macro, which will tell you what the limit is for the particular filesystem that happens to be in use at that particular location.

Don't forget, {NAME_MAX} represents the longest allowable path component, whereas {PATH_MAX} represents the longest allowable path. Both may be needed to choose an appropriate filename size.

It might not be a terrible idea to have some sort of fallback mechanism, in case judicial use of {NAME_MAX} and {PATH_MAX} still results in ENAMETOOLONG; maybe keep cutting the pathname by a third and see if it works sortuva thing.

Micah Cowan <micahcowan>
Project Administrator
Tue 28 Oct 2008 04:51:58 PM UTC, comment #14:

I'm trying to attach the patches that worked for me (never uploaded anything here, so hoping it works). The only changes were to http.c adding a new branch to an 'if' statement that checks if creating a new file fails and why it fails. Added one line to http.h as well. Someone emailed me for this patch, so I thought it best to post it here. Hopefully if anyone tries it out, he/she can let post a message here letting us know if it works. And if Micah takes a look at it, he can let us know if it seems reasonable and that it doesn't break anything else.

(file #16744, file #16745)

Joseph Rios <alotau>
Thu 09 Oct 2008 03:16:57 AM UTC, comment #13:

Micah,

I've made some changes to http.c and util.c which serve my purposes and may serve others as well.

Essentially, I recognize when a file name is too long and then chomp off some number of characters then pass that string to unique_name() for a new file name. This pretty much follows your suggestions I think.

I haven't extensively tested it, nor made the code "pretty" yet, but I am certain it works for my application of wget. Also, it shouldn't break anything else since it only triggers when a long file name error is returned from the fopen() call. If it doesn't work for all cases, I think it is still better than what exists now (i.e. you never get the file at all).

Let me know if you'd like me to clean up the code and submit a patch or provide more details on my fix.

Joseph Rios <alotau>
Thu 09 Oct 2008 01:51:55 AM UTC, comment #12:

My apologies for raising that subject in the wrong place.
I reckon Micah's alternative is better though.

I'm a newbie here so I'll have to find out said mailing list ;-)

David Han <kayhadrin>
Wed 08 Oct 2008 07:05:08 PM UTC, comment #11:

Hi guys, an unrelated bug isn't really the place to discuss new feature ideas; that's really what the mailing list is for.

I would tend to prefer a more generalized solution: send the output to a user-specifiable command, which could then save it in the database or whatever it desires to do.

Micah Cowan <micahcowan>
Project Administrator
Wed 08 Oct 2008 03:34:18 PM UTC, comment #10:

Maybe a flag to choose database saving isn't a bad idea. Linux and Mac usually come with sqlite installed (right?). It's server-less, has a C/command-line API and saves the db as single file wherever you like, so folks wouldn't necessarily have to have a mysql server running. I'm sure this will take quite a bit more thought, but a workaround like this would definitely solve the problem I've been having with long file names.

Joseph Rios <alotau>
Wed 08 Oct 2008 03:44:16 AM UTC, comment #9:

Does anyone know if there's a tool to save wget's output to a database instead?

That may be a good workaround to be able to save any length of page URL and later for backup purpose.

David Han <kayhadrin>
 Spam posted by an anonymous
Wed 20 Aug 2008 08:28:04 PM UTC, comment #8:

Using URLs as the basis of filenames still makes tons of sense when they're not too long, though. Throwing out the baby with the bathwater isn't what is needed. And while long URLs are not an extreme rarity, calling them common is a bit of a stretch (I personally have yet to run across one in my own use; that might change if I were downloading bug reports from Savannah, though: it tends to have really huge URLs).

As to whether it's still useful to base alternatives to too-long filenames on a truncated form of the URL, I would find it so. And why not? It's not hard to do, and it still keeps as much of the original location information as intact as possible. So what advantage is there in eschewing the original URL entirely, and making up some arbitrary name?

Micah Cowan <micahcowan>
Project Administrator
Wed 20 Aug 2008 04:21:38 PM UTC, comment #7:

If the URLs are too long, relative links will still be broken when viewing the downloaded files in a browser, because the files simply won't be there. The attempt to save them using a too-long filename will fail.

I ran into this problem myself. It's the original reason why I posted a comment here.

John Nolan <fossjn>
Mon 21 Jul 2008 06:56:34 PM UTC, comment #6:

That's all well and good; but using URLs as the basis for filenames has the advantage that relative URLs are not broken when you then view the downloaded files in a browser.

That's less of a problem if you use the -k option.

Micah Cowan <micahcowan>
Project Administrator
Mon 21 Jul 2008 06:15:42 PM UTC, comment #5:

According to the RFCs, there is no theoretical limit to the size of a URL. In theory, a URL can be arbitrarily long.

Most browsers and servers have some kind of implementation limit on URL length, but often the implementation limits are generous. Even older mobile phone browsers, which may have severely limited device memory to work with, often allow 1024 bytes or more for a URL. I believe Firefox allows very long URLs.

http://www.boutell.com/newfaq/misc/urllength.html

This contrasts with filesystems, which typically have a much smaller length limit on filenames. Ext2 apparently allows only up to 255 characters for a filename.

In the general case, it really does not make sense to blindly attempt to use a URL as a filename. Long URLs are not even an edge case, they are common.

John Nolan <fossjn>
Mon 21 Apr 2008 06:56:47 PM UTC, comment #4:

I'm unclear how truncating the URL does not result in an artificial file name? Obviously, we'd "unique"-ize it, same as we do for other filenames.

Any rename whatsoever is likely to be useless, unless -k (and probably -E) has been enabled, to retain interlinking between the renamed page and other downloaded pages. Still, for some cases it may at least help to narrow down what the original page was.

The real issue that's likely to result, is that on a followup download with -c or -N, Wget won't know where to look for the local file, so will always redownload (to a new, unique filename - the proposed Session Info Database feature would solve this problem).

Micah Cowan <micahcowan>
Project Administrator
Mon 21 Apr 2008 05:39:18 PM UTC, comment #3:

I'd just like to add a suggestion based on my experience with this bug/issue. It seems that wget is actually able to download the "long file name" files with no problem, so the "truncate URL and try again" may be too harsh. Perhaps an option to "create artificial local file name and try again" would be more appropriate. wget could then download to that artificial file name. That would solve the issue with my wget script. From my experience, it seems that the long file name is due to a long list of cgi script inputs in the URL so truncation would likely be useless.

Anyway, thanks for addressing this issue, it's much appreciated.

Joseph Rios <alotau>
Sun 13 Jan 2008 09:41:19 PM UTC, comment #2:

This has been a problem for several users. It kind of sucks that there's no workaround.

Perhaps, if it gets a "filename too long" error, it could automatically truncate the path and try again? Or possibly add a 'cut out the query string from paths'-type deal. Both would require -k to be useful, though, it seems to me.

Micah Cowan <micahcowan>
Project Administrator
Tue 04 Dec 2007 05:42:57 AM UTC, comment #1:

I'm not sure about the "should not exist logically in wget" bit; systems tend to have maximum filename lengths, and the file here exceeded it. Still, it'd be nice to have Wget handle it better, say by picking a shorter name.

See also bug 21042, which has been made a duplicate of this one (but which also refers to an erroneous status message).

Micah Cowan <micahcowan>
Project Administrator
Tue 04 Dec 2007 05:29:29 AM UTC, original submission:

I am unable to save Java to my computer from wget because of this error:
(File name too long).

It seems there is no bypass for it, and it should not exist logically in wget.

John Doe <johndoe32102002>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #25881:  0001-filename-length-patch-v4-1.14.patch added by rockdaboot (7kB - text/x-diff - fixed up for current 1.13.4)
file #24473:  wget_filename_length.v3-1.13.patch added by frank (6kB - text/x-diff - patch updated for 1.13, check actual target directory)
file #23520:  wget_filename_length.v2-1.12.patch added by frank (7kB - text/x-diff - alotau's latest patch updated for 1.12, bug fixed)
file #17414:  wget_filename_length.v2.patch added by alotau (9kB - application/octet-stream - Final fix?)
file #17409:  wget_file_length.patch added by alotau (8kB - application/octet-stream)
file #16746:  wget.patch added by alotau (4kB - application/octet-stream)
file #16744:  http.c.patch added by alotau (2kB - application/octet-stream)
file #16745:  http.h.patch added by alotau (301B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by nok (Posted a comment)
  • -unavailable- added by nok
  • -unavailable- added by rockdaboot (Updated the item)
  • -unavailable- added by frank (Updated the item)
  • -unavailable- added by robman (Posted a comment)
  • -unavailable- added by gsauthof
  • -unavailable- added by timido (Posted a comment)
  • -unavailable- added by kayhadrin (Posted a comment)
  • -unavailable- added by fossjn (Posted a comment)
  • -unavailable- added by alotau (Posted a comment)
  • -unavailable- added by cy6erbr4in (Updated the item)
  • -unavailable- added by micahcowan (Updated the item)
  • -unavailable- added by johndoe32102002 (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 25 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 29 Sep 2012 11:49:26 AM UTCgscrivanoStatusNeeds Discussion=>Fixed
      Assigned toNone=>gscrivano
      Open/ClosedOpen=>Closed
    Fri 28 Sep 2012 04:35:09 PM UTCgscrivanoAssigned toalotau=>None
    Mon 09 Jul 2012 06:43:07 PM UTCnokCarbon-Copy-=>Added -unavailable-
    Tue 29 May 2012 11:19:19 AM UTCrockdabootAttached File-=>Added 0001-filename-length-patch-v5-1.13.4.patch, #25949
    Fri 18 May 2012 03:05:32 PM UTCrockdabootAttached File-=>Added 0001-filename-length-patch-v4-1.14.patch, #25881
    Sun 27 Nov 2011 11:29:55 AM UTCfrankAttached File-=>Added wget_filename_length.v3-1.13.patch, #24473
    Mon 13 Jun 2011 04:27:50 AM UTCfrankAttached File-=>Added wget_filename_length.v2-1.12.patch, #23520
    Sun 30 Jan 2011 11:42:41 PM UTCgsauthofCarbon-Copy-=>Added gsauthof
    Tue 08 Sep 2009 07:52:26 AM UTCmicahcowanPlanned Release1.14=>1.12.x
    Tue 01 Sep 2009 05:45:58 AM UTCmicahcowanStatusIn Progress=>Needs Discussion
    Sun 30 Aug 2009 04:11:36 AM UTCmicahcowanStatusAwaiting Approval=>In Progress
      Planned Release1.12=>1.14
    Tue 18 Aug 2009 10:07:47 PM UTCmicahcowanPriority6=>7 - High
      Planned Release1.14=>1.12
    Sun 05 Jul 2009 09:50:35 PM UTCmicahcowanStatusReady For Test=>Awaiting Approval
      Planned Release1.12=>1.14
    Fri 06 Feb 2009 07:21:44 PM UTCalotauStatusIn Progress=>Ready For Test
    Fri 06 Feb 2009 07:20:18 PM UTCalotauAttached File-=>Added wget_filename_length.v2.patch, #17414
    Fri 06 Feb 2009 04:37:55 AM UTCalotauAttached File-=>Added wget_file_length.patch, #17409
    Tue 04 Nov 2008 01:35:56 AM UTCmicahcowanAssigned toNone=>alotau
    Wed 29 Oct 2008 02:07:13 AM UTCalotauAttached File-=>Added wget.patch, #16746
    Tue 28 Oct 2008 06:04:30 PM UTCmicahcowanAssigned tocy6erbr4in=>None
    Tue 28 Oct 2008 04:51:58 PM UTCalotauAttached File-=>Added http.c.patch, #16744

    Back to the top


    Powered by Savane 3.1-cleanup