Wed 30 Apr 2008 08:58:22 PM UTC, comment #21:
Fixed for eleven in b91b83a4cc21.
Fixed for mainline in 9cb2563197bc.
|
Wed 30 Apr 2008 08:29:56 PM UTC, comment #20:
Sorry: throw out the first four lines of that patch text.
|
Wed 30 Apr 2008 08:29:05 PM UTC, comment #19:
Problem found, and fixed:
The issue is that the locale-fixes I added for 1.11.1 to fix assertion errors in 1.11, include the use of the variable "bytes_cols_diff" to hold the disparity between the number of bytes used, and the number of column positions used (prior to those fixes, Wget always only counted byte positions). This variable is in turn used to figure out how many spaces Wget should append to the line.
The variable is set by the first use of get_eta(). It didn't affect smallish files because they never had a chance to display the ETA string (and didn't affect the C or typical english locales because the number of columns equals the number of bytes). All that needed to happen was for the ETA string to be calculated-and-displayed for one file, and then the variable would be used to figure out how many extra spaces we need to add to our image. That value, of course, would be wrong for any display cycles where we weren't actually using the ETA string.
The fix was to only use the variable when we've actually printed the ETA string.
Note that the final display, containing the " in " line (" в "), still prints too-few spaces, possibly not overwriting the full image in some cases. The patch above doesn't address that. I may want to rework it a bit to fix that case as well, before I commit anything.
The whole progress-bar code is pretty hack-y, IMO, and any fixes I place atop it to make it behave properly on different locales will also tend to be hacks. A rewrite is planned for some point in the future (but isn't a priority right now).
|
Wed 30 Apr 2008 07:34:00 PM UTC, comment #18:
Note: the example line I gave doesn't display properly on my browser. To be clear: it was a simple invocation of wget, with the same argument, +noformat+ftp://ftp.gnu.org/gnu/wget/wget-1.11.1.tar.gz-noformat-, given twice.
|
Wed 30 Apr 2008 07:32:52 PM UTC, comment #17:
I've since determined that it's nothing to do with the filename. In fact, I'm able to demonstrate that invoking wget with:
will produce the problem on the second download (of the exact same file), but not the first.
It does appear that the file size may make a difference: doing a recursive retrieval of several small files doesn't produce the problem (though retrievals of small files that follow the download of a large file does continue to exhibit the problem).
I'll have to do some tracing to see what's up.
At any rate, if it is a "funky character" problem, then it's one produced by Wget messages; it's certainly not one produced by file names, so wouldn't be solved by escnonprint/quote after all.
|
Wed 30 Apr 2008 09:30:30 AM UTC, comment #16:
The previous output example was with patch applied. This one is without the patch.
ivvmm
(file #15555, file #15556)
|
Tue 29 Apr 2008 08:55:14 PM UTC, comment #15:
Hm... Unlike your previous example, this time Wget is actually issuing a different number of columns. Could you please try with the patch I gave you earlier, if you're not (and, if you are, try reversing it). This appears to be caused by a completely different problem from the other one.
Where do you first see the problem: on the second file downloaded?
Reopening in the meantime.
|
Tue 29 Apr 2008 08:30:34 PM UTC, comment #14:
This was also unexpectedly for me too.
ivvmm
(file #15553, file #15554)
|
Tue 29 Apr 2008 04:05:15 PM UTC, comment #13:
Can you provide an example to corroborate that? In your example so far, the problem is demonstrably being caused by the screwy characters. Note that once those characters appear, all further output will be screwed up until Wget exits, so it may well be that filenames with only ASCII symbols and spaces will display this problem if they appear after a screwy name, but I've doubts as to whether they can trigger the issue on their own. Please provide an example.
|
Tue 29 Apr 2008 01:14:57 PM UTC, comment #12:
Note, this happens even if only ASCII symbols and spaces are in file names.
|
Wed 23 Apr 2008 06:14:49 PM UTC, comment #11:
Okay, so I'm finally able to reproduce the problem; I've used a couple of the exact same filenames that appear in your typescript on my own FTP server (ftp://micah.cowan.name/wget-test/), and can reproduce your results on one of them.
It appears that you're right that it's name related (thought it's not spaces). The filenames contain many characters that are not valid UTF-8 (ISO-8859-5?). None of them should correspond to control characters, but some of them might pair up to form legitimate UTF-8 control characters, or some such. I noticed that the last character (\xEB) of one of the filenames appears to get "combined" on output with the period from ".pdf" that follows it.
When I tried playing around with this combination, with "printf '\xEB.\n'" and such, it actually killed my Xterm. So Xterm definitely doesn't care much for it. Same is true for gnome-terminal. "\xEB." still shouldn't represent a valid UTF-8 character, but anyhoo...
Not sure why it doesn't kill Xterm in our case. It seems likely, though, that there's an interaction between a bug in Xterm, and the bug in Wget (in that it's not properly escaping the filename).
The effect seems to be that it causes carriage returns to be treated as line feeds. I'm still puzzled as to why it only occurs for certain locales, and why "ОВП" apparently negates its effect. It's probably worth finding out whether this does in fact correspond to a bug in Xterm or in code upon which it relies; but in any case, the solution for Wget is clear: we should be escaping the filenames when we output them.
However, using Wget's current escaping function (escnonprint) may be less-than-desirable, as it is not locale-aware in the least, and so would produce poor output for non-English locales. There is current work being done against the mainline development repository to use Gnulib's quoting mechanisms, which do take locale into account. I'm probably going to punt the fix for this for now, so we can do quoting properly in 1.12.
For your own purposes, you could choose to hack it to use escnonprint, which will probably make all non-English filenames look like crap in the log, but will at least protect your terminal and prevent the "newlines" problem you're experiencing.
I'm closing this as a dupe of 21187, and will add a comment there.
NOTE: in trying to see whether this was reproducible with previous versions of Wget, I was surprised to find that Wget 1.10.2 refuses to download the files at all. Guess we've improved over that a little bit. :)
|
Wed 23 Apr 2008 12:29:42 PM UTC, comment #10:
My XTerm is using 'fixed' font, which is not a TTF. I've already thought of it being a problem of teminal and/or font, so ran wget -r ftp:// in Xfce's Terminal, which by default uses DejaVu TTF fonts and got the same.
I am using Slackware-current. The script, which builds the GNU Wget package can be found here: ftp://slackware.osuosl.org/pub/slackware/slackware-current/source/n/wget/wget.SlackBuild
Well, it seems it looks just like yours output. The screenshot is in attachment.
(file #15513)
|
Wed 23 Apr 2008 03:35:14 AM UTC, comment #9:
Well, I checked the typescript, and each row is exactly 79 columns in length, when viewed in a Unicode-capable editor. Both the rows that come before, and those that come after.
What font is your Xterm set to use? That could definitely make a difference, particularly if some of the characters being output by Wget aren't supported by the font.
What distribution and edition of GNU/Linux are you using?
I've prepared an altered version of your typescript file, "cols-test". Please cat it out to your terminal, and scroll up so that the "ОВП" section starts in the middle of your terminal. I want the |-separated version, not the "normal" version at the start of the file. I've attached a screenshot of what I want from you.
This is designed to point out any characters that are taking more columns than they have a right to.
(file #15509, file #15510)
|
Tue 15 Apr 2008 09:05:30 AM UTC, comment #8:
So here it is.
(file #15453)
|
Tue 15 Apr 2008 05:34:26 AM UTC, comment #7:
Here's something that would at least help me to see exactly what Wget is doing output-wise: run your Wget invocation under the "script" command.
Many systems provide this command, and hopefully you have it installed on your system. If it's not, please get a copy (on my Ubuntu system, it is provided by the "bsdutils" package).
Script will spawn a new shell, and record every character that is sent to the terminal, by you or by programs. This includes special escape sequences, and everything. When it's done, please attach the "typescript" file to this bug report.
It's important that you attach the real typescript file, and not a copy/paste from the terminal, as otherwise you'll lose the escape sequences.
If the Wget session contains sensitive invocation, you can try to edit typescript with a editor that will leave any control characters intact, but that can be hard to do. If that doesn't work, perhaps you would be more comfortably emailing the typescript file to me privately.
|
Mon 14 Apr 2008 01:20:34 PM UTC, comment #6:
Sorry, I really don't have an idea where to get such an ftp server, I can only found such only on my LAN.
ivvmm
|
Sun 13 Apr 2008 06:50:34 AM UTC, comment #5:
Sorry, not possible.
Among things that could possibly affect it, there's the file size and download speeds, I think. Except that I would have expected them to make the multiple-lines thing more likely after the "ETA" is added, rather than before.
At this point, I'm going to need a publicly accessible URL for me to reproduce this on, since I'm having trouble with my own tests.
FWIW, the progress-bar code is fairly crappy, and I do plan on doing a rewrite of it at some point, which would hopefully eliminate any remaining potential for the problem you're encountering.
|
Sat 12 Apr 2008 09:40:17 PM UTC, comment #4:
Applied the patch, recompiled, repackaged, reinstalled. Does not help.
Additional information that actually i've noticed only now. Before that I could not determine when exactly did that happen: it happened just "from time to time", and now I am sort of being determined: it happens with those files, which contain spaces in their name.
ivvmm
|
Sat 12 Apr 2008 06:51:27 AM UTC, comment #3:
I'm unable to reproduce it on Ubuntu 7.10 with XTerm (229).
However, the fact that the problem stops when the ETA displays gives me an idea. Can you determine whether the following patch solves the issue?
|
Sat 12 Apr 2008 06:21:53 AM UTC, comment #2:
I am using XTerm terminal emulator
$ xterm -version
X.Org 6.8.99.903(232)
$ echo $COLUMNS
80
$ printf "%$(($COLUMNS - 1))d\n" 1 ; printf "%${COLUMNS}d\n" 1
1
1
So it takes just two lines, one for the first '1' and the second for the second '1'.
ivvmmm.
|
Sat 12 Apr 2008 02:22:02 AM UTC, comment #1:
AFAICT this should only be possible when your terminal is lying about its width in characters (or mishandles Unicode output).
I'm unable to reproduce this (yes, with ru_RU.UTF-8 locale).
What platform are you running on, and what terminal emulator do you use (and what version is it)?
What do you get for:
echo $COLUMNS
and
printf "%$(($COLUMNS - 1))d\n" 1
printf "%${COLUMNS}d\n" 1
(and how many lines do those take up?)
|
Fri 11 Apr 2008 07:46:12 PM UTC, original submission:
I use a UTF-8 Russian locale and it happens in it, and happens in every UTF-8 locale. Wget spans a new line with progressbar every refresh untill ETA is calculated and displayed for the first time. It is very similar to this bug, exposed in oggenc: https://trac.xiph.org/ticket/1321
This does not happen in C locale.
A command I run to produce it:
wget -r ftp://ftpservernameishidden
Sorry, but the output is only in Russian, because this won't happen in UTF-8 US locale.
Удалён `void.progtech.ru/1st Season [en, srt]/110 - The Big House/.listing'.
--2008-04-11 23:42:39-- ftp://void@void.progtech.ru/1st%20Season%20%5Ben%2C%20s
rt%5D/110%20-%20The%20Big%20House/110%20-%20The%20Big%20House.avi
=> `void.progtech.ru/1st Season [en, srt]/110 - The Big House/110 - T
he Big House.avi'
==> CWD не требуется.
==> PASV ... готово. ==> RETR 110 - The Big House.avi ... готово.
Длина: 110387668 (105M)
0% [ ] 0 --.-K/s
0% [ ] 129 872 634K/s
0% [ ] 202 152 428K/s
0% [ ] 296 360 437K/s
0% [ ] 416 592 471K/s
0% [ ] 518 992 477K/s
0% [ ] 615 848 475K/s
0% [ ] 705 960 467K/s
0% [ ] 820 648 476K/s
0% [ ] 942 080 488K/s
0% [ ] 1 078 696 505K/s
1% [ ] 1 191 936 509K/s
1% [ ] 1 309 520 514K/s
1% [ ] 1 422 760 511K/s
1% [ ] 1 553 832 513K/s
21% [=======> ] 24 020 392 567K/s ОВП 2m 35s
|