bugGNU Wget - Bugs: bug #22905, wget progress bar spans new lines...

 
 

bug #22905: wget progress bar spans new lines continuously

Submitter:  None
Submitted:  Fri 11 Apr 2008 07:46:12 PM UTC
   
 
Category:  User Interface Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  None
Originator Name:  ivvmm Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.11.1
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  1.11.2 Planned Release:  1.11.2
Regression:  None Work Required:  0 - Hours
Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 30 Apr 2008 08:58:22 PM UTC, comment #21: 

Fixed for eleven in b91b83a4cc21.
Fixed for mainline in 9cb2563197bc.

Micah Cowan <micahcowan>
Wed 30 Apr 2008 08:29:56 PM UTC, comment #20: 

Sorry: throw out the first four lines of that patch text.

Micah Cowan <micahcowan>
Wed 30 Apr 2008 08:29:05 PM UTC, comment #19: 

Problem found, and fixed:


diff -r 1c2e84c5527c src/progress.c
--- a/src/progress.c    Sun Apr 27 12:11:03 2008 -0700
+++ b/src/progress.c    Wed Apr 30 13:07:43 2008 -0700
@@ -1029,6 +1029,7 @@
diff -r 1c2e84c5527c src/progress.c
--- a/src/progress.c    Sun Apr 27 12:11:03 2008 -0700
+++ b/src/progress.c    Wed Apr 30 13:16:26 2008 -0700
@@ -881,6 +881,8 @@
   int dlbytes_size = 1 + MAX (size_grouped_len, 11);
   int progress_size = bp->width - (4 + 2 + dlbytes_size + 8 + 14);

+  bool printed_eta = true;
+
   if (progress_size < 5)
     progress_size = 0;

@@ -1029,6 +1031,7 @@
       else if (bp->total_length > 0)
         {
         skip_eta:
+          printed_eta = false;
           APPEND_LITERAL ("             ");
         }
     }
@@ -1047,8 +1050,12 @@
       move_to_end (p);
     }

-  while (p - bp->buffer - bytes_cols_diff - size_grouped_diff < bp->width)
-    *p++ = ' ';
+  while (p - bp->buffer -
+         (printed_eta? bytes_cols_diff : 0)
+         - size_grouped_diff < bp->width)
+    {
+      *p++ = ' ';
+    }
   *p = '\0';
 }


The issue is that the locale-fixes I added for 1.11.1 to fix assertion errors in 1.11, include the use of the variable "bytes_cols_diff" to hold the disparity between the number of bytes used, and the number of column positions used (prior to those fixes, Wget always only counted byte positions). This variable is in turn used to figure out how many spaces Wget should append to the line.

The variable is set by the first use of get_eta(). It didn't affect smallish files because they never had a chance to display the ETA string (and didn't affect the C or typical english locales because the number of columns equals the number of bytes). All that needed to happen was for the ETA string to be calculated-and-displayed for one file, and then the variable would be used to figure out how many extra spaces we need to add to our image. That value, of course, would be wrong for any display cycles where we weren't actually using the ETA string.

The fix was to only use the variable when we've actually printed the ETA string.

Note that the final display, containing the "  in  " line ("  в "), still prints too-few spaces, possibly not overwriting the full image in some cases. The patch above doesn't address that. I may want to rework it a bit to fix that case as well, before I commit anything.

The whole progress-bar code is pretty hack-y, IMO, and any fixes I place atop it to make it behave properly on different locales will also tend to be hacks. A rewrite is planned for some point in the future (but isn't a priority right now).

Micah Cowan <micahcowan>
Wed 30 Apr 2008 07:34:00 PM UTC, comment #18: 

Note: the example line I gave doesn't display properly on my browser. To be clear: it was a simple invocation of wget, with the same argument, +noformat+ftp://ftp.gnu.org/gnu/wget/wget-1.11.1.tar.gz-noformat-, given twice.

Micah Cowan <micahcowan>
Wed 30 Apr 2008 07:32:52 PM UTC, comment #17: 

I've since determined that it's nothing to do with the filename. In fact, I'm able to demonstrate that invoking wget with:


wget ftp://ftp.gnu.org/gnu/wget/wget-1.11.1.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.11.1.tar.gz


will produce the problem on the second download (of the exact same file), but not the first.

It does appear that the file size may make a difference: doing a recursive retrieval of several small files doesn't produce the problem (though retrievals of small files that follow the download of a large file does continue to exhibit the problem).

I'll have to do some tracing to see what's up.

At any rate, if it is a "funky character" problem, then it's one produced by Wget messages; it's certainly not one produced by file names, so wouldn't be solved by escnonprint/quote after all.

Micah Cowan <micahcowan>
Wed 30 Apr 2008 09:30:30 AM UTC, comment #16: 

The previous output example was with patch applied. This one is without the patch.

ivvmm

(file #15555, file #15556)

Anonymous
Tue 29 Apr 2008 08:55:14 PM UTC, comment #15: 

Hm... Unlike your previous example, this time Wget is actually issuing a different number of columns. Could you please try with the patch I gave you earlier, if you're not (and, if you are, try reversing it). This appears to be caused by a completely different problem from the other one.

Where do you first see the problem: on the second file downloaded?

Reopening in the meantime.

Micah Cowan <micahcowan>
Tue 29 Apr 2008 08:30:34 PM UTC, comment #14: 

This was also unexpectedly for me too.

ivvmm

(file #15553, file #15554)

Anonymous
Tue 29 Apr 2008 04:05:15 PM UTC, comment #13: 

Can you provide an example to corroborate that? In your example so far, the problem is demonstrably being caused by the screwy characters. Note that once those characters appear, all further output will be screwed up until Wget exits, so it may well be that filenames with only ASCII symbols and spaces will display this problem if they appear after a screwy name, but I've doubts as to whether they can trigger the issue on their own. Please provide an example.

Micah Cowan <micahcowan>
Tue 29 Apr 2008 01:14:57 PM UTC, comment #12: 

Note, this happens even if only ASCII symbols and spaces are in file names.

Anonymous
Wed 23 Apr 2008 06:14:49 PM UTC, comment #11: 

Okay, so I'm finally able to reproduce the problem; I've used a couple of the exact same filenames that appear in your typescript on my own FTP server (ftp://micah.cowan.name/wget-test/), and can reproduce your results on one of them.

It appears that you're right that it's name related (thought it's not spaces). The filenames contain many characters that are not valid UTF-8 (ISO-8859-5?). None of them should correspond to control characters, but some of them might pair up to form legitimate UTF-8 control characters, or some such. I noticed that the last character (\xEB) of one of the filenames appears to get "combined" on output with the period from ".pdf" that follows it.

When I tried playing around with this combination, with "printf '\xEB.\n'" and such, it actually killed my Xterm. So Xterm definitely doesn't care much for it. Same is true for gnome-terminal. "\xEB." still shouldn't represent a valid UTF-8 character, but anyhoo...

Not sure why it doesn't kill Xterm in our case. It seems likely, though, that there's an interaction between a bug in Xterm, and the bug in Wget (in that it's not properly escaping the filename).

The effect seems to be that it causes carriage returns to be treated as line feeds. I'm still puzzled as to why it only occurs for certain locales, and why "ОВП" apparently negates its effect. It's probably worth finding out whether this does in fact correspond to a bug in Xterm or in code upon which it relies; but in any case, the solution for Wget is clear: we should be escaping the filenames when we output them.

However, using Wget's current escaping function (escnonprint) may be less-than-desirable, as it is not locale-aware in the least, and so would produce poor output for non-English locales. There is current work being done against the mainline development repository to use Gnulib's quoting mechanisms, which do take locale into account. I'm probably going to punt the fix for this for now, so we can do quoting properly in 1.12.

For your own purposes, you could choose to hack it to use escnonprint, which will probably make all non-English filenames look like crap in the log, but will at least protect your terminal and prevent the "newlines" problem you're experiencing.

I'm closing this as a dupe of 21187, and will add a comment there.

NOTE: in trying to see whether this was reproducible with previous versions of Wget, I was surprised to find that Wget 1.10.2 refuses to download the files at all. Guess we've improved over that a little bit. :)

Micah Cowan <micahcowan>
Wed 23 Apr 2008 12:29:42 PM UTC, comment #10: 

My XTerm is using 'fixed' font, which is not a TTF. I've already thought of it being a problem of teminal and/or font, so ran wget -r ftp:// in Xfce's Terminal, which by default uses DejaVu TTF fonts and got the same.

I am using Slackware-current. The script, which builds the GNU Wget package can be found here: ftp://slackware.osuosl.org/pub/slackware/slackware-current/source/n/wget/wget.SlackBuild

Well, it seems it looks just like yours output. The screenshot is in attachment.


Anonymous
Wed 23 Apr 2008 03:35:14 AM UTC, comment #9: 

Well, I checked the typescript, and each row is exactly 79 columns in length, when viewed in a Unicode-capable editor. Both the rows that come before, and those that come after.

What font is your Xterm set to use? That could definitely make a difference, particularly if some of the characters being output by Wget aren't supported by the font.

What distribution and edition of GNU/Linux are you using?

I've prepared an altered version of your typescript file, "cols-test". Please cat it out to your terminal, and scroll up so that the "ОВП" section starts in the middle of your terminal. I want the |-separated version, not the "normal" version at the start of the file. I've attached a screenshot of what I want from you.

This is designed to point out any characters that are taking more columns than they have a right to.

(file #15509,

Micah Cowan <micahcowan>
Tue 15 Apr 2008 09:05:30 AM UTC, comment #8: 

So here it is.

(file #15453)

Anonymous
Tue 15 Apr 2008 05:34:26 AM UTC, comment #7: 

Here's something that would at least help me to see exactly what Wget is doing output-wise: run your Wget invocation under the "script" command.

Many systems provide this command, and hopefully you have it installed on your system. If it's not, please get a copy (on my Ubuntu system, it is provided by the "bsdutils" package).


  $ script
  Script started, file is typescript
  $ wget ...
  ...
  $ exit
  Script done, file is typescript


Script will spawn a new shell, and record every character that is sent to the terminal, by you or by programs. This includes special escape sequences, and everything. When it's done, please attach the "typescript" file to this bug report.

It's important that you attach the real typescript file, and not a copy/paste from the terminal, as otherwise you'll lose the escape sequences.

If the Wget session contains sensitive invocation, you can try to edit typescript with a editor that will leave any control characters intact, but that can be hard to do. If that doesn't work, perhaps you would be more comfortably emailing the typescript file to me privately.

Micah Cowan <micahcowan>
Mon 14 Apr 2008 01:20:34 PM UTC, comment #6: 

Sorry, I really don't have an idea where to get such an ftp server,  I can only found such only on my LAN.

ivvmm

Anonymous
Sun 13 Apr 2008 06:50:34 AM UTC, comment #5: 

Sorry, not possible.

Among things that could possibly affect it, there's the file size and download speeds, I think. Except that I would have expected them to make the multiple-lines thing more likely after the "ETA" is added, rather than before.

At this point, I'm going to need a publicly accessible URL for me to reproduce this on, since I'm having trouble with my own tests.

FWIW, the progress-bar code is fairly crappy, and I do plan on doing a rewrite of it at some point, which would hopefully eliminate any remaining potential for the problem you're encountering.

Micah Cowan <micahcowan>
Sat 12 Apr 2008 09:40:17 PM UTC, comment #4: 

Applied the patch, recompiled, repackaged, reinstalled. Does not help.

Additional information that actually i've noticed only now. Before that I could not determine when exactly did that happen: it happened just "from time to time", and now I am sort of being determined: it happens with those files, which contain spaces in their name.

ivvmm

Anonymous
Sat 12 Apr 2008 06:51:27 AM UTC, comment #3: 

I'm unable to reproduce it on Ubuntu 7.10 with XTerm (229).

However, the fact that the problem stops when the ETA displays gives me an idea. Can you determine whether the following patch solves the issue?


diff -r 00eef3f2814a src/progress.c
--- a/src/progress.c    Fri Apr 11 16:34:50 2008 -0700
+++ b/src/progress.c    Fri Apr 11 23:50:06 2008 -0700
@@ -1027,11 +1027,7 @@
           sprintf (p, get_eta(), eta_to_human_short (eta, false));
           move_to_end (p);
         }
-      else if (bp->total_length > 0)
-        {
-        skip_eta:
-          APPEND_LITERAL ("             ");
-        }
+      skip_eta: ;
     }
   else
     {


Micah Cowan <micahcowan>
Sat 12 Apr 2008 06:21:53 AM UTC, comment #2: 

I am using XTerm terminal emulator
$ xterm -version
X.Org 6.8.99.903(232)
$ echo $COLUMNS
80
$ printf "%$(($COLUMNS - 1))d\n" 1 ; printf "%${COLUMNS}d\n" 1
                                                                              1
                                                                               1


So it takes just two lines, one for the first '1' and the second for the second '1'.

ivvmmm.

Anonymous
Sat 12 Apr 2008 02:22:02 AM UTC, comment #1: 

AFAICT this should only be possible when your terminal is lying about its width in characters (or mishandles Unicode output).

I'm unable to reproduce this (yes, with ru_RU.UTF-8 locale).

What platform are you running on, and what terminal emulator do you use (and what version is it)?

What do you get for:

  echo $COLUMNS

and

  printf "%$(($COLUMNS - 1))d\n" 1
  printf "%${COLUMNS}d\n" 1

(and how many lines do those take up?)

Micah Cowan <micahcowan>
Fri 11 Apr 2008 07:46:12 PM UTC, original submission:  

I use a UTF-8 Russian locale and it happens in it, and happens in every UTF-8 locale. Wget spans a new line with progressbar every refresh untill ETA is calculated and displayed for the first time. It is very similar to this bug, exposed in oggenc: https://trac.xiph.org/ticket/1321

This does not happen in C locale.

A command I run to produce it:
wget -r ftp://ftpservernameishidden

Sorry, but the output is only in Russian, because this won't happen in UTF-8 US locale.

Удалён `void.progtech.ru/1st Season [en, srt]/110 - The Big House/.listing'.
--2008-04-11 23:42:39--  ftp://void@void.progtech.ru/1st%20Season%20%5Ben%2C%20s
rt%5D/110%20-%20The%20Big%20House/110%20-%20The%20Big%20House.avi
           => `void.progtech.ru/1st Season [en, srt]/110 - The Big House/110 - T
he Big House.avi'
==> CWD не требуется.
==> PASV ... готово.  ==> RETR 110 - The Big House.avi ... готово.
Длина: 110387668 (105M)

 0% [                                       ] 0           --.-K/s              
 0% [                                       ] 129 872      634K/s              
 0% [                                       ] 202 152      428K/s              
 0% [                                       ] 296 360      437K/s              
 0% [                                       ] 416 592      471K/s              
 0% [                                       ] 518 992      477K/s              
 0% [                                       ] 615 848      475K/s              
 0% [                                       ] 705 960      467K/s              
 0% [                                       ] 820 648      476K/s              
 0% [                                       ] 942 080      488K/s              
 0% [                                       ] 1 078 696    505K/s              
 1% [                                       ] 1 191 936    509K/s              
 1% [                                       ] 1 309 520    514K/s              
 1% [                                       ] 1 422 760    511K/s              
 1% [                                       ] 1 553 832    513K/s              
21% [=======>                               ] 24 020 392   567K/s  ОВП 2m 35s 

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #15555:  typescript added by None (150KiB - application/octet-stream)
file #15556:  timings added by None (28KiB - application/octet-stream)
file #15554:  timings added by None (42KiB - application/octet-stream)
file #15553:  typescript added by None (181KiB - application/octet-stream)
file #15513:  xterm.png added by None (8KiB - image/png)
file #15509:  cols-test added by micahcowan (11KiB - application/octet-stream)
file #15510:  micah-xterm.png added by micahcowan (29KiB - image/png)
file #15453:  typescript added by None (24KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by micahcowan (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Follow 22 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-04-30 micahcowan StatusConfirmed Fixed
        Open/ClosedOpen Closed
        Fixed ReleaseNone 1.11.2
    2008-04-30 micahcowan Work Required1 - Days 0 - Hours
    2008-04-30 micahcowan StatusNeeds Investigation Confirmed
        Open/ClosedClosed Open
        Planned Release1.12 1.11.2
        Work RequiredNone 1 - Days
    2008-04-30 None Attached File- Added typescript, #15555
        Attached File- Added timings, #15556
    2008-04-29 micahcowan StatusDuplicate Needs Investigation
        Planned Release1.11.2 1.12
    2008-04-29 None Attached File- Added timings, #15554
    2008-04-29 None Attached File- Added typescript, #15553
    2008-04-23 micahcowan StatusNeed Info Duplicate
        Open/ClosedOpen Closed
    2008-04-23 None Attached File- Added xterm.png, #15513
    2008-04-23 micahcowan Attached File- Added cols-test, #15509
        Attached File- Added micah-xterm.png, #15510
    2008-04-15 None Attached File- Added typescript, #15453
    2008-04-12 micahcowan StatusNone Need Info
        Planned ReleaseNone 1.11.2

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code