helpThe GNU Bourne-Again SHell - Support: sr #108163, Broken shell line handling when...

 
 

sr #108163: Broken shell line handling when using X clipboard as input

Submitted by:  None
Submitted on:  Tue 16 Oct 2012 02:45:02 PM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 4 - ImportantStatus: None
Privacy: PublicAssigned to: None
Originator Email: -unavailable-Open/Closed: Open
Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 28 Nov 2013 04:47:21 AM UTC, comment #1:

This may not be entirely bash's fault. I did the same test with dash and bash, both pasting into a screen(1) session in a gnome-terminal, from the X selection buffer.

dash got 3812 lines into /tmp/testfile, one into /tmp/testfil, one into /tmp/testfilfile, and one empty /tmp/testfilet-3719.

bash was worse, with I think fewer total lines into files in /tmp, and scattered between more files.

In any case, "worse" or "better" isn't really relevant; if there's any behaviour that's determined by CPU performance, there's a problem.

xterm's man page has a BUGS section which says:
Large pastes do not work on some systems. This is not a bug in xterm; it is a bug in the pseudo terminal driver of those systems. xterm feeds large pastes to the pty only as fast as the pty will accept data, but some pty drivers do not return enough information to know if the write has succeeded.

So presumably gnome-terminal (and/or screen in my case), are writing characters to the pseudo-terminal faster than the shell is reading them, and this is causing a buffer somewhere to wrap around and overwrite data that hasn't been read yet.

So unless there's a way for bash to enable flow control that it isn't taking advantage of, there's nothing bash or libreadline can do to fix this. This is basically a known problem with pasting large buffers into terminals.

If anyone wants to play around with this, I'd suggest writing a test program that slowly reads a line or character at a time from the terminal, in raw mode. If there are any termios(3) calls that will make the terminal emulator's writes block when the buffer is full, then let us know and maybe libreadline can turn them on. If not, all you could do would be bandaid the problem by having libreadline / bash buffer its input so it could scoop up incoming terminal input faster than it could process it.

Having bash / libreadline do another layer of input buffering is just a band-aid, since you'd want to set a limit on how big that buffer can be, or else you could get into perverse situations where bash is stuck on a command, but getting data from a terminal. (e.g. something fell on a keyboard with key repeat on, or some giant binary file gets fed into a terminal accidentally via an ssh usage error or something. Or infinite other ways, only some of them malicious, that you could make bash use up all the virtual memory.)

Peter Cordes <pcordes>
Tue 16 Oct 2012 02:45:02 PM UTC, original submission:

Configuration Information:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu' -DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/share/l
ocale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -D_FORTIFY_SOURCE
=2 -DDEFAULT_PATH_VALUE='/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin' -DSTANDARD_UTILS_PATH='/usr/bin:/bin:/usr/sbin:/sbin' -DSYS_BASHRC='/etc/bash.bashrc' -DSYS_BA
SH_LOGOUT='/etc/bash.bash_logout'
uname output: Linux ntb-fedorpiecka 3.5.6-1-ARCH #1 SMP PREEMPT Sun Oct 7 19:30:49 CEST 2012 x86_64 GNU/Linux
Machine Type: x86_64-unknown-linux-gnu

Bash Version: 4.2
Patch Level: 37
Release Status: release

Description:

Long scripts which I paste into xterm (tested with xterm, Gnome Terminal, LXTerminal and Konsole on Gnome, KDE and LXDE) from clipboard are corrupted by BASH.

The problem starts at around 4096 character scripts but is not bound to 4096 and is random. E.g. sometimes 4150 characters work and sometimes 4050 don't.

Repeat-By:

1. Open script1.txt or script2.txt in a text editor in X
2. Select the whole text
3. Copy selected text to clipboard
4. Open terminal window
5. Paste clipboard into the terminal window

Expected result:
a) script1.txt - /tmp/test contains 4096 lines with string test-LINENUMBER on each line where LINENUMBER equals the number of the line

b) script2.txt - /tmp/test contains 128 lines with string "-128abcdefghijlkmnopqrstuvwxyz128-" on the last line

Result in reality:
a) script1.txt - /tmp/test contains randomly selected subset of lines, /tmp/te contains randomly selected subset of lines and bash tries to use /t too and possibly other files with path which is a substring of "/tmp/test"

b) script2.txt - /tmp/test has last N (N seems random) lines corrupted, sometimes bash doesn't register the final EOF correctly and you have to write it by hand (happens with some script lengths)

I've used these generators to prepare the test scripts:

# script1
for i in `seq 1 4096`; do echo "echo test-$i >> /tmp/testfile"; done > /tmp/script1.sh
# end of script1

# script2
echo "cat > /tmp/testfile <<'EOF'" > /tmp/script2.sh
for i in `seq 1 128`; do echo "-${i}abcdefghijlkmnopqrstuvwxyz${i}-"; done >> /tmp/script2.sh
echo EOF >> /tmp/script2.sh
# end of script2

We were able to reproduce the issue on Arch Linux, Linux Mint, Fedora and Debian with bash 4.1 and 4.2 both through an SSH connection and directly.

The issue cannot be reproduced on FreeBSD with bash 3.2.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #26774:  script1.sh added by None (127kB - text/x-sh)
file #26775:  script2.sh added by None (4kB - text/x-sh)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by pcordes (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 16 Oct 2012 02:45:02 PM UTCNoneAttached File-=>Added script1.sh, #26774
      Attached File-=>Added script2.sh, #26775

    Back to the top


    Powered by Savane 3.1-cleanup