mainThe GNU Bourne-Again SHell - Support: sr #108163, Broken shell line handling when...

 
 

sr #108163: Broken shell line handling when using X clipboard as input

Submitter:  None
Submitted:  Tue 16 Oct 2012 02:45:02 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  4 - Important Status:  Invalid
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 13 Nov 2015 02:38:00 AM UTC, comment #2: 

There was a lot of investigation done on this which eventually identified the culprit as the (Linux kernel's at least) tty driver.

Chet Ramey <chet>
Group administrator
Thu 28 Nov 2013 04:47:21 AM UTC, comment #1: 

This may not be entirely bash's fault.  I did the same test with dash and bash, both pasting into a screen(1) session in a gnome-terminal, from the X selection buffer.

 dash got 3812 lines into /tmp/testfile, one into /tmp/testfil, one into /tmp/testfilfile, and one empty /tmp/testfilet-3719.

 bash was worse, with I think fewer total lines into files in /tmp, and scattered between more files.

 In any case, "worse" or "better" isn't really relevant; if there's any behaviour that's determined by CPU performance, there's a problem.

 xterm's man page has a BUGS section which says:
Large pastes do not work on some systems.  This is not a bug in xterm; it is a bug in the pseudo  terminal  driver of  those  systems.   xterm  feeds  large pastes to the pty only as fast as the pty will accept data, but some pty drivers do not return enough information to know if the write has succeeded.

 So presumably gnome-terminal (and/or screen in my case), are writing characters to the pseudo-terminal faster than the shell is reading them, and this is causing a buffer somewhere to wrap around and overwrite data that hasn't been read yet.

 So unless there's a way for bash to enable flow control that it isn't taking advantage of, there's nothing bash or libreadline can do to fix this.  This is basically a known problem with pasting large buffers into terminals.

 If anyone wants to play around with this, I'd suggest writing a test program that slowly reads a line or character at a time from the terminal, in raw mode.  If there are any termios(3) calls that will make the terminal emulator's writes block when the buffer is full, then let us know and maybe libreadline can turn them on.  If not, all you could do would be bandaid the problem by having libreadline / bash buffer its input so it could scoop up incoming terminal input faster than it could process it.

 Having bash / libreadline do another layer of input buffering is just a band-aid, since you'd want to set a limit on how big that buffer can be, or else you could get into perverse situations where bash is stuck on a command, but getting data from a terminal.  (e.g. something fell on a keyboard with key repeat on, or some giant binary file gets fed into a terminal accidentally via an ssh usage error or something.  Or infinite other ways, only some of them malicious, that you could make bash use up all the virtual memory.)

Peter Cordes <pcordes>
Tue 16 Oct 2012 02:45:02 PM UTC, original submission:  

Configuration Information:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu' -DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/share/l
ocale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -D_FORTIFY_SOURCE
=2 -DDEFAULT_PATH_VALUE='/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin' -DSTANDARD_UTILS_PATH='/usr/bin:/bin:/usr/sbin:/sbin' -DSYS_BASHRC='/etc/bash.bashrc' -DSYS_BA
SH_LOGOUT='/etc/bash.bash_logout'
uname output: Linux ntb-fedorpiecka 3.5.6-1-ARCH #1 SMP PREEMPT Sun Oct 7 19:30:49 CEST 2012 x86_64 GNU/Linux
Machine Type: x86_64-unknown-linux-gnu

Bash Version: 4.2
Patch Level: 37
Release Status: release


Description:

Long scripts which I paste into xterm (tested with xterm, Gnome Terminal, LXTerminal and Konsole on Gnome, KDE and LXDE) from clipboard are corrupted by BASH.

The problem starts at around 4096 character scripts but is not bound to 4096 and is random. E.g. sometimes 4150 characters work and sometimes 4050 don't.


Repeat-By:

1. Open script1.txt or script2.txt in a text editor in X
2. Select the whole text
3. Copy selected text to clipboard
4. Open terminal window
5. Paste clipboard into the terminal window

Expected result:
a) script1.txt - /tmp/test contains 4096 lines with string test-LINENUMBER on each line where LINENUMBER equals the number of the line

b) script2.txt - /tmp/test contains 128 lines with string "-128abcdefghijlkmnopqrstuvwxyz128-" on the last line

Result in reality:
a) script1.txt - /tmp/test contains randomly selected subset of lines, /tmp/te contains randomly selected subset of lines and bash tries to use /t too and possibly other files with path which is a substring of "/tmp/test"

b) script2.txt - /tmp/test has last N (N seems random) lines corrupted, sometimes bash doesn't register the final EOF correctly and you have to write it by hand (happens with some script lengths)


I've used these generators to prepare the test scripts:

# script1
for i in `seq 1 4096`; do echo "echo test-$i >> /tmp/testfile"; done > /tmp/script1.sh
# end of script1

# script2
echo "cat > /tmp/testfile <<'EOF'" > /tmp/script2.sh
for i in `seq 1 128`; do echo "-${i}abcdefghijlkmnopqrstuvwxyz${i}-"; done >> /tmp/script2.sh
echo EOF >> /tmp/script2.sh
# end of script2


We were able to reproduce the issue on Arch Linux, Linux Mint, Fedora and Debian with bash 4.1 and 4.2 both through an SSH connection and directly.

The issue cannot be reproduced on FreeBSD with bash 3.2.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #26774:  script1.sh added by None (127KiB - text/x-sh)
file #26775:  script2.sh added by None (4KiB - text/x-sh)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by chet (Posted a comment)
  • -email is unavailable- added by pcordes (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-11-13 chet StatusNone Invalid
    2012-10-16 None Attached File- Added script1.sh, #26774
        Attached File- Added script2.sh, #26775

    Back to the top

    Powered by Savane 3.14-3b9d.
    Corresponding source code