bugGNU nano - Bugs: bug #66858, spurious tilde when pasting five...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #66858: spurious tilde when pasting five bytes over OpenSSH_for_Windows

Submitter:  Benno Schulenberg <bens>
Submitted:  Fri 28 Feb 2025 04:21:07 PM UTC
   
 
Severity:  3 - Normal Status:  Fixed
Assigned to:  bens Open/Closed:  Closed

Jump to the original submission

Thu 03 Apr 2025 10:46:12 AM UTC, comment #25: 

The fix for this issue was released in nano-8.4.

Benno Schulenberg <bens>
Group administrator
Tue 11 Mar 2025 03:14:05 PM UTC, comment #24: 

Beeping for a newline is maybe unneeded, but nano should definitely beep for any other command code.  So the beep will return in a subsequent commit.  (But at the moment I can't push -- the git repos seem to be down.)

This beeping has made me think about tabs -- tabs inside a paste.  They should not be understood as commands, I think, but as literal tab characters.  See bug #66892.  Maybe you have an opinion on it.]

Benno Schulenberg <bens>
Group administrator
Mon 10 Mar 2025 11:13:44 PM UTC, comment #23: 


comment #22:

>  -- it used to beep for the embedded ^M's.  I'm not sure what I like better.


Yes, I'm not sure either.
Anyway, thank you for all your very hard work on this one.

> Anyway...  Pushed to master, commit bf12c90b.


I did compile and briefly tested the new master today. All worked fine.

Doug Smythies <dougsmythies>
Mon 10 Mar 2025 04:21:08 PM UTC, comment #22: 

Thank you for the meticulous testing, and for hanging in there while I fumbled my way around.

To better test the change, I've dug up an old little laptop from a cupboard, one that still actually beeps for a BELL (\a).  I've modified the patch to not beep any more for pastes at a prompt, but it also means that it doesn't beep any more when pasting multiple lines at a prompt -- it used to beep for the embedded ^M's.  I'm not sure what I like better.

Anyway...  Pushed to master, commit bf12c90b.

Benno Schulenberg <bens>
Group administrator
Mon 10 Mar 2025 03:43:25 AM UTC, comment #21: 


comment #19:

> Does the bell ring anywhere when the paste was actually valid?


Yes, the bell rings for paste into what to search for and paste to file name to save to, which are the only 2 I tried.
A screen shot of my testing table is attached.
I also try to copy the table below, but the formatting might get messed up:

ssh client nano version paste size Bell? comment
win-ssh test 16,528,512 No worked. took 103 seconds
win-ssh 8.3 16,528,512 No worked. took 103 seconds
leonard test 16,528,512 No worked. took 24 seconds. 4.3X
leonard 8.3 16,528,512 No worked. took 23 seconds. 4.5X
win-ssh test 05 through 25 No worked
win-ssh or leonard test 22 Yes paste to file name to save. worked.
win-ssh or leonard 8.3 22 No paste to file name to save. worked.
win-ssh or leonard test 22 Yes paste to search string. worked.
win-ssh or leonard 8.3 22 No paste to search string. worked.
win-ssh or leonard test 5 Yes paste to file name to save. worked.
win-ssh or leonard 8.3 5 No paste to file name to save. worked.
win-ssh or leonard test 5 Yes paste to search string. worked.
win-ssh or leonard 8.3 5 No paste to search string. worked.
PuTTY test 5 Yes paste to search string. worked.
PuTTY test 5 Yes paste to file name to save. worked.
PuTTY 8.3 5 No paste to search string. worked.
PuTTY 8.3 5 No paste to file name to save. worked.



Doug Smythies <dougsmythies>
Sun 09 Mar 2025 05:43:15 PM UTC, comment #20: 


comment #19:

> Thanks for testing, and for confirming that it works.  To be certain: for testing you used the original OpenSSH_for_Windows program?  Not the exe provided by Leonard?


Yes, I was very careful to use the original OpenSSH_for_Windows program. You can also tell from the test details file that I was using the correct program.
 

> Attached is a more complete patch.  Please give it some exercise, pasting things also at prompts and other places.  Does the bell ring anywhere when the paste was actually valid?


Give me a few days to test. I am not sure what is wanted for "pasting things also at prompts and other places". Typically the only other place in nano that I paste into is "the what string to search for" prompt.
 

> (The patch applies cleanly only to current git -- commit 615570cf -- not to anything older.)


Okay. I did a git pull, then created working branch, then applied the patch.

doug@s19:~/temp-git-nano/nano$ git log --oneline
a8f06429 (HEAD -> benno2) input: implement bracketed pastes in a different manner
615570cf (origin/master, origin/HEAD, master) tweaks: remove two commented-out lines and two unused variables
aff9f474 input: remove single-keycode detection from the bracketed-paste routine
31a2cc9d tweaks: swap two fragments of code, to allow unwrapping a line
...

Doug Smythies <dougsmythies>
Sun 09 Mar 2025 04:54:03 PM UTC, comment #19: 

Thanks for testing, and for confirming that it works.  To be certain: for testing you used the original OpenSSH_for_Windows program?  Not the exe provided by Leonard?

Attached is a more complete patch.  Please give it some exercise, pasting things also at prompts and other places.  Does the bell ring anywhere when the paste was actually valid?

(The patch applies cleanly only to current git -- commit 615570cf -- not to anything older.)


(file #56991)

Benno Schulenberg <bens>
Group administrator
Sat 08 Mar 2025 05:12:03 PM UTC, comment #18: 

The patch from comment #17 was applied on top of 9c6a62d3, which is the up to date git master as of today (2025.03.08). The usual "Sequence of hex codes" debug aid was also enabled, via code edit.
In addition to a lot of 5 character pastes, several paste lengths were tried, including the 93,298 character one and an over 1 mega character one.
Most, but not all, of the testing was done with a 5X100% load on the windows computer, which is the highest reasonable load for it's processor.

Summary: Everything was fine.

Details are in the attached file.


(file #56986)

Doug Smythies <dougsmythies>
Sat 08 Mar 2025 09:40:53 AM UTC, comment #17: 

When you asked "Is something else in between doing it?", I thought: Ncurses!  On the mailing list I had noted that some escape sequences are kept together, but others not [1].  Somehow ncurses does not know about "\e[200~" and "\e201~" -- and how could it: they are not in the terminfo description of the terminal.  ...  Then I understood: nano needs to tell ncurses about those sequences, with define_key(), so that ncurses knows them and will try to keep them together.

See the attached patch.  Please try it out on top of current git, with your default OpenSSH program, and with the five-character paste that is almost guaranteed to split the last tilde off into a separate chunk.  If things work as I think they do, then you should see each time a burst of seven hex codes: the start-of-paste code, the five characters, and the end-of-paste code.

(The patch is rough, implementing bracketed pastes only for the edit window, not for the prompts or anything else.  That will come when this attempt works.)

[1] https://lists.gnu.org/archive/html/nano-devel/2025-02/msg00011.html


(file #56985)

Benno Schulenberg <bens>
Group administrator
Fri 07 Mar 2025 03:37:40 PM UTC, comment #16: 


Thanks for all your hard work on this issue.

By the way, there has been activity over on the WIN-SSH discussion [1] also. Until the new longer chunk code is officially released I can just use the .exe the guy provided.

comment #15:

> Yes, we're back to inserting an unwanted tilde plus the "Flawed paste" warning.  But now the hang preventions are more general: they should catch any misbehaving paste.  That's better than a limited workaround for one specific faulty program -- especially since a fine-functioning alternative exists.
>


Is the current WIN-SSH program faulty? I have not found any specification that says that "end of pasted text" ESCAPE sequences can not be split across "chunks". I am not saying there isn't such a specification, I am just saying I haven't found it.

I tested various paste sizes from 5 to 10 characters with my key-stroke timing monitoring program. The time on the sending chunk boundary is always similar in the 20 to 40 mSec range. But from previous work, we know that different chunk sizes are being seen by nano. I don't understand this. Is something else in between doing it? Anyway, I'll ask over on the discussion if it even knows about bracketed paste.

[1] https://github.com/PowerShell/Win32-OpenSSH/discussions/2330

Doug Smythies <dougsmythies>
Fri 07 Mar 2025 09:48:47 AM UTC, comment #15: 

Sure, a lone and broken start-of-paste sequence should never occur.  But... if it were to occur, the resulting behavior is atrocious.  I don't want it.  And the issue that the responsible code works around is also caught by the single-code detection in the bracketed-paste routine -- in a less clean manner, and with a spurious tilde, but it clearly indicates that something is wrong.

Yes, we're back to inserting an unwanted tilde plus the "Flawed paste" warning.  But now the hang preventions are more general: they should catch any misbehaving paste.  That's better than a limited workaround for one specific faulty program -- especially since a fine-functioning alternative exists.

Benno Schulenberg <bens>
Group administrator
Fri 07 Mar 2025 06:17:13 AM UTC, comment #14: 

comment #13:

> Thanks for the detailed testing.
>
> However, testing this stuff some more, I found that sending an incomplete start-of-paste sequence (\e[200~ minus the ~) to the input, nano fully blanks the screen, pauses a second, and then reports an "Unknown sequence" while leaving the screen blank.  This is unacceptable behavior.  So I have reverted commit 555a9878 (that implemented the waiting for a missing tilde).


Isn't the probability of an incomplete "begin of pasted text" very low? I do not know how it would occur.

> Furthermore, the waiting for 40 ms at the end of a paste would affect every paste, for everyone.  And I can feel those 40 ms -- the reaction is not any more as immediate as it used to be.


Fair enough. I have been experimenting with responsiveness verses time waiting also.

> So, the check before waiting 40 ms has to be more specific: nano must detect that an escape sequence is being truncated.  See attached patch.  It detects truncation only immediately after "\e" or after "\e[", and leaves other truncations to the "Flawed paste" error.  (The commented out line is for making the paste simulation with the xdotool work.)
>
> What do you think?


Well, it puts things back the way they were near the start of all this.
Myself, I would ignore the test reaults I got in comment #8, before going back to this, because that situation is much much less probable than the single trailing ~ case.

Doug Smythies <dougsmythies>
Thu 06 Mar 2025 03:47:30 PM UTC, comment #13: 

Thanks for the detailed testing.

However, testing this stuff some more, I found that sending an incomplete start-of-paste sequence (\e[200~ minus the ~) to the input, nano fully blanks the screen, pauses a second, and then reports an "Unknown sequence" while leaving the screen blank.  This is unacceptable behavior.  So I have reverted commit 555a9878 (that implemented the waiting for a missing tilde).

Furthermore, the waiting for 40 ms at the end of a paste would affect every paste, for everyone.  And I can feel those 40 ms -- the reaction is not any more as immediate as it used to be.  So, the check before waiting 40 ms has to be more specific: nano must detect that an escape sequence is being truncated.  See attached patch.  It detects truncation only immediately after "\e" or after "\e[", and leaves other truncations to the "Flawed paste" error.  (The commented out line is for making the paste simulation with the xdotool work.)

What do you think?


(file #56978)

Benno Schulenberg <bens>
Group administrator
Wed 05 Mar 2025 10:47:47 PM UTC, comment #12: 

I tested the patch from comment #10 applied on top of the work from yesterday, ie.e on top of f8de98f7.
Everything behaved as expected. Details attached.

(file #56976)

Doug Smythies <dougsmythies>
Wed 05 Mar 2025 04:16:00 PM UTC, comment #11: 

Agreed about the every keystroke would be delayed part.

The graph of inter-chunk delays I supplied on the email thread was just one test, with no other loads on the windows computer. Attached is a graph of repeated 5 character pastes, showing that, for my computer at least, a bit more max delay should be used. The graph has various loads, but never a severe overload, on the windows computer. Example data capture segment:

   1581    98  4517405  1b <<< It has been 4.5 seconds since the last paste
   1582    98        3  5b
   1583    98        1  32
   1584    99        1  30 <<< The 16 byte chunk boundaries are useless here.
   1585    99        1  30
   1586    99        1  7e
   1587    99        1  31
   1588    99        1  32
   1589    99        1  33
   1590    99        1  34
   1591    99        1  35
   1592    99        1  1b
   1593    99        1  5b
   1594    99        1  32
   1595    99        1  30
   1596    99        1  31
   1597    99    33613  7e <<< 33.6 mSec chunk boundary delay Trailing "~"
   1598    99  3189538  1b <<< It has been 3.2 seconds since the last paste
   1599    99        4  5b
   1600   100        1  32
   1601   100        1  30
   1602   100        1  30
   1603   100        1  7e
   1604   100        1  31
   1605   100        1  32
   1606   100        1  33
   1607   100        1  34
   1608   100        1  35
   1609   100        1  1b
   1610   100        1  5b
   1611   100        1  32
   1612   100        1  30
   1613   100        1  31
   1614   100    25959  7e <<< 26 mSec chunk boundary delay. Trailing "~"
  |       |          |__ time in uSec since the previous character got to the program
  |       |__ buffer number = meaningless and not valid in this test.
  |__ Character number since start of keystroke monitoring program

It will be later today or tomorrow before I can test the patch.


Doug Smythies <dougsmythies>
Wed 05 Mar 2025 08:25:58 AM UTC, comment #10: 

Thank you for all the diligent testing.  It's clear now that waiting only for a trailing tilde is not good enough.  Your idea of waiting 29 milliseconds when there is no more input was good, but would mean that every keystroke would be delayed for 29 ms.  I've improved on this idea by waiting only when already several key codes have been gathered, as this most likely means that a paste is in progress, because nobody can type so fast (for several keystrokes to accumulate in the input stream before nano gets around to handling them -- or the computer is so bogged down that waiting another 29 ms won't be noticed anyway).

So, please try the attached patch.  I've chosen the value 4 as threshold for the waiting codes, because then also my paste-simulating script procuces this:

Last code = 0x30,  from 5
Last code = 0x7e,  from 17
Sequence of hex codes:  1b  5b  32  30  30  7e  41  62  63  64  65  1b  5b  32  30  31  7e
Sequence of hex codes:  18
Sequence of hex codes:  11

instead of this:

Sequence of hex codes:  1b  5b  32  30  30
Sequence of hex codes:  41
Sequence of hex codes:  62  63  64  65
Sequence of hex codes:  1b  5b  32  30  31
Sequence of hex codes:  18
Sequence of hex codes:  11

Of course, the fprintf() is just for debugging purposes.  And your email to the list suggest that 24 ms should be enough: more than the largest delay spike.  But you probably have to raise it to 29 ms.

(file #56973)

Benno Schulenberg <bens>
Group administrator
Tue 04 Mar 2025 04:23:19 PM UTC, comment #9: 

(I used the wording "40 bytes" because in UTF-8 a character can be more than one byte.  It probably would have been clearer to say "40 ASCII characters".)

So... when the pasting system is under load, it's not just the final tilde that can get split off into a separate chunk but probably anything after the Escape character.  So the workaround that I just pushed to master (commit 555a9878) will not solve all of the hangs with OpenSSH_for_Windows.  But the subsequent commit (41c1b962) should allow to break a pasting hang by hitting any key.  Can you confirm the latter when pasting 40 ASCII characters from a system under load?

Benno Schulenberg <bens>
Group administrator
Mon 03 Mar 2025 11:11:12 PM UTC, comment #8: 

By "paste exactly 40 bytes" do you mean 40 characters, which would be more bytes? Anyway, I did 40 characters.

For no load on the windows computer I got: A line of 16 characters that started with the 6 character "begin of pasted text" sequence; Followed by a line of of 36 characters ending with the 6 character "end of pasted text" sequence;

For 5 out of 6 cores 100% loaded (not multi-threaded, so 1 CPU per core, or 6 CPUs total. i5-9800K) I got the exact same.

For 6 out of 6 cores 100% loaded, the nano program froze, and I had to use that trick I learned from one your bug reports or somewhere else of pasting a different sized buffer to unfreeze it. I got: The same first line; 16 characters on the 2nd line; 16 characters on the 3rd line ending with only the ESCAPE and "[" character part of the end sequence; The rest of the end sequence on the third line. When I unfroze nano, then the "201~" also appeared in the pasted text.

For 7 tasks that would each take 100% of 1 CPU, I got the exact same thing.

I'll attach a file with all of the "Sequence of hex codes" debug outputs.

I was using a nano from my clone of the git master, as of a git pull from a few days ago:

c1df6dfd (HEAD -> master, origin/master, origin/HEAD) execute: retain what the user typed before a tool was invoked
 
with just an edit for the "Sequence of hex codes" stuff:

doug@s19:~/temp-git-nano/nano$ git diff
diff --git a/src/winio.c b/src/winio.c
index 2fff0d9e..b4b8c49b 100644
--- a/src/winio.c
+++ b/src/winio.c
@@ -303,12 +303,10 @@ void read_keys_from(WINDOW *frame)
        /* Restore blocking-input mode. */
        nodelay(frame, FALSE);

-#ifdef DEBUG
        fprintf(stderr, "\nSequence of hex codes:");
        for (size_t i = 0; i < waiting_codes; i++)
                fprintf(stderr, " %3x", key_buffer[i]);
        fprintf(stderr, "\n");
-#endif
 }

 /* Return the number of key codes waiting in the keystroke buffer. */

By just waiting longer before deciding there is no further input, everything works fine no matter the load. I.E. this from on the email thread the other day:

diff --git a/src/winio.c b/src/winio.c
index 2fff0d9e..755d775b 100644
--- a/src/winio.c
+++ b/src/winio.c
@@ -290,8 +290,13 @@ void read_keys_from(WINDOW *frame)
                input = wgetch(frame);

                /* If there aren't any more characters, stop reading. */
-               if (input == ERR)
-                       break;
+
+               if (input == ERR) {
+                       napms(29);  /* check for sender slower than here */
+                       input = wgetch(frame); /* check again */
+                               if (input == ERR)
+                                       break;
+               }

                /* When the keystroke buffer is full, extend it. */
                if (waiting_codes == capacity)

Doug Smythies <dougsmythies>
Mon 03 Mar 2025 04:44:44 PM UTC, comment #7: 

I've grepped the code from https://github.com/PowerShell/openssh-portable for possible escape codes (033, 27, 1b, 1B, \e), but nowhere does it appear to check for an escape character in  buffer, so how could it do the chunking observed when pasting 6 to 10 bytes?  It must be done somewhere else.

When you paste exactly 40 bytes, does nano receive the paste in one batch of 52 bytes (including the start and stop sequeneces), or in two batches of 16 plus one of 20 bytes?  Or does that also depend on the load of the sender?

Benno Schulenberg <bens>
Group administrator
Sun 02 Mar 2025 04:01:21 PM UTC, comment #6: 


comment #5:

> Since we already know there is subtle timing involved, I assume that is the difference here.
>


I loaded down my windows computer with CPU burning jobs. And tried a couple of the paste buffer sizes that previously PASSED: 149 now FAILED; 133 now FAILED; 21 now FAILED; 37 now FAILED

Doug Smythies <dougsmythies>
Sun 02 Mar 2025 03:43:50 PM UTC, comment #5: 

Good find. I had searched the issues, but didn't find #771.
Back when I found the problematic paste sizes of 5, 85, 197, and 293 I was trying to find any pattern, without the knowledge of the N*16-11 equation, although I was using the cool "Sequence of hex codes" debugging aid to observe chunk boundaries. I tried some of the paste sizes that should have an issue, but they didn't. I redid the work:

5 FAIL
21 PASS
37 PASS
53 PASS
69 PASS
85 PASS
101 FAIL
117 PASS
133 PASS
149 PASS
165 PASS
181 FAIL
197 FAIL
213 PASS
229 PASS
245 PASS
261 PASS
277 PASS
293 FAIL
309 PASS

Since we already know there is subtle timing involved, I assume that is the difference here.

Doug Smythies <dougsmythies>
Sun 02 Mar 2025 11:17:50 AM UTC, comment #4: 

Clicking along from the Github link you gave in one of your emails, I found this: https://github.com/PowerShell/openssh-portable/pull/771.  Seeing the old buffer size of 16 bytes, I realized that the numbers you mentioned earlier (5, 85, 197, 293) are all multiples of 16 minus 11.  If that buffer size is the cause, then also pastes of 21 bytes, 37 bytes, 53 bytes, and so on, should go wrong.  Can you confirm?

(If yes, then in the next version of OpenSSH_for_Windows pastes will break for 1013 bytes, 2037 bytes, 3061 bytes, and so on.)

Benno Schulenberg <bens>
Group administrator
Sat 01 Mar 2025 04:15:26 PM UTC, comment #3: 

Thanks for testing and confirming success.

Benno Schulenberg <bens>
Group administrator
Sat 01 Mar 2025 03:02:02 PM UTC, comment #2: 

I tested the patch for previously identified problematic paste buffer lengths of 5, 85, 197, and 293 characters. All worked fine. Other general testing, such as pasting large buffers and such also worked fine.

Doug Smythies <dougsmythies>
Sat 01 Mar 2025 11:38:54 AM UTC, comment #1: 

Patch (atteched) that waits a bit for the tilde to arrive, and then consumes it, preventing it from getting pasted.


(file #56960)

Benno Schulenberg <bens>
Group administrator
Fri 28 Feb 2025 04:21:07 PM UTC, original submission:  

(Original report by Doug Smythies in https://lists.gnu.org/archive/html/nano-devel/2025-02/msg00001.html.)

When pasting five bytes over OpenSSH_for_Windows into a nano running on a Linux computer, then a spurious "~" can appear after the paste.  (Before version 8.0 things were worse: nano would hang.)

When pasting five bytes several times quickly after another, then only the first paste is affected -- the subsequent ones are fine.

The root issue seems to be a "chunking" problem in OpenSSH_for_Windows, but... it would be nice to have a workaround for this in nano.

Benno Schulenberg <bens>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #56985:  define-SOP-and_EOP.patch added by bens (3KiB - text/x-patch)
file #56978:  check-for-truncation.patch added by bens (821B - text/x-patch)
file #56974:  paste-5-delays.png added by dougsmythies (26KiB - image/png - inter-chunk delays)
file #56971:  TRAIL-40-f8de98f7.txt added by dougsmythies (5KiB - text/plain - Test results for Benno's comment #9 request. Summary: Yes, confirmed.)
file #56969:  TRAIL-40-VARIOUS-LOADS.txt added by dougsmythies (3KiB - text/plain - test data for comment #8)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dougsmythies (Posted a comment)
  • -email is unavailable- added by bens (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 16 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2025-04-03 bens Open/ClosedOpen Closed
    2025-03-10 bens StatusReady for test Fixed
    2025-03-10 dougsmythies Attached File- Added nano-testing-2025-03-09.png, #56994
    2025-03-09 bens StatusIn progress Ready for test
    2025-03-09 bens Attached File- Added 0001-input-implement-bracketed-pastes-in-a-different-mann.patch, #56991
    2025-03-08 dougsmythies Attached File- Added test-details-2025-03-08.txt, #56986
    2025-03-08 bens Attached File- Added define-SOP-and_EOP.patch, #56985
    2025-03-06 bens Attached File- Added check-for-truncation.patch, #56978
    2025-03-05 dougsmythies Attached File- Added TRAIL-40-patch-report.txt, #56976
    2025-03-05 dougsmythies Attached File- Added paste-5-delays.png, #56974
    2025-03-05 bens Attached File- Added wait-when-paste-is-throttled.patch, #56973
    2025-03-04 dougsmythies Attached File- Added TRAIL-40-f8de98f7.txt, #56971
    2025-03-03 dougsmythies Attached File- Added TRAIL-40-VARIOUS-LOADS.txt, #56969
    2025-03-01 bens StatusReady for test In progress
    2025-03-01 bens Attached File- Added 0001-input-wait-a-bit-for-when-bracketed-paste-sequence-i.patch, #56960
        StatusIn progress Ready for test

    Back to the top

    Powered by Savane 3.15.
    Corresponding source code