GNU GRUB - Bugs: bug #56391, Easily-triggered TCP bug(s)
You are not allowed to post comments on this tracker with your current authentication level.
bug #56391: Easily-triggered TCP bug(s)
Submitter: | Faidon Liambotis <paravoid> | ||
Submitted: | Sun 26 May 2019 02:59:11 AM UTC | ||
Votes: | 1 | ||
Category: | Network | Severity: | Major |
Priority: | 5 - Normal | Item Group: | Software Error |
Status: | None | Privacy: | Public |
Assigned to: | None | Originator Name: | |
Open/Closed: | Open | Release: | Git master |
Release: | Reproducibility: | Every Time | |
Planned Release: | None |
No files currently attached
Depends on the following items: None found
Items that depend on this one: None found
Carbon-Copy List
There is 1 vote so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Follows 1 latest change.
Date | Changed by | Updated Field | Previous Value | => | Replaced by |
---|---|---|---|---|---|
2019-06-18 | jjw | Carbon-Copy | - | ![]() |
Added jjw |
GRUB (git master @ 53e70d30cf0d18e6c28bab0ab8d223a90d3e1b46), PXE booted, with this configuration:
set prefix=(http)/grub
menuentry 'memtest86+' --id memtest {
linux16 /boot/memtest86+.bin
}
Boot the host (say, a QEMU). Reboot and try it again, relatively quickly.
Boot gets stuck at "Welcome to GRUB" and cannot fetch files for a while. This is the simplified version -- I started debugging this when in a complicated menu I was getting a "connection timed out" every now and then when selecting an entry like memtest.
The HTTP server is a Linux 4.9, with nginx, in the same LAN (no firewalls or anything like that). Nothing out of the ordinary there.
It appears that GRUB has a fixed local port that it starts with (21550); the (src IP, src port, dst IP, dst port) tuple remains the same in the two fetches across reboots, so the socket has remained in TIME_WAIT on the server. Thus, the server attempts to reuse the same ack nr, per RFC. All hell breaks loose after that. Multiple bugs I think -- GRUB gets something unexpected but doesn't notice, doesn't RST, and keeps sending the same ACKs over and over. Plus, Wireshark doesn't seem to be happy with the FIN/ACK of the first connection either.
(I took a look at the source, which seems ridden with FIXMEs. Have you thought about using a third-party TCP/IP stack like lwIP or PicoTCP? No idea how easy it would be to integrate this, apologies if this is a stupid idea :)
Packet capture from tshark (tshark -n -o tcp.relative_sequence_numbers:FALSE -r grub.pcap 'tcp.port == 21550'):
<first boot>
1 0.000000 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [SYN] Seq=96 Win=8192 Len=0
2 0.000290 192.168.24.1 → 192.168.24.104 TCP 60 80 → 21550 [SYN, ACK] Seq=150886417 Ack=97 Win=29200 Len=0 MSS=1460
3 0.000689 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [ACK] Seq=97 Ack=150886418 Win=8192 Len=0
4 0.001023 192.168.24.104 → 192.168.24.1 HTTP 162 GET /grub/i386-pc/command.lst HTTP/1.1
5 0.001175 192.168.24.104 → 192.168.24.1 TCP 162 [TCP Retransmission] 21550 → 80 [PSH, ACK] Seq=97 Ack=150886418 Win=8192 Len=108
6 0.001208 192.168.24.1 → 192.168.24.104 TCP 60 80 → 21550 [ACK] Seq=150886418 Ack=205 Win=29200 Len=0
7 0.001335 192.168.24.1 → 192.168.24.104 TCP 60 [TCP Dup ACK 6#1] 80 → 21550 [ACK] Seq=150886418 Ack=205 Win=29200 Len=0
8 0.001576 192.168.24.1 → 192.168.24.104 TCP 590 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
9 0.001581 192.168.24.1 → 192.168.24.104 TCP 590 80 → 21550 [ACK] Seq=150886954 Ack=205 Win=29200 Len=536 [TCP segment of a reassembled PDU]
10 0.001583 192.168.24.1 → 192.168.24.104 TCP 590 80 → 21550 [ACK] Seq=150887490 Ack=205 Win=29200 Len=536 [TCP segment of a reassembled PDU]
11 0.001585 192.168.24.1 → 192.168.24.104 TCP 590 80 → 21550 [ACK] Seq=150888026 Ack=205 Win=29200 Len=536 [TCP segment of a reassembled PDU]
12 0.001587 192.168.24.1 → 192.168.24.104 TCP 590 80 → 21550 [ACK] Seq=150888562 Ack=205 Win=29200 Len=536 [TCP segment of a reassembled PDU]
13 0.001646 192.168.24.1 → 192.168.24.104 TCP 590 80 → 21550 [ACK] Seq=150889098 Ack=205 Win=29200 Len=536 [TCP segment of a reassembled PDU]
14 0.001651 192.168.24.1 → 192.168.24.104 TCP 590 80 → 21550 [ACK] Seq=150889634 Ack=205 Win=29200 Len=536 [TCP segment of a reassembled PDU]
15 0.001653 192.168.24.1 → 192.168.24.104 HTTP 393 HTTP/1.1 200 OK
16 0.001894 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [ACK] Seq=205 Ack=150886954 Win=8192 Len=0
17 0.002365 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [ACK] Seq=205 Ack=150887490 Win=8192 Len=0
18 0.002554 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [ACK] Seq=205 Ack=150888026 Win=8192 Len=0
19 0.002721 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [ACK] Seq=205 Ack=150888562 Win=8192 Len=0
20 0.002885 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [ACK] Seq=205 Ack=150889098 Win=8192 Len=0
21 0.003047 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [ACK] Seq=205 Ack=150889634 Win=8192 Len=0
22 0.003213 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [ACK] Seq=205 Ack=150890170 Win=8192 Len=0
23 0.003393 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [ACK] Seq=205 Ack=150890510 Win=8192 Len=0
24 0.003566 192.168.24.104 → 192.168.24.1 TCP 54 21550 → 80 [FIN, ACK] Seq=205 Ack=150890510 Win=0 Len=0
25 0.003770 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Out-Of-Order] 21550 → 80 [FIN, ACK] Seq=205 Ack=150890510 Win=0 Len=0
26 0.003798 192.168.24.1 → 192.168.24.104 TCP 60 80 → 21550 [ACK] Seq=150890510 Ack=206 Win=29200 Len=0
27 0.003945 192.168.24.1 → 192.168.24.104 TCP 60 [TCP Dup ACK 26#1] 80 → 21550 [ACK] Seq=150890510 Ack=206 Win=29200 Len=0
<reboot here>
71 36.235831 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Port numbers reused] 21550 → 80 [SYN] Seq=93 Win=8192 Len=0
72 36.236082 192.168.24.1 → 192.168.24.104 TCP 60 [TCP ACKed unseen segment] 80 → 21550 [ACK] Seq=150890510 Ack=206 Win=29200 Len=0
73 36.236502 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Dup ACK 0#1] 21550 → 80 [ACK] Seq=94 Ack=0 Win=8192 Len=0
74 36.635996 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Retransmission] 21550 → 80 [SYN] Seq=93 Win=8192 Len=0
75 37.035883 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Retransmission] 21550 → 80 [SYN] Seq=93 Win=8192 Len=0
76 37.036117 192.168.24.1 → 192.168.24.104 TCP 60 [TCP Dup ACK 72#1] [TCP ACKed unseen segment] 80 → 21550 [ACK] Seq=150890510 Ack=206 Win=29200 Len=0
77 37.036428 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Dup ACK 0#2] 21550 → 80 [ACK] Seq=94 Ack=0 Win=8192 Len=0
78 37.435783 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Retransmission] 21550 → 80 [SYN] Seq=93 Win=8192 Len=0
79 37.835738 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Retransmission] 21550 → 80 [SYN] Seq=93 Win=8192 Len=0
80 37.835972 192.168.24.1 → 192.168.24.104 TCP 60 [TCP Dup ACK 72#2] [TCP ACKed unseen segment] 80 → 21550 [ACK] Seq=150890510 Ack=206 Win=29200 Len=0
81 37.836155 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Dup ACK 0#3] 21550 → 80 [ACK] Seq=94 Ack=0 Win=8192 Len=0
82 38.235704 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Retransmission] 21550 → 80 [SYN] Seq=93 Win=8192 Len=0
83 38.635605 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Retransmission] 21550 → 80 [SYN] Seq=93 Win=8192 Len=0
84 38.635810 192.168.24.1 → 192.168.24.104 TCP 60 [TCP Dup ACK 72#3] [TCP ACKed unseen segment] 80 → 21550 [ACK] Seq=150890510 Ack=206 Win=29200 Len=0
85 38.635962 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Dup ACK 0#4] 21550 → 80 [ACK] Seq=94 Ack=0 Win=8192 Len=0
86 39.035594 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Retransmission] 21550 → 80 [SYN] Seq=93 Win=8192 Len=0
87 39.435549 192.168.24.104 → 192.168.24.1 TCP 54 [TCP Retransmission] 21550 → 80 [SYN] Seq=93 Win=8192 Len=0
88 39.435754 192.168.24.1 → 192.168.24.104 TCP 60 [TCP Dup ACK 72#4] [TCP ACKed unseen segment] 80 → 21550 [ACK] Seq=150890510 Ack=206 Win=29200 Len=0
<etc. etc. etc., keeps repeating for a while>