helpGnuTLS - Support: sr #107948, DTLS handshake deadlocks on packet...


sr #107948: DTLS handshake deadlocks on packet loss

Submitted by:  None
Submitted on:  Mon 06 Feb 2012 02:21:05 PM UTC  
Category:  Core library Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  Nikos Mavrogiannopoulos <nmav>
Originator Email:  -email is unavailable- Open/Closed:  Closed
Operating System:  GNU/Linux

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Fri 10 Feb 2012 10:06:56 PM UTC, comment #15: 

I've retested with current git master, and unfortunately, losing complete server hello flights is still deadly. So is losing packets in the server's final flight: the server will report Handshake Success after the flight is sent, but the client might not receive parts and retransmit it's final flight, which the server will drop.

I'm writing a comprehensive test for both drops and reordering, when I'm done, I'll send it to -devel.

Fri 10 Feb 2012 06:51:04 PM UTC, comment #14: 

I've quite improved it since then (some packets were discarded by the sliding window used to detect duplicates). Thus I'd expect the issues you mention not to be there. If they are please let me know.

Nikos Mavrogiannopoulos <nmav>
Project AdministratorIn charge of this item.
Thu 09 Feb 2012 05:19:30 PM UTC, comment #13: 

The deadlocks I was seeing were in fact not deadlocks, but a programming error on my part. Having fixed that, I can't reproduce any deadlocks anymore either, but dropping three server hello flights still kills both varieties of handshake. The server only drops our retransmitted ClientHello packets, and does not retransmit the server hello flight. Same thing goes for dropping client's ChangeCipherSpec: the server won't retransmit it's own flight, but instead drop the retransmitted flights.

Thu 09 Feb 2012 09:02:07 AM UTC, comment #12: 

The handshake termination could be because of timeouts (might need some trimming). I'll verify that. Do you remember on which dropped packet you saw deadlocks?

Contributing test cases for missing packets and/or reordering would be really great. If you are interested I'd invite you to fill the form in the url, and send it to

Wed 08 Feb 2012 11:40:42 PM UTC, comment #11: 

Now the server side of things works nicely with lost packets. Only dropping three server hello flights still kills the handshake and forces the client to time out, but that's fine. Similar result on the client side. Almost all is well, only losing two ChangeCipherSpec packets is still fatal. The handshake does not deadlock, it simply times out, so that is pretty much fine too. All in all, the bug looks squashed for nonblocking.

For blocking, I still see deadlocks, but since can't reproduce, and mini-loss runs fine, I must be doing something wrong there.

Judging from this new robustness against loss, packet reordering should be no problem either. If you wish, I can test that thoroughly as well.

As for my test code: an update with client drop predicates is attached; if you want to use it for the GnuTLS test suite, I'd be happy to rewrite it and grant you whatever rights necessary.

(file #25032)

Wed 08 Feb 2012 10:52:16 PM UTC, comment #10: 

It seems the last flight handling wasn't complete. I've improved it quite now. What I cannot see is the non-deterministic behavior you describe. Your tests were quite interesting. I've tried to emulate it in mini-loss.

Nikos Mavrogiannopoulos <nmav>
Project AdministratorIn charge of this item.
Wed 08 Feb 2012 06:51:45 PM UTC, comment #9: 

Oops. The file I attached did not do quite the right thing (dropped one flight too few), and it didn't include errno.h - for clang compiled it, while gcc will not, for whatever reason.

The actuall diff is small, both final flight drop predicates had an off-by-one on the packet counter.

(file #25029)

Wed 08 Feb 2012 06:42:24 PM UTC, comment #8: 

mini-loss.c and mini-loss2.c run and pass on my machine as well.

For your question: I have observed that GnuTLS does not coalesce handshake packets into single transport layer packets over the loopback interface, so I used a modified version of handshake.cpp with some more advanced fingerprinting for server handshake packets. The same thing could be done for the client, but I haven't tested that yet. Of course I'd be more than happy to find out that this is all my mistake.

Code is attached.

(file #25028)

Wed 08 Feb 2012 05:20:23 PM UTC, comment #7: 

I'm not been able to reproduce that. I've added a test program that tests the behavior of packet loss in blocking mode (mini-loss2.c) and I do not see what you mention. I couldn't also make the handshake.cpp to deadlock.

How did you drop individual handshake packets? (gnutls bundles each flight in a single packet if possible) The issue is quite important. Deadlocks shouldn't occur, so if you have some example code, I'd like to see it.


Nikos Mavrogiannopoulos <nmav>
Project AdministratorIn charge of this item.
Wed 08 Feb 2012 02:49:42 PM UTC, comment #6: 

GnuTLS does behave better now, but unfortunately, there are still problems:

  • losing any, but not all, of ServerHello, ServerKeyExchange, ServerHelloDone, deadlocks the handshake for blocking sessions.
  • losing all of them sometimes deadlocks the handshake for nonblocking sessions. The server drops retransmitted ClientHello, instead of retransmitting it's own flight, in maybe half the cases. In the other half, everything works out fine. Blocking sessions never retransmit, and this deadlock. This seems strange, but I cant find errors in my test program.
  • losing any, but not all, of the server's ChangeCipherSpec, Finished causes deadlocks for all sessions. No retransmission is done by the client when a duplicated handshake message from the next epoch is received.

I must admit that this is getting academic. Also, maybe I'm reading the DTLS RFC incorrectly.

Wed 08 Feb 2012 09:55:56 AM UTC, comment #5: 

I've committed a fix and a test application, which test individual packet loss cases. Let me know if this fixes the issues you notice and/or you notice anything else.

Nikos Mavrogiannopoulos <nmav>
Project AdministratorIn charge of this item.
Tue 07 Feb 2012 10:08:02 PM UTC, comment #4: 

Indeed. I'm already checking it.

Nikos Mavrogiannopoulos <nmav>
Project AdministratorIn charge of this item.
Tue 07 Feb 2012 09:51:53 PM UTC, comment #3: 

That indeed fixes the problem for packets lost in the first flights, but some parts remain:

  • losing the client ChangeCipherSpec will kill the server with unexpected TLS packet.
  • losing any other packet of the final flight will still deadlock the handshake.

The latter will affect both blocking and nonblocking operation; nonblocking DTLS will return GNUTLS_E_AGAIN indefinitely. Can you confirm?

Tue 07 Feb 2012 06:36:15 PM UTC, comment #2: 

Hello, it seems gnutls was being overly strict in checking incoming packets for DTLS. The patch below should fix your issue. Thank you for reporting it.;a=commitdiff;h=35c17f2fd72083bf12c5faa83195d18c3cb759ce

Nikos Mavrogiannopoulos <nmav>
Project AdministratorIn charge of this item.
Tue 07 Feb 2012 10:34:01 AM UTC, comment #1: 

This shouldn't have happened. I'll check it out and let you know. --nmav

Mon 06 Feb 2012 02:21:05 PM UTC, original submission:  

When packets in the middle of any handshake flight are lost, DTLS handshake fails in two situations:

  • in blocking mode, handshake simply deadlocks and despite all timeouts, no progress is made.
  • in nonblocking mode, handshake continues as expected, but fails on completion with GNUTLS_E_UNEXPECTED_PACKET or through more obscure errors like GNUTLS_E_ERROR_IN_FINISHED_PACKET on client side.

Given the rather low probability that packets are lost in exactly this manner, this bug does not seem severe. It is rather annoying though, as it leads to handshakes failing that should have succeeded.

A testcase (rather lengthy) is attached.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #25032:  handshake-all.cpp added by None (7KiB - text/x-c++src)
file #25029:  handshake2.cpp added by None (6KiB - text/x-c++src)
file #25028:  handshake2.cpp added by None (6KiB - text/x-c++src)
file #24993:  handshake.cpp added by None (4KiB - text/x-c++src)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by nmav (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-02-24 nmav StatusReady For Test => Done
        Open/ClosedOpen => Closed
    2012-02-08 None Attached File- => Added handshake-all.cpp, #25032
    2012-02-08 None Attached File- => Added handshake2.cpp, #25029
    2012-02-08 None Attached File- => Added handshake2.cpp, #25028
    2012-02-08 nmav StatusDone => Ready For Test
    2012-02-07 nmav StatusNone => Done
        Assigned toNone => nmav
    2012-02-06 None Attached File- => Added handshake.cpp, #24993

    Back to the top

    Powered by Savane 3.7