From: "Pavel (Narayana OU)" <pd@narayana.im>
Subject: libssl: DTLS drops the MTU after retransmissions even with SSL_OP_NO_QUERY_MTU
To: tech@openbsd.org
Date: Mon, 5 Oct 2026 13:36:44 +0300

Hi everyone!


I'm not a libressl dev, so please pardon me for possible mistakes, but 
it seems I've found a bug while testing how our library (which is XMPP 
client library which uses libssl for DTLS negotiation i.e.) works 
against LibreSSL instead of OpenSSL.

Beware! Content below this line is LLM generated, I have very little 
idea is it a real fix or dirty workaround. But it is so small so I hope 
that you'll review it and in case that it is not a slop -- adopt it in 
upstream.

The patch is almost one-liner.

|- if (s->d1->timeout.num_alerts > 2) { |

|+ if (s->d1->timeout.num_alerts > 2 && |

|+ !(SSL_get_options(s) & SSL_OP_NO_QUERY_MTU)) {|

This is an explanation from Fable 5.1:

>   What happens: libdatachannel sets SSL_OP_NO_QUERY_MTU and its own 
> MTU with
>   SSL_set_mtu(), and its BIO is a custom one that does not know the DGRAM
>   controls (it returns 0 for them). When a DTLS handshake gets no 
> answer at
>   all, which is normal for us (a call to a browser that already closed),
>   libssl retransmits. After the second retransmission 
> dtls1_check_timeout_num()
>   in lib/libssl/d1_lib.c replaces the MTU with the BIO's answer to
>   BIO_CTRL_DGRAM_GET_FALLBACK_MTU, which is 0 here. The third 
> retransmission
>   then hits the OPENSSL_assert(s->d1->mtu >= dtls1_min_mtu()) in
>   dtls1_do_write() and the whole process exits. I would have expected the
>   handshake to fail with SSL_R_READ_TIMEOUT_EXPIRED after 
> DTLS1_TMO_ALERT_COUNT
>   tries, which is what happens once the MTU is left alone.
>
>   The other place that asks the BIO about the MTU, dtls1_do_write(), 
> skips it
>   when SSL_OP_NO_QUERY_MTU is set; dtls1_check_timeout_num() does not 
> check
>   the option. OpenSSL checks it in the same function (since 1.0.2, if 
> I read
>   their history right), so the diff below just does the same.
>
>   I have been building LibreSSL 4.3.2 with this change for a while 
> now: the
>   crash is gone, and the DTLS handshakes and our test suite behave as 
> before.
>   The diff is against -current's d1_lib.c (r1.70); it applies to 4.3.2 as
>   well with an offset.
>
>   Index: lib/libssl/d1_lib.c
>   --- lib/libssl/d1_lib.c
>   +++ lib/libssl/d1_lib.c
>   @@ -305,7 +305,8 @@ dtls1_check_timeout_num(SSL *s)
>         s->d1->timeout.num_alerts++;
>
>         /* Reduce MTU after 2 unsuccessful retransmissions */
>   -     if (s->d1->timeout.num_alerts > 2) {
>   +     if (s->d1->timeout.num_alerts > 2 &&
>   +         !(SSL_get_options(s) & SSL_OP_NO_QUERY_MTU)) {
>                 s->d1->mtu = BIO_ctrl(SSL_get_wbio(s),
>                     BIO_CTRL_DGRAM_GET_FALLBACK_MTU, 0, NULL);
>
>
>   If it helps, the way to see it without WebRTC is a DTLS client over a
>   BIO_s_mem() pair with SSL_OP_NO_QUERY_MTU and SSL_set_mtu(ssl, 
> 1200), whose
>   peer's flight is never delivered, calling DTLSv1_handle_timeout() at 
> each
>   DTLSv1_get_timeout(): the third timeout aborts without the diff and 
> keeps
>   retransmitting with it.
>

Thank you for your time.