OmniOS now ships and enables oqsprovider (quantum-safe crypto
provider for openssl) by default.  One thing this does is add lots
of sigalgs to the TLS client hello.  Wireshark says there are 71 of
them in there.  Trying to connect to a server that uses LibreSSL
fails like this:

$ curl -vi https://openbsd.org/
 
[...]

* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cacert.pem
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS alert, decode error (562):
* OpenSSL/3.3.2: error:0A00041A:SSL routines::tlsv1 alert decode error

because LibreSSL will reject anything with a sigalg extension longer
than 64 bytes.  Increasing this to 256 bytes, as in the diff below,
allows OmniOS clients to connect.  Is this a reasonable thing to do?


Index: ssl_tlsext.c
===================================================================
RCS file: /cvs/src/lib/libssl/ssl_tlsext.c,v
diff -u -p -u -p -r1.154 ssl_tlsext.c
--- ssl_tlsext.c	9 Jul 2024 12:27:27 -0000	1.154
+++ ssl_tlsext.c	7 Jan 2025 00:14:00 -0000
@@ -600,7 +600,7 @@ tlsext_sigalgs_server_process(SSL *s, ui
 
 	if (!CBS_get_u16_length_prefixed(cbs, &sigalgs))
 		return 0;
-	if (CBS_len(&sigalgs) % 2 != 0 || CBS_len(&sigalgs) > 64)
+	if (CBS_len(&sigalgs) % 2 != 0 || CBS_len(&sigalgs) > 256)
 		return 0;
 	if (!CBS_stow(&sigalgs, &s->s3->hs.sigalgs, &s->s3->hs.sigalgs_len))
 		return 0;