From: Theo Buehler <tb@theobuehler.org>
Subject: Re: relayd: check imsg_composev() / imsgbuf_flush() fail
To: Rafael Sadowski <rafael@sizeofvoid.org>, tech@openbsd.org
Date: Mon, 2 Mar 2026 23:38:40 +0100

On Tue, Feb 24, 2026 at 11:22:32AM +0100, Claudio Jeker wrote:
> At some point there was a plan to real hooks in libtls to do this key
> separation possible without these insane engine hacks. I have lost track
> on where that project was left.

Code was added to libtls but (IIRC) it broke RSA key exchange since it
tries to decrypt without having a private key available. A plan to fix
is said to exist. Time to implement it hasn't been found in four years.

ENGINE proper is gone and has been replaced with an RSA method a long
time ago. Trade the engine include for a missing x509.h for X509_free
and X509_pubkey_digest.

I don't know if the minimally tweaked comment is still correct but
the engine layer no longer exists, so cannot be responsible for
requiring a synchronous imsg.

Index: ca.c
===================================================================
RCS file: /cvs/src/usr.sbin/relayd/ca.c,v
diff -u -p -r1.48 ca.c
--- ca.c	2 Mar 2026 19:28:01 -0000	1.48
+++ ca.c	2 Mar 2026 22:16:28 -0000
@@ -31,7 +31,7 @@
 #include <openssl/evp.h>
 #include <openssl/pem.h>
 #include <openssl/rsa.h>
-#include <openssl/engine.h>
+#include <openssl/x509.h>
 
 #include "relayd.h"
 #include "log.h"
@@ -334,7 +334,7 @@ rsae_send_imsg(int flen, const u_char *f
 
 	/*
 	 * Send a synchronous imsg because we cannot defer the RSA
-	 * operation in OpenSSL's engine layer.
+	 * operation in OpenSSL.
 	 */
 	if (imsg_composev(ibuf, cmd, 0, 0, -1, iov, cnt) == -1) {
 		log_warn("%s: imsg_composev", __func__);