From: Job Snijders <job@openbsd.org>
Subject: rpki-client: indicate instances of suboptimal HTTP content encoding
To: tech@openbsd.org
Date: Fri, 6 Jun 2025 21:11:42 +0000

In verbose mode, draw some attention to inefficient transfers by logging
a message when objects larger than a megabyte were transferred without
using gzip or deflate compression.

Example:

	# rm -rf /var/cache/rpki-client/.rrdp
	# rpki-client -v 2>&1 | grep --line-buffered -e downloaded -e Proces -e New | ts
	Jun 06 20:45:45 rpki-client: https://rrdp.ripe.net/172322cf-c642-4e6f-806c-bd2375d8001a/130462/snapshot-b... (2.18.121.80): downloaded 243840080 bytes without content encoding compression
	Jun 06 20:46:09 rpki-client: https://rrdp.lacnic.net/rrdp/66b09eea-03c7-435f-bb29-5bee810bd657/209590/d51... (172.233.4.134): downloaded 61735570 bytes without content encoding compression
	Jun 06 20:47:46 rpki-client: https://rrdp.paas.rpki.ripe.net/1c33ba5d-4e16-448d-9a22-b12599ef1cba/229024/... (104.123.50.160): downloaded 18427607 bytes without content encoding compression
	Jun 06 20:48:38 rpki-client: https://rpki-rrdp.us-east-2.amazonaws.com/rrdp/20aa329b-fc52-4c61-bf53-09725... (108.156.60.53): downloaded 12421100 bytes without content encoding compression
	...

OK?

Index: http.c
===================================================================
RCS file: /cvs/src/usr.sbin/rpki-client/http.c,v
diff -u -p -r1.93 http.c
--- http.c	11 Mar 2025 14:53:03 -0000	1.93
+++ http.c	6 Jun 2025 20:44:49 -0000
@@ -797,7 +797,6 @@ http_inflate_advance(struct http_connect
 
 	if (zctx->zdone) {
 		/* all compressed data processed */
-		conn->gzipped = 0;
 		http_inflate_done(conn);
 
 		if (conn->iosz == 0) {
@@ -911,7 +910,9 @@ http_done(struct http_connection *conn, 
 	if (conn->gzipped) {
 		conn->gzipped = 0;
 		http_inflate_done(conn);
-	}
+	} else if (conn->totalsz > (1024 * 1024))
+		logx("%s: downloaded %zu bytes without content encoding "
+		    "compression", conn_info(conn), conn->totalsz);
 
 	conn->state = STATE_IDLE;
 	conn->idle_time = getmonotime() + HTTP_IDLE_TIMEOUT;