From: Job Snijders <job@openbsd.org>
Subject: Re: rpki-client: indicate instances of suboptimal HTTP content encoding
To: Theo Buehler <tb@theobuehler.org>
Cc: tech@openbsd.org
Date: Tue, 10 Jun 2025 20:50:46 +0000

On Sat, Jun 07, 2025 at 06:52:24AM +0200, Theo Buehler wrote:
> >  	if (zctx->zdone) {
> >  		/* all compressed data processed */
> > -		conn->gzipped = 0;
> 
> I can't convince myself that this is right in the case of chunked
> transfer encoding. This doesn't go straight to http_done but reads
> the next chunk and now conn->gzipped and the decompression state are
> out of sync.

I agree.

How about a more readable approach? Example output:

	$ doas rpki-client -v 2>&1 | grep HTTP
	rpki-client: https://rrdp.lacnic.net/rrdp/66b09eea-03c7-435f-bb29-5bee810bd657/211566/d51... (172.233.4.134): downloaded 61810242 bytes without HTTP compression
	rpki-client: https://rrdp.twnic.tw/rrdp/71d39673-7062-4036-b080-957aa1b854b3/snapshot/176... (103.235.88.190): downloaded 12410151 bytes without HTTP compression
	rpki-client: https://rpki.cnnic.cn/rrdp/d4427960-abb3-49e4-902c-d815a518b867/snapshot/301... (218.241.105.61): downloaded 8930687 bytes without HTTP compression
	rpki-client: https://rpki-repo.registro.br/rrdp/b0982bb6-6f99-448a-9f71-91a86a455e39/9292... (200.160.2.50): downloaded 68567400 bytes without HTTP compression

Some of the RRDP endpoints in the wild combine chunked transfer encoding
with gzipped content encoding.

Kind regards,

Job

Index: http.c
===================================================================
RCS file: /cvs/src/usr.sbin/rpki-client/http.c,v
diff -u -p -r1.93 http.c
--- http.c	11 Mar 2025 14:53:03 -0000	1.93
+++ http.c	10 Jun 2025 20:45:22 -0000
@@ -137,6 +137,7 @@ struct http_connection {
 	int			fd;
 	int			chunked;
 	int			gzipped;
+	int			was_gzipped;
 	int			keep_alive;
 	short			events;
 	enum http_state		state;
@@ -799,6 +800,7 @@ http_inflate_advance(struct http_connect
 		/* all compressed data processed */
 		conn->gzipped = 0;
 		http_inflate_done(conn);
+		conn->was_gzipped = 1;
 
 		if (conn->iosz == 0) {
 			if (!conn->chunked) {
@@ -911,7 +913,14 @@ http_done(struct http_connection *conn, 
 	if (conn->gzipped) {
 		conn->gzipped = 0;
 		http_inflate_done(conn);
+		conn->was_gzipped = 1;
 	}
+
+	if (conn->was_gzipped)
+		conn->was_gzipped = 0;
+	else if (conn->totalsz > (1024 * 1024))
+		logx("%s: downloaded %zu bytes without HTTP "
+		    "compression", conn_info(conn), conn->totalsz);
 
 	conn->state = STATE_IDLE;
 	conn->idle_time = getmonotime() + HTTP_IDLE_TIMEOUT;