From: Theo Buehler <tb@theobuehler.org>
Subject: Re: rpki-client: indicate instances of suboptimal HTTP content encoding
To: Job Snijders <job@openbsd.org>
Cc: tech@openbsd.org
Date: Sat, 7 Jun 2025 06:52:24 +0200

On Fri, Jun 06, 2025 at 09:11:42PM +0000, Job Snijders wrote:
> In verbose mode, draw some attention to inefficient transfers by logging
> a message when objects larger than a megabyte were transferred without
> using gzip or deflate compression.
> 
> Example:
> 
> 	# rm -rf /var/cache/rpki-client/.rrdp
> 	# rpki-client -v 2>&1 | grep --line-buffered -e downloaded -e Proces -e New | ts
> 	Jun 06 20:45:45 rpki-client: https://rrdp.ripe.net/172322cf-c642-4e6f-806c-bd2375d8001a/130462/snapshot-b... (2.18.121.80): downloaded 243840080 bytes without content encoding compression
> 	Jun 06 20:46:09 rpki-client: https://rrdp.lacnic.net/rrdp/66b09eea-03c7-435f-bb29-5bee810bd657/209590/d51... (172.233.4.134): downloaded 61735570 bytes without content encoding compression
> 	Jun 06 20:47:46 rpki-client: https://rrdp.paas.rpki.ripe.net/1c33ba5d-4e16-448d-9a22-b12599ef1cba/229024/... (104.123.50.160): downloaded 18427607 bytes without content encoding compression
> 	Jun 06 20:48:38 rpki-client: https://rpki-rrdp.us-east-2.amazonaws.com/rrdp/20aa329b-fc52-4c61-bf53-09725... (108.156.60.53): downloaded 12421100 bytes without content encoding compression
> 	...
> 
> OK?
> 
> Index: http.c
> ===================================================================
> RCS file: /cvs/src/usr.sbin/rpki-client/http.c,v
> diff -u -p -r1.93 http.c
> --- http.c	11 Mar 2025 14:53:03 -0000	1.93
> +++ http.c	6 Jun 2025 20:44:49 -0000
> @@ -797,7 +797,6 @@ http_inflate_advance(struct http_connect
>  
>  	if (zctx->zdone) {
>  		/* all compressed data processed */
> -		conn->gzipped = 0;

I can't convince myself that this is right in the case of chunked
transfer encoding. This doesn't go straight to http_done but reads
the next chunk and now conn->gzipped and the decompression state are
out of sync.

>  		http_inflate_done(conn);
>  
>  		if (conn->iosz == 0) {
> @@ -911,7 +910,9 @@ http_done(struct http_connection *conn, 
>  	if (conn->gzipped) {
>  		conn->gzipped = 0;
>  		http_inflate_done(conn);
> -	}
> +	} else if (conn->totalsz > (1024 * 1024))
> +		logx("%s: downloaded %zu bytes without content encoding "
> +		    "compression", conn_info(conn), conn->totalsz);
>  
>  	conn->state = STATE_IDLE;
>  	conn->idle_time = getmonotime() + HTTP_IDLE_TIMEOUT;
>