On Tue, Jun 10, 2025 at 08:50:46PM +0000, Job Snijders wrote:
> On Sat, Jun 07, 2025 at 06:52:24AM +0200, Theo Buehler wrote:
> > >  	if (zctx->zdone) {
> > >  		/* all compressed data processed */
> > > -		conn->gzipped = 0;
> > 
> > I can't convince myself that this is right in the case of chunked
> > transfer encoding. This doesn't go straight to http_done but reads
> > the next chunk and now conn->gzipped and the decompression state are
> > out of sync.
> 
> I agree.
> 
> How about a more readable approach? Example output:
> 
> 	$ doas rpki-client -v 2>&1 | grep HTTP
> 	rpki-client: https://rrdp.lacnic.net/rrdp/66b09eea-03c7-435f-bb29-5bee810bd657/211566/d51... (172.233.4.134): downloaded 61810242 bytes without HTTP compression
> 	rpki-client: https://rrdp.twnic.tw/rrdp/71d39673-7062-4036-b080-957aa1b854b3/snapshot/176... (103.235.88.190): downloaded 12410151 bytes without HTTP compression
> 	rpki-client: https://rpki.cnnic.cn/rrdp/d4427960-abb3-49e4-902c-d815a518b867/snapshot/301... (218.241.105.61): downloaded 8930687 bytes without HTTP compression
> 	rpki-client: https://rpki-repo.registro.br/rrdp/b0982bb6-6f99-448a-9f71-91a86a455e39/9292... (200.160.2.50): downloaded 68567400 bytes without HTTP compression
> 
> Some of the RRDP endpoints in the wild combine chunked transfer encoding
> with gzipped content encoding.

I really would prefer if we did not add more log output like this.
Both with and without -v the noise from rpki-client is too high.

If you really want something I would suggest we track the number of bytes
recevied (before and after decompression) to the repo stats.
 
> Kind regards,
> 
> Job
> 
> Index: http.c
> ===================================================================
> RCS file: /cvs/src/usr.sbin/rpki-client/http.c,v
> diff -u -p -r1.93 http.c
> --- http.c	11 Mar 2025 14:53:03 -0000	1.93
> +++ http.c	10 Jun 2025 20:45:22 -0000
> @@ -137,6 +137,7 @@ struct http_connection {
>  	int			fd;
>  	int			chunked;
>  	int			gzipped;
> +	int			was_gzipped;
>  	int			keep_alive;
>  	short			events;
>  	enum http_state		state;
> @@ -799,6 +800,7 @@ http_inflate_advance(struct http_connect
>  		/* all compressed data processed */
>  		conn->gzipped = 0;
>  		http_inflate_done(conn);
> +		conn->was_gzipped = 1;
>  
>  		if (conn->iosz == 0) {
>  			if (!conn->chunked) {
> @@ -911,7 +913,14 @@ http_done(struct http_connection *conn, 
>  	if (conn->gzipped) {
>  		conn->gzipped = 0;
>  		http_inflate_done(conn);
> +		conn->was_gzipped = 1;
>  	}
> +
> +	if (conn->was_gzipped)
> +		conn->was_gzipped = 0;
> +	else if (conn->totalsz > (1024 * 1024))
> +		logx("%s: downloaded %zu bytes without HTTP "
> +		    "compression", conn_info(conn), conn->totalsz);
>  
>  	conn->state = STATE_IDLE;
>  	conn->idle_time = getmonotime() + HTTP_IDLE_TIMEOUT;
> 

-- 
:wq Claudio