From: Claudio Jeker Subject: Re: rpki-client: indicate instances of suboptimal HTTP content encoding To: Job Snijders Cc: Theo Buehler , tech@openbsd.org Date: Wed, 11 Jun 2025 06:11:31 +0200 On Tue, Jun 10, 2025 at 08:50:46PM +0000, Job Snijders wrote: > On Sat, Jun 07, 2025 at 06:52:24AM +0200, Theo Buehler wrote: > > > if (zctx->zdone) { > > > /* all compressed data processed */ > > > - conn->gzipped = 0; > > > > I can't convince myself that this is right in the case of chunked > > transfer encoding. This doesn't go straight to http_done but reads > > the next chunk and now conn->gzipped and the decompression state are > > out of sync. > > I agree. > > How about a more readable approach? Example output: > > $ doas rpki-client -v 2>&1 | grep HTTP > rpki-client: https://rrdp.lacnic.net/rrdp/66b09eea-03c7-435f-bb29-5bee810bd657/211566/d51... (172.233.4.134): downloaded 61810242 bytes without HTTP compression > rpki-client: https://rrdp.twnic.tw/rrdp/71d39673-7062-4036-b080-957aa1b854b3/snapshot/176... (103.235.88.190): downloaded 12410151 bytes without HTTP compression > rpki-client: https://rpki.cnnic.cn/rrdp/d4427960-abb3-49e4-902c-d815a518b867/snapshot/301... (218.241.105.61): downloaded 8930687 bytes without HTTP compression > rpki-client: https://rpki-repo.registro.br/rrdp/b0982bb6-6f99-448a-9f71-91a86a455e39/9292... (200.160.2.50): downloaded 68567400 bytes without HTTP compression > > Some of the RRDP endpoints in the wild combine chunked transfer encoding > with gzipped content encoding. I really would prefer if we did not add more log output like this. Both with and without -v the noise from rpki-client is too high. If you really want something I would suggest we track the number of bytes recevied (before and after decompression) to the repo stats. > Kind regards, > > Job > > Index: http.c > =================================================================== > RCS file: /cvs/src/usr.sbin/rpki-client/http.c,v > diff -u -p -r1.93 http.c > --- http.c 11 Mar 2025 14:53:03 -0000 1.93 > +++ http.c 10 Jun 2025 20:45:22 -0000 > @@ -137,6 +137,7 @@ struct http_connection { > int fd; > int chunked; > int gzipped; > + int was_gzipped; > int keep_alive; > short events; > enum http_state state; > @@ -799,6 +800,7 @@ http_inflate_advance(struct http_connect > /* all compressed data processed */ > conn->gzipped = 0; > http_inflate_done(conn); > + conn->was_gzipped = 1; > > if (conn->iosz == 0) { > if (!conn->chunked) { > @@ -911,7 +913,14 @@ http_done(struct http_connection *conn, > if (conn->gzipped) { > conn->gzipped = 0; > http_inflate_done(conn); > + conn->was_gzipped = 1; > } > + > + if (conn->was_gzipped) > + conn->was_gzipped = 0; > + else if (conn->totalsz > (1024 * 1024)) > + logx("%s: downloaded %zu bytes without HTTP " > + "compression", conn_info(conn), conn->totalsz); > > conn->state = STATE_IDLE; > conn->idle_time = getmonotime() + HTTP_IDLE_TIMEOUT; > -- :wq Claudio