On 2024/05/02 13:59, Job Snijders wrote:
> Dear all,
> 
> If a HTTP caching layer is in use (either a proxy, or if you're fetching
> via a CDN), ideally - the TTL for SHA256.sig is low (we all wants the
> latest snapshots) and the TTL on the *.tgz files higher (if nothing
> changed, serve up cached objects).

TTL means different things re http caches.

Old style caches used to just serve the cached file for up to a certain
time unless the client requests a refresh. That time was often adaptive
based on how recently the object was modified when it was first fetched,
but subject to an overall cache-admin-configured TTL. They didn't check
a cached object with the origin.

More recently (but still many years ago), it became common to allow
serving objects of whatever age from the cache, but if the TTL was
expired then check back with the origin (i.e. with If-Modified-Since
in the GET headers). Slightly higher latency than something which is
fully served from cache, but still faster than refetching from the
origin at whatever speed is available (in the case of cdn.o.o that's
often relatively large files via a trans-(atlantic|pacific) download
so it really wants to be served from cache if possible).

https://www.fastly.com/documentation/guides/concepts/edge-state/cache/stale/

> The below changeset could be a step towards making the best of any
> potential caching layers and also getting fresh sets as soon as they
> appear at the origin, while providing increased chances of consistency
> between *.tgz files and the SHA256.sig. In other words, you don't want
> to end up with a new SHA256.sig and old *.tgz files.
> 
> The trick is to append the SHA256 hash of the SHA256.sig file as query
> string when fetching the set files. This 'needle' should prompt the
> caching layer to do an origin fetch. New SHA256.sig file == new URLs for
> the set files == new cache fill.

I don't think this helps.

SHA256.sig on the origin cannot be relied upon to be in sync with the
tgz files. Part of this was due to non-atomic syncs which AIUI have
now been improved, but another part AFAIK is an artefact of the way in
which builds are done and that's harder to change.

I fetch base snaps and run signify to check the hashes. Despite only
fetching them once a day (so I guess the chances of running into
any individual breakage are probably fairly low) I've had 2 failures
in the last month. (See examples below).

With this "append hash" scheme, if the contents of the tgz files
don't match SHA256.sig, then this doesnt help - wrong tgz files can
still be served and if the "without revalidation" TTL for tgz is
high then this can persist for some time. (Actually it could make
things worse as you won't be waiting for the original part-expired
TTL to reset, you'll be waiting for a new full TTL for the
URL with appended hash).

Instead, setting the TTL for all files in the snapshots directory
to the same relatively low value should have pretty much the same
positive effect as this change, but will also reduce the problem
of overcaching where SHA256.sig doesn't match the tgz files.
(Doing that will result in more I-M-S GETs to the origin than at
present, but they'll mostly be done over a persistent connection so no
3WH and latency won't be much of a problem, and it shouldn't result in
more full file fetches).


27 April:

Signature Verified
BOOTIA32.EFI: OK
BOOTX64.EFI: OK
BUILDINFO: OK
INSTALL.amd64: OK
base75.tgz: OK
cdboot: OK
cdbr: OK
pxeboot: OK
miniroot75.img: FAIL
floppy75.img: FAIL
bsd: FAIL
game75.tgz: FAIL
bsd.rd: FAIL
install75.iso: FAIL
xbase75.tgz: FAIL
bsd.mp: FAIL
comp75.tgz: FAIL
man75.tgz: FAIL
install75.img: FAIL
cd75.iso: FAIL
xshare75.tgz: FAIL
xfont75.tgz: FAIL
xserv75.tgz: FAIL

Today:

Signature Verified
BOOTIA32.EFI: OK
BOOTX64.EFI: OK
BUILDINFO: OK
INSTALL.amd64: OK
base75.tgz: OK
bsd: OK
bsd.mp: OK
bsd.rd: OK
cd75.iso: OK
cdboot: OK
cdbr: OK
comp75.tgz: OK
floppy75.img: OK
game75.tgz: OK
install75.img: OK
install75.img: OK
pxeboot: OK
miniroot75.img: FAIL
xbase75.tgz: FAIL
man75.tgz: FAIL
xshare75.tgz: FAIL
install75.iso: FAIL
xfont75.tgz: FAIL
xserv75.tgz: FAIL