From: "emulti@disroot.org" <emulti@disroot.org>
Subject: 7.6 /etc/rc blocks NFS-mounting /usr for diskless clients on boot
To: tech@openbsd.org
Date: Tue, 1 Apr 2025 14:31:50 +0800

I am doing a project with net-booted diskless OpenBSD/amd64 clients and an NFS shared read-only /usr directory, as described in the diskless(8) manpage.

However, I found that diskless clients are unable to mount /usr during boot, due to pf rules implemented in the standard /etc/rc.

/etc/rc contains a section (starting l466) with pf rules followed by initial mounts with comment "don't kill NFS":

        RULES="$RULES
        pass in proto carp keep state (no-sync)
        pass out proto carp !received-on any keep state (no-sync)"

        if (($(sysctl -n vfs.mounts.nfs 2>/dev/null)+0 > 0)); then
                # Don't kill NFS.
                RULES="set reassemble yes no-df
                $RULES
                pass in proto { tcp, udp } from any port { sunrpc, nfsd } to any
                pass out proto { tcp, udp } from any to any port { sunrpc, nfsd } !received-on any"
        fi
...
...
mount -s /var >/dev/null 2>&1           # cannot be on NFS
mount -s /var/log >/dev/null 2>&1       # cannot be on NFS
mount -s /usr >/dev/null 2>&1           # if NFS, fstab must use IP address

However, the /usr/ mount doesn't make it through pf, I think because portmap is exposing dynamic reserved ports for mountd that are not in the ruleset.
rpcinfo: 
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100004    2   udp    838  ypserv
    100004    2   tcp    669  ypserv
    100007    2   udp    926  ypbind
    100007    2   tcp   1007  ypbind
    100005    1   udp    648  mountd
    100005    3   udp    648  mountd
    100005    1   tcp    965  mountd
    100005    3   tcp    965  mountd
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100026    1   udp    710  bootparam

I couldn't work out a way to get the portmap ports simply, so made the following small change to bring the /usr mount before the pf rules are activated, which allows booting to continue:

--- /etc/rc     Mon Sep 30 22:33:10 2024
+++ ./rc        Tue Apr  1 14:01:16 2025
@@ -463,6 +463,8 @@
                pass in inet6 proto udp from any port dhcpv6-server to any port dhcpv6-client"
        fi
 
+mount -s /usr >/dev/null 2>&1          # if NFS, fstab must use IP address
+
        RULES="$RULES
        pass in proto carp keep state (no-sync)
        pass out proto carp !received-on any keep state (no-sync)"
@@ -486,7 +488,6 @@
 
 mount -s /var >/dev/null 2>&1          # cannot be on NFS
 mount -s /var/log >/dev/null 2>&1      # cannot be on NFS
-mount -s /usr >/dev/null 2>&1          # if NFS, fstab must use IP address
 
 reorder_libs 2>&1 |&

It's still not quite right- I occasionally get boot failures on clients until mountd is reloaded. I suspect this is because /var and /var/log are also on an NFS exported rootfs, in contravention of the comments above.
Is there a better or more elegant way of doing this, or avoiding the issue, or is it worth the probably minor and transient risk of mounting /usr without pf rules running, to restore the functionality of NFS-mounting /usr on boot?

-- 
Chris Billington