From: Showta Ishizaki <zakinko@snowrabbit.org>
Subject: nan(3): a long payload reaches the sign bit
To: tech@openbsd.org
Cc: zakinko@snowrabbit.org
Date: Fri, 11 Sep 2026 06:51:00 +0900

_scan_nan() in lib/libm/src/s_nan.c fills every bit of the words it is
given, so a payload longer than the mantissa reaches the exponent and
the sign.  nan() and nanf() OR in the exponent afterwards, which hides
that part, but the sign survives, and both nanl() variants assign the
exponent and leave the sign alone in the same way:

    nan("9999999999999999")        fff9999999999999
    strtod("nan(9999999999999999)") 7ff9999999999999
    nanf("ffffffff")               ffffffff
    strtof("nan(ffffffff)")        7fffffff
    nanl("ffffffffffffffffffff")   ffff ffffffffffffffff (sign set)

C11 7.12.11.2 defines nan(s) as strtod("NAN(s)"), so the two columns
should agree; the comment above _scan_nan() also says the high order
bits are discarded.  abseil's charconv test compares from_chars, which
follows nan(3), against strtod and caught it.

The diff keeps only the mantissa in nan() and nanf() and clears the
sign in nanl().  I rebuilt lib/libm from the 7.9 source with it and
ran the probe against the new libm.so.10.1 through LD_LIBRARY_PATH:
nan("9999999999999999") is 7ff9999999999999, nanf("ffffffff") is
7fffffff, nanl("ffffffffffffffffffff") has 7fff in the sign and
exponent, and short payloads such as nan("1") are unchanged.  Not built
on -current, though the diff applies to it unchanged.

Index: lib/libm/src/s_nan.c
--- lib/libm/src/s_nan.c.orig
+++ lib/libm/src/s_nan.c
@@ -101,10 +101,11 @@
 	} u;
 
 	_scan_nan(u.bits, 2, s);
+	/* Keep the mantissa only; a long payload must not reach the sign. */
 #if BYTE_ORDER == LITTLE_ENDIAN
-	u.bits[1] |= 0x7ff80000;
+	u.bits[1] = (u.bits[1] & 0x000fffff) | 0x7ff80000;
 #else
-	u.bits[0] |= 0x7ff80000;
+	u.bits[0] = (u.bits[0] & 0x000fffff) | 0x7ff80000;
 #endif
 	return (u.d);
 }
@@ -120,6 +121,6 @@
 	} u;
 
 	_scan_nan(u.bits, 1, s);
-	u.bits[0] |= 0x7fc00000;
+	u.bits[0] = (u.bits[0] & 0x007fffff) | 0x7fc00000;
 	return (u.f);
 }
Index: lib/libm/src/ld80/s_nanl.c
--- lib/libm/src/ld80/s_nanl.c.orig
+++ lib/libm/src/ld80/s_nanl.c
@@ -43,6 +43,7 @@
 	} u;
 
 	_scan_nan(u.bits, 3, s);
+	u.ieee.ext_sign = 0;	/* a long payload must not reach the sign */
 	u.ieee.ext_exp = 0x7fff;
 	u.ieee.ext_frach |= 0xc0000000;	/* make it a quiet NaN */
 
Index: lib/libm/src/ld128/s_nanl.c
--- lib/libm/src/ld128/s_nanl.c.orig
+++ lib/libm/src/ld128/s_nanl.c
@@ -43,6 +43,7 @@
 	} u;
 
 	_scan_nan(u.bits, 4, s);
+	u.ieee.ext_sign = 0;	/* a long payload must not reach the sign */
 	u.ieee.ext_exp = 0x7fff;
 	u.ieee.ext_frach |= 1U << 15;	/* make it a quiet NaN */