From: Otto Moerbeek <otto@drijf.net>
Subject: libc malloc canaries improvement
To: tech@openbsd.org
Date: Sun, 25 Oct 2026 09:34:44 +0100

Hi,

I received the report below from acts1631kjv@proton.me. It improves
malloc canaries so it detects more cases. IMO, this is not an "attack"
being mitigated, as canaries are a mechanism that uses a best effort
approach anyway. But it *is* an improvement of canaries.

This reads fine, passes the existing malloc regression test plus
builds with malloc option S.

OK?
	-Otto

Subject: malloc canaries accept the public junk byte
Date: 30 September 2026

An attacker with a chosen-byte heap overflow can fill malloc's canary
area with 0xdb and make free() accept the overwritten canary. This lets
corruption continue into adjacent heap data without the abort promised
by malloc(3). The bypass applies to programs using malloc option C,
including OpenSSH code that compiles in option S. A heap overflow is
still a prerequisite but the issue defeats its intended detection.

malloc.c:97 defines the public allocation junk byte:

    #define SOME_JUNK       0xdb    /* deadbeef */

fill_canary() writes the low byte of the process-wide random value into
up to 32 bytes of allocation slack. However, validate_canary() at
malloc.c:1227 accepts either that byte or SOME_JUNK:

    if (*p != (u_char)mopts.chunk_canaries &&
        *p != SOME_JUNK) {
            wrterror(d, "canary corrupted ...");
    }

The exception makes 0xdb a deterministic universal canary value.
An overflow can write 0xdb over the slack and then continue with
arbitrary data in the next heap object. free() sees an intact canary.

Option S expands to CFGJ at malloc.c:541. OpenSSH enables S in
usr.bin/ssh/xmalloc.c:25:

    const char * const malloc_options = "S";

The exception also masks an initialization inconsistency. In omalloc()
and omemalign(), page-backed allocations with junk level 2 fill the
slack with SOME_JUNK and skip fill_canary() through an else-if. Small
allocations receive the random canary after junk validation. The check
nevertheless accepts 0xdb for every allocation size.

This reproducer requests 100 bytes, which uses a 112-byte chunk, and
overwrites all 12 canary bytes:

    #include <sys/resource.h>

    #include <err.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>

    #define REQUEST_SIZE 100
    #define CHUNK_SIZE 112
    #define PUBLIC_JUNK_BYTE 0xdb

    int
    main(int argc, char **argv)
    {
            volatile unsigned char *p;
            unsigned char byte;
            size_t i;

            if (argc != 2)
                    errx(1, "usage: %s junk|bad", argv[0]);
            byte = strcmp(argv[1], "junk") == 0 ?
                PUBLIC_JUNK_BYTE : 0;

            if ((p = malloc(REQUEST_SIZE)) == NULL)
                    err(1, "malloc");
            for (i = REQUEST_SIZE; i < CHUNK_SIZE; i++)
                    p[i] = byte;
            free((void *)p);
            printf("overflow byte 0x%02x accepted\n", byte);
            return 0;
    }

I compiled it as follows:

    cc -O2 -Wall -Wextra -Werror \
        -fno-builtin-malloc -fno-builtin-free \
        -o /tmp/canary_junk_bypass canary_junk_bypass.c

The public junk byte passes under both C and S:

    $ MALLOC_OPTIONS=C /tmp/canary_junk_bypass junk
    overflow byte 0xdb accepted
    $ echo $?
    0
    $ MALLOC_OPTIONS=S /tmp/canary_junk_bypass junk
    overflow byte 0xdb accepted
    $ echo $?
    0

Zero is a control case and aborts as expected:

    $ MALLOC_OPTIONS=C /tmp/canary_junk_bypass bad
    canary_junk_bypass(...) in free(): canary corrupted \
        ...[100]@100/112
    Abort trap
    $ echo $?
    134

The appended diff makes the random canary authoritative. It excludes
SOME_JUNK when choosing that byte, removes the validator exception, and
writes the random canary after junking in omalloc() and omemalign().

The performance change is limited to C with junk level 2. Page-backed
and highly aligned allocations receive one additional write of at most
32 bytes after they have already been junk-filled. Small allocations and
other option combinations keep their existing write behavior. Canary
validation also loses one comparison per checked byte.

diff --git a/lib/libc/stdlib/malloc.c b/lib/libc/stdlib/malloc.c
index ca5aa9f7097..083eaca70e4 100644
--- a/lib/libc/stdlib/malloc.c
+++ b/lib/libc/stdlib/malloc.c
@@ -569,6 +569,7 @@ omalloc_init(void)
 		do {
 			mopts.chunk_canaries = arc4random();
 		} while ((u_char)mopts.chunk_canaries == 0 ||
+		    (u_char)mopts.chunk_canaries == SOME_JUNK ||
 		    (u_char)mopts.chunk_canaries == SOME_FREEJUNK);
 }
 
@@ -1235,7 +1236,7 @@ validate_canary(struct dir_info *d, u_char *ptr, size_t sz, size_t allocated)
 	q = p + check_sz;
 
 	while (p < q) {
-		if (*p != (u_char)mopts.chunk_canaries && *p != SOME_JUNK) {
+		if (*p != (u_char)mopts.chunk_canaries) {
 			wrterror(d, "canary corrupted %p[%tu]@%zu/%zu%s",
 			    ptr, p - ptr, sz, allocated,
 			    *p == SOME_FREEJUNK ? " (double free?)" : "");
@@ -1363,7 +1364,8 @@ omalloc(struct dir_info *pool, size_t sz, int zero_fill)
 				else
 					memset(p, SOME_JUNK,
 					    psz - mopts.malloc_guard);
-			} else if (mopts.chunk_canaries)
+			}
+			if (mopts.chunk_canaries)
 				fill_canary(p, sz - mopts.malloc_guard,
 				    psz - mopts.malloc_guard);
 		}
@@ -2276,7 +2278,8 @@ omemalign(struct dir_info *pool, size_t alignment, size_t sz, int zero_fill)
 			    SOME_JUNK, psz - sz);
 		else
 			memset(p, SOME_JUNK, psz - mopts.malloc_guard);
-	} else if (mopts.chunk_canaries)
+	}
+	if (mopts.chunk_canaries)
 		fill_canary(p, sz - mopts.malloc_guard,
 		    psz - mopts.malloc_guard);