Memory Management: Decisions, Risks and Open Questions

Part of the memory management rewrite plan: architecture · observability · maintainability · phases · decisions and risks.

1. Where does the portable kernel fit? kernel/core/ has a memory model that is host-tested and initialised after userland has already run, so it has no runtime role. L0 and L2 are exactly the sort of thing it should own. This plan puts them in kernel/mm/ for now; whether that is kernel/core/ is a decision for P1.
2. Does the TLB shootdown design survive contact with L1? The known problem is that a synchronous barrier cannot be met while syscall masks interrupts for its whole duration. L1 concentrates the question in one place; it does not answer it. The honest fix is likely to stop masking interrupts for the whole syscall, which is a scheduler change, not a memory one.

These are the points where I would otherwise choose on your behalf. I will stop and ask at each one rather than pick.

Anything not on this list and not specified above, I will implement as written. If the plan turns out to be wrong about something, I stop and say so rather than improvise a different design.

Decisions taken

D1 — kernel/mm/, decided 2026-08-30

A dedicated module, compiled into the kernel image and into the host test binary. The name says what it contains and it inherits nothing.

The alternative was kernel/core/, which would have given the portable kernel the runtime job it has never had - and that argument got stronger this week, because vibeos_kmain now runs before userland rather than after it. It was rejected because kernel/core/ already holds the boot state, the log, the timer and the interrupt controller; adding memory to it turns a module into a container.

This does not close the "two kernels" question recorded in the plan. It is noted there as still open, deliberately, rather than being quietly answered by where a file was put.

D2 — 16 bytes per frame, decided 2026-08-30

The complete descriptor: owners, state, flags, backing, lru_next, lru_prev. 1.7 MiB against the 440 MiB this kernel sees, or 0.4%.

The smaller forms - 8 bytes without the reclaim lists, 4 bytes with only what the repair needs - were rejected for the same reason: each buys back well under a megabyte and costs a second migration of the structure and its tests, at P4 or P6, in a subsystem whose whole problem has been changing the same thing four times. The point of this rewrite is not to do that again.

D9 — the ownership contract of an allocation

Decided A, 2026-08-30. Landed in 29a6e29. What follows is the question as it was put, kept because the reasoning is the record of why the kernel counts the way it does.

It surfaced while writing the wrappers, and it is not a detail: today hw_alloc_page hands back a frame with a reference count of zero, and the count only becomes one when hw_map_page maps it. The new layer hands back a frame with one owner, because "allocated and owned by nobody" is exactly the state that made the old scheme unsafe - it is indistinguishable from free.

Both are coherent; they are not compatible, and the mismatch is silent in the worst way. Wire the wrappers without settling it and every user page ends up with two owners instead of one (allocation plus first mapping), so nothing is ever reclaimed - a leak. Settle it the other way carelessly and a kernel page table, which is allocated and freed without ever being mapped, is freed at a count that was never incremented - a double free.

Option A — an allocation gives the caller one owner (Linux's model)

alloc returns owners = 1. The caller holds that reference. Mapping the frame takes another; unmapping drops it. A caller that allocates a frame purely to map it drops its own reference once the mapping exists, exactly as alloc_page/put_page do.

• Correct for kernel page tables with no change: allocate, use, put, freed.
• Every path that allocates a frame for a mapping gains one put. There are about nine such sites.
• A missed put leaks one frame and shows up in meminfo. A leak is the failure direction this whole rewrite chose on purpose.

Option B — an allocation gives the caller nothing, mapping owns the frame

alloc returns owners = 0, as today. Only mappings count.

• No call site changes at all.
• Keeps the state that has caused every defect in this subsystem: a frame that is allocated and owned by nobody reads as free to anything that asks, which is precisely what the invariant "a frame is freed only at zero" was written to make impossible.
• Kernel page tables live permanently at zero owners, so the count cannot be used to check anything about them.

My recommendation is A, and it is not close. B is cheaper this week and re-creates the exact ambiguity that cost four fixes; A costs nine put calls and makes every wrong one a measurable leak rather than a corruption that surfaces in an unrelated program three boots later.

What I need from you is a yes to A, or a reason for B. Until then P1 stops at step 2, which is where it is: the layer is built, host-tested and compiled in, and nothing calls it yet.

D9 as implemented

An allocation returns a frame with one owner, and that owner is the caller.

• A caller that allocates in order to map - process images, stacks, mmap, brk, the copy-on-write copy - hands the frame to the address space and releases its own reference immediately after the mapping succeeds. Nine sites, each carrying a comment naming this decision.
• A caller that allocates memory for the kernel itself - page tables, the PML4, kernel stacks - never maps it, so its one reference is the only one and hw_free_page releases it.
• Releasing the last reference and reclaiming the frame are now one operation. There is no longer any moment in which a frame has no owners and is not yet free, which is the state every defect in this subsystem has lived in.

Three consequences worth writing down, because each was found by doing it rather than by planning it:

1. Contiguous allocation had to exist before any of this could be wired in. The bump allocator underneath served the exec staging buffers and the two-page kernel stacks, and it has to stop serving anything at all: while two allocators hand out frames from one region, one of them is wrong about what is free. vibeos_frame_alloc_contig covers those callers.

2. A second allocator was already live and invisible. The GUI back buffer - several megabytes - was taken from the bump allocator after the frame layer had adopted the region. The frame table still described those frames as free and handed them to a process; the desktop then rendered over its memory. No detector fired, because nothing was freed early. The bump allocator is closed the moment the layer comes up, so the next such call fails visibly.

3. meminfo was not one picture. It derived allocated as total minus free, which swallowed every reserved frame, and took reserved from somewhere else again. One walk of the frame table answers it now and the boot gate fails if the states do not sum to the total.

D3 — a bit in the entry, decided 2026-08-30 (and the bit was wrong)

A software bit in the page-table entry, not a side table. VIBEOS_PTE_OWNED.

The argument for a side table was that a future use of the bit would break it. The argument against, which won, is that a side table is a second structure that has to agree with the first - and this subsystem's entire history is two structures disagreeing about who owns a frame. The whole point of the bit is that the record and the thing it describes cannot drift apart, because they are the same word of memory.

The plan named bit 9 and bit 9 was not free. PTE_COW in the x86-64 code is 0x200, which is bit 9. Sharing them makes every mapped page read as copy-on-write, so a write fault on a genuinely read-only page would be resolved by granting the write instead of killing the process - a program could write to its own text segment and be told it worked.

Sixteen clean boots showed nothing, because the two bits only disagree on a path that needs a fault on a page a program is not allowed to write. It was caught by reading the definition while wiring up protect, which is luck, not method. What replaces the luck is a _Static_assert that the two bits are different, confirmed to fire by putting them back together.

Ownership is bit 11. The risk the side-table argument raised is real and is now answered by the compiler rather than by care.

D4 — seven commits, decided 2026-08-30

Landed as six, and the phase's own history is the argument for it. Each step was verified before the next began, and two of them failed in ways that would have been far harder to attribute inside one large commit:

• Step 2 wedged nothing but hid the bit collision above.
• Step 5 wedged the boot outright, and the cause was not the code the step was about - it was that moving allocation into the layer moved it out from under the memory lock the architecture had been taking at the call sites.

Both were found by having a green commit immediately before them to compare against. A single commit for the phase would have presented both at once, in a diff of about a thousand lines, with a silent wedge as the only symptom.

D5 — ten per cent, and measured, decided 2026-08-30

The boot may get up to 10% slower per phase, and the boot gate times it and fails past that.

The measurement is the decision, more than the number is. A budget nobody checks is not a budget, and a slowdown noticed three phases later cannot be attributed to the phase that caused it - which in this subsystem means it gets attributed to whichever phase somebody happens to be reading. The gate records the elapsed time to BOOT_OK and compares it against a recorded baseline.

The user's framing, which is the right one: optimising the system as a whole is its own milestone, a long way out. This budget exists to stop a phase making things quietly worse in the meantime, not to make anything fast.

D6 — two caches, for now, decided 2026-08-30

P4 builds a page cache for memory. The block cache in kernel/fs/ stays where it is and keeps doing what it does.

They hold different things keyed differently - one by sector, one by offset within a file - and merging them pulls the filesystem into a plan that does not otherwise touch it. The argument that settled it is narrower than tidiness: this subsystem's every serious defect has been two structures disagreeing about one resource, and creating that situation on purpose, mid-rewrite, to save a layer of indirection is the wrong week for it.

The question is worth reopening once P4 exists and the duplication can be measured rather than guessed at. It is recorded here so that reopening it is a decision rather than a rediscovery.