Navian Open Source · module 02 · resource-leak / soak testing

navian-memcheck — prove your memory plateaus.

Assert that a workload's memory levels off after warmup — catching the reachable-but-unbounded growth that leak detectors are built to ignore. A cargo test assertion, plus a soak CLI for whole binaries.

$ cargo add navian-memcheck
Caught a 52 GB bug in CI — with zero leaked bytes
a leak detector sees nothing; this fails the build
$ cargo test --test memory_stays_bounded

── the buggy engine ────────────────────
navian-memcheck: FAIL — still growing:
  back-half slope 620,389 B/sample > budget 4,096 B/sample
────────────────────────────────────────

# add the eviction, re-run:
$ cargo test --test memory_stays_bounded
   test memory_stays_bounded ... ok  # memory plateaus
Where it fits

The memory bug leak detectors are built to ignore.

Ask an AI agent to keep per-session state and it writes the insert; unless the lifecycle is explicit, it omits the eviction. Every byte stays reachable, so a leak detector sees nothing wrong — and it only becomes a problem after millions of events, in production. navian-memcheck asserts the property that was actually violated: after warmup, live memory levels off. It catches the three shapes that growth takes:

Temporal

Climbs with runtime

At steady load, memory keeps rising the longer the process runs — a structure that fills and never drains.

Unbounded

Scales with load

Ten times the customers, ten times the memory — and it never comes back down when the load subsides.

Type

Wrong Big-O

Memory grows faster than its input — double the load for quadruple the memory. A retained buffer with the wrong complexity.

ToolWhat it findsCatches reachable, unbounded growth?
Valgrind · LeakSanitizermemory that was lost — no longer referencedno — every byte here is still reachable
Heap profiler (dhat · heaptrack)where bytes live, as a flamegraphshows it, but only as output a human reads
navian-memcheckreachable memory that never plateausyes — a capacity contract, asserted in CI
The library · cargo test

One assertion. No allocator to install.

Add the crate and write one test. It drives your real handler in a loop, samples live memory on a fixed cadence, fits a least-squares line through the back half of the run, and fails if the slope exceeds your budget. You write the workload; the crate owns the loop, the sampling, and the verdict.

// Temporal — drive your real per-event work; prove memory plateaus after warmup.
navian_memcheck::assert_plateau(200_000, |i| engine.handle_event(next_event(i)));

// Unbounded — memory stays under a cap as the driver scales.
assert_bounded(64 * MB, (1..=10).map(|n| n * 100_000), |load| live_bytes_at(load));

// Type — memory grows at most linearly in its driver (no O(n^2) blowup).
assert_linear_in(&growth_points, /* max bytes per entity */ 128.0);

By default it samples process RSS — zero setup, no allocator to install — and the slope budget is your noise knob. An optional jemalloc feature gives a clean in-process live-heap signal when you want precision.

The CLI · navian-memcheck-cli

A whole binary, under load, with no test code at all.

For a whole program there's nothing to write — point the soak runner at your build under load. It runs the target in its own process group, polls the group's RSS on an interval, applies the same plateau assertion, and exits non-zero so it gates CI directly.

soak a command, gate CI on the plateau
$ cargo install navian-memcheck-cli

$ navian-memcheck soak --cmd "./pulse --load" \
    --duration 20m --interval 5s \
    --slope-budget 2mb --max 6gb

  sampling group RSS every 5s for 20m …
  FAIL — still growing:
    back-half slope 620,389 B/sample > budget 2mb
  exit 1  →  CI red
  1. No code, no allocator. cargo install navian-memcheck-cli, then soak any command via --cmd. RSS is read from the OS — nothing to link into the target.
  2. It drives the clock, you set the budget. --duration / --samples bound the run, --interval the cadence, --slope-budget the allowed back-half growth, and --max a hard RSS ceiling.
  3. Exit codes gate CI. 0 plateaued (and stayed under --max), 1 still growing or over the cap, 2 crashed / too few samples / usage error.
  4. Reproduce and bisect. Run it from a seed, or under navian-dst, and the failing run reproduces exactly — bisect the growth to the event sequence that caused it.

The target runs in its own process group and the whole tree's RSS is summed. A process that re-groups (setsid, a double-fork daemonize) leaves the group and stops being counted — soak its foreground/no-detach mode instead.

Evidence

The 52 GB bug that wasn't a leak.

One of our engines climbed to 52 GB and the kernel killed it — but nothing leaked. Every byte was reachable: a per-session map that grew and never evicted. This tool is what we built to catch it, in a minute of CI instead of weeks in production.

52 GB
reachable growth, zero leaked bytes — invisible to LeakSanitizer and Valgrind, caught here as a slope
seconds
a weeks-long climb surfaces in CI: drive the handler directly, CPU-bound, with an in-memory stand-in for disk
Apache-2.0
library + CLI, RSS and jemalloc samplers — on crates.io and docs.rs

Honest about scope (navian-memcheck)

  • It's a detector, not a localizer: it fails the build because memory grew, and tells you the slope — to find the exact line, hand off to a heap profiler like dhat or heaptrack.
  • RSS is noisy — the slope budget is the noise knob; use the jemalloc sampler when you want a clean in-process signal.
  • It measures a capacity contract on memory that is legitimately held — not correctness of what's stored.
  • Don't soak a process that daemonizes / re-groups; its memory leaves the counted process group.

Add one soak test and gate your CI.

Turning it on changes nothing in your production code — one cargo test assertion, or one CLI line for a whole binary.