PSI: The Right Instrument for Memory Oversubscription

There is a particular embarrassment that only happens in infrastructure: a machine that politely refuses work while doing essentially nothing. On 3 September one of our hosts had eight microVMs booked against it, was carrying 1,770 MB of actual resident memory, and turned down a request to wake a customer's app because the arithmetic said it was full. It was not full. Across two hosts the fleet sat at roughly 4% real utilisation and 100% booked capacity. The arithmetic was counting what guests had been promised rather than what they had touched.

I'm Ajay, and I build PandaStack: Firecracker microVMs that restore from a snapshot on every create, each with its own guest kernel under KVM. The fix was to change what admission counts — charge a new VM against measured working set, not its baked size. That is overcommit, and overcommit is only responsible engineering if you can tell the difference between comfortably oversubscribed and one page fault away from the OOM killer. This post is about the instrument that draws that line: PSI, Pressure Stall Information, in the kernel since Linux 4.20 and still absent from most fleets' dashboards.

Why load average and free -m cannot answer the question

Load average is not a measure of CPU utilisation. On Linux it counts tasks that are runnable OR in uninterruptible sleep — the D state — and reports an exponentially damped moving average of that count over roughly one, five and fifteen minutes. Both decisions are hostile to capacity work. First, the conflation: a load average of 40 might mean forty threads fighting over eight cores, or eight threads computing happily while thirty-two wait on a slow disk. Including uninterruptible sleep was a deliberate Linux choice made decades ago so I/O-bound machines did not look idle, and it succeeds at that — it just means the metric answers 'is something going on around here' rather than any question you can act on.

Second, the missing unit. Load average counts tasks, so its meaning depends on core count and workload mix: forty is catastrophic on a two-core box and unremarkable on a 96-core one. The damping compounds this — a savage 20-second stall barely moves the one-minute figure, while a spike that ended five minutes ago still inflates it. There is no threshold on load average that means the same thing on two machines, which is exactly what an admission gate needs.

The memory side is subtler. Everyone knows to read MemAvailable rather than the free column. MemAvailable is better and it is still not a measurement — it is a kernel-side estimate: page cache minus a reserve the kernel does not believe it can reclaim, plus reclaimable slab, minus watermarks, with heuristics on top. The documentation says as much. It is an educated guess about what a new allocation could obtain without swapping, and it will cheerfully report gigabytes available on a host already spending real wall-clock time thrashing — the memory is nominally reclaimable, it is just refaulted straight back in after eviction.

That is the failure mode that matters for overcommit. Thrashing is not a shortage of pages; it is a shortage of pages worth keeping. A host in that state has plenty of available memory by the accounting and makes no forward progress, because every task is waiting for a page it had two seconds ago. No count of bytes describes this. What describes it is time — and time lost to waiting is precisely what none of these instruments measure.

What PSI actually measures

PSI, contributed by Meta and merged in Linux 4.20, asks a different question: what fraction of wall-clock time did tasks lose to waiting on a resource? Not how much of the resource is left, not how many tasks are queued — how much productive time the machine gave up. It exposes three files, /proc/pressure/cpu, /proc/pressure/memory and /proc/pressure/io, all the same shape.

$ cat /proc/pressure/memory
some avg10=0.72 avg60=0.31 avg300=0.11 total=41283991
full avg10=0.00 avg60=0.02 avg300=0.01 total=9127740
#    ^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^
#    |         |                        |
#    |         |                        monotonic counter, MICROSECONDS
#    |         |                        of stall since boot -- the number
#    |         |                        you should actually scrape
#    |         percent of wall time stalled, averaged over the
#    |         trailing 10 / 60 / 300 seconds
#    "some" = at least one task was stalled on memory
#    "full" = EVERY runnable task was stalled: zero forward progress

$ cat /proc/pressure/cpu
some avg10=2.31 avg60=1.88 avg300=1.44 total=88231004
# The CPU file has historically reported only a "some" line at the system
# level. Newer kernels expose "full" for CPU inside cgroups; at the top
# level it is definitionally near zero -- if a task is on-CPU, the CPU is
# not idle. Check what your kernel actually prints before alerting on it.

$ cat /proc/pressure/io
some avg10=0.04 avg60=0.02 avg300=0.05 total=2201933
full avg10=0.01 avg60=0.00 avg300=0.02 total=903112

The unit is a percentage of wall time, so it is comparable across machines with different core counts, memory sizes and workloads. A memory 'some' of 12 means the same thing on a 4-core VM as on a 96-core host: twelve percent of the last window had at least one task unable to proceed because of memory. That comparability is why PSI works as a fleet-wide threshold and load average does not.

some is the warning; full is the fire

The two lines are not severity tiers of one measurement — they are different measurements, and swapping them is the most common PSI mistake. 'some' aggregates time during which at least one task was stalled. On a busy multi-tenant host this is almost never zero and should not be: one tenant faulting in a page while forty others compute is a machine working correctly. A rising 'some' means reclaim is starting to cost somebody something — your early warning, the point at which you might stop admitting new work, not the point at which anything is broken.

'full' aggregates time during which every non-idle task was stalled — nobody making progress, the machine doing nothing but waiting. On memory, sustained non-zero 'full' is the thrashing signal, and it is the state where MemAvailable still reports gigabytes while the box has stopped being a computer. A clean empirical version from our own testing: capping a VM's memory.high well below its working set on a host with no swap pinned memory PSI avg10 near 96 while the workload simply never finished. With zswap enabled, the identical burn completed in about 45 seconds at a PSI around 23, roughly 1.79 GiB compressed through the pool. Same allocation, same cap. The only thing separating 'degrading gracefully' from 'dead' was time spent stalled — and PSI was the only instrument that said so.

Scrape the total counter, not the pre-averaged windows

The avg10/avg60/avg300 fields are convenient and they will lie to you about short spikes, for the same reason load average does: they are pre-averaged over a window you did not choose. A three-second full-pressure event — long enough to blow every latency SLO you have — contributes about a third of a percent to avg10 by the time a 15-second scrape reads it, and nothing to avg60. Averaging a spike is how you make it disappear.

The 'total' field is the honest one: a monotonically increasing count of microseconds of stall since boot, and the raw material the averages are computed from. Scrape it as a counter, rate it over whatever window your alerting needs, and you get a stall fraction at a resolution you control. Dividing a microsecond-per-second rate by one million gives the same 0-to-1 quantity the avg fields express as a percentage.

# node_exporter's pressure collector exposes the total counters as
# node_pressure_memory_waiting_seconds_total  ("some")
# node_pressure_memory_stalled_seconds_total  ("full")
# node_pressure_cpu_waiting_seconds_total
# node_pressure_io_{waiting,stalled}_seconds_total
#
# Rate them yourself. The result is a fraction of wall time in [0,1],
# so 0.15 == "15% of the window was stalled" == avg10-style units / 100.

# Fraction of the last 1m at least one task waited on memory:
rate(node_pressure_memory_waiting_seconds_total[1m])

# Fraction of the last 1m NOTHING made progress -- the thrash signal:
rate(node_pressure_memory_stalled_seconds_total[1m])

# Catch short spikes the pre-averaged avg10 field smooths away:
max_over_time(
  (rate(node_pressure_memory_stalled_seconds_total[30s]))[10m:30s]
)

# Page when the host has been genuinely stuck, not merely busy.
# Treat this number as a STARTING POINT to tune against your own fleet,
# not as a measured optimum -- the right value depends on workload mix,
# swap configuration, and what your users actually notice.
- alert: HostMemoryThrashing
  expr: rate(node_pressure_memory_stalled_seconds_total[1m]) > 0.02
  for: 5m

Per-cgroup PSI: who is causing this?

Host-wide PSI tells you the machine is suffering. On a multi-tenant host that is half an answer, and the missing half decides what you do about it. cgroup v2 exposes the same three files inside every cgroup — memory.pressure, cpu.pressure, io.pressure — same format, scoped to that subtree.

This is the attribution mechanism. When host pressure rises you can walk the per-VM cgroups and rank them by their own stall totals, separating the workload causing the pressure from those merely suffering it. Those are frequently different tenants, and getting them backwards means squeezing the victim while the culprit carries on. On our hosts every microVM gets its own cgroup with the memory controller delegated at creation, so those files exist for every guest from birth.

# Rank per-VM cgroups by their OWN memory stall, to separate the
# workload causing pressure from the ones merely suffering it.
cd /sys/fs/cgroup/pandastack || exit 1

for cg in vm-*; do
  [ -r "$cg/memory.pressure" ] || continue

  # "some" line: total= is microseconds of stall since the cgroup existed.
  stall=$(awk '/^some/{sub(/total=/,"",$4); print $4}' "$cg/memory.pressure")
  cur=$(cat "$cg/memory.current")
  high=$(cat "$cg/memory.high")

  printf '%-24s stall_us=%-14s current=%-12s high=%s\n' \
    "$cg" "$stall" "$cur" "$high"
done | sort -t= -k2 -rn | head -10

# Sample twice a couple of seconds apart and diff the totals for a live
# rate -- the cumulative number is dominated by whatever happened at boot.

# Worth knowing: on kernels 6.1 and later a cgroup.pressure file can switch
# per-cgroup PSI accounting OFF for a subtree. It is on by default; if a
# cgroup's pressure files read permanently zero, check that before
# concluding the workload is healthy.

One caveat that costs people an afternoon: a cgroup's memory.pressure reflects stalls attributed to its own reclaim, not its contribution to global pressure. A cgroup with a tight memory.high shows high local pressure while behaving perfectly — it is absorbing its own reclaim as instructed — while a cgroup with no limit can push the whole host into global reclaim with a modest file of its own. Read local pressure as 'is this workload constrained' and global as 'is this machine in trouble'.

Using PSI as an admission gate

The control loop is deliberately boring. A threshold on memory 'full' is a hard stop: while it is above the line, admit nothing new. A threshold on 'some' is a soft brake: keep admitting, but apply backpressure — shrink cold workloads, defer non-urgent work, stop prefetching.

Then hysteresis, because a loop that reacts symmetrically to a threshold crossing will oscillate, and an oscillating admission gate is worse than none. The pattern we use is a Schmitt trigger: the level may RISE the instant the raw signal says so, but may only FALL after the raw signal has stayed lower for two consecutive ticks. The asymmetry is the point. Reacting to danger should be instant; declaring it over should require evidence.

For concreteness: our host controller ticks every 10 seconds, reads MemAvailable and memory PSI together, and classifies the host as OK, ELEVATED or CRITICAL. The default trip points are 'some' avg10 above 10 or MemAvailable under 15% of total for ELEVATED, and 'full' avg10 above 5 or MemAvailable under 8% for CRITICAL, with the percentage lines clamped to absolute ceilings so a large host does not idle in ELEVATED while holding gigabytes of real headroom. Every one of those numbers is an environment variable with a default we picked, not a measured optimum — a starting point to tune against your own fleet. What is not arbitrary is the structure: two signals, two levels, asymmetric transitions, a kill switch.

Triggers and poll(), versus polling on an interval

Reading the file on a timer is fine for a slow gate like admission — a 10-second tick is appropriate when the decision it feeds is 'may this VM be created'. It is not fine for reacting to the onset of thrashing: by the time your next tick arrives the host may have been stalled for nine seconds.

For that, PSI has a proper notification interface. You write a trigger specification into the pressure file — resource type, a stall threshold in microseconds, a window in microseconds — and poll() the descriptor. The kernel wakes you when accumulated stall within a window crosses the threshold, at a granularity no polling loop could match. The window must fall in the kernel's accepted range (roughly 500 ms to 10 s), the threshold cannot exceed the window, and the descriptor must stay open for the trigger to live.

/* Wake up when memory stalls SOME task for >=150ms in any 1s window.
 * Verify the accepted window range and file permissions on your own
 * kernel: see Documentation/accounting/psi.rst. */
#define _GNU_SOURCE
#include <fcntl.h>
#include <poll.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

int main(void)
{
    /* Host-wide. For one workload, open a cgroup's memory.pressure
     * instead -- same interface, same trigger format. */
    int fd = open("/proc/pressure/memory", O_RDWR | O_NONBLOCK);
    if (fd < 0) { perror("open (CONFIG_PSI? psi=1?)"); return 1; }

    /* "<some|full> <stall_us> <window_us>" -- threshold <= window,
     * window within the kernel's permitted range. */
    const char *trig = "some 150000 1000000";
    if (write(fd, trig, strlen(trig)) < 0) { perror("write"); return 1; }

    struct pollfd pfd = { .fd = fd, .events = POLLPRI };

    for (;;) {
        int n = poll(&pfd, 1, -1);       /* block until the kernel says so */
        if (n < 0) { perror("poll"); return 1; }

        if (pfd.revents & POLLERR) {     /* trigger destroyed */
            fprintf(stderr, "psi trigger died\n");
            return 1;
        }
        if (pfd.revents & POLLPRI) {
            /* Memory stalled someone for >=15% of the last second.
             * Stop admitting; shed the coldest work; re-read the file
             * for current avg/total values before deciding more. */
            fprintf(stderr, "memory pressure event\n");
        }
    }
}

This is roughly how userspace OOM daemons of the oomd family work: watch PSI, act at a threshold you chose, and kill something you selected rather than letting the kernel decide under duress. The kernel OOM killer is an unsentimental capacity planner — it always arrives, it always frees exactly enough memory, and it has no opinion about which of your customers deserved to keep running. Any policy you prefer must execute before it does, on a signal that leads the collapse rather than confirming it.

The other half: admit against working set, not against promises

A stall signal alone does not buy density. It buys a safe way to fail. Density comes from changing what admission counts — the part we shipped in early September.

The structural problem is specific to snapshot-restored microVMs. A Firecracker guest gets a fixed memory size when its snapshot is baked and cannot change it at restore — our first-party templates bake 4 GiB. But Firecracker faults guest pages in lazily, so a restored guest touches a small fraction of what it was promised. Twelve days of shadow instrumentation across two hosts, some 27,200 one-minute samples, put the ratio of resident to committed memory at a median of 0.084 and 0.064, with p95 values of 0.270 and 0.202. VMs were using six to nine percent of what they had booked, and admission was charging them for a hundred.

So a 32 GiB host hit its ceiling at roughly seven 4 GiB VMs no matter how idle they were, which is how you get the incident I opened with. Working-set admission replaces the committed sum with something closer to reality: measured resident memory for live guests, a reserve for creates admitted but not yet measured, and a fixed headroom band, against the same budget. The per-create reserve is a fraction of the baked size with a floor — 0.25 and 512 MB by default — so a 4 GiB app is admitted against 1 GiB: about three times its median real footprint, and four times denser than committed accounting allowed.

Three details matter more than the formula. The in-flight reserve: a create admitted but not yet measured must count for something, or a burst of concurrent creates all admit against the same stale zero and the host learns the truth at page-fault time. Ours settle on first measurement and expire on a TTL. The class split: managed databases are charged their full committed size and never overcommitted, because that is what we publish, and a database is exactly the workload for which 'degrades under pressure' is the wrong answer. And the headroom band is pinned to the water line at which the pressure ladder starts acting, so admission can never by itself push a host into an elevated state.

The scheduler follows the same discipline: the agent computes admittable capacity once, with the arithmetic the local gate uses, and publishes it on its heartbeat rather than letting the control plane re-derive the formula. Two implementations of one rule is a bug with a delivery date. And a create refused for capacity does not error the customer's app — it parks in a waiting state and retries with backoff, because a transient refusal is not a failure.

The canary told the story cleanly. On a host with a 31,066 MB budget, twelve creates booking 46,080 MB of baked memory were all admitted, ran as twelve live VMs at 997 MB of total resident memory, and left roughly 25 GB still admittable, with the pressure ladder reporting OK throughout. Under committed accounting that host refuses at number seven.

What PSI will not do for you

It is a good instrument, not a magic one, and the limits are worth knowing before you build a control loop on it.

• It has to be compiled in, and may have to be switched on. PSI needs CONFIG_PSI, and some distributions ship CONFIG_PSI_DEFAULT_DISABLED, requiring psi=1 on the kernel command line before anything is collected. A missing /proc/pressure means the feature is absent; files reading permanently zero mean it is compiled in but off. Verify on the kernel you run.
• It tells you a stall happened, never which page. PSI is an aggregate over time; it will not name the allocation that triggered reclaim or the mapping that is thrashing. For that you still need /proc/vmstat, per-cgroup memory.stat and refault counters. It is the trigger for an investigation, not the investigation.
• It lags a sudden allocation storm. A process that maps and touches a large amount of memory in a few hundred milliseconds can wreck a host before enough stall accumulates to cross any sane threshold — feedback signals are behind by construction. This is exactly why the in-flight admission reserve exists: it is the feed-forward half.
• Fast storage flattens the curve until it doesn't. A host with quick NVMe swap and zswap absorbs a lot of reclaim without spending much wall time waiting — good news, and it means the curve stays flat and then turns very sharp. Do not calibrate thresholds on storage unlike the storage they will run on.
• It is host-side and does not see inside your guests. From the host, guest memory is ordinary anonymous memory belonging to a VMM process. If a guest kernel is thrashing inside its own 4 GiB — reclaiming page cache, swapping internally, running its own OOM killer — that stall is invisible to host PSI: those pages are resident and nothing on the host is waiting. The guest's own /proc/pressure is the only thing that sees it. Collect both if the in-guest experience is yours to answer for.

Overcommit without a stall signal is just hoping

Put the halves together and the design falls out. Admit optimistically, against what workloads measurably use, with a reserve covering the window before you can measure a new one. Then let PSI say stop: a soft brake on 'some' that shrinks the cold tail and slows intake, a hard stop on 'full', hysteresis so the gate does not chatter, and a waiting path so a deploy arriving mid-squeeze waits rather than fails.

Neither half is sufficient alone. Working-set admission without a stall signal bets that your measured ratio holds — and that ratio is a property of your tenants' behaviour, which they may change on a Tuesday without consulting you. A PSI gate without working-set admission is an excellent instrument bolted to a machine that still refuses work at 4% utilisation, which is where we started. Density comes from measuring reality; safety comes from measuring pain.

None of this makes a create faster — that is snapshot restore, and ours sits at a p50 of 179 ms whether the host is 4% or 70% booked. What changes is how many creates a host accepts before it says no, and whether that no arrives because the machine is in distress or because a spreadsheet said so. One of those is capacity management. The other is arithmetic in a high-visibility vest.

MemAvailable tells you what the kernel thinks it could free. PSI tells you what your users are already paying for the fact that it hasn't. Only one of those is a measurement.