EEVDF vs CFS: What the New Linux Scheduler Means for MicroVM Density

Somewhere between Linux 6.5 and 6.6, the thing that decides which of your threads runs next got replaced. Not tweaked — replaced. CFS, the Completely Fair Scheduler that had been the default since 2007 and which every piece of performance folklore on the internet is written against, was swapped for EEVDF: Earliest Eligible Virtual Deadline First. The commit landed, distros shipped it, and for most people nothing visibly happened, which is the highest compliment you can pay a scheduler change.

For me it was not invisible. I'm Ajay, and I build PandaStack, where the runnable entities on a host are not a tidy set of application threads but the vCPU threads of a few hundred deliberately oversubscribed Firecracker microVMs, most of them idle, a few of them suddenly not. That population is unusual enough that the scheduler's choice of who-runs-next is a product decision, not a footnote. So I went and read the thing. This post is what CFS actually did, what EEVDF actually does, which knobs quietly evaporated, and what any of it means when your workload is a fleet of tiny VMs.

What a CPU scheduler is actually deciding

A general-purpose scheduler answers two questions that people constantly merge into one. First: over a long window, what proportion of CPU should each runnable thread receive? Call that the share question. Second: right now, at this instant, given several threads that all want a core, which one goes first and for how long? Call that the order question.

Share is about fairness and it is the easy one — you attach a weight to each entity and divide. Order is about latency, and it is where all the interesting failure modes live. A thread that wakes up every millisecond to do 50 microseconds of work and a thread that wants to burn a core for the next forty minutes can have identical shares and radically different requirements. The first one cares enormously about when it runs. The second one genuinely does not care at all, as long as it gets its total.

CFS answered the share question beautifully and the order question by accident. EEVDF's whole thesis is that these are two questions and you should be allowed to answer them separately.

How CFS worked: virtual runtime and a red-black tree

CFS modelled an ideal machine that runs every runnable thread simultaneously at 1/N speed, then tried to approximate it on real hardware. Each schedulable entity carried a virtual runtime — vruntime — which advanced as the entity ran, but scaled by its weight: a high-weight thread's vruntime crawled forward, a low-weight thread's sprinted. Weight came from the nice value through a lookup table where each nice level is roughly a 1.25x multiplier, so nice -20 and nice +19 sit about 1,900x apart in share terms.

The scheduling rule was then one line long, and this is genuinely why CFS was loved: always run the entity with the smallest vruntime. Keep them in a red-black tree ordered by vruntime, pick the leftmost node, run it, advance its vruntime, reinsert. O(log n) picks, no heuristic tables, no priority-boost hacks like the O(1) scheduler had. Fairness emerged from the ordering rule instead of being bolted on.

But notice what's missing: nowhere in that rule does an entity get to say anything about latency. Everything a thread could express, it expressed through weight — and weight means share. So CFS grew a second layer to handle order: sched_latency_ns, a target period in which every runnable thread should get a turn; sched_min_granularity_ns, a floor on slice length so that at high thread counts you stopped context-switching yourself to death; sched_wakeup_granularity_ns, deciding whether a waking task was allowed to preempt the current one. These were global sysctls. One number, for every workload on the machine.

That is the structural weakness. A short interactive wakeup and a long batch hog were the same kind of object to CFS, and the only way to bias the machine toward one was to move a global knob that also affected the other. Latency tuning became a folk practice: people copied sysctl blocks between servers, half of them tuned for a 2012 desktop, and the results were roughly as reproducible as you'd expect. Various out-of-tree patch sets existed precisely because the mainline model had no place to put the sentence 'this task is latency-sensitive but does not need more CPU'.

What EEVDF changes: lag, eligibility, and virtual deadlines

EEVDF comes from an academic line of work on proportional-share scheduling that predates CFS by about a decade, and it keeps the fairness model while giving each task a way to express its latency requirement independently of its share. It became the default for the fair class in Linux 6.6.

Lag and the eligibility rule

The core accounting concept is lag: for each entity, the difference between the service it should have received by now under the ideal fair-share model and the service it has actually received. Positive lag means the scheduler owes you time. Negative lag means you've run ahead of your fair share and are, briefly, in debt.

An entity is eligible when its lag is non-negative — that is, when it has not yet consumed its fair share up to this point in virtual time. Entities that have run ahead are simply not candidates until virtual time catches up with them. This one rule does a lot of work: it is what stops a thread that just burned a long slice from immediately competing again, and it is what makes the fairness bound provable rather than emergent. Lag also gets preserved across sleep and wake in the 6.6+ implementation, which closed a long-standing CFS trick where a task could game the system by sleeping and re-entering with a favourable vruntime.

Request size and the virtual deadline

The second concept is the request size, or time slice: how much CPU this entity wants in one go. Each entity gets a virtual deadline computed as its eligible time plus its request size divided by its weight. Among all eligible entities, the scheduler runs the one with the earliest virtual deadline. That's the whole pick rule, and like CFS's it is a single sentence — but now there are two independent inputs, weight and request size, instead of one.

This is the part worth sitting with, because it is the actual payoff. Request size divides into the deadline. A task that asks for a SHORT slice gets a NEARER virtual deadline and therefore gets picked sooner — without asking for a larger share of CPU. It runs promptly and briefly. A task that asks for a long slice gets a distant deadline and gets scheduled less often, but when it runs it runs for longer, which is exactly what a compile job wants: fewer context switches, better cache behaviour, and total throughput unchanged. Two workloads, same weight, same long-run share, opposite latency behaviour. CFS had no way to say that.

Per-task control over that request size is exposed through sched_attr — the sched_runtime field, for ordinary SCHED_NORMAL tasks, as the mechanism that subsumed the older latency-nice proposals. Be careful here: the exact interface, the clamping range, and the kernel version in which the per-task slice became settable are things you should verify against the kernel you are actually running rather than against this post. The EEVDF core landed in 6.6; the userspace-visible slice control arrived in the releases after it. Check sched_setattr(2) and your kernel's scheduler documentation.

/* Ask for a SHORT slice: earlier virtual deadline, same CPU share.
 * Interface and clamping range are kernel-version dependent -- verify
 * against sched_setattr(2) and your kernel's docs before relying on it. */
#define _GNU_SOURCE
#include <sched.h>
#include <linux/sched/types.h>   /* struct sched_attr */
#include <sys/syscall.h>
#include <unistd.h>

int main(void) {
    struct sched_attr attr = {
        .size          = sizeof(attr),
        .sched_policy  = SCHED_OTHER,   /* the fair class; EEVDF on 6.6+ */
        .sched_nice    = 0,             /* weight -- the SHARE question */
        .sched_runtime = 500000,        /* 0.5 ms request size, in ns:
                                           the ORDER/latency question */
    };
    /* pid 0 == this thread */
    return syscall(SYS_sched_setattr, 0, &attr, 0u);
}

CFS vs EEVDF, dimension by dimension

• Ordering rule — CFS: run the runnable entity with the smallest virtual runtime, where vruntime advances inversely to weight; a red-black tree keyed on vruntime makes the pick O(log n). EEVDF: filter to entities whose lag is non-negative (eligible), then among those run the one with the earliest virtual deadline, where deadline = eligible time + request size / weight.
• Latency handling — CFS: no per-task latency concept at all; responsiveness is a side effect of global slice heuristics (sched_latency_ns, min_granularity, wakeup_granularity) applied identically to every thread on the box. EEVDF: latency is a first-class per-task input via the request size — a smaller slice yields an earlier deadline and quicker scheduling, at no change to the task's share.
• Tunables — CFS: a pile of global sysctls under kernel.sched_* that every tuning blog on the internet has an opinion about, none of which can distinguish two workloads on the same host. EEVDF: far fewer knobs, mostly debug-only under /sys/kernel/debug/sched/, with the intended per-workload control moved to the per-task slice and to cgroup weights.
• Fairness guarantee — CFS: fairness is emergent from the vruntime ordering and holds well in the long run, but the bound on how far any single task can drift from its ideal share is a property of the heuristics rather than something the algorithm proves. EEVDF: the eligibility rule gives a bounded-lag guarantee by construction — that is precisely what the eligibility test is for.
• What it means for oversubscribed vCPUs — CFS: a vCPU thread waking to service a virtio interrupt is indistinguishable from a vCPU thread grinding through a build, so tuning for the first degrades the second and vice versa. EEVDF: those two are finally expressible as different requests against the same share, which is the exact shape of a dense microVM host — many tiny latency-sensitive wakeups, a few long compute burns, all mixed on the same runqueues.

The tunables that vanished (and the runbooks that didn't)

Here is the operational bit that bites people. The classic CFS sysctls are gone. They were moved out of /proc/sys into debugfs some releases before EEVDF landed, and then EEVDF removed or replaced the ones whose underlying concepts it no longer has: there is no sched_latency_ns target period any more, and no sched_min_granularity_ns floor, because slices are now per-entity rather than derived from a global period. What you find instead, under /sys/kernel/debug/sched/ on a kernel built with scheduler debug enabled, is a smaller set including a base slice value that seeds the default request size.

So half the scheduler tuning advice on the internet now edits sysctls that no longer exist — which at least fails loudly, since sysctl returns an error for an unknown key rather than silently accepting it. The genuinely dangerous version is the config-management repo that has been shipping those settings for six years and whose failure is a log line nobody greps for. Before you trust any tuning runbook, find out which scheduler you're on:

# 1. Which kernel is this host actually running?
uname -r
# 6.6 and later: EEVDF is the default for the fair class (SCHED_NORMAL).
# 6.5 and earlier: CFS. Distro kernels backport things, so treat this as
# a strong hint, not proof -- check your vendor's changelog if it matters.

# 2. Do the classic CFS sysctls still exist here?
for k in sched_latency_ns sched_min_granularity_ns sched_wakeup_granularity_ns; do
  printf '%-32s ' "$k"
  sysctl -n "kernel.$k" 2>/dev/null || echo '(absent)'
done
# On a modern kernel you should expect three "(absent)" lines. If your
# config management is still writing these, it has been failing silently.

# 3. What IS exposed? Debug knobs live in debugfs (CONFIG_SCHED_DEBUG),
#    and are debug knobs -- not a supported tuning API.
mount | grep -q debugfs || mount -t debugfs none /sys/kernel/debug
ls /sys/kernel/debug/sched/ 2>/dev/null

# e.g. the base slice that seeds the default request size, in nanoseconds:
cat /sys/kernel/debug/sched/base_slice_ns 2>/dev/null || echo 'not present on this kernel'

Why a microVM host is the workload EEVDF was drawn for

A Firecracker vCPU is an ordinary host thread sitting in a KVM_RUN ioctl loop. When the guest computes, the thread is on-CPU. When the guest idles, the thread blocks and costs essentially nothing. That is the entire basis of microVM density: you can allocate far more vCPUs than you have cores because the runnable set at any instant is a small fraction of the allocated set. On PandaStack, guests are baked with 8 vCPUs as burst capacity precisely because that capacity is shared rather than reserved, and CPU is billed by the active CPU-seconds actually burned rather than by what was allocated.

Now look at what those vCPU threads are doing when they wake. A large fraction of wakeups are tiny: the guest kicked a virtio queue, a packet arrived, a timer fired, the VMM needs to service an exit and hand control back. Microseconds of work, and the guest is stalled for the entire round trip. That is textbook latency-sensitive-but-small-slice — the exact profile EEVDF's request size was introduced to express. Meanwhile, on the same host, another guest is running a webpack build that wants the longest slice it can get and does not care in the slightest when it gets scheduled.

Under CFS these two competed as identical objects and the only dial was global. Under EEVDF they are at least representable as different requests. I want to be honest about the size of the effect though, because this is where people over-claim: the difference does not show up as more throughput. Aggregate throughput on a busy host is governed by cores and by how much work there is, and no scheduler creates cycles. It shows up in the tail — p99 latency and jitter — and inside the guest it shows up as steal time, the time a vCPU was runnable and did not get a core. If you are hunting for a scheduler-shaped problem, the symptom is a p99 that looks nothing like the p50, not a throughput number that moved.

cpu.weight sets the share; EEVDF sets the order

This is the conflation I see most often, so let me be blunt about it. cgroup v2's cpu.weight is the share question. It maps onto the scheduler's weight for the group's entities and it decides what proportion of contended CPU a cgroup receives relative to its siblings. cpu.max is the ceiling question — a hard bandwidth quota over a period, still implemented by the bandwidth controller that CFS brought with it and which EEVDF inherited unchanged. Neither of these is the order question. EEVDF decides who runs next and for how long, within whatever share cpu.weight has apportioned.

Which means: switching to EEVDF does not change your cgroup fairness model, and changing cpu.weight will not fix a latency problem caused by slice sizing. They are orthogonal levers and you can pull both. On a dense host, cpu.weight is still the right tool for keeping one busy tenant from starving a quiet one, and cpu.max is still the right tool for stopping a runaway guest from eating the box. The number you should actually be watching in cpu.stat is throttling, because a throttled cgroup produces guest-visible stalls that look exactly like scheduler unfairness and are not.

CG=/sys/fs/cgroup/pandastack/vm-abc

# SHARE: proportional weight, only bites under contention.
# cgroup v2 range is 1..10000, default 100.
cat "$CG"/cpu.weight

# CEILING: hard bandwidth quota. "max 100000" == uncapped.
# "200000 100000" == at most 2 CPUs' worth per 100ms window.
cat "$CG"/cpu.max

# The number that actually explains mystery stalls:
cat "$CG"/cpu.stat
# usage_usec ...        <- CPU actually burned by this group
# nr_periods ...
# nr_throttled ...      <- how many periods hit the cpu.max ceiling
# throttled_usec ...    <- total time this group was frozen at the ceiling
#
# nr_throttled climbing means your cpu.max is the constraint, NOT the
# scheduler. Fix the quota before you go reading scheduler source.

Measure runqueue latency instead of guessing

The question you almost always want answered is not 'which scheduler am I on' but 'how long do my threads sit runnable before they get a core'. That is runqueue latency, and it is directly measurable. Two tools, both cheap enough to run on a production host for a short window.

perf sched records scheduler tracepoints and gives you per-task wakeup-to-run latency. Record for a bounded window; the trace file grows fast on a busy host.

# Record scheduler events for 10 seconds across the whole host.
perf sched record -- sleep 10

# Per-task summary: average and maximum wakeup-to-run delay.
perf sched latency --sort max

# Firecracker names its vCPU threads fc_vcpu N, so the rows you care
# about on a microVM host are easy to pick out:
perf sched latency --sort max | grep -E 'fc_vcpu|firecracker'

# Timeline view -- useful for seeing one thread repeatedly passed over:
perf sched timehist | head -40

# Cleanup: perf.data is large.
rm -f perf.data perf.data.old

For continuous or lower-overhead observation, an eBPF histogram is better. bcc ships runqlat, which does exactly this; here is the shape of it in bpftrace so you can see there is no magic in it — timestamp on wakeup, subtract on switch-in, bucket the delta.

# Runqueue latency histogram (microseconds), log2 buckets.
# The prebuilt version is bcc's runqlat; this is the same idea inline.
bpftrace -e '
tracepoint:sched:sched_wakeup,
tracepoint:sched:sched_wakeup_new
{
    @qtime[args.pid] = nsecs;
}

tracepoint:sched:sched_switch
{
    if (args.prev_state == 0) {          /* still runnable: requeued */
        @qtime[args.prev_pid] = nsecs;
    }
    $t = @qtime[args.next_pid];
    if ($t) {
        @usecs = hist((nsecs - $t) / 1000);
        delete(@qtime[args.next_pid]);
    }
}

END { clear(@qtime); }'

# Read it as a distribution, not an average. A fat tail out past a few
# milliseconds on a host whose guests are meant to feel interactive is
# your signal -- and it will show up inside those guests as steal time.

What I'd actually do about it

1. Find out which scheduler you're on before reading any advice about it. uname -r, then check whether the classic sysctls exist. A tuning document that doesn't state a kernel version is describing a machine that may not resemble yours.
2. Delete dead sysctls from your config management. If your Ansible role has been setting kernel.sched_min_granularity_ns since 2019, it has been failing on modern kernels for a while. Removing it changes nothing about behaviour and everything about how much you trust the rest of the role.
3. Do not cargo-cult scheduler knobs. The EEVDF-era design intent is fewer global dials and more per-task expression. If you find yourself writing to debugfs on a fleet, ask what per-workload property you were trying to express and whether cgroup weights or a per-task slice expresses it properly.
4. Use cgroup v2 for share and ceiling, deliberately. cpu.weight for proportional fairness under contention, cpu.max when you need a hard ceiling, and watch nr_throttled in cpu.stat so you can tell a quota problem from a scheduler problem. These behave the same under both schedulers.
5. Be careful with pinning. Pinning a vCPU thread to a core buys predictability by permanently spending that core, which is the density model you presumably built the platform for. Pin the small set of workloads that genuinely need low jitter and let everything else float.
6. Measure the tail, not the mean. Runqueue latency distribution on the host, steal time in the guest. Both are cheap and both answer 'is the scheduler the problem' far faster than reading source.
7. Keep the host and guest kernels straight in your head. Our guests run 5.10 and are scheduling their own processes with CFS; the host arbitrating between vCPU threads is a different kernel with a different scheduler. Conclusions do not transfer between the two layers.

A better vocabulary, not a free lunch

EEVDF is not a performance upgrade and nobody in the kernel community claimed it was. It's a better vocabulary. CFS could say 'this thread deserves more CPU' and nothing else, so every latency requirement on the machine had to be smuggled through that one sentence, and the tuning culture that grew around it was a decade of people shouting share numbers at an order problem. EEVDF separates the two and gives the fairness half a bound you can actually prove.

For a dense microVM host that separation is the right shape, because the workload genuinely is bimodal: hundreds of tiny latency-sensitive virtio wakeups sharing runqueues with a handful of long compute burns. It will not make a create faster — that's snapshot restore, not scheduling, and ours sits at a p50 of 179ms — and it will not conjure cores. What it changes is which of your tenants notices when the box gets busy. That's a tail-latency story, and tail latency is where multi-tenant platforms are actually judged.

No scheduler creates cycles. All it can do is decide who waits, and for how long. EEVDF's contribution is letting a task say which of those two it cares about.