Running Customer Git Hooks in Isolated microVMs

A git hook is a shell script your user wrote that your server agreed to run. That sentence is the whole post; everything below is the consequences of taking it literally. There is no CVE to patch, because the execution is the feature — hooks exist precisely so somebody other than the platform author decides what happens on a push. The question was never whether to run their code. It is what is standing next to it while it runs.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, so read this as opinionated. It is not hypothetical for me: PandaStack's git-driven app hosting clones a customer's repository, resolves their runtimes with mise, runs their install and build commands, and starts their process. All of that is repo-controlled code, and none of it runs on a host — it runs in a per-app microVM restored from a baked snapshot.

Shape one: policy hooks you host for a tenant

If you run a git-hosting product, an internal platform with governance requirements, or anything with a "custom push policy" feature, you will eventually let tenants install a server-side hook. Reasonable: organisations want to reject commits missing a ticket reference, block force-pushes to protected branches, refuse oversized files, or scan for credentials before they reach a shared history — and each is easiest to express as a script.

The mechanics fit in a paragraph. On a push, receive-pack runs pre-receive once with one line per updated ref on stdin: old SHA, new SHA, ref name. A non-zero exit rejects the entire push atomically, and stderr is relayed to the pusher prefixed with "remote:". The update hook runs per ref; post-receive runs after the refs move. Git also quarantines incoming objects and points the hook at them via an environment variable, so a rejected push leaves nothing behind.

Now the inventory. That script runs as your git user, on a machine whose disk holds every other tenant's bare repositories. It can read another customer's refs and the SSH host key, fork children that outlive the push, drop a payload in /tmp for the next one, and on most default cloud setups curl the metadata endpoint for a role. None of that is an exploit — those are ordinary operations available to any process, and this one you started on purpose.

#!/usr/bin/env bash
# pre-receive -- runs on YOUR server, as YOUR git user, on every push.
# stdin: one line per ref: "<old-sha> <new-sha> <refname>"
# Non-zero exit rejects the WHOLE push; stderr is shown to the pusher.
set -euo pipefail

ZERO=0000000000000000000000000000000000000000
fail() { echo "POLICY: $*" >&2; exit 1; }

while read -r old new ref; do
  [ "$new" = "$ZERO" ] && continue            # branch deletion, nothing to scan

  if [ "$old" = "$ZERO" ]; then range="$new"  # new branch: everything reachable
  else                          range="$old..$new"
  fi

  # 1. No force-pushes to main. Cheap, bounded, well-behaved.
  if [ "$ref" = "refs/heads/main" ] && [ "$old" != "$ZERO" ]; then
    git merge-base --is-ancestor "$old" "$new" || fail "no force-push to main"
  fi

  # 2. Reject newly-introduced blobs over 5 MiB.
  #    Note the process substitution: written as a pipeline into "while",
  #    the loop body runs in a SUBSHELL, "fail" exits the subshell, and your
  #    push is happily accepted. Policy hooks fail open for exactly this
  #    reason -- a shell quirk, quietly deciding your merge policy.
  while read -r type size path; do
    [ "$type" = blob ] || continue
    [ "$size" -gt 5242880 ] && fail "$path is $size bytes (limit 5 MiB)"
  done < <(git rev-list --objects "$range" --not --all |
           git cat-file --batch-check='%(objecttype) %(objectsize) %(rest)')

  # 3. "Just grep every added line for secrets." This is the line item that
  #    walks a 4 GB monorepo's object graph while the pusher watches a
  #    spinner and every other push to this repo waits behind the ref lock.
  git log -p --no-color "$range" |
    grep -nE 'AKIA[0-9A-Z]{16}|-----BEGIN [A-Z ]*PRIVATE KEY-----' &&
    fail "credential-shaped string in pushed diff"
done

exit 0

Notice which part is dangerous. Not step one — step three. The classic production failure here is not an exploit; it is a well-intentioned secret scanner that walks the object graph on every push. On a small repo it takes 200ms and nobody thinks about it again. On a 4 GB monorepo, after somebody pushes a branch rebased off main three months ago, it pegs a core, thrashes the page cache, holds the ref lock, and turns git push into a thirty-second stall for everyone. A platform incident caused by a customer being conscientious.

The threat model everyone plans for is a malicious hook. The incident you actually get is a sincere one, written by a security-minded engineer, that is accidentally O(repository) on every push.

Shape two: git clone is not the boundary you think it is

Both halves need stating clearly. The reassuring half: git clone does not run the remote repository's hooks. Hooks live in .git/hooks, which is not part of the transferred object graph, so a fresh clone gets the sample hooks from your own init template directory — and clone does not import the remote's .git/config either. Clone a hostile repository, do nothing else, and nothing of theirs has executed. A lot of security theatre is built on not knowing that.

The half that matters: nobody clones and then stops. The entire premise of a build server is that after the clone, you run the repository's code — and none of those steps look like "a hook" in the mental model people carry.

# The reassuring half -- verify it yourself, it is genuinely true:
git clone --depth 1 https://git.example.com/tenant/repo /work/repo
ls /work/repo/.git/hooks              # only *.sample, from YOUR templatedir
git -C /work/repo config --local -l   # remote.origin.url and little else

# The half that matters -- a build server, doing its job, one step at a time:
cd /work/repo
git submodule update --init --recursive   # fetches URLs listed in .gitmodules
git lfs pull                              # endpoint comes from repo .lfsconfig,
                                          # smudge filter runs on checkout
mise install                              # .tool-versions / mise.toml decide
                                          # what gets downloaded and executed
make setup                                # a Makefile is a shell script with tabs
npm ci                                    # preinstall / install / postinstall
npm run build                             # the repo's own build command

# Six repo-controlled command surfaces. Exactly zero of them are "hooks".
# All six run code whose author is your customer, on whatever box ran them.

One at a time. Submodule update fetches URLs the repository chose — an SSRF primitive aimed at your internal network before a build step runs. git-lfs is installed globally on most build images, so a repo's .gitattributes can route files through the smudge filter while its own .lfsconfig names that filter's endpoint. mise reads .tool-versions and downloads toolchains. A Makefile is a shell script with tabs. And npm's lifecycle scripts are, without much competition, the industry's most successful remote code execution feature.

One more class, whenever a directory rather than a URL crosses your boundary. Config-driven execution — core.hooksPath, core.pager, core.sshCommand, filter and diff drivers — lives in git config, which clone does not import. But if a repository directory reaches you via a tarball, an rsync or a restored volume, that config arrives with it, and the next innocuous git command runs whatever it names.

Why a container is a weak boundary for this specific job

I use containers constantly; they are excellent at packaging, resource accounting, and separating workloads that are cooperating rather than adversarial. For hostile repository code they are structurally weak, and the usual reassurance — "a container escape is hard" — answers a question nobody asked.

The payload does not need to escape. Without touching the boundary, a postinstall script or a Makefile target can read the environment — which on a build runner holds your registry token, deploy key and probably a database URL — read mounted caches, and open a socket to the node's metadata endpoint, which on default configurations answers unauthenticated HTTP from inside the network. The isolation model was never what stood between the attacker and the credential. Proximity was, and a shared build node has none.

The subtler reason is that container hardening has almost nothing to deny here. Seccomp and AppArmor narrow an application to the operations it legitimately needs — and a build step legitimately needs to spawn processes, write files, open sockets and compile native code. When your defence is "deny what this program does not need" and this program needs everything, you are holding a policy file full of allows. And every container on the node shares one kernel with every other tenant.

The design: a disposable machine per push, per build

The fix is boring, which is usually a good sign. Give each hook execution and each build its own virtual machine — a Firecracker guest with its own kernel under KVM — holding nothing worth taking, reaching nothing worth reaching, destroyed once the exit code is read.

1. Bake a snapshot, do not build an image per run. One template with git, the toolchains and your policy runner, baked once. Every create restores it rather than cold-booting: about 179ms p50 and 203ms p99 end to end, the restore step itself near 49ms. The first-ever cold boot is around 3 seconds, paid once.
2. Hand the guest only the objects it needs, never the bare repository. For a pre-receive check the interesting set is the pushed range — git's quarantine already isolates it — so bundle that range host-side and copy it in.
3. No long-lived credentials inside: not the git user's key, not the registry token, not a cloud role. If the guest needs a private repo, mint a short-lived token on demand and let it die with the machine.
4. Default-deny egress, enforced outside the guest. Each PandaStack sandbox gets its own network namespace with a veth pair and TAP device — 16,384 /30 subnets pre-allocated per agent — so the allowlist is host-side filtering, not a setting the guest could be asked nicely to respect.
5. A wall-clock timeout and a memory ceiling, both platform-enforced, so the monorepo-grep hook kills its own VM instead of your push queue.
6. Get the decision out as an exit code plus captured output, then destroy the machine: zero accepts, non-zero rejects, stderr goes back to the pusher.

from pandastack import Sandbox

HOOK_BUDGET_SECONDS = 90


def check_push(repo_url: str, bundle_path: str, refs_stdin: str) -> tuple[bool, str]:
    """Run one tenant's pre-receive hook in a machine we are about to delete.

    Returns (accept, message). The message is what the pusher sees after
    "remote:" -- so it has to be theirs, not ours, and it has to be bounded.
    """
    # ttl_seconds is the backstop, not the primary timeout. If this process
    # dies mid-check, the platform still reaps the guest. Cleanup that
    # depends on your own code running is cleanup that fails during an
    # incident, which is the only time it mattered.
    sbx = Sandbox.create(template="base", ttl_seconds=HOOK_BUDGET_SECONDS + 60)

    try:
        # The guest sees ONLY the pushed range, as a bundle. Not the bare
        # repo, not the other tenants' repos sitting on the same host disk,
        # and not a credential that could fetch any of them.
        sbx.filesystem.write("/work/push.bundle", open(bundle_path, "rb").read())
        sbx.exec("git init -q /work/repo && "
                 "git -C /work/repo fetch -q /work/push.bundle '+refs/*:refs/*'")

        # Their script. Their exit code. Our machine, for another 90 seconds.
        sbx.filesystem.write("/work/repo/hooks/pre-receive", tenant_hook_source())
        sbx.exec("chmod +x /work/repo/hooks/pre-receive")

        r = sbx.exec(
            "cd /work/repo && ./hooks/pre-receive",
            stdin=refs_stdin,               # "<old> <new> <ref>" lines
            timeout_seconds=HOOK_BUDGET_SECONDS,
        )

        if r.exit_code == 0:
            return True, ""

        # 124 = timed out, 137 = SIGKILL, i.e. the hook OOMed its own guest
        # walking a monorepo. Both are policy failures with a clear message,
        # not a wedged push queue and a pager at 3am.
        reason = {
            124: "policy hook exceeded its 90s budget",
            137: "policy hook exceeded its memory budget",
        }.get(r.exit_code, r.stderr[-4000:])
        return False, reason

    finally:
        sbx.destroy()   # the hook, its children, its /tmp, and its kernel

Egress: the honest tension between "no network" and "npm install"

For a policy hook the answer is easy: deny everything. A check that validates commit messages, enforces branch rules and scans a diff needs zero packets, so give it zero. The rare case where the ideal policy and the practical one are the same policy.

For a build it is harder, and anyone who says otherwise has not run one. npm ci needs a registry, pip needs an index, mise needs a Node tarball, Go needs a module proxy. "No network" is not on the menu; a small allowlist of package endpoints is. Explicitly not on it: the metadata endpoint, your internal network, your artifact store, your control-plane API, the open internet. That one deny turns an exfiltration payload into a build step that fails a DNS lookup.

The stronger version is a registry mirror that authenticates upstream on the guest's behalf, so the sandbox never holds a token at all — a credential that was never mounted cannot be exfiltrated by any hook, however clever. And be honest about the residual: an allowlisted registry is still a channel, and a determined payload can encode a stolen secret into a package name. This bounds risk rather than eliminating it, which is worth doing because it turns a one-line curl into an engineering project.

Timeouts and memory: making the monorepo-grep hook boring

Give every hook run two limits, both enforced outside the guest. A wall-clock budget on the exec, so a hook walking the object graph dies at a number you chose. And a memory ceiling that is the guest's rather than the host's — a Firecracker VM's RAM is fixed at the machine level, so a hook buffering a multi-gigabyte pack file OOMs itself and nothing else. On PandaStack that size comes from the template the snapshot was baked at, which has a useful consequence: the ceiling is a property of the machine, not a cgroup setting somebody can forget to apply.

Put a platform-enforced TTL underneath as a backstop: the timeout covers the hook misbehaving, the TTL covers your dispatcher misbehaving during a deploy or a node eviction. Then make the timeout a product behaviour rather than an error — exit 124 becomes a rejection message saying the hook exceeded its budget. Half the value of this design is that a resource failure stops being your incident and becomes their build log.

Secrets: the token that never enters the guest

For hooks the credential question has a clean answer: the guest never talks to git. You bundle the range host-side and copy it in, so there is no token, no deploy key, no clone URL with a secret in it.

For builds against a private repository somebody has to authenticate. PandaStack's app hosting mints a GitHub App installation token on demand for the clone and never persists it — not baked into a template, not written into a snapshot, not stored on the deployment record. It is short-lived, so its value decays on a clock rather than on your rotation discipline, and scoped to the installation, so its blast radius is the repositories the customer already granted. Better still, clone host-side and hand the guest a working tree with no remote credential — a build needs the code, not the ability to fetch it again.

Fork: five policy checks from one identical starting state

This is where the microVM model stops being a tax and starts paying you back. Policy is rarely one check: a mature platform wants a commit-message linter, a secret scanner, a license check, a file-size rule and a signature check — five checks against the same tree, four of them independent.

Run them sequentially in one VM and you serialise five workloads and lose isolation between them: a scanner that leaves a 2 GB temp file breaks the license check behind it. Run five VMs from scratch and you pay five clones, the expensive part on a monorepo. Fork is the third option: clone once, snapshot the post-clone state, fork it five ways. Guest memory is copy-on-write and the rootfs is a reflink clone, so each fork is O(metadata), not O(repository) — 400–750ms same-host, 1.2–3.5s cross-host.

import concurrent.futures as cf
from pandastack import Sandbox

CHECKS = {
    "commit-msg":  "./policy/commit-message.sh",
    "secrets":     "./policy/scan-secrets.sh",
    "licenses":    "./policy/check-licenses.sh",
    "file-size":   "./policy/max-blob-size.sh",
    "signatures":  "git verify-commit --raw HEAD",
}


def run_policy_suite(repo_url: str, ref: str) -> dict[str, tuple[int, str]]:
    # Clone ONCE. On a big monorepo this is the expensive step, and it is the
    # step we are about to stop repeating.
    base = Sandbox.create(template="base", ttl_seconds=900)
    base.exec("git clone --depth 1 --branch " + ref + " " + repo_url + " /work/repo")
    base.exec("cd /work/repo && cp -r /opt/policy ./policy")

    # Freeze the post-clone state. Every check now starts from a byte-identical
    # machine, which also makes a failure reproducible: re-fork and re-run.
    warm = base.snapshot()
    base.destroy()

    def one(name: str, cmd: str) -> tuple[str, tuple[int, str]]:
        # Copy-on-write: guest memory is shared until written and the rootfs
        # is a reflink clone, so fork number five costs the same as number one.
        sbx = warm.fork(ttl_seconds=300)
        try:
            r = sbx.exec("cd /work/repo && " + cmd, timeout_seconds=240)
            return name, (r.exit_code, (r.stderr or r.stdout)[-2000:])
        finally:
            sbx.destroy()

    with cf.ThreadPoolExecutor(max_workers=len(CHECKS)) as pool:
        return dict(pool.map(lambda kv: one(*kv), CHECKS.items()))


# Wall clock is now the slowest single check plus one fork, instead of the
# sum of five. And a check that fork-bombs, OOMs, or fills the disk takes
# down a machine with a five-minute lifespan -- not the other four.

The reproducibility is worth as much as the speed. When a check fails on a Friday afternoon and nobody can reproduce it, "re-fork the snapshot and run it again" is a real answer, because the snapshot is the environment — an option you do not have when the check ran on a shared runner that has since drifted.

Shared git host vs container vs microVM per push

Softest boundary to hardest. The container column describes general architectural properties and common defaults, not a benchmark — a well-configured platform with per-job network policy and ephemeral runners closes several of these gaps, so verify against your own platform's docs. The only measured numbers here are PandaStack's.

• What the hook can read — Shared git host: every tenant's bare repositories on that disk, the SSH host key, the git user's environment, whatever the last push left in /tmp. Container per hook: the job's environment and mounts, usually more than the hook needs. microVM per push: a bundle of the pushed range, because that is all you put there.
• Reachable network by default — Shared git host: whatever the host reaches, typically the internal network and the metadata endpoint. Container per hook: much the same, absent a per-job policy. microVM per push: its own network namespace, host-enforced default-deny, metadata endpoint unreachable.
• A hook that pegs a CPU for 40 seconds — Shared git host: everyone pushing to that repo waits behind the ref lock. Container per hook: cgroups cap CPU and RAM if set, but IO, page cache and kernel locks are shared. microVM per push: a hard vCPU and RAM boundary; the hook busts its own budget and the tenant gets a rejection.
• Blast radius if the boundary is attacked — Shared git host: there is no boundary; the script is already on the machine you care about. Container per hook: namespaces and seccomp on a shared kernel, and a syscall policy has little it can honestly deny a build. microVM per push: an own-kernel guest under KVM, so a full compromise owns a machine that was about to be deleted — kernel and all state destroyed with it, leaving no residue for the next push to inherit.
• Cost per run — Shared git host: effectively zero, which is why nobody changes it. Container per hook: milliseconds on a cached image, seconds on a cold pull. microVM per push: about 179ms p50 and 203ms p99 to restore a baked snapshot, roughly 3 seconds for the one-time cold boot, 400–750ms for a same-host fork.

What this looks like in production

PandaStack's app hosting is this design applied to the build half. A push fires an HMAC-verified webhook; we provision a fresh microVM on the base template, shallow-clone the exact commit with an on-demand installation token for private repos, run mise install so .nvmrc and .tool-versions pick the toolchain, run install and build, health-check the port, then flip traffic and destroy the old machine. Every step runs repo-controlled code, in the guest.

The payoff is not primarily the security story. It is that a build which OOMs, fork-bombs or hangs forever is a failed deployment with a log attached rather than a degraded node — and because the flip happens only after the health check passes, a build that destroys its own machine never becomes an outage for the version still serving.

The summary

Server-side hooks are code your tenants wrote, running as your git user, on a box holding everyone's repositories. Client-side execution is subtler: git clone genuinely does not run the remote's hooks or import its config — and then submodule update, lfs filters, mise, make and npm's install lifecycle run repo-controlled commands anyway, because running the repository's code is the point of a build server.

A container is the wrong shape here — not because containers are bad, but because the payload never needs to escape one and a build legitimately needs the syscalls you would otherwise deny. What holds is a disposable machine per push and per build: its own kernel, a bundle instead of a repository, no long-lived credentials, default-deny egress with a small package allowlist, and a timeout and memory ceiling that make a runaway hook the tenant's problem.

Snapshot-restore makes that affordable at 179ms p50; fork makes it pleasant. You give up warm process state, and you gain a system where the worst thing a customer's shell script can do is destroy a machine that had ninety seconds left to live. Which is, more or less, the deal you thought you were getting when you shipped the hooks feature.