One microVM Per Sync Run: Isolating SCIM Connectors

There is a moment in every B2B SaaS company's life where the security questionnaire arrives and the word "SCIM" is in it. Somebody promises the deal team it is a two-week integration. It is a two-week integration in the same sense that a bank vault is a two-week door. What you are actually agreeing to build is a service that holds a long-lived, high-privilege credential for another company's identity provider, and then reads their entire employee roster with it, on a schedule, forever.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, and directory sync keeps showing up in my inbox with the same fingerprints on it. A single connector worker, a table of encrypted tokens, a cron loop, and a growing pile of per-customer special cases in the middle of the mapping code. It works. That is the problem — it works right up until it doesn't, and the failure mode is not "sync was late." This post is the version where each sync run gets its own microVM, and why that turns out to be the cheap option.

You are not holding an API key. You are holding their company.

Engineers file IdP credentials mentally next to Stripe keys and S3 access keys. They do not belong there. A Stripe key can move the customer's money. An Okta API token or an Entra ID application credential with directory read — often directory write, because provisioning is bidirectional the moment someone asks for group writeback — is a lens onto the org chart of the entire company, including the parts they have not announced yet.

• Every employee: legal name, work email, personal email in a surprising number of tenants, manager, department, cost center, employee number, sometimes phone and location.
• The full group graph, which is an org chart plus an access-control map. Group names leak roadmap, M&A, and layoffs — "Project Halibut - Diligence" is a group somebody created on a Tuesday.
• Joiner and leaver events in near real time. Knowing who was terminated an hour ago, before IT finishes the offboarding checklist, is a genuinely useful thing for an attacker to know.
• With write scope: the ability to create accounts, modify group membership, or flip attributes that downstream systems treat as authorization inputs.
• For on-prem LDAP: a bind DN that frequently has far more reach than anyone documented, because the person who created it in 2019 granted read on the whole tree to make the ticket go away.

The shared connector worker is the worst place on earth to keep these

The default architecture is one fleet of sync workers, each pulling jobs off a queue, each decrypting whichever tenant's credential the job names. At any instant, that process has one token in memory. Over a day it has held all of them. The relevant question is not "how many at once" — it is "what does one hour of arbitrary code execution in this process get you," and the honest answer is: a token-harvesting loop, because the process is already a token-harvesting loop with a legitimate purpose.

Then consider how code execution shows up in a connector, specifically. Connectors parse hostile-shaped JSON from a hundred different SCIM implementations. They follow URLs the tenant configured, which is a server-side request forgery primitive wearing a tie. They pull in an XML parser for the SAML metadata path and an LDAP library nobody has audited and a retry helper with 40 transitive dependencies. And they log — enthusiastically, at DEBUG, during an incident, with the request headers attached, at which point your logging middleware is an identity-provider credential exfiltration service with excellent uptime and full-text search.

The second problem: every customer's directory is a different shape

If the security story were the only issue you could paper over it with a smaller blast radius and better secret hygiene. But directory sync has a shape problem that pushes the code in exactly the wrong direction. SCIM is a standard the way charging cables are a standard. Everyone implements the core schema, nobody implements the same extensions, and the interesting attributes are always in the extensions.

• Custom schema URNs per tenant, carrying the attribute you actually need — cost center, employment type, the flag that decides whether this person gets the expensive seat.
• Pagination that varies by IdP and sometimes by tenant configuration: cursor tokens, 1-indexed startIndex, a Graph nextLink, or an LDAP paged-results control that dies halfway if the server's page size limit is lower than yours.
• Filter dialects. The SCIM filter grammar is real and mostly implemented, where "mostly" is doing load-bearing work in that sentence.
• Group names with characters your parser has never met: emoji, right-to-left marks, embedded newlines, a slash in a group whose name you were about to use as a path segment, and the eternal customer whose department field is a JSON string somebody pasted in 2021.
• On-prem LDAP schemas that predate you. Nested groups seven levels deep, memberOf that is not populated, UIDs that are not unique, and a tenant who has decided that objectClass=person is only advisory.
• Rate limits that differ by tenant plan, so the backoff behaviour that is polite for one customer is a self-inflicted 429 storm for another.

So the connector gets tenant-specific. First a config flag. Then a small if-statement. Then a per-tenant transform module, because the alternative was telling a seven-figure account that their LDAP tree is wrong. Now you have code paths that exist for exactly one customer running inside the process that holds all the other customers' credentials, and you are deploying changes to it on a Thursday. The isolation argument and the customization argument are the same argument arriving from two directions.

The shape: one microVM per tenant, per sync run

The pattern that holds up is boring to describe and load-bearing in practice. A sync run is a job: fetch a tenant's directory state, diff it, apply changes, checkpoint. Give that job its own machine. Boot it, hand it exactly one tenant's credential at the moment it starts, let it talk to exactly one IdP hostname, and destroy it when the run ends. The credential's window of existence outside your secret store is the duration of one sync, inside a VM that contains nothing else.

The reason this is affordable rather than aspirational is that the VM does not have to be expensive. On PandaStack a sandbox is created by restoring a pre-baked snapshot rather than cold-booting — p50 179ms, p99 203ms, with the restore step itself around 49ms — and only the first-ever boot of a template costs about 3 seconds. A per-run VM stops being a budget line and starts being a function call. Verify the equivalent numbers for whatever platform you use against its own documentation; the architectural point is that per-run isolation only survives contact with reality if provisioning is cheap enough that nobody proposes reusing the VM "just for the retries."

1. Scheduler picks a tenant and a mode (full or delta) and mints a short-lived, narrowly scoped credential from your secret store — ideally one that expires before the VM's TTL does.
2. Create a sandbox from the connector template, tagged with tenant, run id, IdP and mode in metadata, with a TTL sized to the mode.
3. Write non-secret config and the last-known cursor into the guest filesystem. Config on disk is fine. Credentials on disk are not.
4. Exec the connector with the credential passed in at exec time, under a hard timeout that is shorter than the TTL.
5. Read results back out over the filesystem API — a changeset, a run log, a manifest — rather than having the guest post them somewhere.
6. Apply the changeset in your own control plane, advance the cursor only on a clean finish, and kill the VM. The TTL is the backstop for every path where step six does not run.

# sync_run.py -- one microVM per tenant per sync run. The IdP credential
# exists inside exactly one VM, for exactly one run, and is never written
# to the guest's disk.
import json
from pandastack import Sandbox

from secrets_store import mint_idp_token   # short-lived, tenant-scoped
from tenants import load_tenant_config

CONNECTORS = {
    "okta": "/opt/connectors/okta_scim.py",
    "entra": "/opt/connectors/entra_graph.py",
    "google": "/opt/connectors/google_directory.py",
    "ldap": "/opt/connectors/ldap_sync.py",
}

# A full sync of a large directory is slow and that is fine. A HUNG sync is
# not slow, it is permanent, so both the exec and the VM get a deadline.
BUDGET = {"delta": (540, 900), "full": (5400, 7200)}   # exec secs, ttl secs


def run_sync(tenant_id: str, mode: str = "delta") -> dict:
    cfg = load_tenant_config(tenant_id)
    exec_budget, ttl = BUDGET[mode]

    sbx = Sandbox.create(
        template="scim-connector",   # baked from base, connectors preinstalled
        ttl_seconds=ttl,
        metadata={
            "tenant": tenant_id,
            "purpose": "directory-sync",
            "idp": cfg["idp"],
            "mode": mode,
            "run_id": cfg["run_id"],
            # The one hostname this VM is permitted to reach. Written here
            # so "which VM was allowed to talk to acme.okta.com" is a query.
            "egress_allow": cfg["idp_host"],
        },
    )
    try:
        # Public config and the resume cursor are fine on disk.
        sbx.filesystem.write("/work/config.json", json.dumps(cfg["public"]))
        sbx.filesystem.write("/work/cursor.json", json.dumps(cfg["cursor"]))

        token = mint_idp_token(tenant_id, ttl_seconds=ttl)
        connector = CONNECTORS[cfg["idp"]]

        r = sbx.exec(
            f"IDP_TOKEN={token} /opt/run-sync.sh {cfg['idp_host']} {mode} {connector}",
            timeout_seconds=exec_budget,
        )

        manifest = json.loads(sbx.filesystem.read("/work/out/manifest.json"))
        result = {
            "tenant": tenant_id,
            "run_id": cfg["run_id"],
            "exit_code": r.exit_code,
            "duration_ms": r.duration_ms,
            "records": manifest["records"],
            "changeset_sha256": manifest["sha256"],
            # Redacted in-guest before we ever read it -- see run-sync.sh.
            "log": sbx.filesystem.read("/work/out/run.log").decode()[-16000:],
        }
        if r.exit_code == 0:
            result["changes"] = sbx.filesystem.read("/work/out/changes.jsonl")
            result["cursor"] = json.loads(
                sbx.filesystem.read("/work/out/cursor.json")
            )
        return result
    finally:
        sbx.kill()   # credential, process memory, partial state: all gone

Note what is not in that function: no shared connection pool, no global rate limiter holding another tenant's state, no in-process cache that could serve tenant A's group list to tenant B's transform. The isolation is not a policy someone has to remember; it is the absence of a place for the mistake to happen.

Credential at exec, never at rest

The credential goes in as an environment variable on the exec, not as a file the connector reads and not as something baked into the template. That matters because the guest's disk is a thing that can be snapshotted, forked, or — on the bad day — dumped. Config, cursors and results can live on disk; the token should exist only in the memory of the process that needs it, in a VM that is destroyed at the end of the run. If your secret store can mint per-run credentials, do that too: a token that expires in fifteen minutes turns "we cannot rule out exfiltration" into "here is the expiry timestamp."

Egress: exactly that tenant's IdP hostnames, and nothing else

Isolation stops a compromised connector from reading other tenants' credentials. Egress control decides whether a compromised connector can send the one credential it does hold anywhere useful. These are different controls and you need both, because the whole value of an IdP token to an attacker is realised over the network.

• Default-deny outbound. Allowlist the tenant's IdP hostnames for the life of that run and nothing else — not your own API, not the package registry, not a metrics endpoint someone will inevitably want to add.
• Pin DNS to a resolver you control and log every query. A connector resolving a domain that is not the tenant's IdP is the highest-signal alert in this entire system.
• Block the cloud instance metadata endpoint explicitly. A link-local address that hands out credentials to anyone who asks is a poor neighbour for a process parsing untrusted JSON.
• No route to your internal network. The connector needs the IdP and a filesystem; it does not need your control-plane database, and the shortest path from SSRF to incident runs through an internal service that trusted the caller's IP.
• For on-prem LDAP, the allowlist is a private endpoint or tunnel to one customer's network — which is also the strongest argument for per-tenant VMs, because you cannot give one shared worker private connectivity to four hundred separate corporate networks and still call anything isolated.
• Give each run its own network namespace so the allowlist is per-VM rather than per-host. PandaStack pre-allocates 16,384 /30 subnets per agent for exactly this reason: every sandbox already has its own network to be lonely in.

#!/bin/bash
# /opt/run-sync.sh -- baked into the connector template, runs INSIDE the
# tenant's VM. Everything it needs is local; the only things that leave are
# the files under /work/out.
set -uo pipefail

IDP_HOST="$1"     # e.g. acme.okta.com -- the only host egress permits
MODE="$2"         # full | delta
CONNECTOR="$3"
mkdir -p /work/out

# 1. Verify the allowlist BEFORE we do anything while holding a token.
#    A connector that can reach the open internet is a connector that can
#    POST a bearer token to the open internet.
if curl -s --max-time 5 -o /dev/null https://example.com; then
  echo "FATAL: egress is open, refusing to run" >&2
  exit 78
fi
if ! getent hosts "$IDP_HOST" > /work/out/resolved.txt; then
  echo "FATAL: cannot resolve $IDP_HOST" >&2
  exit 78
fi

# 2. Hard timeout inside the guest as well as outside it. Full syncs are
#    slow; hung syncs are forever.
if [ "$MODE" = "full" ]; then LIMIT=5100; else LIMIT=480; fi
timeout --signal=TERM --kill-after=30 "$LIMIT" python3 "$CONNECTOR" --mode "$MODE" --config /work/config.json --cursor-in /work/cursor.json --cursor-out /work/out/cursor.json --out /work/out/changes.jsonl 2> /work/out/run.log
STATUS=$?
echo "exit=$STATUS mode=$MODE host=$IDP_HOST" > /work/out/status.txt

# 3. Redact before anything is readable from outside. The connector should
#    never print a token. "Should never" is a wish, not a control.
sed -E -i 's/(Bearer|SSWS) [A-Za-z0-9._~+=-]{8,}/\1 <redacted>/g; s/"(token|password|client_secret)": *"[^"]*"/"\1": "<redacted>"/g' /work/out/run.log

# 4. A manifest the customer's compliance team can read without trusting
#    our word for what happened.
python3 - <<'PY' > /work/out/manifest.json
import hashlib, json, os, time
p = "/work/out/changes.jsonl"
blob = open(p, "rb").read() if os.path.exists(p) else b""
print(json.dumps({
    "records": blob.count(b"\n"),
    "sha256": hashlib.sha256(blob).hexdigest(),
    "finished_at": int(time.time()),
}))
PY

exit $STATUS

Long full syncs versus fast incremental deltas

These are two different workloads wearing the same name, and treating them as one job is where per-run VMs get a bad reputation. A delta sync — poll for changes since a cursor, apply a handful of updates — is short, frequent, and should be cheap. A full reconciliation walks the entire directory, pages through everything, and reconciles against your own state; it is rare, slow, and bounded by the IdP's rate limits rather than by anything you control. Give them different TTLs, different exec budgets, different concurrency limits, and different alerting. A full sync taking an hour is normal. A delta sync taking an hour means something is wrong and nobody has noticed because the alert threshold was set for the full sync.

Idempotency and replay: a half-applied sync is worse than a failed one

The VM will die mid-run. The IdP will 429 you at page 40 of 60. The host will be drained. This is fine — provided the design treats a partial run as a non-event rather than as a partial truth. The failure that hurts is the one where the connector applied 300 of 500 changes, advanced the cursor, and left you permanently convinced you are in sync while a slice of the directory quietly drifts.

• Separate fetching from applying. The VM produces a changeset; your control plane applies it. A crash during fetch costs you a re-fetch, which is boring, and that is the goal.
• Advance the cursor only on a clean, verified finish. Never on a timeout, never on a partial page, never in a finally block written by someone in a hurry.
• Make every apply idempotent and keyed on a stable external id — the IdP's immutable user id, not email, because email changes and people get married.
• Give every run an id, put it in the VM metadata, and stamp it on every mutation. Replaying run 4471 should be provably a no-op the second time.
• Treat a full sync as a reconciliation with a diff you can inspect, not a blind overwrite. If the diff says you are about to deactivate 900 users, the correct behaviour is to stop and ask a human.
• Retry the run, not the record. A fresh VM restoring from a snapshot is fast enough that re-running the whole delta is simpler and safer than resuming a half-dead one.

The attribute-mapping expression is user-supplied code with a job title

Sooner or later an enterprise customer asks for a custom attribute mapping. Not a dropdown — an expression. "Set our role field to the second path segment of their distinguishedName, unless department is Contract, in which case use the extension attribute." Product ships a little expression box. It looks like configuration. It is a program, written by someone outside your company, that runs on your infrastructure against identity data. If it can call eval, or read files, or open a socket, you have added a remote code execution feature to your provisioning system and given it a settings icon.

# mapping_dryrun.py -- a customer's attribute-mapping expression is user
# code. Evaluate it inside the tenant's own VM, on a FORK, with no network
# and no credential anywhere in scope, and show them the diff.
import json
from pandastack import Sandbox


class MappingError(Exception):
    pass


# The evaluator runs in-guest. The restricted __builtins__ below is a
# guardrail against typos, NOT a security boundary -- the VM is the
# boundary. Anyone telling you a Python sandbox is the boundary has not
# read enough CPython.
EVAL = '''
import json, resource

resource.setrlimit(resource.RLIMIT_CPU, (5, 5))
resource.setrlimit(resource.RLIMIT_AS, (256 * 1024 * 1024,) * 2)
resource.setrlimit(resource.RLIMIT_NOFILE, (32, 32))

code = compile(open("/work/mapping.py").read(), "<tenant-mapping>", "exec")
safe = {"len": len, "str": str, "int": int, "sorted": sorted, "lower": str.lower}

out = []
for line in open("/work/sample.jsonl"):
    user = json.loads(line)
    scope = {"user": user, "result": None}
    exec(code, {"__builtins__": safe}, scope)
    out.append({"id": user["id"], "mapped": scope["result"]})

json.dump(out, open("/work/mapped.json", "w"))
'''


def dry_run_mapping(parent: Sandbox, mapping_src: str, sample_jsonl: str):
    """parent = a VM that already holds this tenant's fetched directory
    sample. Fork it so a mapping that wedges costs one child, not the run."""
    child = parent.fork()          # same-host fork lands in 400-750ms
    try:
        child.filesystem.write("/work/mapping.py", mapping_src)
        child.filesystem.write("/work/sample.jsonl", sample_jsonl)
        child.filesystem.write("/work/eval.py", EVAL)

        r = child.exec("python3 /work/eval.py", timeout_seconds=60)
        if r.exit_code != 0:
            # Hand the customer THIS, verbatim. They wrote the expression;
            # they are the only person who can fix it.
            raise MappingError(r.stderr[-2000:])

        return json.loads(child.filesystem.read("/work/mapped.json"))
    finally:
        child.kill()               # infinite loops included


# Save the mapping only after a human has looked at the diff. "This change
# reassigns 412 users out of the Engineering role" is a sentence somebody
# should read before it becomes an access-control event.
before = dry_run_mapping(parent, current_mapping, sample)
after = dry_run_mapping(parent, proposed_mapping, sample)
changed = [b["id"] for b, a in zip(before, after) if b["mapped"] != a["mapped"]]
print(f"{len(changed)} of {len(before)} sampled users change role under this mapping")

The dry run matters as much as the sandbox. A mapping expression is an authorization input; changing it silently reassigns roles across a customer's whole workforce at the next sync. Running it against a sample in a forked VM and showing the diff before saving turns "we deployed your mapping" into "here is what your mapping does, please confirm" — which is a better product and, not coincidentally, a better audit trail.

Deprovisioning: the failure with the longest fuse

Every other sync error is loud. Somebody's account did not get created, they cannot log in, they file a ticket within the hour. Deprovisioning is the mirror image: when an offboarding event fails to apply, nothing happens. No error surfaces to a user, because the user is gone. The account stays live, the session stays valid, and the discovery event is a security review nine months later, or a former employee's laptop, or a breach report that lists an account belonging to someone who left in March. Treat a silently dropped deactivation as an incident class of its own — because it is one, it just has a very long fuse. Alert on the absence of expected deactivations, reconcile them explicitly in the full sync, and make "we could not confirm this user is deactivated" a page rather than a log line.

Four places to run a tenant's connector

Same job, four topologies. Characterizations of any specific product's isolation, networking and billing behaviour should be verified against that vendor's own documentation — the details differ by configuration and they change.

• Credential blast radius — Shared worker fleet: every tenant's token passes through one process, so one code-execution bug is a fleet-wide identity-provider compromise. Container per tenant on a shared host: better, but a kernel-level escape still reaches the sibling containers holding everyone else's tokens. Long-lived VM per tenant: strong, and you now run one machine per customer forever. microVM per sync run: one token, one VM, one run, hardware-virtualization boundary, destroyed at the end.
• Tenant-specific code — Shared worker fleet: special cases accumulate inside the process that holds all the credentials, and every deploy is a fleet-wide risk. Container per tenant on a shared host: per-tenant images, which is real progress until you count how many you now build. Long-lived VM per tenant: clean, but drift is guaranteed and nobody rebuilds them. microVM per sync run: per-tenant config and mapping layered on a shared template, rebuilt from the snapshot every run, so drift has nowhere to accumulate.
• Egress control — Shared worker fleet: the allowlist must be the union of every tenant's IdP, which means every tenant's connector can reach every other tenant's IdP. Container per tenant on a shared host: per-container policy is achievable and frequently misconfigured. Long-lived VM per tenant: per-VM policy, correct and static. microVM per sync run: per-run policy scoped to one hostname, expiring with the VM, so a stale allowance cannot outlive the tenant that justified it.
• Long full syncs — Shared worker fleet: one tenant's two-hour reconciliation starves the queue for everyone else. Container per tenant on a shared host: better isolated, still competing for the same host memory and CPU. Long-lived VM per tenant: fine, and idle 99% of the time you are paying for it. microVM per sync run: a long TTL for full runs and a short one for deltas, with the machine existing only while the work does.
• Cleanup and forensics — Shared worker fleet: nothing to clean, and correspondingly nothing to examine, since the evidence is interleaved with 399 other tenants' logs. Container per tenant on a shared host: remove the container, plus whatever it did to the shared host, which by definition you cannot enumerate. Long-lived VM per tenant: state persists, which is good for debugging and bad for everything else. microVM per sync run: kill it and the credential, memory and partial state go together — with metadata and a per-run manifest as the record that survives.

Audit evidence, per run, that the customer's compliance team can read

The last thing per-run isolation buys you is the one that closes deals. Enterprise security teams ask what your connector did with their credential, and the shared-worker answer is a shrug wrapped in a SOC 2 report. The per-run answer is a record with edges: this run id, this tenant, this VM, this credential minted at this time and expired at that one, these hostnames reachable, this many records read, this changeset hash, this exit code. Put the identifying fields in the sandbox metadata and "show me every machine that held Acme's Okta token in Q3" becomes a query instead of a research project.

• Run id, tenant id, IdP, mode, and the credential's identifier and expiry — never the credential itself, in any log, ever.
• The egress allowlist as it was applied, plus every denied connection attempt. A denial log is the only evidence that the control was live rather than configured.
• Record counts and a hash of the changeset, computed inside the guest, so "what did this run actually read" is answerable without replaying it.
• Redacted run logs, redacted in-guest before anything reads them out — redaction on the collector is redaction after the token has already been on the wire.
• Deactivations attempted versus confirmed, as a first-class metric with its own alert. This is the number the customer's auditor will ask about, and it is the one people forget to emit.
• The VM's lifecycle: created, exec started, exec finished, destroyed, or destroyed by TTL. "Destroyed by TTL" appearing regularly is a signal your happy path is not running.

None of this is free. You are now operating a fleet of short-lived VMs, a per-run credential minting path, and a template pipeline, and a machine that no longer exists is harder to debug than a worker you can attach a profiler to. In exchange, the worst day stops being existential. A dependency compromise in your connector is a bad week and one customer's rotation, not four hundred disclosure emails explaining that you may have handed away their identity system. That is the trade, and once you have written the sentence you would have to send otherwise, it stops being a close call.