Per-Tenant Isolation for CRM Data Enrichment

A GTM platform is, underneath the dashboards, a machine that takes a customer's contact list and makes it worth more. Pull the accounts out of their CRM, scrape the company websites, hit three or four third-party data APIs for firmographics and technographics, run an LLM over the result to classify and score each record, write the scores back into their Salesforce or HubSpot. Repeat nightly, for every customer, forever.

The architecture that falls out of that description on the first pass is a job queue and a pool of enrichment workers. Each job carries a tenant_id. The worker looks up that tenant's credentials, decrypts them, does the work, writes back, moves to the next job. It works, it's easy to reason about, and it is one of the most credential-dense and highest-consequence pieces of software you will ever operate — because every one of those credentials belongs to someone else, and the write side of the pipeline mutates a system of record that your customer's revenue team lives in.

I'm Ajay; I build PandaStack, a Firecracker microVM platform. This post is about why enrichment specifically is a bad fit for the shared-worker shape, and what changes when the unit of isolation becomes one microVM per tenant-job rather than one row in a table. I'll be concrete about the failure modes, including the one that isn't a security incident at all and is still the worst thing that can happen to you.

Take inventory of what one enrichment run holds

Before the threat model, the asset list, because it is longer than the design doc admitted:

• The tenant's CRM OAuth tokens. A Salesforce refresh token with the api and refresh_token scopes is, functionally, a permanent read-write session against that company's system of record. HubSpot private-app tokens are the same story with a different name.
• The tenant's own third-party data API keys. This is the part people underestimate. Customers bring their own Clearbit, Apollo, ZoomInfo, Cognism, or People Data Labs keys, because they already pay for those seats and they want the credits spent against their own account. You are now the custodian of a competitor-sensitive set of vendor keys, per customer.
• The tenant's LLM API key, or yours spent on their behalf. Either way it's a metered credential with a spend limit attached and no per-request scoping.
• The contact records themselves. Names, work emails, phone numbers, job titles, and whatever the customer stuffed into custom fields. This is personal data under GDPR whether or not the deck called it "B2B data."
• Whatever intermediate files the job wrote. The 40MB CSV pulled out of the CRM, the scraped HTML cache, the JSONL of API responses, the parquet the scoring step read. All of it sitting on a worker's disk in /tmp with a name derived from the tenant ID.

Now note the property that makes this different from a generic background job: the code touching all of that has to be somewhat general, because it handles every customer's weird CRM schema, and it is the same code path for every tenant. One process, N customers' keys, N customers' contact data, in memory, concurrently.

The credential blast radius is the core problem

In a shared worker pool, the boundary between customer A's ZoomInfo key and customer B's job is a variable scope. That's it. Not a namespace, not a cgroup, not a policy — a variable scope, enforced by everyone on the team writing correct code every time, including in the retry path, the error handler, and the one-off script somebody ran at 2am during an incident.

Here is what actually leaks these, in rough order of how often I've seen it:

1. The exception reporter. An enrichment job throws inside an HTTP client — a 429 from a data vendor, a malformed JSON response, a timeout scraping some company's website that returns a 200 with a 3MB JavaScript blob. Your APM helpfully attaches local variables and the environment block to the stack trace. The customer's API key is now in a third-party SaaS, retained for ninety days, searchable by your whole engineering org. Nobody calls this a breach and it is one.
2. The environment block. If you configure a job by setting CLEARBIT_API_KEY on the worker process, every library in that process can read it, and any process running as the same UID can read it out of /proc. That includes the subprocess your scraper spawned and the concurrent job belonging to a different customer.
3. Debug logging. Someone debugs a broken HubSpot integration by logging the whole config object. It ships. The token is in your log pipeline, replicated to the warehouse, and in the backups of both.
4. The dependency tree. Scraping and parsing pull in a lot of packages. A postinstall script or an import-time side effect in one transitively-pulled library runs with the worker's full privileges — access to its memory, its environment, and the network. No exploit is required; you invited it in.
5. Cross-tenant memory. A connection pool keyed wrong, a cached client object reused across jobs, a native extension bug. Language runtimes are tidy, not isolating.

The worst bug: the wrong tenant's data in the wrong tenant's CRM

Everything above is a security problem, and security problems at least have an incident process. The failure I'd actually lose sleep over in this category isn't a breach. It's a correctness bug with a write side.

The shape: a job for tenant A somehow ends up holding tenant B's CRM client, or tenant B's batch of contacts, and the write-back step runs. Maybe a cached Salesforce client keyed on instance URL rather than on tenant. Maybe a retry that re-read the job payload from a stale variable. Maybe a worker that pooled connections and handed the wrong one back. Maybe the classic: a backfill script written under time pressure that filtered on the wrong column.

A tenant_id column is a promise your ORM makes and your intern's one-off backfill script breaks. The column is real, the discipline around it is not enforced by anything.

What makes this the worst bug in the category is the combination of three properties:

• It's silent. Nothing errors. Salesforce happily accepts a valid API call with valid field values. Your job reports success, your metrics are green, your customer's dashboard shows the enrichment run completed. The only signal is a human noticing that a lead score looks wrong, which happens weeks later if at all.
• It's unrecoverable in practice. Once you've written into a customer's system of record, you can't cleanly take it back. Field history is polluted. Workflow rules fired. Automations sent emails. Lead-routing assigned owners. A sequence started. You cannot un-send the outbound email that your bad score triggered.
• It's a data leak with a write instead of a read. Tenant B's contact data is now inside tenant A's CRM, permanently, and it got there through a legitimate authenticated API call that your own product made. There is no attacker to blame in the postmortem.

In a shared worker, defending against this means defending every code path that could possibly conflate two tenants — which is every code path, since both tenants' objects are in scope simultaneously. In a per-tenant-job microVM, the defence is structural: the guest holds exactly one tenant's credentials and exactly one tenant's contact batch. There is no second CRM client to accidentally grab, because there is no second tenant on the machine. A mix-up bug still writes wrong data, but it can only write it back into the CRM whose token is present, which is the correct CRM.

Isolation doesn't make your code correct. It makes the set of things a bug can reach small enough that the worst outcome is bounded by construction rather than by review.

Egress, scraping, and whose IP got blocked

The enrichment pipeline's middle step is usually scraping — fetch the company homepage, the careers page, the pricing page, maybe a LinkedIn-adjacent source, and feed the text to the classifier. From a shared worker pool, all of that traffic leaves on the same handful of egress IPs.

Two consequences, both of which show up as support tickets rather than as alerts:

• One aggressive tenant rate-limits everybody. A customer uploads a 400,000-row list and asks for a full re-enrichment. Your workers hammer a Cloudflare-fronted target from three IPs. Those IPs get challenged, then blocked. Every other customer's enrichment run now fails on the same target, and your on-call sees "scraper broken" with no tenant attribution in the failure.
• You can't answer the abuse email. Someone's ops team writes in complaining about crawl volume from your IP range. From a shared pool you know the aggregate, not the tenant. "We'll look into it" is the only honest answer you can give, and it's the answer that ends with your ASN on a blocklist.

Per-tenant-job VMs change the unit of network identity. On PandaStack each sandbox gets its own Linux network namespace with a veth pair and TAP device — there are 16,384 pre-allocated /30 subnets per agent host — so "this job's traffic" is a thing with an address, not a share of a pool. That gives you three practical levers you don't otherwise have:

1. Per-tenant egress policy. Default-deny outbound, with an allowlist of the data-vendor APIs, the CRM host, and the scrape targets this job is actually supposed to reach. A scraping job that tries to POST to an unexpected host is the highest-signal event in the whole system — it's either a broken allowlist or a finding, and you want to see either the same day.
2. Attribution by construction. Route each tenant's egress through an address or proxy pool you can map back to a tenant, so an abuse report resolves to a customer and a job ID instead of to a shrug. It also means you can rate-limit the one tenant who's causing the problem rather than globally throttling to protect yourself from them.
3. Blast-radius containment on blocks. When a target does block you, it blocks the address the offending tenant was using. Everyone else's runs keep working, which converts a platform outage into one customer's problem with one customer's fix.

Customer-supplied scoring logic is untrusted code

Every GTM platform eventually ships custom scoring, because the ICP definition that works for a dev-tools company doesn't work for a medical-device company, and no amount of dropdowns covers it. So you add a Python or JavaScript hook: "write a function that takes an enriched record and returns a score." It's the feature that turns a rigid product into a platform.

It is also arbitrary code execution, sold as a configuration option. Once a tenant supplies the function, the only remaining question is what that function can reach — and in a shared worker, what it can reach is every other tenant's credential, in the environment, one os.environ away. The LLM-generated variant is no better; if your product writes the scoring function from a natural-language prompt, you've automated the production of code you never reviewed.

The mitigations people reach for first — a restricted interpreter, an AST allowlist, stripping dangerous builtins — are the ones that get bypassed. Sandboxing a language runtime from inside the same language runtime is a losing game with a long, entertaining history of CVEs. The boundary that holds is a kernel boundary, and the reason to prefer a microVM over a container here is the usual one: a container shares the host kernel, so the wall between two tenants' scoring functions is namespaces and optimism, while a Firecracker guest runs its own kernel under KVM and exposes a handful of virtio devices instead of the full Linux syscall surface.

Residency: a tenant's contacts must stay in one region

Contact data is personal data. A German customer's enrichment run processing their contact list on a us-east worker is a transfer, and "our job queue picks whichever worker is free" is not a lawful basis. This turns up in enterprise procurement long before it turns up in a regulator's letter, usually as a security questionnaire asking where the data is processed and whether you can prove it.

In a shared worker pool, the honest answer is "wherever the scheduler put it," and retrofitting region affinity means partitioning the queue, the worker fleet, the caches, and the credential store — a large project that touches everything. When the unit of work is a VM you create per job, residency becomes a placement decision at create time: this tenant's jobs are created on agents in eu-west, their contact batch is written into a guest in eu-west, the intermediate files live on that guest's disk in eu-west, and the machine is destroyed there. The audit story is a per-job record with a region and a machine ID rather than a paragraph about queue configuration.

There's a longer treatment of the placement mechanics in /blog/multi-region-sandbox-placement-data-residency. The point for this post is that per-job machines make residency a property you can state per record, which is exactly the granularity the questionnaire asks for.

The cached contact CSV that outlives the job

Enrichment jobs are file-heavy by nature. You pull a batch out of the CRM into a CSV because paginating a REST API 400,000 times is not a plan. You cache scraped HTML so a retry doesn't re-crawl. You write a JSONL of vendor API responses so the scoring step can read it without re-spending credits. Then the job finishes and, on a long-lived worker, all of that is still sitting on a shared disk.

The cleanup code exists, of course. It's in the finally block. It doesn't run when the process is OOM-killed, when the pod is evicted mid-run, when the host reboots, or when the job crashes in a way that skips the handler — which is precisely the set of situations where an incident is already in progress and nobody is thinking about /tmp. Six months later a routine review finds a worker with 40GB of other customers' contact exports on it, dating back to whenever that node last got replaced.

An ephemeral VM makes the cleanup unconditional. The disk is the guest's disk, it exists for the duration of the job, and it goes away with the machine whether the job succeeded, failed, crashed, or hung until the TTL reaper got it. "We delete the intermediate files" stops being a policy statement and becomes a description of the machine's lifecycle. And a TTL is a much better cleanup mechanism than a finally block, because a TTL fires on exactly the cases the finally block misses.

The architecture: one microVM per tenant-job

The shape in one sentence: your control plane holds encrypted credentials and decides what runs; for each tenant's enrichment batch it creates a fresh microVM, injects exactly that tenant's credentials, writes exactly that tenant's contact batch in, runs the enrichment, reads the results out over a host-side channel, and lets the machine die.

Note the direction of the write-back in what follows. The guest produces scored records; it does not call Salesforce itself in this version. That's a deliberate choice I'll defend in a moment.

# enrich.py -- one tenant, one batch, one machine, one lifetime.
import json
from pathlib import Path

from pandastack import Sandbox

# Our own enrichment driver: fetches vendor APIs, scrapes, calls the LLM, then
# hands each record to the tenant-supplied scoring function.
RUNNER_SRC = (Path(__file__).parent / "runner.py").read_text()


def run_enrichment(tenant_id: str, batch_id: str,
                   contacts: list[dict],
                   tenant_creds: dict[str, str],
                   scoring_script: str) -> dict:
    """Enrich one tenant's contact batch in a machine that holds nothing else."""

    sbx = Sandbox.create(
        template="base",
        # TTL is the backstop that makes cleanup unconditional. If our process
        # dies between create and kill, this is what stops a VM holding a live
        # CRM token and 40MB of contacts from sitting around until someone
        # notices the bill.
        ttl_seconds=600,
        metadata={"tenant": tenant_id, "batch": batch_id, "kind": "enrichment"},
    )
    try:
        # 1. The work goes in over the host-side filesystem API. The guest never
        #    receives a token that can talk back to our control plane.
        sbx.filesystem.write("/work/contacts.json", json.dumps(contacts))
        sbx.filesystem.write("/work/score.py", scoring_script)  # tenant-supplied
        sbx.filesystem.write("/work/enrich.py", RUNNER_SRC)     # ours

        # 2. Credentials arrive on the exec call, not baked into the image and
        #    never written into a snapshot. Only THIS tenant's keys, and only
        #    for the duration of this one command. Nothing else on this machine
        #    can read them because nothing else is on this machine.
        env = " ".join(f"{k}={v}" for k, v in tenant_creds.items())
        r = sbx.exec(
            f"cd /work && env {env} python3 enrich.py "
            f"--contacts contacts.json --score score.py --out scored.json",
            timeout_seconds=540,
        )

        # 3. Results come back out through the host side. We do the CRM
        #    write-back from the control plane, against the tenant we asked
        #    for -- see below for why.
        scored = json.loads(sbx.filesystem.read("/work/scored.json"))
        return {
            "tenant_id": tenant_id,
            "batch_id": batch_id,
            "ok": r.exit_code == 0,
            "records": scored,
            "stderr": r.stderr[-8000:],
        }
    finally:
        # 4. The machine, its RAM, its disk, its network namespace, the cached
        #    HTML, the contact CSV, and the credentials all cease to exist.
        sbx.kill()

Three details in there are doing most of the work, and they're worth calling out because each maps to a failure mode above.

Credentials on the exec call, never in a snapshot

Passing the credential on the exec rather than setting it at create time bounds its window by a command instead of by a machine. More importantly: never bake a tenant's credential into a snapshot. A Firecracker snapshot's memory file is a byte-for-byte freeze of guest RAM, so anything the guest held at capture time — a decrypted refresh token, a TLS session key, the vendor API key you exported two seconds earlier — is inside that file and inside every machine restored from it. Snapshot the template with its dependencies installed, which is where the latency actually lives; inject the credential after restore.

Do the CRM write-back from the control plane

This is the design decision I'd argue about, so here's the reasoning. Letting the guest hold the CRM token and write back directly is simpler and saves a hop. It also puts a permanent read-write credential for your customer's system of record inside the same machine that is executing customer-supplied scoring code and parsing scraped HTML. Given that the worst outcome in this whole category is a bad write into a CRM, I'd rather the write path be the smallest, most boring, most reviewed piece of code you own, running where no untrusted code does, and taking the tenant identity from the job record rather than from anything the guest returned.

If your enrichment genuinely needs interactive CRM reads mid-run — a lookup-then-decide loop rather than a batch — then scope the token as narrowly as the CRM permits (a Salesforce integration user with field-level access to exactly the objects you touch), give it the shortest session your flow tolerates, and treat the guest as holding a live credential, which means the egress allowlist and the audit record both become mandatory rather than nice.

The tenant's scoring script is just a file in the guest

Notice there's no restricted interpreter, no AST allowlist, no monkey-patched builtins. The tenant's script is written into the guest as a plain file and executed by a plain Python. It can import whatever it likes, open sockets the egress policy allows, and read every environment variable on the machine — all of which belong to the tenant who wrote it. The isolation is the machine, not the language.

Shared worker vs container-per-job vs microVM-per-tenant-job

• Credential blast radius — Shared worker with a tenant_id column: every tenant's CRM tokens and vendor API keys pass through one process, so one crash report, one debug log line, or one malicious dependency exposes all of them. Container per job: better, but secrets injected as env still sit behind a shared-kernel boundary, and a container escape reaches every neighbouring job on the host. MicroVM per tenant-job: only one tenant's credentials ever enter the guest; the others were never present to steal.
• Wrong-tenant write-back — Shared worker: both tenants' CRM clients are live in the same address space, so a cache keyed wrong or a stale variable in a retry path silently writes A's data into B's CRM. Container per job: one job per container removes concurrency inside the process, but the container still holds whatever the orchestrator injected and a queue bug can still hand it the wrong payload. MicroVM per tenant-job: the guest holds one token and one batch, so a mix-up bug can only write into the CRM whose token is present — and moving the write to the control plane makes it a reviewed path rather than an incidental one.
• Egress attribution — Shared worker: all scraping leaves on a shared pool of IPs, so one tenant's aggressive crawl gets everybody rate-limited or blocked, and an abuse report can't be resolved to a customer. Container per job: usually shares the host's network namespace or a bridged pool, so attribution is per-host at best. MicroVM per tenant-job: each sandbox has its own netns, veth, and TAP (16,384 pre-allocated /30 subnets per agent), so egress policy and IP attribution are per-job properties and a block hits one tenant, not the platform.
• Untrusted-code safety — Shared worker: a customer-supplied scoring function runs next to every credential you hold; in-language sandboxes are bypassable and have the CVE history to prove it. Container per job: namespaces and seccomp, sharing one kernel — better, and still a kernel-bug-away from the host. MicroVM per tenant-job: its own guest kernel under KVM with a virtio-only device surface, which is the boundary AWS puts between Lambda customers.
• Start cost — Shared worker: effectively zero, which is the whole reason the shape is tempting. Container per job: tens to hundreds of milliseconds, plus image pull on a cold node. MicroVM per tenant-job: on PandaStack, p50 179ms end-to-end (p99 ~203ms) because every create restores a baked snapshot rather than cold-booting; a true cold boot is ~3s and happens once, at bake time.
• Cleanup guarantees — Shared worker: cleanup is a finally block, which doesn't run on OOM-kill, eviction, or host reboot, so contact CSVs accumulate on shared disks. Container per job: the writable layer goes with the container, but mounted caches and host volumes persist. MicroVM per tenant-job: the disk is the machine's disk and dies with it, and a TTL reaps the cases where your own cleanup path never ran.

Set the 179ms against the workload before deciding it's expensive. An enrichment batch is dominated by network I/O you don't control: vendor APIs with per-second rate limits, page loads from company websites that were never fast, and an LLM classification pass measured in seconds per chunk. A fifth of a second of machine setup is noise against the first HTTP request the job makes. The economics are also better than the always-on version, because an idle tenant costs nothing — you create on demand instead of keeping a warm worker per customer.

What to record per run

When a customer asks what your product did in their Salesforce last Tuesday — and enterprise customers ask, usually during procurement rather than after an incident — you want a per-run record, not a log-grep expedition. Tag the sandbox at create time so lifecycle events carry the attribution, and record:

• Which tenant and which batch, plus the machine ID and its create and destroy timestamps. That interval is the exact window in which their credentials could have been used at all.
• Which credentials were injected — by identity, never by value. The vendor account, the CRM integration user, the OAuth scope set. This is what joins your record to the audit trail on their side.
• Which region the machine ran in. This is the residency answer, per run, and it costs nothing to record at create time.
• A hash of the scoring script and the resolved dependency set. "We ran the version you approved" is only provable if you wrote down what you ran.
• Every egress denial. A scraping job that tried to reach an unexpected host is either a broken allowlist or a finding, and both need a human the same day.
• The write-back diff: which CRM object IDs were touched, which fields, and the before value. This is the record that turns a wrong-tenant write from unrecoverable into merely expensive.

What a microVM does not fix

Being direct about the limits, because the failure mode of a post like this is a team that ships the VMs and thinks it's done:

• Over-scoped CRM tokens. If your onboarding tells customers to connect with a full-admin Salesforce user because a narrow permission set generated support tickets, a perfectly isolated ten-minute run still has full write access to their org for ten minutes. Isolation bounds who can steal a credential; scope bounds what it's worth. Scope is a product decision, and it's usually the higher-leverage one.
• Bugs in your own logic. The boundary is between tenants, not between you and the customer. A scoring pipeline that computes garbage and writes it back is fully authorised and executes perfectly. Isolation is not correctness — it just stops incorrectness from crossing tenants.
• The data you legitimately fetched. Enrichment's whole purpose is to pull contact data in and produce a scored copy. Wherever your control plane stores that copy is now personal data with the same residency and retention obligations as the source. Isolating the run doesn't isolate the output.
• Your credential store. If the encrypted tokens at rest are decryptable by a service account half your fleet holds, the per-job VM is a strong door on a building with an open loading bay. The mint side is now the most valuable target you own.
• Compliance with the vendor's terms. Scraping from per-tenant IPs makes attribution honest; it doesn't make a crawl permitted. Read the terms of the sources you fetch, and rate-limit per tenant because it's correct, not just because it's containable.

The summary

CRM data enrichment concentrates four uncomfortable things in one job: other companies' credentials, other companies' personal data, customer-supplied code, and a write path into a system of record. The shared-worker shape puts all four in one process for all tenants at once, where an exception reporter, a debug log, or a package postinstall is enough to leak the credentials with no exploit involved, and a mis-keyed client cache is enough to write tenant A's contacts into tenant B's Salesforce — silently, and past the point of recovery. Make the unit of isolation one microVM per tenant-job: inject one tenant's credentials on the exec call and never into a snapshot, write one tenant's batch in and read the results out over a host-side channel, default-deny egress to the vendors and targets the job actually needs so scraping is attributable per tenant, place the machine in the tenant's region so residency is a per-run fact, keep the CRM write-back on the control plane where no untrusted code runs, and let a TTL reap the machine so the cached contact CSV isn't a policy statement. At a p50 create of 179ms it costs less than the job's first HTTP request. Then go and narrow the CRM scope you asked for at onboarding, because that's the control that decides what any of it was worth.