Running Jobs With Customer-Supplied Cloud Credentials

There is a button in your product that says "Connect your AWS account." Behind it, a customer pastes an access key pair, or uploads a GCP service account JSON, or types a Snowflake password into a form. They do this because they want the thing your product does: sync their warehouse, scan their account for misconfigurations, run a Terraform plan, drop a nightly report into their own S3 bucket. And from that moment you are holding, in production, a credential that can read or write another company's infrastructure.

The uncomfortable part is not the storage. Everyone gets the storage roughly right — envelope-encrypted in KMS or Vault, decrypted on use. The uncomfortable part is the ten minutes when the credential is decrypted and in the memory of a worker process that is also, at that same moment, running four hundred other customers' jobs. That process has one address space, one filesystem, one environment block, and one exception reporter. The wall between customer A's credential and customer B's job is a variable scope.

I'm Ajay, I build PandaStack, a Firecracker microVM platform, so I spend a lot of time on the question of where exactly a boundary sits and what it is actually made of. This post is about the shape I think is correct for this specific problem: a microVM per job, credential injected into a machine that exists only for that one customer's work and is destroyed with it. I'll also be direct about the parts a microVM does not fix, because there are some, and one of them is the part most teams get wrong first.

Take an honest inventory of what you're holding

Before the threat model, the asset list. It is usually longer than the security review assumed, because these accrete one integration at a time:

• AWS long-lived access keys, or a cross-account role ARN plus an external ID. The role is better. Both end up as an STS session in your worker's memory.
• GCP service account JSON. A file on disk by convention, which is the worst possible convention here, because now the credential has a path and anything that can read the filesystem can read it.
• Snowflake, BigQuery, Databricks, or plain Postgres credentials for warehouse sync. Often with broader grants than anyone intended, because the customer's data engineer just wanted the integration to work on a Friday.
• Stripe restricted keys, GitHub App installation tokens, Slack bot tokens, Salesforce OAuth refresh tokens. Lower blast radius individually, and a very interesting collection when a single process holds a thousand of them.
• Kubeconfigs and SSH keys, for anything that claims to "manage your cluster" or "run this on your fleet."

Notice what these have in common. They are bearer credentials for someone else's systems, they are used by code that has to be somewhat general (because it handles every customer's weird setup), and the code path that uses them is the same code path for every tenant. That combination is the whole problem.

How a credential actually escapes a shared worker

The threat model people write down is "an attacker gets RCE on the worker." That's true but it's the boring end. Most real leaks of this kind are not dramatic. They're ordinary, and there are five paths worth naming:

1. The environment block. If you set AWS_SECRET_ACCESS_KEY on the worker process to configure a job, every other piece of code in that process can read it via os.environ, and any process running as the same user can read it from /proc/<pid>/environ. That includes a subprocess the job shells out to, and it includes a subprocess belonging to a completely different tenant's job on the same box.
2. The exception reporter. Your APM or crash reporter helpfully attaches local variables and environment to every stack trace. A customer's job throws inside the AWS SDK, and their secret key is now in a third-party SaaS, searchable, retained for ninety days, visible to your whole engineering org. This is the single most common way I've seen these credentials leave a company, and nobody ever calls it a breach.
3. The logs. Somebody debugs a broken integration by logging the whole config object. It ships. The credential is now in your log pipeline, replicated to your warehouse, and in the backups of both.
4. The dependency tree. Your job runner installs packages — maybe the customer's own requirements.txt, maybe just yours. A postinstall script or an import-time side effect in one transitively-pulled package runs with full access to that process's memory and environment. It does not need an exploit. You invited it in and gave it a shell.
5. Cross-tenant memory. Even without an escape, a use-after-free, a buffer over-read, a bug in a native extension, or a mis-scoped connection pool can hand one request's bytes to another request. Language runtimes are not tenant boundaries; they're just tidy.

Here's the depressing demonstration. On a shared host, this is a complete credential harvest, and it requires no vulnerability in anything:

# Running as the same UID as your job workers -- e.g. from a subprocess
# a customer's own job legitimately spawned, or a package postinstall.
# No exploit involved. This is just Linux working as designed.

for p in /proc/[0-9]*; do
  tr '\0' '\n' < "$p/environ" 2>/dev/null |
    grep -E '^(AWS_|GOOGLE_|GCP_|SNOWFLAKE_|STRIPE_|GITHUB_|DATABASE_URL)'
done

# And the file-shaped ones, since the convention is a path on disk:
find /tmp /run /home -maxdepth 3 -name '*.json' -newermt '-10 minutes' \
  -exec grep -l 'private_key' {} + 2>/dev/null

# Output: every credential every concurrent job on this host is using.

"We use a separate IAM role per customer" is necessary, not sufficient

This is the standard answer and it is a good one. Cross-account AssumeRole with an external ID, one role per customer, least-privilege policy attached — yes, do all of that. It is strictly better than long-lived keys and it should be your integration's front door.

But notice what it protects. Separate roles mean that customer A's credential cannot be used against customer B's account. That's a real property and it bounds the damage of a leak. What it does not do is prevent the leak, because all N of those distinct sessions are still materialized, one after another or concurrently, inside the same worker process. A compromise of that worker yields not one credential but a stream of them: every session minted for every job that runs while the attacker is resident. Separate roles turn "one breach, all customers" into "one breach, all customers, but with better attribution in the incident report."

The two controls are orthogonal and you want both. Scoping decides what a stolen credential can do. Isolation decides whether it can be stolen in the first place, and by whom.

The per-job microVM shape

The pattern is simple enough to state in one sentence: create a fresh microVM, hand it exactly one customer's credential, run exactly that customer's job, take the result out over a host-side channel, destroy the machine. The credential's lifetime and the machine's lifetime are the same interval, and that interval belongs to one tenant.

The reason this is a microVM and not a container is the same reason as always: a container shares the host kernel, so the boundary between two jobs is namespaces and a whole lot of hope. A container is a polite suggestion to the kernel. A Firecracker microVM boots its own guest kernel under KVM and exposes a handful of virtio devices instead of the full Linux syscall surface, which is the boundary AWS themselves put between Lambda customers. For the harvest script above, the difference is total: inside a per-job VM, /proc contains one job's processes, because there is nothing else on that machine.

# run_job.py -- one customer's credential, one machine, one lifetime.
from pandastack import Sandbox

def run_customer_job(customer_id: str, job_id: str, creds: dict[str, str],
                     script: str) -> dict:
    sbx = Sandbox.create(
        template="base",
        # TTL is a backstop, not the control. If your process dies between
        # create and kill, this is what stops a VM holding a live credential
        # from sitting around until someone notices the bill.
        ttl_seconds=1800,
        metadata={"customer_id": customer_id, "job_id": job_id},
    )
    try:
        # Job code goes in over the host-side filesystem API. The guest
        # never gets a token that can talk back to the control plane.
        sbx.filesystem.write("/work/job.py", script)

        # Credentials arrive on the exec call, not baked into the image and
        # not written into a template. They exist for the duration of this
        # one command, in a process tree that contains nothing else.
        env = " ".join(f"{k}={v}" for k, v in creds.items())
        r = sbx.exec(f"env {env} python3 /work/job.py",
                     timeout_seconds=900)

        return {
            "customer_id": customer_id,
            "job_id": job_id,
            "exit_code": r.exit_code,
            # Read results out through the host-side channel. Do NOT let the
            # guest push results anywhere itself -- that would need an egress
            # path and a credential of yours, which is the thing we removed.
            "result": sbx.filesystem.read("/work/out.json").decode(),
            "stderr": r.stderr[-8000:],
        }
    finally:
        # The machine, its RAM, its disk, and its network namespace go away.
        # So does the credential, whatever the job did with it.
        sbx.kill()

Two details in there do most of the work. The credential is passed on the exec call rather than set on the sandbox at create time, so its window is bounded by a command instead of by a machine. And the VM holds no identity of yours — no platform API key, no registry token, no object storage credential — so a fully compromised job has stolen exactly one customer's credential and nothing that would let it reach a second customer.

Environment variable or file, and the snapshot trap

Inside a single-tenant VM, the environment-block objection mostly evaporates: there is no neighbouring process to read /proc, because the only processes are this job's. Env vars are fine here, and they're the shape most cloud SDKs expect. For file-shaped credentials like a GCP service account JSON, write it to a tmpfs mount rather than the rootfs, so it never touches a block device that could be reflinked or snapshotted, and unlink it as soon as the SDK has loaded it.

# Inside the guest, before the job runs. tmpfs, mode 0700, and gone
# the moment the job exits -- including if it crashes.
mount -t tmpfs -o size=1m,mode=0700 tmpfs /run/cred
trap 'umount -l /run/cred 2>/dev/null' EXIT

install -m 0600 /dev/null /run/cred/sa.json
cat > /run/cred/sa.json    # written over the exec channel, not via argv
export GOOGLE_APPLICATION_CREDENTIALS=/run/cred/sa.json

python3 /work/job.py
# umount fires here. The pages were never on a disk to begin with.

This is worth internalising because the tempting optimisation is exactly the wrong one. "Set up the customer's environment once, snapshot it, restore per job" is a great idea right up to the moment the environment you snapshotted includes an authenticated session. Then you have built a credential distribution mechanism with excellent latency characteristics.

Egress: the credential still has to not leave

Isolation stops a compromised job from reading its neighbours. It does nothing about the job reading its own credential — which it legitimately can, it's right there in the environment — and posting it to a server the attacker controls. If the job can reach the internet, the credential can reach the internet. That's a one-line curl.

So the second control is a default-deny egress policy with an allowlist of the endpoints this specific job actually needs. Cloud-credential jobs are unusually well suited to this, because the destination list is short and derivable: the customer's cloud API endpoints in the customer's region, and nothing else. On PandaStack each sandbox gets its own Linux network namespace, veth pair, and TAP device — 16,384 pre-allocated /30 subnets per agent host — so "this job may talk to sts.amazonaws.com and s3.eu-west-1.amazonaws.com and nowhere else" is a property of one VM rather than a firewall rule somebody has to remember to delete afterwards.

• Default-deny outbound — Start from nothing leaves. Allowlist per job, generated from the integration's declared endpoints, not from a wildcard added during an incident three quarters ago.
• Treat DNS as an exfil channel — An open port 53 to arbitrary resolvers is a covert channel with a credential-sized payload budget. Point the guest at a controlled resolver that only answers allowlisted names.
• Block the metadata endpoint — 169.254.169.254 and metadata.google.internal hand out whatever ambient identity the host has. That is your identity, not the customer's, and a job that can read it has escalated from one tenant's credential to your platform's.
• Log the denials — A blocked outbound connection from a data-sync job is not noise. It is either a broken allowlist or a finding, and both are things you want to see the same day.

There's more depth on the mechanics in /blog/controlling-network-egress-untrusted-code. The short version: isolation and egress control are two different controls solving two different halves, and shipping only the first is a common and expensive mistake.

The real endgame: make the stolen thing expire

Everything above reduces the probability and the blast radius of a leak. Short-lived credentials reduce the value of a successful one, and they're the control I'd fight hardest for if I could only have one. A credential that expires in fifteen minutes and is scoped to one bucket prefix is a materially different incident from a long-lived key pair with an AdministratorAccess policy attached because that was easier to get approved.

The concrete version on AWS: your customer grants a role to your account with an external ID; you AssumeRole per job with an explicit session policy that intersects down to what this specific job needs, a short duration, and session tags that record which customer and which job this session is for. Those tags land in the customer's own CloudTrail, which means they can audit you — a feature, not a cost.

# mint.py -- one session per job, minted on YOUR side, handed to the VM.
# The customer's role trust policy requires the external ID; the session
# policy narrows the role's permissions further for this one job.
import json, boto3

def mint_for_job(customer, job_id, bucket, prefix, ttl=900):
    sts = boto3.client("sts")

    # Intersection semantics: the effective permissions are role policy AND
    # this document. You can only narrow, never widen -- which is exactly
    # the property you want when the job is about to run someone's code.
    session_policy = {
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
            "Resource": f"arn:aws:s3:::{bucket}/{prefix}/*",
        }],
    }

    r = sts.assume_role(
        RoleArn=customer["role_arn"],
        ExternalId=customer["external_id"],
        RoleSessionName=f"job-{job_id}"[:64],
        Policy=json.dumps(session_policy),
        # Session tags show up in the CUSTOMER's CloudTrail. They can see
        # exactly which of their jobs each API call belonged to.
        Tags=[
            {"Key": "pandastack:customer", "Value": customer["id"]},
            {"Key": "pandastack:job", "Value": job_id},
        ],
        # Shorter than the job's timeout is wrong; a little longer is right.
        DurationSeconds=ttl,
    )["Credentials"]

    return {
        "AWS_ACCESS_KEY_ID": r["AccessKeyId"],
        "AWS_SECRET_ACCESS_KEY": r["SecretAccessKey"],
        "AWS_SESSION_TOKEN": r["SessionToken"],
    }

Mint on your side, never inside the guest. The guest gets the derived session, not the thing that can mint sessions. If the job VM held the long-lived credential that calls AssumeRole, you'd have handed it the ability to mint a fresh credential every fifteen minutes forever, which rather defeats the point of the expiry.

Audit: one record per job, on both sides of the boundary

When a customer asks "what did your product do in my account last Tuesday" — and enterprise customers will ask, usually during procurement — you want a per-job record rather than a log-grep expedition. Tag the sandbox with the customer and job ID at create time so platform-side usage and lifecycle events carry the attribution, and record the credential's identity alongside it.

• Which credential — the role ARN and the STS session name or assumed-role ID, not the secret. This is what joins your record to the customer's CloudTrail.
• Which machine — the sandbox ID and its create and destroy timestamps. This is the exact window in which the credential could have been used at all.
• What ran — a hash of the job script or config, plus the resolved dependency lockfile if the job installs anything. "We ran the version you approved" is only provable if you recorded what you ran.
• What egress was permitted — the allowlist as applied, and every denial. A job that tried to reach an unexpected host is the highest-signal event in the whole system.
• Result and exit code, with the stderr tail redacted. Redaction on the way in, not on the way out of your log store, because your log store already replicated it.

The question is never "could this credential have been misused." It is "can you show me the interval in which it existed, the machine it existed on, and the code that was running there." A VM per job answers all three by construction.

The honest cost and latency trade-off

A VM per job sounds expensive, and whether it is depends entirely on your definition of "job." The startup cost on PandaStack is a snapshot restore rather than a cold boot: p50 179ms end to end, p99 around 203ms, with the snapshot load step itself in the 49–80ms range. A genuine cold boot, which happens only the first time a template is spawned before its snapshot is baked, is about 3s.

Set that 179ms against the workload. A warehouse sync is minutes. A Terraform plan is minutes. An account scan is tens of seconds at best because the cloud APIs you're calling are rate-limited and slow. Against any of those, a fifth of a second of setup is measurement noise, and you get a hardware-enforced boundary for it. The economics are also better than the always-on version, because an idle customer costs nothing: you don't keep a worker warm per tenant, you create on demand and kill when done.

The other honest cost is operational. Per-job VMs mean you now have machine lifecycles to reason about: leaked sandboxes if your process dies between create and kill (hence the TTL backstop), a timeout that must be larger than your longest legitimate job or your own reaper becomes the thing that kills a half-finished sync, and results that have to come out over a host-side channel instead of the job just writing to your database directly. That last one is a feature disguised as friction — it's why the guest holds no credential of yours — but it is friction, and you should budget for it.

What a microVM does not fix

The most important limitation, stated plainly: a microVM does nothing about a credential that was over-scoped when it was issued. If your onboarding docs tell customers to attach AdministratorAccess because your integration guide was written in a hurry and narrowing it generated too many support tickets, then a perfectly isolated, perfectly egress-controlled, fifteen-minute session still has full control of their account for fifteen minutes. Isolation bounds who can steal it. Scope bounds what it's worth. You need both, and scope is the one that's a product decision rather than an infrastructure one.

Three other things it doesn't fix, so nobody is surprised later:

• Your storage layer. If the encrypted credentials at rest are decryptable by a service account that half your fleet holds, the per-job VM is a strong door on a building with an open loading bay. The mint-side of this system is now the most valuable target you have; treat it accordingly.
• Malicious use by your own code. The boundary is between tenants, not between you and the customer. A bug in your sync logic that deletes the wrong bucket is fully authorised and executes perfectly. Isolation is not correctness.
• Data the job legitimately reads. A sync job's whole purpose is to pull the customer's data out; whatever it exports lands wherever you put it. Isolating the credential doesn't isolate the data the credential fetched, and the second problem is usually bigger than the first.

The summary

If your product asks customers for cloud credentials, the credential's danger window is not while it sits encrypted at rest — it's the minutes it spends decrypted in a worker shared with every other tenant, where an environment block, an exception reporter, or a package postinstall is enough to leak it with no exploit involved. Move that window into a microVM that exists for one job: mint a short-lived, session-policy-narrowed credential on your side, pass it on the exec call rather than baking it into any image or snapshot, default-deny egress to the endpoints the job actually needs, record the machine's lifetime and the session identity for audit, and kill the VM when the work is done. At a p50 create of 179ms, that boundary costs less than the first cloud API call the job makes. Then go and narrow the IAM policy you asked for in the first place, because that's the control that decides what the whole thing was worth stealing.