How to Vet a Code Execution Vendor's Security

If you're running AI-generated code, you've handed a security boundary to a vendor. Whatever your agent decides to execute, some other company's isolation is what stands between that and everyone else's data — including, in the multi-tenant case, yours.

Most security reviews of these vendors go badly, in a specific way: they ask for a SOC 2 report, receive one, and stop. SOC 2 tells you a company has processes. It tells you close to nothing about whether the thing executing untrusted code can be escaped. Those are different questions and only one of them is about your actual risk.

I build PandaStack, which is one of these vendors, so read this with that in mind. I've tried to write the questions I'd want a customer to ask me, including the ones where our honest answer isn't the flattering one.

Question 1: What is the isolation boundary, precisely?

This is the question that matters most and the one with the widest range of real answers. "Sandboxed" and "isolated" are marketing words covering wildly different security properties.

What you want to know is what an escape has to defeat. Roughly, in increasing order of strength:

• A language-level sandbox (restricted interpreter, seccomp filter, chroot). The boundary is the kernel's syscall surface plus whatever the runtime forgot to block. Historically this is where escapes come from.
• A container with hardening. Better, but tenants still share one kernel. A kernel privilege-escalation bug is a cross-tenant event.
• A user-space kernel that intercepts syscalls. Meaningfully reduces the host kernel surface at some performance cost.
• A hardware-virtualized VM. Each tenant gets its own kernel; an escape has to get through the hypervisor, which is a much smaller and more scrutinized surface.

Ask directly: does each tenant's code get its own kernel, or is a kernel shared? A vendor who answers clearly is telling you something real. A vendor who answers "we use industry-standard sandboxing" has told you nothing, and you should assume the weaker option until they say otherwise.

Question 2: What can the code reach on the network?

Isolation from other tenants isn't the whole risk. Code that can reach anything on the internet can exfiltrate whatever you put in the sandbox, mine cryptocurrency on your account, or attack third parties from an IP that has your vendor's name on it.

Worth asking:

• Is egress open by default, and can it be restricted per workload?
• Can sandboxes reach the vendor's own internal services or metadata endpoints? Cloud instance metadata is a classic pivot and should be unreachable from guest code.
• Can two sandboxes reach each other over the network, or is each one on its own isolated segment?
• What happens when abuse is detected — is there active monitoring, and what's the response?

On the last point, be suspicious of a vendor who says abuse never happens. Anyone offering free or cheap compute is a target for cryptocurrency mining, and the honest answer is that it happens, it's detected, and here's how fast. We block known mining protocol traffic at the network layer and trace incidents from a sandbox back to the account behind it, because we've had to.

Question 3: What happens to data after the sandbox exits?

Sandboxes are usually ephemeral, which people take to mean data evaporates. Sometimes it does. Ask what "ephemeral" actually means here.

• When a sandbox is deleted, is its disk securely discarded, or just unlinked and eventually reused?
• If snapshots or forks are supported, where do those live and how long are they retained? A snapshot is a complete copy of memory — including any secret that was in memory.
• Are logs of executed commands and output retained? For how long, and who can read them?
• If the platform stores state in object storage, is it encrypted at rest with a key you can reason about?

The snapshot question catches people out. If a platform can fork a running sandbox, then somewhere there's a file containing that sandbox's entire RAM. That's a genuinely useful feature and it's also a copy of your secrets sitting on a disk. Ask how long it's kept.

Question 4: How do secrets get in, and where do they end up?

Your agent needs an API key. How does it get into the sandbox? Environment variables set at creation are the common answer, and the follow-up matters more: are those values logged anywhere, do they appear in support tooling, and are they written to disk inside the guest where a subsequent snapshot would capture them?

A pattern worth asking for: secrets delivered at runtime with restrictive file permissions, not baked into images or snapshots. If a vendor bakes a template snapshot with your credentials in guest memory, every sandbox created from it starts with those credentials resident.

Question 5: What does the compliance answer actually cover?

Now the compliance conversation, in its proper place — after the technical questions, not instead of them.

SOC 2 Type II tells you an auditor tested that a company's stated controls operated over a period. That's genuinely useful for questions like whether access is reviewed and whether there's an incident process. It does not evaluate whether the hypervisor is configured correctly or whether tenant isolation holds. A vendor can hold a clean SOC 2 and have a weak execution boundary; those findings live in different documents.

So ask for both, and ask separately:

• Compliance: which attestations are held, by which auditor, covering which period, and covering which systems? Scope is where these get slippery — an attestation covering the corporate environment but not the compute platform is common and nearly useless for this decision.
• Technical: has the isolation boundary been penetration tested by a third party, and can you see the summary?
• Legal: is there a published DPA, and a current list of subprocessors? For GDPR this is not optional, and a vendor who can't produce a subprocessor list hasn't thought about it.
• Disclosure: is there a security.txt or a published contact for reporting vulnerabilities, and a stated response commitment?

Question 6: what can you verify yourself?

Some of this you can check in ten minutes rather than taking anyone's word for it.

from pandastack import Sandbox

sbx = Sandbox.create(template="code-interpreter", ttl_seconds=300)

# Own kernel, or shared with the host? A microVM reports its own.
print(sbx.exec("uname -a").stdout)

# Can guest code reach cloud instance metadata? This should fail.
print(sbx.exec("curl -s -m 3 http://169.254.169.254/ ; echo exit=$?").stdout)

# What else is visible from inside? A container often shows host-ish
# cgroup and mount details a VM does not.
print(sbx.exec("cat /proc/1/cgroup; ls /dev").stdout)

sbx.kill()

Run that against every vendor you're evaluating. It won't prove a boundary is sound, but it will quickly separate a real VM from a container with marketing on top, and the metadata check catches a genuinely dangerous misconfiguration.

The takeaway

The question isn't whether a vendor has a compliance badge. It's what an escape has to defeat, what the code can reach while it's running, and what's left behind when it's done. Those questions have concrete answers, and how readily a vendor gives them tells you a lot on its own.

A vendor who tells you plainly what they don't have is giving you more information than one whose page implies everything. Apply that standard to us too.