all posts

Isolating Per-Tenant CSV and Bulk Import Pipelines in MicroVMs

Ajay Kumar··10 min read

Every B2B SaaS product grows an import button. It's usually the second thing enterprise buyers ask about, right after SSO, and it is almost always built in a hurry by whoever lost the sprint planning coin toss. It ends up as a small endpoint that accepts a file, a worker that parses it, and a loop that inserts rows. Shipped in a week, never revisited. And it is, quietly, the single most hostile input surface in the entire product: arbitrary bytes, arbitrary size, arbitrary encoding, uploaded by anyone who can start a free trial, parsed by libraries you have never read, inside a worker that holds your production database credentials.

I'm Ajay; I built PandaStack. This post is about treating bulk import as what it actually is — untrusted-input processing on behalf of one tenant — and giving each import job its own disposable Firecracker microVM. Along the way: the resource-exhaustion family (zip bombs, billion laughs, the four-gigabyte single line), CSV formula injection and where the sanitization boundary belongs, spreadsheet parsers as a C dependency surface nobody audits, the encoding and locale bugs that silently corrupt a million rows, memory fences between tenants, per-tenant transform scripts as the untrusted code they are, and how to fail closed instead of leaving a half-imported customer database behind.

The framing I'd like you to leave with: a customer's CSV is not data, it is an executable specification for how your import worker will spend memory, CPU, and file descriptors. Most of them are polite. The interesting ones are legally valid CSV files and morally denial-of-service payloads, and nothing in the RFC forbids that combination.

The front door you left unlocked

Think about what your import path actually accepts compared to the rest of your API. Every other endpoint takes a JSON body with a schema, size limits, a validator, and a rate limiter. The import endpoint takes a file. Not a shape — a file. Then it hands that file to a parser whose job is to be maximally accommodating about malformed input, because customers export from Excel, from a 2009 ERP, from a Google Sheet with merged cells, and from a system that emits CP-1252 with a UTF-8 BOM stapled on the front.

So the parsing layer is deliberately permissive, the file is deliberately unconstrained, and the process doing the work is deliberately privileged — it needs write access to the tenant's tables, which in most architectures means write access to the table, which means it holds a credential that reaches every tenant. That combination is the whole problem in one sentence.

  • Unbounded size and shape — "CSV" describes a delimiter convention, not a size, a row count, a column count, or a line length. A file can be a hundred columns wide, ten million rows deep, or one row containing four gigabytes of unquoted text with no newline in sight.
  • Unbounded formats behind one file extension — .xlsx is a ZIP archive full of XML. .csv might be TSV, might be semicolon-delimited European CSV, might be UTF-16LE. Users rename things. Your content-sniffing code is now a parser selector driven by attacker-controlled bytes.
  • Permissive parsers by design — a strict parser fails the customer's real export and generates a support ticket. So you use the lenient one, which means the code path exercised by a hostile file is the code path least likely to reject anything.
  • Privileged workers — the importer writes to your primary datastore. Whatever compromises or wedges it is sitting next to the credentials that reach everyone else's rows.
  • Shared fate — a pooled worker fleet means tenant A's pathological file and tenant B's ordinary 5,000-row contact list are competing for the same RSS budget under the same OOM killer, which does not care whose import was reasonable.
Most upload endpoints are treated as a data problem. They are an execution problem wearing a data problem's clothes.

The first category of trouble doesn't involve any malicious code at all. These are well-formed documents that happen to make a parser allocate absurd amounts of memory or CPU. They're worth enumerating precisely, because each one defeats a different naive defense.

Decompression bombs

An .xlsx is a ZIP container. So is .ods, so is .docx, and a great many import endpoints also accept a plain .zip of CSVs because a customer asked once. A few kilobytes of highly compressible zeros expands to gigabytes on extraction, and nested archives multiply that. Your upload size limit — the one you're proud of, the one that says 25 MB — bounds the compressed size and tells you nothing at all about the decompressed size. If your validation is "file under 25 MB, proceed," you have validated the wrapper and not the contents.

XML entity expansion and its friends

Inside that ZIP is XML, and XML has entities, and entities can reference other entities. The billion-laughs attack is ten nested entity definitions that expand to a billion strings from a file you could send in a text message. Most modern XML parsers disable entity expansion by default now — but "most" and "by default" are doing heavy lifting there, and the setting travels badly across the four different XML libraries your dependency tree pulled in. External entity resolution (XXE) is the sibling: an entity that references a local file path or an internal URL, turning your import worker into a reader of files it was never meant to open. In a shared worker with metadata-service access on a link-local address, that is a very direct path to a credential.

The four-gigabyte single line

My favorite, because it defeats streaming. Everyone's advice for large imports is "stream it, don't load the whole file into memory," and that advice is correct right up until the file contains no newline. A streaming reader that iterates line by line will happily buffer until it finds a delimiter, and if the delimiter is four gigabytes away, your carefully streamed parser has just materialized four gigabytes in a single allocation. The quoted-field variant is worse: an opening quote with no closing quote means the parser is now in "inside a quoted field" state for the remainder of the file, and every newline it passes is data rather than a record boundary. One stray double-quote character turns a ten-million-row file into a single ten-million-row cell.

Quadratic blowups and cardinality bombs

Then there's the long tail. A spreadsheet with a sparse cell at row 1,048,576 makes some parsers allocate the full rectangle. Thirty thousand columns, each triggering a column-type inference pass. Pathological regex in your own validation rules meeting an input crafted for catastrophic backtracking. A CSV where every value is unique and your dedupe step builds a hash set the size of the file. None of these are exotic — most of them arrive by accident from real customers exporting from real systems, which is precisely why you cannot treat them as an attack you'll get around to.

The load-bearing insight: you cannot validate your way out of this class from the outside. Determining whether a file will blow up a parser generally requires running the parser. The only reliable defense is to run it somewhere the blowup is contained and cheap to lose — with a hard memory ceiling, a hard time ceiling, and no neighbors.

CSV formula injection: the attack on your customer's laptop

This one is different in an instructive way, because the victim isn't your server. Spreadsheet applications interpret a cell beginning with `=`, `+`, `-`, `@`, or a leading tab/carriage return as a formula. If a tenant imports a contact list containing a cell like `=cmd|' /C calc'!A0`, and your product later lets someone export that list back to CSV, and a user opens the export in Excel, and they click through the warning dialog — which they will, because they requested this file and it came from your trusted product — then a command has executed on their workstation. Formula injection is a persistence-and-delivery attack that uses your application as the courier.

Even without command execution, formulas can exfiltrate quietly. A `HYPERLINK` cell, a `WEBSERVICE` call, or an external data reference can push the contents of neighboring cells to an attacker-controlled URL the moment the sheet opens. Your export was the payload delivery mechanism; your database was the storage.

The design question is where the sanitization boundary sits, and this is where teams reliably get it wrong. Sanitizing on export is too late and too scattered — you'll have four export paths, a reporting feature, an email digest, and a third-party integration, and one of them will forget. Sanitizing in the application layer means every service that touches the field has to know the rule. The right boundary is the import guest: the throwaway machine that already has the untrusted bytes in it, that already exists solely to make hostile input safe, and that produces a normalized, typed, boring record on the way out.

Store the raw value, store the neutralized value, and know which one you're handing to which consumer. Prefixing with an apostrophe or a zero-width character mutates data, and a customer whose product SKU legitimately starts with a minus sign will notice. Decide it once, in the guest, and record what you did.

Your spreadsheet parser is a large pile of C

Parsing CSV in pure Python or Go is comparatively boring. Parsing XLSX is not. Follow the dependency chain of any mainstream spreadsheet library and you land in native code fairly quickly: ZIP decompression, an XML parser (libxml2 and its long CVE history), a date/time conversion layer, sometimes a compiled fast-path for cell parsing, and — the part people forget — image decoders. Spreadsheets embed pictures. Charts. OLE objects. A worksheet can carry a PNG, a JPEG, a TIFF, or an EMF, and if any part of your pipeline generates previews or extracts embedded media, you have just fed an attacker-supplied image into an image decoder, which is historically one of the most reliable places to find memory-corruption bugs in the entire software industry.

  • Memory-unsafe by construction — a decompressor, an XML parser, and an image decoder are three of the classic homes for heap overflows. You are running all three against bytes a stranger chose.
  • Transitively invisible — you audited your direct dependencies. The image decoder came in four levels down, through a rendering helper, through a chart-export feature you don't use.
  • Slow to patch, fast to exploit — native library CVEs land in your dependency tree via a base image rebuild, which is a schedule, not a control.
  • Language-level sandboxing doesn't apply — seccomp helps, but a memory-safety bug in a C library reached from a managed runtime is an escape from the managed runtime's guarantees by definition.
  • And the boring one — the parser doesn't have to be exploited to hurt you. It just has to segfault mid-import while holding a database transaction open.

A microVM is the correct answer here for the same reason it's the correct answer for running untrusted code generally: the guest has its own kernel, and a heap overflow in an XML parser inside a hardware-virtualized guest gets you a compromised throwaway machine containing exactly one tenant's spreadsheet. That is a bad afternoon rather than a breach notification.

Encoding and locale: how to silently corrupt a million rows

Now the failure mode that produces no alert, no error, and no crash — just wrong data that a customer discovers three months later. Import pipelines are absolutely riddled with ambient environment dependencies, and a shared worker fleet's environment is whatever the base image happened to ship.

  • Encoding detection is a guess — a file with no BOM could be UTF-8, CP-1252, Latin-1, or Shift-JIS. Charset detection libraries are statistical and confidently wrong on short files. Guess CP-1252 for a UTF-8 file and every accented character becomes mojibake, permanently, in your database.
  • Locale-dependent case conversion — the famous one: uppercasing the ASCII letter i under a Turkish locale yields a dotted capital İ, not I. Any code that normalizes an email address, a country code, or a header name with a locale-aware uppercase will produce different keys depending on the host's `LANG`. This has broken real deduplication logic in real products.
  • Decimal separators and thousands grouping — `1.234` is one thousand two hundred thirty-four in a German export and one point two three four everywhere else. Locale-aware number parsing turns a currency column into an off-by-one-thousand invoice. Locale-naive parsing turns it into a validation failure. Only one of those is loud.
  • Date order ambiguity — 03/04/2026 is two different days depending on who exported it. Any pipeline that infers date format from the data will infer correctly for the first thousand rows and then meet a row where the day exceeds twelve.
  • Timezone and DST — a naive timestamp interpreted in the host's local timezone rather than the tenant's produces a consistent, plausible, wrong answer, and it changes twice a year.
  • Unicode normalization — NFC versus NFD means two visually identical customer names hash differently, so your idempotency key and your dedupe both silently fail.

The fix is to make the environment an explicit, pinned input rather than a background condition. This is where snapshot-restore earns its place beyond security: a PandaStack template's first spawn is a real cold boot of about 3 seconds, and every create afterward restores that baked snapshot — roughly 49ms for the restore step, 179ms p50 end to end (~203ms p99). Every import job therefore starts from a byte-identical machine: same locale data, same timezone database, same ICU version, same parser versions. Pin `LC_ALL=C.UTF-8` and `TZ=UTC` in the guest, pass the tenant's locale expectations in as explicit parameters, and re-running an import six months from now behaves the same way it did the first time.

# Inside the import guest. The environment is a pinned INPUT, not ambient state.
#
# Every locale-sensitive operation in the pipeline reads these. Setting them
# explicitly is what stops a base-image bump from re-interpreting a customer's
# decimal separators on a Tuesday.
export LC_ALL=C.UTF-8      # no Turkish-i, no locale-aware number parsing
export LANG=C.UTF-8
export TZ=UTC              # naive timestamps resolve identically, forever
export PYTHONHASHSEED=0    # stable iteration order for anything hash-backed
export PYTHONUTF8=1

# Hard fences, belt-and-braces with the guest's own vCPU/RAM budget.
ulimit -v 3145728          # 3 GiB address space -- the parser dies, not the host
ulimit -f 4194304          # 4 GiB max file size written (staging, extraction)
ulimit -t 900              # 900s CPU -- catastrophic backtracking gets reaped
ulimit -c 0                # no core dumps of a file we're about to destroy

# Check the DECOMPRESSED size before extracting anything. The upload limit
# bounded the wrapper; this bounds the contents.
if [ "$(unzip -Zt import.xlsx | awk '{print $3}')" -gt 2147483648 ]; then
  echo "decompression bomb: refusing" >&2
  exit 65
fi

cd /work && python3 -m importer.run \
  --input import.xlsx \
  --declared-encoding "$DECLARED_ENCODING" \
  --decimal-separator "$DECIMAL_SEPARATOR" \
  --date-order "$DATE_ORDER" \
  --out staged.ndjson

Memory fences: tenant A's 9GB file should not be tenant B's problem

Here is the operational failure that actually pages people, and it involves no attacker whatsoever. A customer uploads a genuinely enormous export — nine gigabytes, because they've been in business twenty years and they exported everything. Your pooled import workers pick it up. Memory climbs. The kernel's OOM killer wakes up and does what it does: it picks the process with the fattest score and reaps it. That process might be the offending import. It might just as easily be an unrelated tenant's ordinary 5,000-row contact import, which dies two-thirds of the way through, mid-transaction, having already inserted four thousand contacts.

So one customer's honest bulk export becomes another customer's partially-imported database, and the second customer's support ticket says "the import says it succeeded but half my contacts are missing," which will consume an entire engineer for a day. There is no code review that prevents this. It's a property of sharing a memory space.

A microVM per import job puts a real fence there. The guest gets a fixed vCPU and RAM allocation baked into its snapshot; a pathological file exhausts that guest's memory and the guest's own OOM killer reaps the guest's own parser. Nobody else notices. Add a TTL so a wedged parse is reclaimed rather than lingering, and add the `ulimit` fences above so the failure happens predictably inside the guest rather than as an abrupt kernel-level execution.

from pandastack import Sandbox
import json

def run_import(tenant_id: str, job_id: str, upload_bytes: bytes,
               mapping: dict, transform_script: str | None) -> dict:
    """Parse ONE tenant's upload in a machine that exists only for this job.

    Nothing in this guest can reach another tenant's data, because nothing
    else is in it: no database credential, no shared cache, no sibling job.
    The guest STAGES records. It never writes to the production datastore.
    """
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=1800,        # a wedged parse cannot run all night
        metadata={
            "tenant": tenant_id,
            "job": job_id,
            "kind": "bulk-import",
        },
    )
    try:
        # The hostile bytes go in. Note what does NOT go in with them:
        # the warehouse DSN, the master API key, any other tenant's rows.
        sbx.filesystem.write("/work/import.xlsx", upload_bytes)
        sbx.filesystem.write("/work/mapping.json", json.dumps(mapping, sort_keys=True))

        # Per-tenant column-mapping logic can escalate into a real script.
        # Treat it as untrusted code, because that is exactly what it is.
        if transform_script:
            sbx.filesystem.write("/work/transform.py", transform_script)

        out = sbx.exec(
            "cd /work && bash fences.sh && python3 -m importer.run "
            "--input import.xlsx --mapping mapping.json "
            "--sanitize-formulas --out staged.ndjson --report report.json",
            timeout_seconds=1500,
        )

        # FAIL CLOSED. A segfaulting parser, an OOM, a timeout, or a bomb
        # detection all land here, and all produce exactly zero rows.
        if out.exit_code != 0:
            raise ImportFailed(job_id, out.exit_code, out.stderr[-4000:])

        report = json.loads(sbx.filesystem.read("/work/report.json"))
        staged = sbx.filesystem.read("/work/staged.ndjson")
        return {"report": report, "staged": staged}
    finally:
        # The guest, the customer's file, the extracted XML, the temp
        # files, and any transform script all cease to exist together.
        sbx.kill()

Column-mapping rules are untrusted code with a friendly name

Import features follow a predictable evolution. Version one is fixed columns. Version two adds a mapping UI, because customers' headers never match yours. Version three adds a transform per column — trim, uppercase, split full name into first and last. Version four adds conditionals. Version five adds a formula field. Version six adds a scripting hook, because an enterprise customer's product codes need a checksum digit computed and no DSL you design will ever cover that.

By version six you are executing customer-authored code inside a process that holds a credential reaching every tenant's rows, and it's called "advanced field mapping" in the UI, which is a lovely name for arbitrary code execution. The same reasoning applies as with customer-defined ETL transforms — I wrote about that shape in /blog/microvm-per-tenant-etl-pipeline-isolation, and about user-supplied automation scripts in /blog/sandbox-user-uploaded-scripts-automation. Expression-language sandboxes leak through attribute access and function registries; the moment customers want a date library or a checksum helper, the evaluator sandbox is over. Give the script a real kernel and nothing else, and throw the machine away when it's done.

# Runs INSIDE the guest. Two jobs: neutralize formula injection on the way
# in, and refuse to guess about anything that could corrupt data silently.

FORMULA_LEADERS = ("=", "+", "-", "@", "\t", "\r")

def neutralize(value: str) -> tuple[str, bool]:
    """Return (safe_value, was_modified).

    We do NOT mutate in place and forget about it -- the raw value is staged
    alongside, so a SKU that legitimately starts with '-' is still
    recoverable and the audit trail says what we touched and why.
    """
    if value and value.startswith(FORMULA_LEADERS):
        return "'" + value, True      # leading apostrophe: Excel treats as text
    return value, False


def decode_strictly(raw: bytes, declared: str) -> str:
    """Never let a charset detector make a silent decision for a million rows.

    Strict decode against the encoding the TENANT declared. If it fails, the
    import fails loudly and we ask a human, rather than writing mojibake to
    the database and finding out about it in a support ticket in October.
    """
    try:
        return raw.decode(declared, errors="strict")
    except UnicodeDecodeError as e:
        raise EncodingMismatch(
            f"declared {declared!r} but byte {e.object[e.start]:#04x} "
            f"at offset {e.start} is invalid -- refusing to guess"
        )


def parse_decimal(text: str, separator: str):
    """Explicit separator, passed in per tenant. No locale, no inference.
    '1.234' means ONE THOUSAND or ONE POINT TWO THREE FOUR and the file
    itself will never tell you which."""
    if separator == ",":
        text = text.replace(".", "").replace(",", ".")
    return Decimal(text)   # Decimal, not float: this is somebody's money

Stage in the guest, commit in the orchestrator

The worst import outcome isn't a rejected file. It's a half-applied one. A parser that dies at row 600,000 of a million leaves the tenant with 600,000 records, no error state they can act on, and a UI that either says "failed" (so they retry, and now they have 1,600,000 records, 600,000 of them duplicated) or says "succeeded" (so they don't notice until reconciliation). Partial imports are the reason import features acquire a reputation for being cursed.

The discipline that fixes it is the same one that fixes billing runs, which I covered in /blog/microvm-per-tenant-billing-usage-metering: the sandbox is pure. It parses, validates, normalizes, sanitizes, and writes a staged artifact. It has no database credential, so it structurally cannot write half a customer's contacts. The trusted orchestrator receives the staged output, checks it as a whole, and commits it in one transaction keyed on an idempotency key derived from the inputs.

  1. Parse to a staging artifact, not to your tables. The guest emits newline-delimited records plus a report: rows read, rows valid, rows rejected with reasons, columns inferred, sanitization actions taken, and a digest of the whole set.
  2. Derive the idempotency key from the input, not the attempt. `(tenant_id, sha256(file_bytes), mapping_version)` identifies the work. A retry of the same upload produces the same key and becomes a no-op; a deliberate re-import after a mapping fix produces a different one, on purpose.
  3. Validate wholesale before committing anything. If 4% of rows failed validation, that is a decision for the customer to make in a preview screen, not a decision for your worker to make silently at row 40,000.
  4. Commit in one transaction with a uniqueness constraint on the key. For very large imports, commit in bounded batches but track a durable high-water mark per batch, so a resumed import continues rather than restarts — and never leave a batch boundary that isn't recorded.
  5. Surface the report, not just a status. "Imported 812,144 of 1,000,000 rows; 187,856 rejected: 187,000 missing required email, 856 invalid date" is a support ticket that never gets filed.
// The TRUSTED orchestrator. Note what it holds that the guest never does:
// the database handle. The sandbox physically could not have written a row.

type ImportReport = {
  rowsRead: number;
  rowsValid: number;
  rowsRejected: number;
  sanitizedCells: number;
  digest: string;          // sha256 over the canonical staged records
};

async function commitImport(
  tenantId: string,
  fileSha: string,
  mappingVersion: number,
  staged: string,
  report: ImportReport,
) {
  // Same upload + same mapping => same key. Retries collapse; a deliberate
  // re-import after fixing the mapping gets a new key, deliberately.
  const idempotencyKey = `${tenantId}:${fileSha}:v${mappingVersion}`;

  // Whole-file decision, made once, by a human policy -- not by a worker
  // that quietly gave up at row 40,000.
  if (report.rowsRejected / report.rowsRead > REJECT_THRESHOLD) {
    return quarantineForReview(tenantId, idempotencyKey, report);
  }

  const records = parseNdjson(staged);
  if (records.length !== report.rowsValid) {
    // The guest's own accounting disagrees with what it handed us. Something
    // was truncated. Commit nothing.
    throw new Error(
      `staged ${records.length} records but report claims ${report.rowsValid}`,
    );
  }

  await db.transaction(async (tx) => {
    const inserted = await tx.insertImportRun(
      { tenantId, idempotencyKey, report },
      { onConflict: "doNothing" },   // unique index on idempotencyKey
    );
    if (!inserted) return;           // a duplicate run: no-op, not a re-import
    await tx.bulkInsert(tenantId, records);
  });
}

Streaming progress back out (because imports are slow and users are anxious)

A large import takes minutes, and a progress bar that sits at zero for four minutes is indistinguishable from a broken product. Per-job guests make this easier rather than harder, which surprises people who expect isolation to cost observability.

Have the importer write structured progress events to stdout — one JSON object per phase and per batch — and stream them out of the guest with a streaming exec, forwarding to the browser over SSE or a websocket. Because the guest runs exactly one tenant's job, every line on that stream belongs to that job. There's no filtering by tenant ID across an interleaved worker log, and no risk that a log line you show a customer came from someone else's import. Same for resource accounting: CPU seconds and peak memory for that guest describe that import, so when a customer asks why theirs was slow, you can show them rather than tell them.

  • Emit phase transitions — `extracting`, `sniffing_encoding`, `inferring_types`, `validating`, `staging`. Users forgive slow far more readily than they forgive silent.
  • Emit bounded-interval progress, not per-row — a progress event per 5,000 rows or per second, whichever is less frequent. Per-row events turn your own progress channel into the bottleneck.
  • Emit rejections with row numbers and reasons as they happen, so the preview screen fills in progressively instead of arriving as a wall at the end.
  • Keep the report as the durable artifact — the stream is for the human watching; the report is what you store, show later, and attach to the support ticket.
  • Log the guest's identity in the run record — one job, one machine, one log stream, one resource profile. Attribution is free when the unit of isolation matches the unit of work.

Shared worker pool vs microVM per import

  • Resource exhaustion — Shared worker pool: a decompression bomb or a 4GB single line drives the pool into OOM, and the kernel reaps whichever process looks tastiest, including an unrelated tenant's ordinary import mid-transaction. MicroVM per import: fixed vCPU/RAM baked into the guest plus a hard TTL, so the pathological file exhausts its own machine and nobody else's, and the guest is reclaimed automatically.
  • Parser exploitation — Shared worker pool: a heap overflow in libxml2 or an image decoder yields code execution in a process holding credentials that reach every tenant. Container per import: better blast containment, but the kernel is shared, so a kernel-reachable bug in the parsing path is a host problem. MicroVM per import: hardware-virtualized guest with its own kernel; a successful exploit owns a throwaway machine containing one tenant's spreadsheet.
  • Credential blast radius — Shared worker pool: the importer needs write access to the tenant table, which in practice means write access to every tenant's rows. MicroVM per import: the guest holds no database credential at all — it stages records and hands them out; the trusted orchestrator does the writing.
  • Determinism — Shared worker pool: locale, timezone data, ICU version, and parser versions are whatever the current base image shipped, so a rebuild can silently change how decimal separators are read. MicroVM per import: every job restores the same pinned snapshot generation, byte-identical, with the locale and timezone fixed as explicit inputs.
  • Partial-failure behaviour — Shared worker pool: a crash at row 600,000 leaves 600,000 rows behind and an ambiguous status, and the retry duplicates them. MicroVM per import: the guest is pure, so a crash produces zero rows; the orchestrator commits once, transactionally, keyed on idempotency.
  • Observability — Shared worker pool: progress and error logs interleave across tenants, so attribution means grepping by tenant ID and hoping. MicroVM per import: one job, one machine, one stdout stream, one honest resource profile.
  • Cost of isolation — Shared worker pool: effectively free per job, which is why everyone starts here and why every import feature eventually acquires an incident. MicroVM per import: on PandaStack a create is 179ms p50 (~203ms p99) because every create restores a baked snapshot — roughly 49ms for the restore step — rather than cold-booting (~3s, and only at bake time). That is not a container image pull; it is a fifth of a second in front of a job that takes minutes.

That last row is the one that makes the whole design practical rather than aspirational. "One VM per import job" sounds extravagant until the create is a fifth of a second. If provisioning cost thirty seconds, you'd pool workers and accept the shared fate; at 179ms, isolation is cheaper than the JSON serialization of the report. Capacity works out too: each PandaStack agent pre-allocates 16,384 /30 network subnets, so concurrent imports are bounded by host CPU and memory rather than by network slot exhaustion — which matters when a customer's Monday-morning migration means two hundred imports arrive at once.

One snapshot-specific gotcha that bites this workload in particular: a restored guest wakes believing it is bake day. If your importer stamps `imported_at`, resolves a relative date like "last quarter," or validates that a date isn't in the future, a frozen clock produces confidently wrong rows. Sync the clock on resume, and pass every date your validation depends on in as an explicit parameter. An import pipeline should never ask the machine what day it is.

The summary

The import button is the part of your product where you accept arbitrary bytes from anyone with an account and hand them to permissive parsers built on decades of C, inside a worker holding credentials that reach every customer you have. It's the highest-risk surface in most B2B SaaS products and it is almost always the least-defended, because it looks like a data-entry feature rather than an untrusted-code-execution feature.

One disposable microVM per import job addresses the whole family at once. Resource attacks — bombs, billion laughs, the four-gigabyte line — exhaust a guest with a fixed memory budget and a hard TTL instead of taking down a pool. Parser exploitation lands in a hardware-virtualized guest with its own kernel and one tenant's file in it. Formula injection gets neutralized at the one boundary that already exists to make hostile input safe, before it can reach a database and then someone's laptop. Encoding and locale become pinned, explicit inputs on a byte-identical snapshot rather than whatever the base image shipped this month. Per-tenant transform scripts get a real machine and nothing else. And because the guest is pure — it stages records and holds no database credential — a crash produces nothing at all, while the orchestrator commits once, transactionally, keyed on idempotency, so a retry can never double a customer's contact list.

You leave a door unlocked because you assume everyone who walks through it is a customer. Mostly they are. The fix isn't to interrogate everyone at the threshold — it's to make the room behind the door empty, cheap, and easy to burn down.

For adjacent patterns: untrusted webhook payload handling is in /blog/sandbox-untrusted-webhook-transformations, per-tenant object storage boundaries in /blog/microvm-per-tenant-object-storage-isolation, and keeping the underlying tenant datastores separate in the first place in /blog/per-tenant-database-isolation.

Frequently asked questions

Why isn't an upload size limit enough to protect a CSV import pipeline?

Because a size limit bounds the bytes you received, not the work those bytes cause. A few kilobytes of ZIP can expand to gigabytes on extraction, which matters immediately since .xlsx is a ZIP container. A ten-line XML file with nested entity definitions can expand to a billion strings. A 200 MB CSV with a single unclosed quote character makes the parser treat the entire remainder of the file as one field, so a streaming reader that looks safe on paper materializes hundreds of megabytes in one allocation. In general, determining whether a file will blow up a parser requires running the parser, which is why the reliable defense is not smarter pre-validation but running the parse inside a bounded, disposable guest with a hard memory ceiling and a hard TTL, where the blowup is contained and cheap to lose.

What is CSV formula injection and where should sanitization happen?

Spreadsheet applications interpret a cell starting with =, +, -, @, or a leading tab or carriage return as a formula. If a customer imports a value like =cmd|' /C calc'!A0, it sits harmlessly in your database until someone exports that data back to CSV and opens it in Excel — at which point a command can execute on their workstation, or a HYPERLINK/WEBSERVICE cell can quietly exfiltrate neighboring cells to an attacker-controlled URL. Your product is the delivery mechanism. Sanitizing on export is the wrong boundary because you will have several export paths, a reporting feature, and an integration, and one of them will forget. Do it in the import guest — the throwaway machine that already holds the untrusted bytes and exists to turn them into safe, normalized records. Store both the raw and neutralized values and record what you changed, so a SKU that legitimately begins with a minus sign is still recoverable.

How can a locale setting silently corrupt an import?

Because locale-aware operations produce different, plausible, wrong answers rather than errors. Uppercasing the ASCII letter i under a Turkish locale yields a dotted capital İ, so any normalization of emails, country codes, or header names produces different keys depending on the host's LANG — which quietly breaks deduplication. Decimal separators are worse: 1.234 is one thousand two hundred thirty-four in a German export and one point two three four elsewhere, and the file itself cannot tell you which. Date order (03/04/2026), timezone interpretation of naive timestamps, and Unicode NFC-versus-NFD normalization all have the same character: no crash, no alert, just wrong data discovered months later. The defense is to make the environment an explicit pinned input — LC_ALL=C.UTF-8, TZ=UTC, tenant expectations passed as parameters — on a snapshot-restored guest that is byte-identical on every run rather than a shared worker whose base image drifts.

Why should the import sandbox not have database credentials?

Because a guest that cannot write to your datastore structurally cannot leave a half-imported database behind. The worst import outcome is not a rejected file, it is a parser that dies at row 600,000 of a million: the tenant now has 600,000 records and an ambiguous status, and if they retry they get 1,600,000 with 600,000 duplicates. Keep the guest pure — it parses, validates, normalizes, sanitizes, and emits a staged artifact plus a report of rows read, valid, rejected, and sanitized. The trusted orchestrator receives that output, checks the whole thing (including whether the record count matches the guest's own accounting), and commits it in a single transaction with a uniqueness constraint on an idempotency key derived from the tenant, a hash of the file bytes, and the mapping version. A retry of the same upload then collapses to a no-op instead of doubling the data, and a crash in the guest yields exactly zero rows.

Isn't one microVM per import job wasteful compared to a worker pool?

It would be if provisioning were expensive, which is the assumption worker pools were built on. On PandaStack a create is 179ms p50 and about 203ms p99, because every create restores a baked Firecracker snapshot — roughly 49ms for the restore step — rather than cold-booting; the ~3s cold boot happens once, at bake time. That is a fifth of a second in front of a job that typically takes minutes, so isolation costs less than a rounding error on the total. Capacity holds up too: each agent pre-allocates 16,384 /30 network subnets, so concurrency is bounded by host CPU and memory rather than by network slots, which is what you want when a customer's migration Monday delivers two hundred imports at once. What you buy for that fifth of a second is a hard memory fence between tenants, a hardware-virtualized boundary around a parser stack built on decades of C, a pinned deterministic environment, and clean per-job attribution for logs and resource accounting.

Run code in a microVM in one API call.

49ms p50 cold start. Fork, snapshot, and scale to zero.

Start free
Written by Ajay Kumar, Founder, PandaStack.