Per-Partner Isolation for B2B EDI and File Integrations

Nobody puts B2B file integration on a conference slide. There is no launch post for "we onboarded a 3PL over SFTP." And yet this is the layer that actually moves the physical world: purchase orders and advance ship notices between a retailer and its suppliers, 837 claims and 835 remittances between a provider and a payer, NACHA files between a company and its bank, container manifests between a freight forwarder and a port. It runs on formats ratified before most of the engineers maintaining them were born, and it runs at 3am because that is when the partner's mainframe finishes its batch window.

I'm Ajay; I built PandaStack. This post is about a specific structural mistake almost every integration platform makes: processing every trading partner's files in one shared worker fleet. That design couples partners who have no business relationship with each other, turns third-party transform code into an insider threat, and makes "a bad partner" into "a bad night for everyone." The fix is one ephemeral microVM per partner per batch, and the interesting part isn't the security argument — it's how much operational pain it removes.

What a partner integration actually looks like

If you have never worked inside one of these, the honest shape of it is worth spelling out, because the architecture arguments only make sense once you accept how irregular the input is. "We support X12" is a sentence with roughly the same information content as "we support files."

• Every partner has a dialect. X12 and EDIFACT are standards in the sense that they define an envelope and a segment vocabulary. What a partner actually sends is governed by their companion guide — a PDF describing which segments they use, which qualifiers they mean, which optional element is mandatory for them, and which field they have quietly repurposed to carry something the standard has no slot for.
• Delimiters are per-partner too. Segment terminator, element separator, sub-element separator: the interchange header declares them, and partners choose differently. Anyone who has hardcoded a tilde has learned this the interesting way.
• Every partner has a bespoke transform. Not configuration — code. A mapping script that turns their dialect into your canonical model, written by an integration consultant on a fixed-bid contract, tested against the six sample files the partner sent during onboarding, and untouched since.
• Transport is whatever the partner had in 2011. SFTP drops into a directory, AS2 posts with an MDN receipt, a legacy VAN mailbox, sometimes a literal email attachment. Files land on a schedule you do not control and cannot change.
• There is a temporary patch, and it is load-bearing. Partner 4417 sends dates as MMDDYY except in one segment where it's YYMMDD, so there is an `if partner_id == 4417` branch with a comment that says TODO and a date four years in the past. It has never failed. Nobody will touch it.
• Acknowledgements are part of the contract. A 997 or 999 functional acknowledgement, an AS2 MDN, a CONTRL message — the partner's system is waiting on it, and "we processed it but never acked" is indistinguishable from "we lost it" from their side.

None of that is dysfunction. That is what integrating with fifty independent companies looks like, and it is not going to become uniform because you wrote a nicer abstraction. The design question is not how to make partners the same. It is where to put the boundary so that their differences stay their own.

Why one shared worker is the wrong shape

The default architecture is a queue and a pool of worker processes. A file lands, a message goes on the queue, a worker picks it up, loads the partner's map, transforms, writes to your system, sends the ack. It is a reasonable first design and it fails in four distinct ways, each of which I have watched happen.

The 900MB interchange nobody warned you about

A partner's system has a bad day — a reprocessing job, a full resend, a bug that repeats a loop — and instead of the usual four megabytes you get nine hundred, or four megabytes containing one unterminated segment so your parser treats the entire file as a single element. The worker's memory climbs. The kernel's OOM killer wakes up and reaps whichever process looks fattest, which may be the offending batch or may be a completely unrelated partner's ordinary 800-line invoice run, dying two-thirds of the way through with rows already committed.

So one partner's incident becomes every partner's incident, at 3am, in a batch window that closes. And the defense people reach for — wrapping the parse in a `try/except MemoryError` — does not work, because by the time the allocator fails you are already in an unrecoverable state, and because the process that dies is chosen by the kernel on the basis of resident size rather than on the basis of whose fault it was. A memory ceiling has to be enforced by something outside the process. A hypervisor is exactly that thing.

The transform script is third-party code you accepted

The mapping script is the part everyone under-classifies. It was written by an outside consultant, often to the partner's specification, sometimes handed to you by the partner directly. It is arbitrary code, it runs in your worker, and your worker holds the credentials for every other partner's endpoints, your ERP connection, and the object storage bucket holding everyone's drops. Nobody wrote a data-exfiltration path; one emerged from running third-party code next to a credential set with total reach. It is the same structural bug as customer-defined ETL transforms, which I wrote up in /blog/microvm-per-tenant-etl-pipeline-isolation, wearing a supply-chain hat instead of a self-serve one.

The shared drop directory

Then there is the filesystem. Shared workers mean a shared staging area — `/var/spool/edi/inbound`, or the equivalent bucket prefix — where every partner's decrypted files sit next to each other while they wait to be processed. A transform script that reads a path it was not given, or a parser exploited through a malformed segment, is one directory listing away from another partner's purchase orders, pricing, and volumes. Those are competitors. The commercial damage of partner A reading partner B's order volumes does not require anyone to be malicious, only for a glob pattern to be one character too greedy.

PII and PHI in the shared error log

The quietest one. When a parse fails, what does your worker log? Usually the offending segment, because that is the only thing that makes the failure debuggable. In a healthcare integration that segment contains a member ID, a name, a date of birth, a diagnosis code. In a payments integration it contains an account number. That log line goes to the shared log stream, which goes to the shared search index, which is visible to whoever supports the integration platform — and if you run this as a service on behalf of your own customers, partner B's support engineer can now search partner A's claims data. Error logs are a data-residency and PHI surface that almost nobody scopes, because logs feel like operations rather than data.

Every partner integration is an accepted third-party dependency with a file interface. Most teams review the contract and never review the blast radius.

One ephemeral guest per partner, per batch

The alternative is to make the unit of isolation match the unit of work, which here is obvious and specific: one partner, one batch. Spin a microVM when a batch arrives, mount only that partner's files, give it only that partner's credentials, let its memory ceiling be enforced by the hypervisor, and delete it once the acknowledgement is durably recorded. The guest holds no connection to your system of record — it emits a canonical, validated artifact, and a trusted orchestrator does the writing.

• Only that partner's bytes are in the machine. Not a shared spool with a permission model — an empty filesystem into which exactly one interchange was written.
• Only that partner's egress is reachable. Default-deny, with a single allowed destination: the host that partner acks to. A transform script that decides to POST somewhere is stopped by a firewall, not by a code review.
• Memory and CPU are budgeted by the VMM. The 900MB interchange exhausts a guest with a fixed RAM allocation and takes down exactly one partner's batch — its own.
• Logs are pre-scoped by construction. One guest, one partner, one stdout stream. There is no filtering step to get wrong, and no way for a segment dump to land in a stream someone else can read.
• The runtime is per-partner. The guest's image is the partner's pinned runtime, not the fleet's.
• It ceases to exist. The decrypted file, the temp directory, the transform script, the credential, and the machine are destroyed together at the same instant.

The reason this is practical rather than aspirational is provisioning cost. On PandaStack a create is 179ms p50 and roughly 203ms p99, because every create restores a baked Firecracker snapshot on demand — around 49ms for the restore step — rather than cold-booting. The ~3s cold boot happens once, at bake time. Two hundred milliseconds in front of a batch that takes minutes is not a tradeoff you have to think about. If provisioning cost thirty seconds, you would pool workers and accept shared fate, which is precisely why everyone did.

from pandastack import Sandbox
import hashlib
import json

def process_partner_batch(partner: dict, batch_id: str,
                          interchange: bytes, transform_src: str) -> dict:
    """Run ONE trading partner's batch in a machine that exists only for it.

    What is deliberately NOT in this guest: any other partner's files, the
    ERP credential, the shared spool directory, and the log stream that the
    rest of the platform writes to. The guest is pure -- it emits canonical
    documents and an ack, and the trusted orchestrator does the writing.
    """
    file_sha = hashlib.sha256(interchange).hexdigest()

    sbx = Sandbox.create(
        # Per-partner runtime pinning lives here: partner 4417 stays on the
        # image its 2019 mapping script was written against, and everyone
        # else moves on without a migration meeting.
        template=partner["template"],          # e.g. "base" or a pinned bake
        ttl_seconds=1800,                       # a wedged parse cannot run all night
        metadata={
            "partner": partner["id"],
            "batch": batch_id,
            "kind": "edi-inbound",
            "doc_type": partner["doc_type"],   # 850, 810, 837, ORDERS, ...
            "file_sha256": file_sha,
        },
    )
    try:
        # The partner's bytes, their profile, and their map. Nothing else.
        sbx.filesystem.write("/work/inbound.edi", interchange)
        sbx.filesystem.write("/work/profile.json", json.dumps({
            "segment_terminator": partner["segment_terminator"],
            "element_separator": partner["element_separator"],
            "component_separator": partner["component_separator"],
            "date_format": partner["date_format"],   # explicit; never inferred
            "companion_rules": partner["companion_rules"],
        }, sort_keys=True))

        # This is third-party code. Treat it as such: it gets a kernel of its
        # own and a firewall, not a linter and a hopeful comment.
        sbx.filesystem.write("/work/transform.py", transform_src)

        # Lock egress to this partner's endpoint BEFORE the map runs.
        sbx.exec(f"bash /work/egress.sh {partner['ack_host_ip']} {partner['ack_port']}",
                 timeout_seconds=30)

        # Snapshot the INPUT state -- decrypted, staged, pre-transform. A
        # failed batch is replayed by forking this, not by re-fetching the
        # file and hoping the world looks the same as it did at 3am.
        replay_point = sbx.snapshot()

        out = sbx.exec(
            "cd /work && python3 -m edi.run --input inbound.edi "
            "--profile profile.json --transform transform.py "
            "--out canonical.ndjson --ack ack.997 --report report.json",
            timeout_seconds=1500,
        )

        # FAIL CLOSED. An OOM, a segfault in the parser, a timeout, or a
        # rejected envelope all land here and all produce zero documents.
        if out.exit_code != 0:
            return {
                "status": "failed",
                "file_sha256": file_sha,
                "replay_point": replay_point,          # fork this to retry
                # Truncate hard: this stderr contains partner segments, which
                # means member IDs, DOBs, account numbers. It goes to THIS
                # partner's log, and nowhere else.
                "stderr": out.stderr[-4000:],
            }

        return {
            "status": "ok",
            "file_sha256": file_sha,
            "report": json.loads(sbx.filesystem.read("/work/report.json")),
            "canonical": sbx.filesystem.read("/work/canonical.ndjson"),
            "ack": sbx.filesystem.read("/work/ack.997"),
        }
    finally:
        # File, temp files, transform, credential, machine: gone together.
        sbx.kill()

Default-deny egress, with one hole shaped like the partner

The transform script needs the network for approximately one thing: talking to that partner's endpoint, and often not even that, since your orchestrator can handle the ack. Everything else it might reach — your internal services, the cloud metadata endpoint, another partner's AS2 host, an S3 bucket, the open internet — is either a mistake or an exfiltration path. So the correct posture is default-deny with a single explicit allowance, applied before the map executes.

Two details matter more than the rule syntax. First, resolve the partner's hostname in the trusted orchestrator and pass an IP into the guest; a hostname resolved inside the guest is a DNS-controlled target, and "allow whatever this name points at right now" is a rebinding hole with extra steps. Second, block the link-local metadata address explicitly even when you think default-deny covers it, because you will eventually add a temporary allow rule for debugging and forget which order the chains evaluate in. The general treatment of this is in /blog/controlling-network-egress-untrusted-code; what follows is the partner-shaped version.

#!/usr/bin/env bash
# /work/egress.sh -- run INSIDE the guest, before the partner's map executes.
#
#   $1 = partner endpoint IP  (resolved by the TRUSTED orchestrator, not here:
#        a hostname resolved in the guest is an attacker-steerable target)
#   $2 = partner endpoint port (443 for AS2, 22 for SFTP, ...)
set -euo pipefail
PARTNER_IP="$1"
PARTNER_PORT="$2"

# Default deny. Not "deny the bad things" -- deny, then name the one exception.
iptables -P OUTPUT DROP
iptables -P FORWARD DROP
iptables -F OUTPUT

# Loopback only. The map talks to itself and to one host on the planet.
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Belt and braces: the cloud metadata service, explicitly, above everything.
# Default-deny already covers it. Someone will add a debug rule at 3am anyway.
iptables -I OUTPUT 1 -d 169.254.0.0/16 -j REJECT --reject-with icmp-admin-prohibited

# RFC1918 stays closed. The transform has no business inside your VPC --
# not your ERP, not your queue, and emphatically not another partner's host.
iptables -A OUTPUT -d 10.0.0.0/8     -j REJECT --reject-with icmp-admin-prohibited
iptables -A OUTPUT -d 172.16.0.0/12  -j REJECT --reject-with icmp-admin-prohibited
iptables -A OUTPUT -d 192.168.0.0/16 -j REJECT --reject-with icmp-admin-prohibited

# The single hole, shaped exactly like this partner and nobody else.
iptables -A OUTPUT -d "$PARTNER_IP" -p tcp --dport "$PARTNER_PORT" -j ACCEPT

# No DNS allowance at all: every destination this batch may reach is already
# an IP literal. If the map needs to resolve something, that is a finding.

# Fences the map cannot argue with, under a RAM ceiling the VMM enforces.
ulimit -v 3145728    # 3 GiB address space -- the parser dies, the host does not
ulimit -t 900        # 900s CPU -- a pathological regex gets reaped
ulimit -c 0          # no core dumps of a file we are about to destroy

export TZ=UTC LC_ALL=C.UTF-8   # EDI dates are ambiguous enough already

Ordering, retries, and the replay you will need at 4am

Isolation is the easy half. The half that determines whether you sleep is what happens when a batch fails, because in B2B integration a failed batch is not a lost request you can shrug at — it is a purchase order that a warehouse is waiting on, and the partner will not resend just because you asked nicely. Some will, once, from a human, in four hours.

The rule that makes this tractable: snapshot the input state, not the output. Once the file is fetched, decrypted, and staged, and the profile and map are in place, take a snapshot. That is your replay point. A retry forks it — 400-750ms on the same host, 1.2-3.5s cross-host — and re-runs the transform against byte-identical input in a byte-identical environment. You are not re-fetching from an SFTP server that may have rotated the file out, not re-decrypting with a key that may have rolled, and not relying on the partner still having the interchange. And when the failure turns out to be your bug rather than theirs, you fix the map and replay the exact bytes that broke it.

1. Key idempotency on the interchange, not on your attempt. `(partner_id, interchange_control_number, sha256(file_bytes))` identifies the work. Partners re-send the same ISA13 far more often than anyone expects — after a VAN hiccup, after their own retry logic fires, after a human reruns the batch job — and a duplicate must collapse to a no-op rather than duplicating a purchase order.
2. Sequence within a partner, parallelize across them. Document order matters inside one relationship: an 850 must land before the 860 that amends it. It never matters between two partners who have never heard of each other. So the correct concurrency model is a per-partner ordered lane and unlimited fan-out across lanes, which is exactly what per-partner guests give you for free.
3. Make the guest pure. It emits canonical documents, a report, and an ack — it holds no connection to your system of record. A crash therefore produces zero documents rather than half a partner's order file, which is the difference between a retry and a reconciliation project.
4. Ack only after the commit is durable. The acknowledgement is the partner's proof you have the data. Sending a 997 before your transaction commits means a crash between the two makes you the only party who knows the order is missing.
5. Quarantine rather than drop. A batch that fails validation goes to a per-partner quarantine with its replay point attached, visible in an operator UI, with the rejected segments and reasons. "Refused, here is exactly which segment and why" is a conversation with the partner. "It didn't work" is a week.

// Fan out one sandbox per partner batch. Ordered within a partner, parallel
// across partners -- because partner A's file cannot depend on partner B's.

type Batch = { partnerId: string; batchId: string; controlNumber: string };

async function runNightly(batches: Batch[]) {
  // Group into per-partner lanes. Within a lane, strict order: an 850 has to
  // land before the 860 amending it. Across lanes, no ordering exists at all.
  const lanes = new Map<string, Batch[]>();
  for (const b of batches) {
    (lanes.get(b.partnerId) ?? lanes.set(b.partnerId, []).get(b.partnerId)!).push(b);
  }

  const results = await Promise.allSettled(
    [...lanes.values()].map(async (lane) => {
      const done = [];
      for (const batch of lane) {          // sequential inside the lane
        // Same interchange control number + same bytes => already handled.
        // Partners resend constantly; a duplicate must be a no-op, not a
        // second purchase order for the same 4,000 units.
        if (await alreadyProcessed(batch.partnerId, batch.controlNumber)) {
          done.push({ ...batch, status: 'duplicate' });
          continue;
        }
        // One guest per batch. A partner whose file OOMs its own machine
        // stalls its own lane and nothing else in the building.
        done.push(await processInSandbox(batch));
      }
      return done;
    }),
  );

  // A rejected lane is ONE partner having a bad night. That is the whole
  // point: the failure is already scoped to the party responsible for it.
  for (const r of results) {
    if (r.status === 'rejected') await pageIntegrationOncall(r.reason);
  }
  return results;
}

Pinning an old runtime for one partner, not for everyone

This is the benefit that sells the architecture internally, because it is not about security at all — it is about being allowed to upgrade things.

In a shared fleet, every partner's map runs on one runtime, so the fleet's version is a negotiation with your least-maintained integration. Partner 4417's script uses a library that was abandoned in 2018 and does not build on a modern interpreter. Nobody is going to rewrite it: the consultant is gone, the partner is a top-ten account, and the script has run correctly every night for six years. So the whole fleet stays on the old runtime, and that constraint spreads outward — you can't take the security patch, can't use the new parser, can't onboard the partner who needs the modern TLS stack. One dead integration holds the platform hostage, which is how a platform becomes a legacy platform.

When the unit of isolation is a guest per partner, the runtime becomes a per-partner property. Partner 4417 gets a template baked with their ancient stack, pinned by name in their profile, and it stays frozen as long as the relationship lasts. Everyone else gets the current image. Onboarding a partner that needs something unusual is a new bake rather than a fleet-wide migration, and the archaeological integration stops being an argument you have to win. That is the same immutable-baseline property that makes batch workloads reproducible generally, which I covered in /blog/microvm-batch-job-isolation.

Shared worker pool vs microVM per partner batch

• Shared worker pool — Blast radius: one partner's 900MB or malformed interchange drives the pool into OOM and the kernel reaps whichever process is fattest, killing unrelated partners' overnight runs mid-commit. Third-party code: the partner's mapping script executes next to credentials reaching every other partner and your ERP. Data at rest: a shared spool directory where every partner's decrypted files sit side by side. Logs: parse failures dump partner segments — member IDs, DOBs, account numbers — into one shared stream. Runtime pinning: impossible; the fleet runs at the version of your least-maintainable integration forever. Replay: re-fetch from the partner's SFTP and hope the file and the environment are still what they were at 3am.
• Container per partner batch — Blast radius: cgroup memory limits contain the OOM to one batch, which is a genuine improvement over the pool. Third-party code: a shared kernel, so a kernel-reachable bug in the parsing path is a host problem, and the syscall surface is the real boundary. Data at rest: per-batch volumes are achievable but depend on getting the mount and permission model right every time. Logs: scoped by convention and a logging config, so one misconfigured sidecar re-merges everyone. Runtime pinning: yes — per-image runtimes are the thing containers are actually good at. Replay: re-run the image with the same inputs, assuming you kept them.
• MicroVM per partner batch — Blast radius: RAM and vCPU budgeted by the VMM, so a pathological interchange exhausts its own guest and its own guest only; no `try/except MemoryError` required and no shared OOM killer to arbitrate. Third-party code: a hardware-virtualized guest with its own kernel, so a successful exploit owns a throwaway machine containing exactly one partner's file. Data at rest: an empty filesystem into which one interchange was written, destroyed with the machine. Logs: one guest, one partner, one stdout — scoping is structural rather than configured. Runtime pinning: per-partner templates, so an abandoned 2018 stack is one partner's problem instead of the fleet's ceiling. Replay: fork the pre-transform snapshot (400-750ms same-host, 1.2-3.5s cross-host) and re-run against byte-identical input in a byte-identical environment.

The cost line is the one that decides it. A create is 179ms p50 and about 203ms p99 because every create restores a baked snapshot rather than cold-booting, and each PandaStack agent pre-allocates 16,384 /30 network subnets, so a nightly window where two hundred partners drop files within the same ten minutes is bounded by host CPU and memory rather than by network slots. The isolation costs a fifth of a second on a job measured in minutes. The shared pool costs you one shared incident per year, minimum, at the hour when nobody is awake.

The summary

B2B file integration has all the properties of untrusted-code execution and none of the reputation. The input is arbitrary bytes from an outside company, on a schedule you don't control. The transform is third-party code you accepted as a deliverable. The parser is permissive by necessity, because a strict one rejects the partner's real files. And the whole thing runs in a worker holding credentials that reach every partner you have, writing failures containing PHI into a log stream that everyone can search.

One ephemeral microVM per partner per batch collapses that whole family. The malformed interchange exhausts a machine with a hypervisor-enforced ceiling instead of taking down the pool. The consultant's mapping script gets its own kernel and a firewall with exactly one hole in it, shaped like that partner's endpoint. Only that partner's files exist in the guest, so a greedy glob finds nothing to read. Error logs are scoped by construction rather than by a filter someone has to maintain. The abandoned 2018 runtime is pinned for the one partner who needs it instead of freezing the fleet. And because the guest is pure and the replay point is a snapshot of the input state, a failed batch is a fork away from a clean retry rather than a phone call asking a partner to resend.

The partner who changed a field width without telling anyone is still going to change a field width without telling anyone. That is not a problem architecture can solve. What architecture can decide is whether their 3am mistake is their outage or everybody's.

For adjacent patterns: the self-serve version of hostile file input is in /blog/microvm-per-tenant-csv-import-pipeline-isolation, the general multi-tenant isolation model is in /blog/microvm-saas-multi-tenant-isolation, and network containment for untrusted code is in /blog/controlling-network-egress-untrusted-code.