How to build a multi-tenant audit log your customers' auditors will accept

The audit log is the feature that arrives in the security questionnaire, three weeks before a contract, phrased as a single innocent line: "Does the platform maintain an audit trail of administrative actions?" Everyone answers yes. Then someone opens the codebase and finds a `log.Info` call with a nicely formatted sentence in it, retained fourteen days, sampled under load, living in the same index as three million health-check lines.

I'm Ajay; I built PandaStack, a Firecracker microVM platform where customers create sandboxes, run code inside them, deploy apps and hold managed databases. Those are exactly the actions someone eventually needs to prove happened, or prove did not. Here is how to build the trail properly, including the part of it we still do not ship as a product.

Application logs and audit trails are two different products

This is the distinction that gets conflated, and conflating it produces a system that is bad at both jobs. Application logs exist so that you can debug your platform. Audit trails exist so that your customer can answer a question about their own tenant, usually to a third party who is skeptical by profession.

Different audience, different lifetime, different guarantees:

• Audience — Application logs: your on-call engineer at 3am. Audit trail: the customer's security team, and eventually their auditor.
• Completeness — Application logs: best-effort. Sampling, dropped lines under backpressure and lossy buffering are all acceptable engineering tradeoffs. Audit trail: a missing event is indistinguishable from a deleted one, which is the whole problem.
• Schema — Application logs: free-text, evolving, whatever the developer felt like writing that day. Audit trail: a stable, versioned, machine-readable vocabulary that will still parse in four years.
• Retention — Application logs: days to weeks, whatever your bill tolerates. Audit trail: months to years, and the number is usually written into a contract.
• Mutability — Application logs: rewrite, reindex, re-ingest, backfill freely. Audit trail: append-only, and demonstrably so.
• Access — Application logs: your staff, not your customers. Audit trail: the customer first, hard-scoped to their tenant, with export they control.

If you try to serve both from one pipeline you get an audit trail that drops events under load and a debugging log you are contractually forbidden from deleting. Keep them separate. They can share transport; they must not share guarantees.

What an audit event actually has to carry

An audit event answers six questions: who, on behalf of which tenant, did what, to which thing, with what outcome, and from where. Miss any one of them and the record is decorative.

{
  "id": "01J9M2Q7X4V8K3PZ6R0T5N1YHD",
  "org_id": "9f1c1c4e-6b8f-4a2e-9a7b-3d2f5c8e1a44",
  "seq": 184392,
  "occurred_at": "2026-08-30T14:22:07.913Z",
  "action": "sandbox.exec",
  "outcome": "denied",
  "actor": {
    "type": "api_key",
    "id": "key_7f2a91",
    "label": "github-actions-ci",
    "on_behalf_of": "user_5512"
  },
  "target": {
    "type": "sandbox",
    "id": "sbx_c41d9a2b",
    "template": "code-interpreter"
  },
  "reason": "org_quota_exceeded",
  "request": {
    "id": "req_01J9M2Q7X2",
    "source_ip": "203.0.113.44",
    "user_agent": "pandastack-python/0.4.1"
  },
  "metadata": { "command_sha256": "3b1f...", "cwd": "/workspace" },
  "prev_hash": "a91c...",
  "hash": "7de0..."
}

The actor field everyone forgets

Actors come in three kinds and most schemas model two. There is a human, acting through a session. There is an API key, acting for a machine — and note that in the example above the key carries an `on_behalf_of`, because "the CI key did it" is a less useful answer than "the CI key that Dana minted did it". And then there is the platform itself.

The `system` actor is the one that gets left out, and it generates the most confusing support tickets. When our idle reaper hibernates a sandbox, when the health monitor restarts an app that failed two consecutive probes, when a host is drained and a database is rebuilt somewhere else — something happened to the customer's resources and no human touched a keyboard. If your trail cannot express that, every one of those events either vanishes or gets falsely attributed to the last human who logged in.

So: `actor.type` is one of `user`, `api_key`, `system`. For `system`, put the subsystem in `actor.id` — `idle-reaper`, `app-health-monitor`, `db-failover` — and put the triggering condition in metadata. "Why did my sandbox disappear at 2am" should be a query, not an investigation.

Record denials, especially denials

The instinct is to log successful state changes, because those are the ones that changed something. For security review, the denials are worth more.

A hundred successful sandbox creates from a CI key is a Tuesday. Forty denied attempts to read another org's sandbox, from one key, in ninety seconds, is the most interesting thing that will happen to your platform this month — and if you only record allowed actions, it is invisible. Same for a revoked key that keeps trying, or a member repeatedly attempting an owner-only billing change.

So `outcome` is a first-class field with at least `allowed`, `denied` and `error`, and every authorization check that returns 403 emits an event with a machine-readable `reason`. The cost is a few extra rows. The benefit is that "show me everything this credential was refused" is a one-line query instead of a research project.

Naming the actions: a vocabulary, not a sentence

Free-text descriptions rot. "User deleted sandbox" becomes "Sandbox was deleted by user" in a refactor, then "Removed sandbox" when someone tidies the wording — and your customer's SIEM rule grepping for "deleted sandbox" has been matching nothing for eight months. Nobody notices, because a rule not firing looks identical to nothing happening.

Use a `resource.verb` vocabulary, defined in one file, treated as an API contract. For a platform shaped like ours:

• sandbox.create, sandbox.destroy, sandbox.exec, sandbox.fork, sandbox.snapshot
• app.deploy, app.rollback, app.env.update, app.delete
• database.create, database.credentials.rotate, database.clone, database.delete
• org.member.invite, org.member.accept, org.member.role_change, org.member.remove
• apikey.create, apikey.revoke
• billing.plan.change, billing.payment_method.update
• audit.read, audit.export

Three rules keep it usable. Add verbs, never rename them — a renamed action is a breaking change for every customer detection rule downstream. Keep the verb present-tense and imperative rather than past-tense, so the string is a stable identifier rather than prose. And put the human-readable sentence in your UI's rendering layer, generated from the action plus its fields, never stored. The stored record is data; the sentence is a view, and views are allowed to change.

Where to write it, and the transactional trap

This is where the design gets genuinely difficult, and there are exactly three options.

Write after commit is the obvious one and it is wrong in a specific, unfixable way: the window between the business transaction committing and the audit write landing is a window in which a crash loses the event permanently. The action happened. The record does not exist. There is no way to reconstruct it, and worse, you cannot even tell that you lost it — the gap is silent.

Write inside the same transaction is correct and durable: either both rows land or neither does. The cost is coupling your hot path to your audit path. The audit table's write latency becomes your API's write latency, index bloat becomes a create-sandbox outage, and the table you are contractually forbidden from deleting from now sits in your most latency-sensitive code. For us that path is measured in milliseconds — a snapshot-restore create runs about 179ms at p50 — and I am not enthusiastic about hanging an unbounded, never-pruned table off it.

The outbox pattern is the middle road and it is what I would build. Inside the business transaction you insert into a small, hot, aggressively pruned `audit_outbox` table — a cheap insert, atomic with the action. A separate worker drains it into the durable store, computing the hash chain, and deletes the outbox row only after the durable write succeeds. That gives you at-least-once delivery, so make the drain idempotent on event id, and it gives you a queue-depth metric. Alert on that depth: a stalled audit pipeline is a compliance incident that presents as a completely healthy-looking product.

Append-only, enforced by grants rather than by good intentions

"We never update the audit table" is a policy. Policies are enforced by code review, and code review is a process that runs at 5pm on a Friday. Enforce it with database grants instead, so that the application's role is physically incapable of modifying history.

CREATE TABLE audit_events (
  id           uuid        PRIMARY KEY,
  org_id       uuid        NOT NULL,
  seq          bigint      NOT NULL,          -- per-org chain position
  occurred_at  timestamptz NOT NULL,          -- when it happened
  recorded_at  timestamptz NOT NULL DEFAULT now(),  -- when we wrote it
  action       text        NOT NULL,          -- 'sandbox.exec'
  outcome      text        NOT NULL CHECK (outcome IN ('allowed','denied','error')),
  reason       text,
  actor_type   text        NOT NULL CHECK (actor_type IN ('user','api_key','system')),
  actor_id     text        NOT NULL,
  actor_label  text,
  on_behalf_of text,
  target_type  text,
  target_id    text,
  source_ip    inet,
  user_agent   text,
  request_id   text,
  metadata     jsonb       NOT NULL DEFAULT '{}'::jsonb,
  prev_hash    char(64),
  hash         char(64)    NOT NULL,
  UNIQUE (org_id, seq)
) PARTITION BY RANGE (occurred_at);

-- Read patterns: a tenant's timeline, filtered by action or by actor.
CREATE INDEX ON audit_events (org_id, occurred_at DESC, id);
CREATE INDEX ON audit_events (org_id, action, occurred_at DESC);
CREATE INDEX ON audit_events (org_id, actor_id, occurred_at DESC);

-- The application can add to history. It cannot change it.
REVOKE ALL ON audit_events FROM app_rw;
GRANT INSERT, SELECT ON audit_events TO app_rw;
-- Deliberately not granted: UPDATE, DELETE, TRUNCATE.

-- Retention is a partition DETACH, run by a separate privileged role that
-- the API never authenticates as.
REVOKE ALL ON SCHEMA audit FROM app_rw;
GRANT USAGE ON SCHEMA audit TO app_rw;

Two details matter more than they look. `occurred_at` and `recorded_at` are separate columns because they are different facts, and when they diverge — a backlogged outbox, a clock problem — you want to see it rather than have it collapse silently into one number. And range-partitioning by time means expiring old data is a `DETACH PARTITION` run by an operator role, not a `DELETE` run by your API. Retention should look like an administrative act, because it is one.

The first thing a competent attacker does after getting in is edit the log. This is a strong argument for the log not being editable.

Tamper evidence, without over-engineering it

Grants stop your application. They do not stop someone with database superuser access, which after a serious compromise is exactly who you are worried about. The cheap, proportionate answer is a hash chain: each event's hash covers its own canonical content plus the hash of the previous event in that tenant's chain. Delete an event, edit a field, reorder two rows, and every subsequent link fails to verify.

import hashlib
import json

CHAIN_VERSION = "v1"  # freeze this; changing canonicalisation breaks old chains


def canonical(event: dict) -> bytes:
    """Byte-for-byte reproducible years from now: sorted keys, no whitespace."""
    body = {k: v for k, v in event.items() if k not in ("hash", "prev_hash")}
    return json.dumps(body, sort_keys=True, separators=(",", ":")).encode("utf-8")


def link(prev_hash: str | None, event: dict) -> str:
    h = hashlib.sha256()
    h.update(CHAIN_VERSION.encode())
    h.update((prev_hash or "GENESIS").encode())
    h.update(canonical(event))
    return h.hexdigest()


def verify(events: list[dict]) -> int | None:
    """Return the seq of the first broken link, or None if the chain is intact.

    `events` must be one org's chain, ordered by seq ascending.
    """
    prev = None
    for e in events:
        if e["prev_hash"] != prev or e["hash"] != link(prev, e):
            return e["seq"]
        prev = e["hash"]
    return None

Chain per tenant, not globally. A global chain forces every write through one serialisation point — a throughput ceiling you will hit, and cross-tenant coupling you do not want. Per-org chains serialise only within an org, and they let one tenant's export be verified standalone without handing them anything belonging to anyone else.

The cheap upgrade: once an hour, write each tenant's latest chain hash to a separate store with different credentials — a bucket with object lock, a different account, anything the application role cannot reach. Verification then does not depend on trusting the database that holds the log.

Retention is per-tenant and contractual

The default answer everyone reaches for is ninety days, because that is what the logging vendor's pricing tier suggested. It is the wrong shape of answer, because retention is not a technical parameter. It is a term in a contract.

One customer's regulator expects years. Another wants the minimum you can defend, because data you hold is data they can be compelled to produce. A third has a data-processing agreement saying deletion within thirty days of termination, and means it. A single global number violates at least one of those, and you find out which during a renewal.

So make retention a per-org field with a sane default, enforced by the partition-drop job, and record retention changes in the audit log itself — `org.audit_retention.change` is exactly the kind of action someone will want to prove was not quietly shortened before an incident. Keep a floor: refuse to go below whatever period your own incident response actually needs, because a tenant who sets retention to one day is also a tenant who will ask you what happened last week.

Exposing it: an API, and an export

An audit log nobody can query is an expensive way to feel compliant. The read path has two consumers: a human in your dashboard filtering to last Tuesday, and a security team that will never open your UI and wants the events in their own SIEM.

For the API: filterable by time range, action, actor and target; keyset-paginated rather than offset-paginated, because offsets over an append-only table shift under you as new events arrive; and scoped to the caller's org in the query itself, not in a wrapper someone can forget.

-- Keyset page. :org_id comes from the authenticated principal, never from
-- a request parameter. The (occurred_at, id) pair is the cursor.
SELECT id, seq, occurred_at, action, outcome, reason,
       actor_type, actor_id, actor_label,
       target_type, target_id, source_ip, request_id, metadata
FROM audit_events
WHERE org_id = :org_id                        -- non-negotiable, always present
  AND occurred_at >= :since
  AND occurred_at <  :until
  AND (:action IS NULL OR action = :action)
  AND (:actor_id IS NULL OR actor_id = :actor_id)
  AND (occurred_at, id) < (:cursor_ts, :cursor_id)
ORDER BY occurred_at DESC, id DESC
LIMIT 200;

That `org_id` predicate is the entire security model of the feature. An audit log with a tenant-scoping bug is a data breach with extra steps — a purpose-built index of everything interesting every other customer has ever done, complete with source IPs. Test it adversarially: assert that org A's caller sees zero rows from org B, on every read endpoint, with the org id also supplied as a query parameter to confirm it is ignored. Better, use row-level security, so the predicate is enforced by the database rather than by every developer who adds an endpoint.

The part we do not ship, stated plainly

PandaStack does not today offer a managed audit-log export — there is no Splunk or Datadog connector, no S3 delivery stream, no signed evidence bundle you can hand an auditor. I would rather say that than let you discover it in a procurement call. What exists today is the operational surface: orgs and members, API keys with prefixes you can attribute actions to, sandbox lifecycle, deploys, and database credential rotation.

The workaround is real: build the collector on your side. Poll the platform APIs on a schedule, keyed by a cursor you persist, normalise into your own event schema, and write into your own append-only store using the design above. You end up owning the retention policy and the export format, which for a company with an actual compliance obligation is where you wanted them anyway.

# Cursor-based collector sketch. Persist LAST_SEEN; never restart from zero.
LAST_SEEN=$(cat ./state/cursor 2>/dev/null || echo "2026-08-01T00:00:00Z")

curl -s -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  "https://api.pandastack.ai/v1/sandboxes" \
| jq -c --arg since "$LAST_SEEN" '
    .items[]
    | select(.created_at > $since)
    | {occurred_at: .created_at, action: "sandbox.create",
       target_type: "sandbox", target_id: .id,
       metadata: {template: .template, status: .status}}
  ' >> ./out/audit.ndjson

date -u +%Y-%m-%dT%H:%M:%SZ > ./state/cursor

That is deliberately crude — it reconstructs events from resource state rather than receiving them, so it will miss anything that was created and destroyed between polls. Treat it as the floor, not the design. If you want the events pushed rather than polled, webhooks are the right transport and the same schema applies on the receiving end.

Reading the audit log is an auditable action

The last piece is the one that feels like a joke until you need it. `audit.read` and `audit.export` are themselves audit events. Who looked, when, at what filter, over what time range.

There are two reasons. The plain one: an attacker who reaches the audit log reads it before they do anything else, because it is a map of your customer's infrastructure and staff, and the read leaves no trace anywhere else. The subtler one: your own staff. If a support engineer can view a customer's audit trail to debug a ticket, the customer is entitled to know that happened. Being able to say "here is every time a PandaStack employee looked at your log, and why" is worth more in a security review than most of the controls people spend months on.

It does mean your log now contains events about reading itself, which recurses exactly one level and then stops, because reading generates a read event and that read event is not itself read. Nobody has ever asked me a follow-up question about this.

If you are starting Monday

Write the action vocabulary first, in one file, before any code. Twenty verbs covering the actions a customer would care about: credential changes, membership changes, anything destructive, anything that touches money, anything that runs code. Create the table with the grants above on day one, because retrofitting append-only onto a table your ORM already has UPDATE on is a migration nobody enjoys.

Then instrument the authorization layer rather than the handlers. That is what makes coverage tractable: if every allow and deny flows through one function, you emit from one place and get denials for free, instead of chasing twelve endpoints and forgetting the thirteenth. Add hash chaining once the schema settles — it is fifty lines, and much harder to retrofit, because you cannot chain events you already wrote without hashes.

Ship the read API before the pretty timeline UI — the customer who cares about this will pipe it into their own tooling regardless. And confirm the specifics — retention periods, what counts as sufficient evidence, whether your controls satisfy the framework you are assessed against — with your own auditor rather than with a blog post. Including this one.