How to ship app logs to your own stack when the platform has no log drain

Let me start with the gap rather than bury it. PandaStack does not have a managed log drain. There is no field where you paste a Datadog endpoint and have your application's logs appear there. It is a real limitation and it is on the list.

What exists is a log stream you can follow over HTTP, and a runtime with outbound network access. Those two things are enough to build either of the two patterns below, and one of them is what I would recommend to a serious team even on a platform that does have a drain — because a drain that reads your stdout is strictly worse than your application emitting structured events on purpose.

What you have to work with

Your application's stdout and stderr are captured to a file inside the guest and served by an endpoint that supports following as a server-sent-event stream.

# your process's stdout/stderr, followed live
curl -N -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  "https://api.pandastack.ai/v1/apps/$APP_ID/runtime-logs?follow=1"

# the last chunk, without following
curl -sS -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  "https://api.pandastack.ai/v1/apps/$APP_ID/runtime-logs"

Two properties of this stream shape everything below. It is per-app, so a fleet means one connection per app. And it is a tail of a file inside a guest that gets replaced on every deploy, so it is not durable storage — which is exactly why you want the logs somewhere else.

Pattern 1: ship from inside the app (recommended)

The application writes structured events and sends them to your collector itself. No platform feature involved, nothing to configure on our side, and it works identically wherever the app runs — your laptop, our platform, a VM you own.

First, make the logs structured. If you are shipping plain text lines to a log aggregator, you are paying for storage of strings you will later parse with regular expressions, badly.

// logger.ts
import pino from "pino";

export const log = pino({
  level: process.env.LOG_LEVEL ?? "info",
  base: {
    service: "checkout-api",
    env: process.env.NODE_ENV,
    deploy: process.env.PANDASTACK_DEPLOY_ID,  // set it in the app env
  },
  formatters: { level: (label) => ({ level: label }) },
  timestamp: pino.stdTimeFunctions.isoTime,
});

// usage
log.info({ orderId, amountCents, latencyMs }, "order placed");
log.error({ err, orderId }, "payment capture failed");

The `base` object is the part that pays for itself. Every line carries the service, the environment and the deploy identifier, so "show me errors from the deploy we shipped at 14:00" is a filter rather than an archaeology project. Put the deploy id into the app's environment as part of your deploy step and it comes along for free.

Then transport. The most portable option is an OTLP endpoint, because everything ingests it — Datadog, Grafana's stack, Honeycomb, anything OpenTelemetry-shaped — and swapping vendors becomes a URL change instead of a code change.

import { logs } from "@opentelemetry/api-logs";
import { LoggerProvider, BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";

const provider = new LoggerProvider();
provider.addLogRecordProcessor(
  new BatchLogRecordProcessor(
    new OTLPLogExporter({
      url: process.env.OTLP_LOGS_ENDPOINT,
      headers: { "api-key": process.env.OTLP_API_KEY! },
    }),
    { maxQueueSize: 2048, scheduledDelayMillis: 5000 }
  )
);
logs.setGlobalLoggerProvider(provider);

Batch, do not send per line. And keep writing to stdout as well as shipping. When your collector is down or misconfigured — which is when you most need logs — the platform's own stream is the fallback that still works.

Pattern 2: a sidecar that tails the stream

Sometimes you cannot change the application. It is a vendored binary, or a legacy service, or you have forty apps and no appetite for forty pull requests. Then you run one small process outside the app that follows the log stream over HTTP and forwards it.

This is a functions-shaped job and it is about thirty lines. The mechanism is a long-lived SSE connection, a batch buffer, and reconnect logic — with the reconnect being the part everyone underestimates.

async function drain(appId: string) {
  const buf: string[] = [];

  setInterval(async () => {
    if (!buf.length) return;
    const batch = buf.splice(0, buf.length);
    await fetch(process.env.LOKI_URL!, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        streams: [{
          stream: { app: appId, source: "pandastack" },
          values: batch.map((l) => [String(Date.now() * 1e6), l]),
        }],
      }),
    }).catch((e) => console.error("forward failed", e));
  }, 5000);

  for (let attempt = 0; ; attempt++) {
    try {
      const res = await fetch(
        `${API}/v1/apps/${appId}/runtime-logs?follow=1`,
        { headers: { Authorization: `Bearer ${process.env.PANDASTACK_API_KEY}` } }
      );
      attempt = 0;                                   // reset only after connecting
      const reader = res.body!.pipeThrough(new TextDecoderStream()).getReader();
      for (;;) {
        const { value, done } = await reader.read();
        if (done) break;
        for (const line of value.split("\n")) if (line.trim()) buf.push(line);
      }
    } catch (e) {
      console.error("stream dropped", e);
    }
    await new Promise((r) => setTimeout(r, Math.random() * Math.min(2 ** attempt * 500, 30_000)));
  }
}

Three things in there are not decoration. The attempt counter resets after a successful connection, not after a successful read, so a connection that establishes and immediately drops still backs off. The backoff is jittered, so twenty of these restarting after a platform blip do not reconnect in lockstep. And forwarding failures are logged rather than thrown, because a sidecar that crashes when the log vendor has a bad minute is worse than no sidecar.

The honest limitation: this tails from the current position. Lines emitted while the sidecar was disconnected are gone. For audit logging that is disqualifying and you want pattern one. For operational visibility it is usually fine.

Which one

• You control the code and you care about the logs: pattern one. Structured events with real fields, batched to an OTLP endpoint, stdout kept as the fallback. This is better than any drain, because a drain can only ever re-parse strings your application already threw away the structure of.
• You cannot change the app, or you want coverage across a fleet without touching each one: pattern two. Accept the gap-on-reconnect.
• You have compliance retention requirements: pattern one, and verify delivery. An at-most-once tail does not satisfy an auditor and you should not pretend otherwise.
• You just want to look at what is happening right now: neither. Follow the stream directly from your terminal and move on with your day.

The thing that will surprise you is the bill

Log volume grows faster than traffic, because the natural response to an incident is to add logging and nobody ever removes it. Two habits keep this in hand.

Sample the boring things. Request logs for successful requests can be sampled at one in a hundred without losing anything you actually use them for; errors and slow requests go through unsampled. And put the level behind an environment variable so you can turn debug on for one app for one hour during an incident, then turn it off. A permanently debug-level fleet is how a logging bill quietly becomes larger than a compute bill.

Also: egress is metered. Shipping every line of a chatty service to an external endpoint is real bytes leaving the platform. Batching and sampling are not just cost controls for the log vendor's invoice.