Tracelane security and data residency

​

Data residency

​

Sigstore Rekor — an external dependency, deliberately not a sub-processor

Anchoring publishes each audit batch’s Merkle root to the public Sigstore Rekor transparency log (log2025-1.rekor.sigstore.dev). It is a live external dependency and is named here so it is visible. It is intentionally absent from our sub-processor tables. Rekor receives a hash — a Merkle root — and never a payload: no prompt or completion text, no identifiers, no ledger row contents. The audit payload is metadata-only by construction ({model, trace_id, warn_aft_id[, business_reference]}) and is what the root is computed over, so a root discloses nothing about a data subject and cannot be reversed into one. On that basis there is no processing of personal data to disclose, which is why Rekor appears as a dependency rather than as a sub-processor. That reasoning is recorded so it is not re-litigated each time the list is reviewed. It would be revisited if the anchored content ever stopped being a bare hash. Cloudflare R2 is a storage location, not a bare-hash dependency. Aged span parts are moved to a ClickHouse cold volume backed by Cloudflare R2 (EU jurisdiction), so Cloudflare holds customer span data and is a sub-processor for it on Tracelane Cloud. Region selection is not offered. Tracelane Cloud runs a single deployment: neither the span schema nor the control-plane schema carries a region dimension, so a tenant cannot be pinned to a region and no per-region isolation is claimed. If you need to control the jurisdiction your data sits in, self-host — you choose where ClickHouse and Postgres run.

​

Workspace deletion and backups

An owner can archive a workspace in Settings by confirming its name. The action marks its API keys revoked; authorization caches can delay enforcement. The scheduled purge selects workspaces archived for more than 30 days, then removes classified tenant rows from the online telemetry and control-plane stores. Failed purges need operator review and deletion-status confirmation. Audit ledger rows, chain state and anchors remain. Identity-provider deletion and subscription cancellation are best-effort steps that need separate confirmation if they fail. The online purge does not rewrite shared backups per tenant. Backups include an EU-jurisdiction copy and a secondary copy in Cloudflare’s default jurisdiction; their pruning and lifecycle rules are separate from the workspace recovery window. Primary EU storage therefore does not mean EU-only processing or immediate erasure of every copy. For a deletion-status confirmation or per-subject erasure request, contact [email protected]. See the service-provider inventory and deletion statement.

​

Provider key handling

Provider API keys (Anthropic, OpenAI, etc.) are envelope-encrypted at rest with AES-256-GCM (ring AEAD). The workflow:

1. Each provider key is sealed with AES-256-GCM under the deployment envelope master key (TRACELANE_BYOK_MASTER_KEY) or, when configured, a tenant data key wrapped by the customer’s KMS.
2. The ciphertext is bound with Additional Authenticated Data (AAD) = provider-key:<tenant_id>:<provider_id>, so a ciphertext cannot be replayed under another tenant or provider.
3. The ciphertext (plus 16-byte GCM tag) is stored in Postgres. The plaintext key never persists.
4. For dispatch, the provider key is decrypted in memory. A bounded cache can retain the plaintext across requests; its secret value is zeroized on drop.

You may also register a Customer-Managed Key public-key fingerprint in Settings → BYOK Keys for audit and labeling. That registry does not itself configure gateway KMS encryption. The separate customer KMS API can wrap a tenant data key for provider-key envelopes. See BYOK. Provider keys never appear in:

• Logs (a tracing redaction layer scrubs Authorization, x-api-key, Bearer …, OpenAI org IDs org-…, AWS access keys AKIA…)
• Spans or trace attributes
• Error messages or stack traces
• Internal error/telemetry records (redacted before storage)

​

Transport

• TLS 1.3 on the edge→origin hop: the origin accepts nothing lower. The public edge (gateway.tracelane.dev, app.tracelane.dev) negotiates TLS 1.2 or later — a TLS 1.2 client is accepted there.
• mTLS on the OTLP ingest receiver — SDK and collector connections to ingest are mutually authenticated against SPIFFE identities. The gateway publishes spans to ingest over NATS on the internal network, which is not mTLS.
• HSTS enforced on the dashboard domain.
• No unencrypted HTTP accepted in production (redirect to HTTPS).

​

Tenant isolation

Tenant isolation is structural, not policy-enforced:

• Every ClickHouse query starts WHERE tenant_id = ? — CI grep blocks any new SQL without it (see scripts/ci/check-tenant-isolation.py).
• tenant_id is always resolved from the validated credential (the WorkOS organizationId claim is mapped to the workspace’s tenant UUID), never from the request body, URL parameter, or MCP tool argument.
• Postgres tables use foreign-key cascades — deleting a tenant row deletes all associated API keys, CMK keys, and billing records atomically.
• Isolation is enforced in the query layer, not by a storage-engine policy. Every ClickHouse table — the span hot tier and the prompt-promotion tables alike — is isolated by the mandatory WHERE tenant_id = ? bind (TenantQuery) plus the CI grep above. No ClickHouse row-level policies are in force on any table, and we would rather say so than credit a control you cannot verify.

​

PII handling

User-supplied span content (prompt/completion text) is wrapped in <UNTRUSTED_USER_DATA>…</UNTRUSTED_USER_DATA> before any internal LLM reads it.

​

Prompt content

The gateway records prompt and response text only when a workspace owner turns it on. It is off by default. Recording the request and recording the response are two separate switches. What is recorded when it is on:

• Request text on POST /v1/chat/completions and POST /v1/messages: the messages you send, and a top-level system field.
• Response text on POST /v1/chat/completions, buffered and streamed — the text your application actually received, including the arguments of any tool the model called. It is recorded after the response guardrails run: text a guardrail redacted is stored redacted, and a blocked response stores no response text. POST /v1/messages does not record response text, and neither does an answer served from the response cache (its request text is recorded).
• Nothing from POST /v1/embeddings, and nothing from requests made before the setting was turned on.

The following message-selection rules describe proxy capture. OTLP content uses the field caps described in OpenTelemetry capture. Limits. Each field is cut at the operator’s maximum field size (64 KiB on Tracelane Cloud) and the cut is marked …[truncated]. For a long conversation the most recent messages that fit are kept, and the span records how many earlier messages were left out; a single message with many parts keeps its newest parts and says how many earlier ones were left out. Inline image data (a data: URL) is not kept, only its media type; an image link is kept like text. Before anything is stored, on every plan, known credential patterns (API keys, tokens, private keys) and personal-data patterns (email addresses, phone numbers, card and social-security numbers) are replaced with [REDACTED:<type>] markers. Where and for how long. The text is stored on the trace’s span in ClickHouse (EU, Germany) and is deleted with the trace, at the end of your plan’s queryable history (on Tracelane Cloud: Free 30 days, every paid tier 730 days; a self-hosted deployment deletes on the schedule its operator configures). Turning the setting off stops recording new text; text already recorded stays until its trace expires. Text you copy into a dataset stays there until you delete the item or the dataset; text copied into a review queue is kept for at most 30 days. Who can change it, and the record of changes. Only the workspace owner, signed in (an API key cannot). Every change is written to the workspace’s tamper-evident audit ledger — event workspace.content_capture.set, with the old and new values and who made the change — so you can show when recording was on. The setting is GET / PUT /v1/workspace/capture on the gateway, and a control on the Settings → Workspace page. On a self-hosted deployment the operator can also turn recording on for named workspaces with a trace_content: block in tracelane.yaml (workspace IDs and the maximum field size). That block does not ask the workspace owner and writes no ledger event. The gateway applies this combined decision to proxy requests and OTLP spans. The direct mTLS ingest receiver uses the workspace decision only; without a control plane it keeps metadata and drops content. The SDK provider adapters normally record model, tokens and timing rather than message bodies. Framework instrumentation has its own capture settings; see LangGraph. The historical Helicone importer sends request and response bodies present in the export. Review instrumentation settings and export content before sending sensitive data. An enabled response cache retains completion bodies. For buffered POST /v1/chat/completions, the gateway can keep a completion in process memory and reuse it within the operator TTL. If your workspace has a usable embedding provider, it can also send message text to your own embedding provider and store the completion plus embedding in ClickHouse. The in-memory tier empties on gateway restart. Native messages, embeddings and streaming chat do not use this response cache. Send x-tracelane-cache: bypass on any plan to skip both lookup and storage. Eligible paid plans can request use or ttl=<hours>; the effective reuse age is the minimum of the requested value, plan ceiling and operator TTL, reported in x-tracelane-cache-ttl-hours and x-tracelane-cache-bound. Both exact and semantic lookup enforce that age. A reuse limit does not promise immediate physical deletion of stored rows or backups. These controls do not change a provider’s own prompt-cache policy. /settings/gateway shows the active configuration; GET /v1/gateway/settings exposes the same authenticated read. OTLP clients, including the history importer, can send gen_ai.input.messages / gen_ai.output.messages to POST /v1/traces. These reach stored trace content only when the corresponding input/output setting is on. Disabled fields are removed before publication, with a marker distinguishing withheld text from text the producer never sent. The gateway cache has a nominal 15-minute TTL; failed refreshes can retain the last known choice longer. The direct ingest cache uses a 30-second TTL and drops content on policy lookup failure. See OpenTelemetry capture for field classification and limits. An inline output PII-redaction pass ([REDACTED:<type>]) is on the roadmap. PR7 today is the entitlement-gated predictive trajectory guard, not an always-on span-output scrubber.

​

Authentication

• Dashboard: WorkOS AuthKit — SSO (Google, Microsoft, GitHub), SAML, email OTP.
• Gateway: tlane_* API keys (peppered HMAC-SHA256 lookup hash + Argon2id verifier stored; raw key shown once) or WorkOS JWT (the organizationId claim is resolved to the workspace’s tenant UUID).
• MCP server: tlane_* API key via env var.

​

SSRF protection

The gateway’s outbound HTTP client blocks all private + link-local destinations: RFC1918 (10/8, 172.16/12, 192.168/16), 100.64/10 (CGNAT), 169.254/16 (metadata services), IPv6 loopback and ULA (::1, fc00::/7). Only http/https are permitted. Redirects are disabled entirely on the SSRF-hardened client — a redirect to a private address can never be followed.

​

Vulnerability scanning

Three independent scanners run on a schedule — OSV-Scanner and Grype daily, Syft weekly. They are not per-commit: the tradeoff is written into security-scan.yml’s own header, which trades per-push detection for a ≤24h-delayed nightly scan.

​

Supply chain

• All GitHub Actions are SHA-pinned — no tag-pinned actions anywhere.
• Sigstore Cosign keyless signing — live on tagged release binaries as of v0.2.3, alongside the published container image and the weekly repository SBOM. Every release binary ships a .cosign.bundle beside it; verify with cosign >= 3.0.6:
  cosign verify-blob \
    --bundle gateway-x86_64-unknown-linux-gnu.cosign.bundle \
    --certificate-identity-regexp="https://github.com/tracelane/tracelane/.*" \
    --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
    gateway-x86_64-unknown-linux-gnu
  
• Build provenance attested via GitHub attest-build-provenance — live on tagged release binaries as of v0.2.3 and on the container image. Release-binary provenance is published as a GitHub attestation (https://slsa.dev/provenance/v1), naming the builder as .github/workflows/release.yml@refs/tags/<tag> and binding the binary’s SHA-256:
  gh attestation verify gateway-x86_64-unknown-linux-gnu --repo tracelane/tracelane
  
• Trusted Publishing OIDC for npm and PyPI — no long-lived tokens. crates.io is configured but not yet used (no crate published).
• CycloneDX SBOM — attached to every tagged release as of v0.2.3 (sbom.cyclonedx.json, CycloneDX 1.6, Cosign-signed with its own bundle), generated weekly for the repository, and attached to the container image.
• OpenSSF Scorecard target ≥ 9.0.
• Container base: Chainguard Wolfi only (cgr.dev/chainguard/rust, cgr.dev/chainguard/static, cgr.dev/chainguard/node, cgr.dev/chainguard/python).

​

Reporting vulnerabilities

See SECURITY.md for our responsible disclosure policy. We target a 72-hour acknowledgement SLA and 30-day patch SLA for critical vulnerabilities.

​

Related

• BYOK
• Audit ledger
• Architecture
• Self-hosting