Troubleshooting Tracelane traces and gateway errors

​

Debug checklist

# 1. Is the gateway running and healthy?
curl $TRACELANE_GATEWAY_URL/health
# Expected: {"status":"ok","service":"tracelane-gateway"}

# 2. Is the API key valid?
curl $TRACELANE_GATEWAY_URL/v1/chat/completions \
  -H "Authorization: Bearer $TRACELANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-haiku-4-5","messages":[{"role":"user","content":"ping"}],"max_tokens":5}'
# Expected: a valid chat completion response

# 3. Are traces landing?
curl "$TRACELANE_GATEWAY_URL/v1/audit/export?since=$(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-5M +%Y-%m-%dT%H:%M:%SZ)" \
  -H "Authorization: Bearer $TRACELANE_API_KEY" | head -3
# Expected: NDJSON lines (empty if no traffic in the last 5 minutes is OK)

​

Common errors

​

401 Unauthorized

• Confirm TRACELANE_API_KEY starts with tlane_.
• Check the key hasn’t been revoked in Settings → API Keys.
• Revocation takes effect at the gateway within 60 seconds; a key that has been idle for over a minute may be accepted once more on its next use while the gateway re-checks it.
• For JWT auth: confirm the organizationId claim is present and the JWKS URL is reachable from the gateway.

# Decode the JWT to inspect claims (no signature check)
echo $JWT | cut -d. -f2 | base64 -d 2>/dev/null | python3 -m json.tool

​

403 Forbidden — entitlement check failed

​

429 Too Many Requests

import time, httpx, random

for attempt in range(5):
    res = client.messages.create(...)
    if res.status_code != 429:
        break
    wait = int(res.headers.get("retry-after", 2 ** attempt))
    time.sleep(wait + random.uniform(0, 1))

​

502 Bad Gateway from provider

​

predictive.verdict = block — guardrail finding

# Get full explanation via the MCP tool
#   explain_guardrail_block(trace_id, span_id)

​

Traces not appearing in the dashboard

1. SDK not initialized — confirm tracelane.init(endpoint=…, api_key=…) runs at startup (both arguments are required; there is no env-var auto-read).
2. Client not wrapped — constructing a client does not capture it. Call the matching instrument_* on each client after init(). auto_instrument() only covers anthropic, openai, litellm, and claude_code — everything else needs an explicit instrument_* call.
3. Wrong endpoint — on Cloud, endpoint is https://gateway.tracelane.dev (the SDK appends /v1/traces), not the dashboard URL. Self-hosting, it is the OTLP receiver you run.
4. 403 insufficient_scope — the key is missing the ingest scope. The response body names it. Mint a key with Ingest ticked in Settings → API Keys; keys minted before scopes existed are unaffected.
5. Ingest lag — spans can take up to 5 seconds to appear after the request. Wait 10 seconds and refresh.
6. Sampling rate — if sample_rate < 1.0, only a fraction of requests are captured.

from tracelane import shutdown
shutdown()  # flushes the batch span processor

​

ClickHouse connection errors (self-host)

Error: ClickHouse: Failed to connect to ch.example.com:8123

• Confirm CLICKHOUSE_URL is reachable from the gateway container.
• For Neon ClickHouse: confirm the connection string uses HTTP, not HTTPS, unless your cluster terminates TLS.
• Check docker compose logs clickhouse for startup errors.

​

WorkOS webhook not provisioning tenants

1. In WorkOS dashboard → Webhooks, confirm the webhook is pointing to https://gateway.<your-domain>/v1/webhooks/workos and is active.
2. Check gateway logs for workos_webhook entries:
   cargo run -p gateway 2>&1 | grep workos_webhook
   
3. Confirm WORKOS_WEBHOOK_SECRET matches what’s shown in the WorkOS dashboard.
4. Replay the failed webhook events from the WorkOS dashboard.

​

Enable debug logging

# Gateway — verbose JSON logs
RUST_LOG=debug cargo run -p gateway

# SDK — Python (the OTLP exporter logs to the root logger)
import logging
logging.basicConfig(level=logging.DEBUG)
import tracelane
tracelane.init(endpoint="http://localhost:4318", api_key="tlane_...")

# Gateway (the SDKs do not read a log-level variable)
TRACELANE_LOG_LEVEL=debug cargo run -p gateway

​

Get help

• GitHub Issues: github.com/tracelane/tracelane/issues
• Discussions: github.com/tracelane/tracelane/discussions
• Email: [email protected] (Business+ SLA applies)

​

Related

• Self-hosting
• API reference
• Security