Skip to Content
InfrastructureObservability

Observability

Distributed tracing goes to Honeycomb, and the goal is one trace per user action across the two clouds: a portal request on Cloudflare and the API Lambda work it triggers on AWS appear as a single trace, not two headless fragments — because a cross-cloud incident is undebuggable when each side only sees its own half. Getting there took three deliberate pieces — a real root span in the API, an instrumented Worker entry for the portal, and manual (not automatic) trace propagation between them. The last one is a hard-won production lesson.

The trace path

  • apps/api opens a proper root span per invocation and continues an incoming traceparent, so portal-originated requests join the caller’s trace instead of starting headless ones.
  • The portal Worker is instrumented by wrapping its entry point: apps/portal/src/otel-worker-entry.ts wraps TanStack Start’s real server entry (@tanstack/react-start/server-entry) with instrument() from @microlabs/otel-cf-workers, exporting to https://api.honeycomb.io/v1/traces with the x-honeycomb-team header from the HONEYCOMB_API_KEY Worker secret (see Secrets), service name opsdna-portal.
  • Propagation is opt-in per call site: server/traceparent-header.ts reads the active span via spanContext() and builds the traceparent header manually wherever the portal fetches the API.

Two portal footnotes worth knowing: src/ssr.tsx is dead code for both build targets (verified empirically with a log marker that never appeared) — the real entry is the TanStack virtual server-entry module, which is exactly why the otel wrapper lives in its own file. And a Cloudflare-native tracing approach was evaluated and dropped in favor of otel-cf-workers.

The instrumentGlobalFetch trap

The portal’s otel config sets:

instrumentation: { instrumentGlobalFetch: false }

Do not turn instrumentGlobalFetch back on. The library’s automatic mode monkey-patches globalThis.fetch to wrap every outbound call — and in production that broke the WorkOS SDK’s session refresh with TypeError: Illegal invocation (the SDK’s internal fetch usage lost its required this binding once wrapped). Users couldn’t authenticate; the fix shipped same-day via wrangler rollback. The library’s wrap code even looks correct on inspection, so the exact mechanism was never fully pinned — which is precisely why the mitigation is categorical: with global instrumentation off, tracing only touches fetches we explicitly opt into via traceparentHeader(), never WorkOS’s or any other library’s.

The general lesson: on Workers, prefer explicit per-call-site propagation over global monkey-patching. It costs one helper call per outbound fetch and eliminates an entire class of third-party breakage.

Adding tracing to a new call path

When the portal grows a new server-side fetch to the API, propagate the trace explicitly:

import { traceparentHeader } from "../server/traceparent-header";
 
const res = await fetch(`${env.API_BASE_URL}/some/route`, {
  headers: { ...traceparentHeader() },
});

Nothing else is needed — instrument() already gives that code real spanContext() access, and apps/api continues whatever traceparent arrives.

Other signals

  • Cloudflare Worker observability ("observability": { "enabled": true } in wrangler.jsonc) provides Workers-native logs/analytics for both the prod and staging portal workers, independent of Honeycomb.
  • CloudWatch alarms ship with the CDK stacks: DatabaseStack carries RDS alarms with an SNS alert topic, and the per-route Lambda factory (infra/lib/constructs/route-lambda.ts) wires alarms for each API route — see the infrastructure overview.
  • Staging traces flow the same way as prod (the staging worker has its own HONEYCOMB_API_KEY secret), so trace regressions can be caught on staging before promotion — see Environments.
Last updated on
Secrets & ConfigGlossary