Observability & cost

Frontier’s operational observability provides engineers with insight into the system’s behavior, performance, and costs. It encompasses structured logging, distributed tracing, metrics collection, error tracking, and AI-specific telemetry, aiming to give a clear picture of the system’s health and resource consumption.

Logs, Traces, and Metrics

All logs and traces within Frontier’s backend systems are primarily routed to Axiom, a logging and observability platform.

Logs and traces are segregated into separate Axiom datasets based on environment (development, staging, release candidate, production, demo) and type (logs vs. traces). For example, production logs go to frontier-call-server-logs-prd, and production traces go to frontier-call-server-traces-prd.

The Call Server, composed of multiple Cloudflare Workers, leverages Cloudflare Workers Observability to route deployed environment logs, with a 100% sampling rate. Traces from Cloudflare Workers Observability are head-sampled at 10% in production and 100% in other environments. Additionally, an app-level OpenTelemetry trace pipeline, using @microlabs/otel-cf-workers, exports to a different Axiom dataset (frontier-agent-{DOPPLER_CONFIG}), also sampled at 10% in production.

Mid-Migration

These two parallel trace pipelines are slated for consolidation into a single pipeline, with the app-level OpenTelemetry exporter being retained and the Cloudflare-native traces destination being deprecated.

Durable Objects and other long-lived execution paths post their logs to a dedicated logging worker, which then ingests them into Axiom. The Dashboard, a Next.js application, sends its logs to frontier-dash-{env} datasets in Axiom, utilizing the @axiomhq/logging and @axiomhq/nextjs libraries. All Axiom log events adhere to a canonical AxiomLogEvent schema, including contextual fields like call_id, org_id, request_id, correlation_id, and user_id.

Silent Log Loss Risk

The logging worker currently returns an HTTP 200 success code even if Axiom ingest fails. This means that if the AXIOM_TOKEN expires or is invalid, all Durable Object logs could be silently dropped without any alert. A failure counter and canary monitoring system are proposed to address this.

Data Quality Risk

A known “body-blob” issue exists where structured logs are stringified into a single field, hindering efficient querying and analysis. A fix to emit objects directly for better indexing is planned.

GAP — founder supplies

Specific figures for Axiom ingest volume, data retention policies, and associated spend are not verifiable from the codebase.

Error Tracking

Error tracking for the Call Server relies on Sentry, integrated via the @sentry/cloudflare SDK. The Call Server has its own dedicated Sentry project (frontierx-call-server) and a unique DSN (Data Source Name), distinct from the Dashboard’s Sentry configuration. Sentry’s trace sampling rate for the Call Server is set to 10% in production environments and 100% otherwise.

To prevent alert fatigue, Frontier implements severity-based routing for errors: only CRITICAL and HIGH severity errors are forwarded to Sentry for alerting, while LOW and MEDIUM severity errors are recorded in Axiom logs only. Sentry issues are fingerprinted using stable identifiers (error name, worker name, error slug) rather than call_id to group similar errors effectively. However, call_id, org_id, request_id, and correlation_id are attached as searchable tags and enable deep-linking directly to corresponding Axiom logs.

Error reporting within Durable Objects is designed to be non-blocking. The reportError function uses a fire-and-forget Sentry.flush(2000) mechanism to ensure that error events are dispatched even if Durable Object WebSocket connections are abruptly closed.

GAP — founder supplies

Details regarding Sentry plan/quota, the configuration status of Sentry Alert Rules, and whether these rules are actively live in production are not verifiable from the codebase.

Product and LLM Telemetry

Product analytics for the Dashboard application (@frontierx/dash) is managed using PostHog. This PostHog instance is self-hosted and proxied at p.frontierxai.com. Frontier adopts a conservative approach to analytics: it is opt-out by default, consent-gated, disables session recording, and captures pageviews manually. User profiles (person_profiles) are limited to identified_only. Server-side LLM telemetry is not captured via PostHog.

LLM usage and cost accounting are crucial for Frontier. A Cloudflare AI Gateway named frontier-ai is integrated into the Call Server. All calls to OpenAI and Cloudflare Workers AI are routed through this gateway. The frontier-ai gateway is instrumental in logging prompt, response, latency, and COST for each request, along with cf-aig-metadata for correlation.

Crucial for Cost of Goods Sold (COGS)

The Cloudflare AI Gateway is designed to become the single source of truth for LLM cost accounting, which is essential for deriving Frontier’s Cost of Goods Sold (COGS) per call.

Currently, several LLM providers, including Together, OpenRouter, Perplexity, and Google Gemini, bypass the frontier-ai gateway. Routing all LLM providers through this gateway for unified cost accounting is a planned roadmap item (T2).

In the Call Server’s current implementation, OpenRouter is utilized only as an embedding fallback provider (baai/bge-large-en-v1.5), used when Cloudflare Workers AI times out or fails. It is not currently the primary LLM gateway or router for general LLM calls. The fallback includes retry logic for specific HTTP error codes (503, 429, 529).

For automated code review, Qodo, Frontier’s PR code-review bot, runs on Amazon Bedrock. The specific Bedrock model ID is configured via the QODO_BEDROCK_MODEL environment variable.

GAP — Token/Cost Accounting Not Yet Live

Per-call token and cost accounting from telemetry is not yet delivered. The OpenTelemetry work aimed at providing wall-clock timings, LLM call counts, tokens, and cost-per-call “isn’t delivering that yet.” In-app LLM spans currently hardcode llm.provider='openai' and do not capture token or cost attributes. This is a roadmap item (T2/T3).

The designed method to compute COGS involves making the frontier-ai AI Gateway the single LLM ingress and cost source of truth. It will pass cf-aig-metadata (including call_id and org_id), allow for custom costs for models without public pricing, and export the gateway’s OpenTelemetry GenAI spans or Logpush into Axiom, correlated by call_id.

Alerting and Service Level Objectives (SLOs)

Environment variables for NTFY alerts (e.g., NTFY_TOKEN, NTFY_DEV_ALERTS_TOPIC) are provisioned. However, there is no in-application NTFY sender code currently wired into the codebase for programmatic alerting.

Mid-Migration — Alerting & SLOs are Spec-Only

Frontier’s Service Level Objectives (SLOs) and a comprehensive alerting system are currently spec-only and not yet implemented. The team acknowledges that they “find out about staging/production breakage from users and QA” and explicitly operates today with “no dashboards, monitors, or alerts.” Therefore, dashboards, monitors, SLO tracking, and per-call cost derivation are considered roadmap items rather than actively operational features.

Defined (target, not measured) SLOs, as documented in specifications, include:

• Transcript delivery p99 under 5 seconds (over 30 days).
• Question-detection availability of 99.5% (over 7 days).
• Quick-answer availability of 99% (over 7 days).
• Sentry high/critical error budget below 50 per day (production).
• Log delivery 99.9% within 2 minutes.
• WebSocket uptime of 99.9% (over 7 days).

Critical alert conditions, as defined in the target specification, would route to paging and warnings to a Slack channel (#dev-alerts). Examples include: a logging worker experiencing more than 5 ingest failures per 5 minutes, Sentry reporting more than 10 critical severity errors per 5 minutes, WebSocket connection failures exceeding 20% per 5 minutes, or a sub-worker showing zero logs for over 10 minutes.

GAP — founder supplies

Actual uptime and SLO attainment percentages are not verifiable from the codebase. Additionally, it needs to be confirmed whether NTFY alerting is wired anywhere outside the main repository (e.g., via external scripts or Cloudflare cron jobs).