Authority Doc / Public V1 Observability

The Observability Plan defines how CoachMe developers detect, correlate, diagnose, and respond to operational failure.

This developer-only authority sets the smallest production baseline one engineer can implement and operate across the NestJS API and workers, coach web, coach mobile, client mobile, PostgreSQL, Redis/BullMQ, object storage, media workers, identity, and registered external providers.

AuthorityDeveloper operationsWeb + mobileLaunch + MaturityPublic V1

Note

Launch direction remains conditional on approved provider decisions, ready provider evidence, Privacy-owned retention, Roles-owned access, and Environment-owned topology. Telemetry export failure must not fail otherwise accepted product work or replace durable authority-required evidence.

Scope And Authority Boundary

Observability defines operational evidence and first response; adjacent authorities define the product behavior being observed.

The Observability Plan owns:

• telemetry event naming, base fields, safe context, and redaction implementation;
• structured-log severity and emission contracts;
• metric names, instrument types, units, bounded dimensions, and operational objectives;
• cross-signal searchability and propagation of identifiers defined by adjacent authorities;
• error and crash reporting behavior, safe context, release tags, and source-map evidence;
• monitoring and synthetic checks against Environment-owned health contracts;
• operational dashboards, alert conditions, severity, and notification behavior;
• developer-facing diagnosis lookups and first-response evidence;
• operational-incident detection, technical timeline evidence, and hand-off triggers;
• observability-specific feature checklists and required verification evidence;
• the explicit Launch versus Maturity observability split.

The Observability Plan does not own:

• workflow states, business outcomes, or canonical recovery behavior;
• REST headers, response envelopes, error envelopes, request ids, receipts, or idempotency rules;
• event envelopes, transactional outbox persistence, retry, replay, discard, ordering, or dead-letter behavior;
• offline device identity, operation envelopes, receipts, conflicts, replay, upload sessions, cache, or purge behavior;
• product-user, support, administrator, or developer permissions for sensitive diagnostic access;
• audit-record schema, immutability, required access events, legal/security evidence, or audit retention;
• privacy classification, telemetry retention periods, legal hold, deletion, or privacy-incident response;
• environment topology, configuration, secrets, health endpoint semantics, deployment, backup, restore, or rollback;
• external provider or operator-controlled component selection, region, terms, pricing, credentials, provider retention, or provider launch evidence;
• package versions, root command names, test runners, CI command shape, dependency updates, or release-note tooling;
• test execution frequency, merge policy, coverage gates, final release evidence, or production approval.

Failure to export optional operational telemetry must not fail an otherwise accepted product operation. This rule does not apply to authority-required audit evidence, transactional outbox persistence, deletion receipts, sync receipts, command receipts, or other durable business, privacy, legal, or security records.

Requirement Tiers

Page-local delivery tiers do not replace provider decisions or evidence statuses.

This authority uses two presentation tiers and keeps them separate from provider-decision and evidence status.

Requirement Tiers table 1

Tier	Meaning	Launch effect
Launch (Public V1)	Binding Public V1 production requirement.	Missing mandatory evidence blocks production launch.
Maturity (post-launch observability)	Approved improvement after the initial launch baseline and its named trigger.	Does not block launch and cannot weaken a Launch rule.

This authority may display the short chips Launch and Maturity, but these are page-local presentation tiers mapped to Public V1 and post-launch observability. They are not new global product-scope or readiness statuses. The Integration Register continues to own Approved, Conditional, Rejected, and Deferred provider decisions plus Open, Ready, and Blocked provider evidence.

Every substantive mixed-tier matrix includes a Tier column. A matrix may omit the repeated column only when its heading explicitly scopes every row to Launch or Maturity, as with the canonical Launch metric catalog and Launch alerts.

Observability Principles

These principles apply to every Public V1 signal.

Public V1 uses these principles:

1. Diagnose user impact, not only infrastructure state.
2. Use stable identifiers to move from a user-visible failure to safe operational evidence.
3. Minimize and redact telemetry before export; provider-side scrubbing is a second layer, not the first control.
4. Keep operational telemetry separate from business records, receipts, audit records, and product analytics.
5. Keep metrics bounded and aggregate; high-cardinality identifiers belong only in restricted log or error context.
6. Alert only when the active engineer can take a defined first action.
7. Instrument a feature as part of its implementation rather than after the feature is complete.
8. Keep the launch stack small enough for one engineer to understand, test, and maintain.
9. Treat telemetry export failure as degradation, while durable authority-required evidence follows its owning contract.
10. Prefer read-only diagnosis. Retry, replay, discard, data repair, restore, and user-data access require the separately authorized mechanism.

Cost optimization is not a standalone Launch objective. Bounded metric labels, excluded noisy request classes, and disabled unused telemetry remain engineering-safety requirements because they prevent broken queries, runaway series, and unnecessary exposure.

Registered Observability Stack

Observability selects signal scope; companion authorities approve providers, topology, retention, credentials, and package lifecycle.

This authority selects the instrumentation package families and owns their signal scope, while companion authorities approve external services, topology, credentials, package lifecycle, and retention.

NestJS API and workers
  |-- Pino JSON to stdout
  |-- OpenTelemetry application metrics
  +-- Sentry exceptions

Docker/runtime
  +-- Grafana Alloy
       |-- forwards Pino and approved runtime logs
       +-- forwards application and runtime metrics
             +-- Grafana Cloud logs, metrics, dashboards, alerts, synthetics

Coach web + coach mobile + client mobile
  |-- Sentry errors, crashes, releases, source maps
  +-- privacy-safe connectivity/app-health summaries through CoachMe contracts

The companion-authority decisions are:

Registered Observability Stack table 1

Stack part	Owning authority	Current contract
Pino, OpenTelemetry metrics SDK, and Sentry SDK signal scope	Observability Plan	These package families are selected for the stated Launch signals; client OTel, Sentry logs, session replay, profiling, and performance traces are prohibited.
Grafana Cloud	Integration Register	Currently Conditional with Open evidence until region, plan, data processing, retention, quotas, access, outage behavior, and launch evidence are verified.
Sentry SaaS	Integration Register	Currently Conditional with Open evidence until region, plan, data processing, retention, quotas, scrubbing, access, outage behavior, and launch evidence are verified.
Grafana Alloy	Integration Register and Environment Runbook	The collector is Approved with Open evidence and uses the Environment-owned protected restricted Docker API proxy; deployment awaits host-access, resource-limit, port, credential, heartbeat, restart, and exporter-failure proof.
Package installation, build/source-map integration, version locks, update policy	Toolchain Contract	Records the exact installable package families, source-map integration, version locks, and coordinated update evidence without owning signal semantics.
Collector placement, endpoints, tokens, DSNs, source-map token, environment labels, local log rotation	Environment Runbook	Owns one staging/production Grafana stack, collector topology, configuration, secret placement, environment isolation, local log rotation, and runtime hardening without publishing values.

Alloy uses the Environment-owned protected restricted Docker API proxy and never receives a direct Docker-socket mount, including a read-only mount. It runs non-root where supported, drops unused capabilities, uses a read-only filesystem, has bounded memory, exposes no public receiver/UI ports, and uses write-only credentials scoped to the required Grafana signals.

Identity And Correlation Contract

Identifiers remain owned by their source authorities; this plan defines safe observability use.

The plan consumes identifiers from their owning authorities and defines only their observability use and cross-tool searchability.

Identity And Correlation Contract table 1

Identifier	Authority and lifetime	Observability use
requestId	API-owned for one HTTP request; the API validates an acceptable client value or generates/replaces it and always returns the authoritative value.	Returned as the normal user-safe diagnostic reference and attached to backend logs and Sentry context.
correlationId	API-issued; one complete bounded product-workflow instance across its requests, authentication return, jobs, events, and provider calls.	Primary cross-signal lookup for the whole workflow; distinct from any W3C trace id.
eventId / causationId	Event And Outbox-owned.	Join event publication and consumption evidence without redefining the event envelope.
jobId	Event And Outbox-owned normalized BullMQ runtime lookup identifier; one queued job, with attempt number distinguishing executions.	Find one queued job and, with attempt number, its completion/failure evidence; never a metric label or domain identifier.
clientOperationId / deviceId / receiptId	Offline Sync and API-owned.	Diagnose mobile replay, acceptance, rejection, conflict, upload refresh, and recovery using the existing receipt path.
workspaceId / relationshipId / internal userId	Existing ownership and permission context.	Allowed in minimized restricted log/error context when needed and approved by Roles; never a metric or Loki index label. Prefer relationshipId over a generic clientId.
authenticated session id	Identity-owned and valid only for the current authentication context.	May persist across app restarts while the auth session remains valid, but is not exported to Grafana or Sentry at Launch.
W3C traceId / spanId	Reserved for the Maturity tracing implementation.	Never used as a synonym for correlationId.

A bounded workflow has an explicit product start and terminal handoff or outcome. Invitation acceptance keeps one API-issued correlation id across invitation validation, authentication return, account binding, relationship creation, and onboarding-context creation. An upload workflow keeps one correlation through intent, byte confirmation, scanning/processing, and ready/failed outcome. A notification workflow keeps one correlation from accepted domain event through normalized provider receipt. The id does not become a permanent multi-day coach-client relationship id; later independent workflows use their own correlation ids and remain linked by canonical domain ids and receipts.

Public clients must not invent an authoritative correlation id. The API Contract creates or resolves it from trusted workflow state or a previously API-issued receipt, and the Event And Outbox Standard propagates it across outbox records and jobs. Internal calls and provider adapters preserve it. Both authorities distinguish correlationId from a W3C trace id and reject or replace untrusted client-authored values.

The mobile apps reuse the Offline Sync Protocol's server-issued device registration. It persists across app restarts only while its installation, app surface, user, and authentication context remain valid. The Observability Plan does not create a second permanent device identity or a permanent observability session.

Structured Logging Contract

Pino emits allow-listed JSON to standard output from the NestJS API and workers.

Pino emits JSON to standard output from the NestJS API and workers. Grafana Alloy collects the stream, applies approved filtering and bounded index labels, and forwards it to the registered backend.

Launch levels are fatal, error, warn, and info. debug is permitted locally and in controlled staging diagnosis but disabled in production. Dynamic production debug toggles are Maturity work.

Required base fields when applicable:

Structured Logging Contract table 1

Field	Rule
timestamp	UTC emission time.
level	Stable severity.
event.name	Stable dot-delimited operational event name, not a domain-event name.
message	Short developer-facing summary with no arbitrary payload interpolation.
service.name / service.version	Stable runtime name and deployed release.
deployment.environment.name	Environment-owned enum: local, test, staging, or production.
app.surface / app.version / app.build	Required when web or mobile context applies.
operation.name / outcome	Bounded operation and normalized outcome.
duration_ms	Required for completed timed operations.
identifiers	Only the ids relevant to that event; never require every id on every row.
error.code / error.type	Stable code and exception type on failures.
stack	Restricted scrubbed stack on server failures; never local variables or attached payloads.

Canonical Launch operational event names include:

Structured Logging Contract table 2

Event	Producer	Required use
http.request.completed	API	One completion summary for each qualifying request.
job.attempt.completed	Worker	One successful/no-op attempt result.
job.attempt.failed	Worker	One retryable or terminal attempt failure.
outbox.claim.completed	Outbox publisher	One normalized claim transition with bounded outcome.
outbox.publish.completed	Outbox publisher	One normalized publish transition with bounded outcome.
outbox.retry.scheduled	Outbox publisher	One retry transition after a failed attempt.
outbox.dead_letter.created	Outbox publisher	One terminal dead-letter transition.
client.diagnostic.received	API/sync adapter	Receipt of a bounded privacy-safe web/mobile diagnostic summary.
telemetry.export.failed	Collector/export adapter	Export failure or dropped telemetry evidence when locally observable.

The API uses a custom allow-listed request-completion object/serializer rather than default pino-http request/response serializers. It explicitly excludes client IP, remote port, full User-Agent/device fingerprint, request/response header objects, raw URL/query, and default response-header serialization. Security-specific IP evidence, if later required, belongs to separately approved security/audit handling.

Additional emission rules:

• exclude normal health, static-asset, and synthetic traffic from per-request info logs;
• emit one completion or failure event for each background-job attempt;
• inherit required claim, publish, retry, and dead-letter evidence from Event And Outbox;
• inherit required sync and receipt evidence from Offline Sync;
• use route templates only, never raw URLs or query strings;
• log database operations only when slow or failed, using a bounded operation/model name and duration;
• do not log SQL text, SQL parameters, row values, result samples, or sensitive result counts;
• do not log request or response bodies by default;
• do not treat production console output from web/mobile apps as a durable channel.

Prohibited content includes headers, cookies, tokens, passwords, OTPs, provider secrets, provider payloads, signed URLs, raw filenames, messages, form answers, coach notes, nutrition/wellness content, media, local SQLite payloads, request/response bodies, and audit payloads. Redaction uses application-side allow-list construction plus known key removal; Pino key-path redaction and provider-side scrubbing are defense in depth and cannot make arbitrary strings safe.

Operational logs are not audit records. An operational log may carry an audit-event id for lookup, but it never replaces the append-only audit fact or its access rule.

Metrics Contract

OpenTelemetry emits bounded backend application metrics; clients do not run OpenTelemetry at Launch.

OpenTelemetry emits application metrics from the API and workers. Grafana Alloy adds approved runtime/container/collector metrics and forwards the bounded set to Grafana Cloud. Web/mobile clients do not run OpenTelemetry at Launch.

Metrics use counters for monotonic outcomes, histograms for durations or reported distributions, and observable gauges for current server-side depth, health, pressure, or oldest age. Time units are seconds unless the pinned OpenTelemetry semantic-convention version requires another unit.

Allowed custom dimensions are the exact bounded attributes in the catalog. Custom keys use the coachme.* namespace; HTTP keys use the exact names from the OpenTelemetry semantic-convention version pinned by Toolchain. Resource attributes such as service.name and deployment.environment.name are not repeated as arbitrary custom dimensions. User, workspace, relationship, device, request, correlation, event, job, operation, receipt, URL, filename, email, phone, IP, and free-text values are prohibited as metric dimensions.

Canonical Launch metric catalog:

Metrics Contract table 1

Metric	Tier	Instrument / unit	Producer	Allowed attributes	Authority hand-off
http.server.request.duration	Launch	histogram / s	API OTel HTTP instrumentation	http.request.method, http.route, http.response.status_code, conditional error.type, url.scheme	API owns route/error semantics; status class is derived in queries.
coachme.auth.attempts	Launch	counter / {attempt}	API/auth adapter	coachme.operation.name, coachme.outcome, coachme.provider.name	Roles/API/Integration own acceptance and provider behavior.
coachme.invitation.attempts	Launch	counter / {attempt}	API	coachme.operation.name, coachme.outcome, coachme.abuse.category	Workflow/API own invitation states and errors.
coachme.client.connectivity.outcomes	Launch	counter / {outcome}	API from safe client summary	coachme.app.surface, coachme.app.version, coachme.platform.name, coachme.connectivity.outcome, coachme.failure.category	API/Offline Sync own transport; Observability owns aggregation.
coachme.queue.jobs	Launch	observable gauge / {job}	Worker/queue adapter	coachme.queue.name, coachme.queue.status	Environment owns queue inventory; Event owns behavior.
coachme.queue.oldest_runnable.age	Launch	observable gauge / s	Worker/queue adapter	coachme.queue.name	Excludes scheduled/delayed jobs.
coachme.outbox.events	Launch	observable gauge / {event}	Outbox publisher	coachme.outbox.status, coachme.event.family	Event And Outbox owns lifecycle.
coachme.outbox.oldest_available.age	Launch	observable gauge / s	Outbox publisher	none	Event And Outbox owns retry/replay/discard.
coachme.outbox.transitions	Launch	counter / {transition}	Outbox publisher	coachme.transition.name, coachme.outcome, coachme.event.family	Covers claim/publish/retry/dead-letter; Event And Outbox owns lifecycle semantics.
coachme.sync.operations	Launch	counter / {operation}	API/sync adapter	coachme.app.surface, coachme.operation.family, coachme.outcome	Offline Sync owns acceptance/rejection/conflict.
coachme.sync.reported_queue.depth	Launch	histogram / {operation}	API from mobile diagnostic summary	coachme.app.surface, coachme.operation.family	Distribution recorded when a summary arrives; not current device state.
coachme.sync.reported_queue.oldest.age	Launch	histogram / s	API from mobile diagnostic summary	coachme.app.surface, coachme.operation.family	Distribution recorded when a summary arrives; disconnected devices contribute no stale state.
coachme.sync.recovery.failures	Launch	counter / {failure}	API from mobile diagnostic summary	coachme.app.surface, coachme.failure.type	Covers restart recovery and cache purge failure.
coachme.upload.operations	Launch	counter / {operation}	API/media worker	coachme.upload.purpose, coachme.outcome	API/Offline/Integration/Privacy own behavior.
coachme.upload.processing.oldest.age	Launch	observable gauge / s	Media worker	coachme.processing.class	Integration owns processor degradation.
coachme.provider.requests	Launch	counter / {request}	Provider adapter	coachme.provider.name, coachme.operation.name, coachme.outcome	Integration Register owns provider semantics/fallback.
coachme.provider.request.duration	Launch	histogram / s	Provider adapter	coachme.provider.name, coachme.operation.name, coachme.outcome	Does not expose raw provider errors.
coachme.privacy.jobs	Launch	counter / {job}	Privacy worker	coachme.job.type, coachme.outcome	Privacy owns lifecycle and incident handling.
coachme.privacy.oldest_pending.age	Launch	observable gauge / s	Privacy worker	coachme.job.type	No subject identifiers.
coachme.backup.latest_success.age	Launch	observable gauge / s	Environment backup monitor	coachme.backup.class	Environment owns cadence, backup, and restore.
coachme.backup.failures	Launch	counter / {failure}	Environment backup monitor	coachme.backup.class, coachme.failure.category	No backup paths or credentials.
coachme.telemetry.export.failures	Launch	counter / {failure}	Alloy/export monitor	coachme.telemetry.signal, coachme.destination.class, coachme.failure.category	Detects observability degradation.

Process, container, PostgreSQL, Redis, and Alloy self-metrics use the stable collector/exporter names selected and pinned in Environment evidence. The Observability page lists the exact names used by its queries after the approved collection method is known; it does not invent aliases that conflict with collector semantics.

The client diagnostic path is a Launch contract without client OpenTelemetry:

• API Contract defines a best-effort, rate-limited, no-content client diagnostic summary for coach web and mobile surfaces.
• Offline Sync defines how mobile stores only bounded counters/status and sends the latest summary on the next successful sync/health exchange.
• Allowed fields are app surface/version/build, platform, connectivity state, bounded network-failure category, operation family, local queue depth, oldest local queued-operation age, restart-recovery outcome, and cache-purge outcome.
• The API validates every submitted dimension against fixed enums and the registered release/build set before metric emission. Unknown, unsupported, or spoofed app versions/builds, surfaces, platforms, connectivity outcomes, failure categories, and operation families map to explicit bounded fallback values; arbitrary client strings are never labels.
• The summary contains no free text, URLs, payloads, user content, filenames, or business values and is not stored as a new durable product record.
• The API converts accepted summaries to the catalogued aggregate metrics and a minimized client.diagnostic.received event.
• Web repeated connectivity failures may also create a scrubbed Sentry event. Mobile state that has never reconnected remains unknowable. Dashboards show report count plus p95/max queue depth and oldest age over a stated recent reporting window; each observation represents a device that reconnected and reported during that window. Queries use only observations inside that window; no current-state device gauge or per-device operational store is created.

Launch coverage also consumes, without redefining, the mandatory Offline Sync diagnostic families and every Event And Outbox claim/publish/retry/dead-letter transition. Sentry crash, release, and issue views stay in Sentry and are not recreated as duplicate Grafana metrics.

Distributed Tracing Decision

Distributed tracing is Maturity, not a Launch requirement.

Distributed tracing is Maturity, not a Launch requirement.

The reasons are:

• CoachMe is a modular monolith with a small worker estate rather than a large service mesh;
• structured logs plus stable request, correlation, event, job, operation, device, and receipt ids cover the launch diagnosis path;
• OpenTelemetry JavaScript logs remain in development and browser instrumentation remains experimental;
• parallel full traces in Sentry and Grafana would create duplicated data and two diagnosis paths;
• tracing introduces sampling, span review, propagation, privacy, storage, and query work that one engineer does not need before launch.

The Environment Runbook requires Launch observability signals for production and promotion and classifies distributed tracing as Maturity rather than an unconditional environment requirement.

If the maturity trigger occurs because causal diagnosis across the API, outbox, queues, Redis, PostgreSQL, or providers is repeatedly inadequate, the future implementation must:

• choose one performance-trace backend;
• use W3C Trace Context;
• keep traceId distinct from correlationId;
• set an explicit sampler before production;
• instrument reviewed operations rather than enabling an unreviewed full auto-instrumentation bundle;
• repeat provider, privacy, retention, and access review;
• add trace-specific redaction and verification evidence.

Error And Crash Reporting

Sentry owns unexpected exception and crash views; Grafana remains the operational logs and metrics path.

Sentry is the selected error-reporting direction, subject to its Integration Register row becoming Approved and its mandatory launch evidence becoming Ready.

Use four projects:

• backend API and workers, distinguished by service/runtime context;
• coach web;
• coach mobile;
• client mobile.

Launch behavior:

• capture unhandled exceptions, crashes, and explicit unexpected failures at 100 percent;
• generate/upload or otherwise enable production TypeScript source mapping for the API/worker, coach web, and both mobile apps;
• attach environment, release, app surface, platform, stable error code, bounded operation name, request id, and correlation id as safe context;
• permit the minimized internal ids approved by Roles in event context, not indexed tags; never export an authenticated session id;
• set sendDefaultPii: false and includeLocalVariables: false;
• use allow-listing beforeSend hooks plus breadcrumb filtering;
• enable provider-side data scrubbing as a second layer;
• permit a normalized connectivity/network-failure event with bounded categories and no URL, IP, header, payload, or free text;
• disable Sentry logs, performance tracing, session replay, screenshots, attachments, user-feedback payloads, and profiling at Launch;
• never attach request/response bodies, headers, cookies, tokens, signed URLs, provider payloads, messages, forms, nutrition/wellness content, media, or local database values.

Sentry outage or exporter failure must not block product operations. Local Pino logs continue to standard output, and dropped external error reports are treated as observability degradation rather than a business retry trigger.

Health, Readiness, And Synthetics

Environment owns endpoint semantics; Observability owns monitoring and alert use.

The Environment Runbook owns endpoint paths, dependency membership, public response shape, topology, check cadence, and remediation. The coordinated Environment clarification defines:

• API liveness as process-only;
• API readiness as PostgreSQL plus only the dependencies required to accept requests safely;
• worker readiness as PostgreSQL plus Redis/BullMQ;
• object storage, Keycloak, media processors, and external providers as separately visible component/degraded checks rather than blanket API-unready conditions;
• minimal public health responses and protected detailed dependency output;
• the normalized environment enum local, test, staging, and production;
• bounded local Docker log rotation and a disk-pressure signal;
• a collector heartbeat and exporter self-metrics through the reviewed Alloy collection boundary.

The Observability Plan owns:

• collection and interpretation of those health outcomes;
• Grafana dashboards and alerts based on them;
• named public HTTP synthetic checks against coach web and API every 60 seconds;
• an external TLS-expiry signal;
• detection of missing Alloy heartbeat, exporter failures/dropped telemetry, and critical alert-query No Data or evaluation errors;
• exclusion of normal health and synthetic traffic from per-request info logs;
• a health-signal drill proving an alert reaches the configured email destination.

Mobile/web operational health is covered through Sentry releases/crashes, API outcomes, the safe client diagnostic summary, sync metrics, connectivity outcomes, and provider receipts. Public V1 does not build a continuous mobile user-journey synthetic.

Operational Objectives And Alert Rules

These are internal engineering objectives, not customer SLAs.

Launch objectives are internal engineering objectives, not customer contracts or commercial SLAs.

Operational Objectives And Alert Rules table 1

Objective	Launch target	Notes
Named public-check availability	99.5 percent per check over a rolling 30-day window	Evaluated separately for coach web and API; requires an approved provider plan with at least 30 days of aggregate metric retention.
Normal JSON API latency	p95 below 1 second	Excludes health, static, synthetic, upload/download byte transfer, and explicitly long-running routes.

Public V1 does not define a separate generic server-success objective or a formal asynchronous SLO. Queue, outbox, sync, upload, and provider age/failure alerts provide the actionable launch signal. Formal SLO review and error budgets remain Maturity.

Every alert record contains evaluation interval, lookback window, pending period, recovery condition, and No Data/query-error behavior. The default evaluation interval is 60 seconds. Pending is zero unless the executable condition specifies consecutive evaluations or a sustained duration. When a row omits query-error behavior, query error is alerting for an immediate alert and warning for a warning alert. Qualifying HTTP traffic excludes health, static assets, synthetics, byte-transfer routes, and explicitly long-running operations. Each ratio rule below defines its own bounded numerator and denominator and does not fire below its absolute minimum count. Recovery requires two consecutive healthy evaluations unless the rule documents a safer condition.

Launch alerts:

Operational Objectives And Alert Rules table 2

Alert	Initial executable condition	No-data/recovery posture	Notification
Public synthetic unavailable	All selected probes for one named 60-second check fail for 3 consecutive evaluations.	Missing check data is alerting; recover after 2 successful evaluations.	Immediate email.
API server errors	At least 5 5xx responses and more than 5 percent of qualifying requests in a 10-minute lookback.	No traffic is normal; query error is alerting.	Immediate email.
API latency degradation	p95 above 2 seconds for a 15-minute lookback with at least 20 qualifying requests.	No traffic is normal; query error is warning.	Email warning.
Authentication system failure	At least 5 unexpected failure outcomes and more than 20 percent of all completed coachme.auth.attempts with the same coachme.operation.name and coachme.provider.name in a 10-minute lookback for 2 consecutive evaluations.	Normal bad credentials and expected denials are excluded; no auth traffic is normal; query error is alerting.	Immediate email.
Invitation abuse/rate limiting	At least 20 abuse or rate-limit outcomes in 10 minutes, excluding normal expired/revoked/consumed outcomes.	No traffic is normal.	Email warning.
Critical dependency/readiness	A named required readiness check fails for 2 consecutive 60-second evaluations.	Missing series is alerting.	Immediate email.
User-visible queue delay	coachme.queue.oldest_runnable.age exceeds 600 seconds for 5 minutes.	Missing worker/queue series is alerting.	Email warning.
Outbox publication delay	coachme.outbox.oldest_available.age exceeds 300 seconds for 5 minutes.	Missing publisher series is alerting.	Immediate email.
New dead letter	Any increase in coachme.outbox.transitions where coachme.transition.name is dead_letter over a 5-minute lookback.	No matching dead-letter transition series is normal; missing publisher/transition telemetry is covered by Telemetry pipeline degraded. Recover after 2 consecutive evaluations with zero increase. Operator review remains a required playbook action; replay/discard remains Event-owned.	Immediate email.
Sync degradation	At least 5 failed/rejected/conflicted coachme.sync.operations and more than 20 percent of all completed sync operations in the same coachme.operation.family over a 15-minute lookback for 2 consecutive evaluations.	No operations is normal; query error or a missing expected API series is warning.	Email warning; immediate only when the core sync path is unavailable.
Upload degradation	At least 5 failed coachme.upload.operations and more than 20 percent of all completed uploads in the same coachme.upload.purpose over a 15-minute lookback for 2 consecutive evaluations.	No uploads is normal; query error or a missing expected worker series is warning.	Email warning; immediate only when the core upload path is unavailable.
Provider degradation	At least 5 failed non-auth coachme.provider.requests and more than 20 percent of all completed provider requests for the same coachme.provider.name and coachme.operation.name over a 15-minute lookback for 2 consecutive evaluations.	No provider calls is normal; query error or a missing expected adapter series is warning; auth-provider operations are covered by Authentication system failure.	Email warning; immediate only when the required provider path is unavailable.
Reported client-connectivity degradation	At least 5 failure outcomes and more than 20 percent of all submitted coachme.client.connectivity.outcomes for the same coachme.app.surface over a 15-minute lookback for 2 consecutive evaluations.	No reports is normal because disconnected clients may be unable to report; query error is warning.	Email warning.
Production backup failed or stale	Any scheduled failure, or coachme.backup.latest_success.age exceeds the Environment-owned expected interval plus documented grace.	Missing success series is alerting.	Immediate email.
Telemetry pipeline degraded	Alloy heartbeat absent for 5 minutes, sustained exporter failures/drops for 5 minutes, or critical alert rules return No Data/query error.	External synthetics remain independent.	Immediate email.
Host disk pressure	Reviewed filesystem usage exceeds 85 percent for 10 minutes or local log-rotation evidence fails.	Missing host series is warning.	Email warning.
New Sentry fatal/regression	New fatal issue or configured regression rule.	Sentry integration health is separately monitored.	Immediate email.
TLS certificate expiry	Remaining validity below 14 days.	Missing certificate check is warning.	Email warning.

Alerts go to the active engineer by email. Release-related alerts include the release/build identifier, failed signal, affected surface and environment, first and latest timestamps, dashboard or Sentry link, relevant diagnosis playbook, and rollback or containment instructions. A release-critical signal is one attributable to the current release that indicates security or privacy exposure, cross-client access, data loss, failed migration, authentication outage, or failure of the canonical release-critical journey. A high noncritical signal is sustained user-visible degradation without those impacts. During the Test And Release Gate's two-hour observation window, an attributable release-critical signal is acknowledged and dispositioned within one hour. Outside that window, the internal critical-response target is 24 hours; a high noncritical signal is reviewed within two business days. These are internal operating targets, not a customer-facing 24/7 SLA.

All thresholds are initial. Recheck them after staging evidence and after the first 30 production days. Tuning a threshold requires a documented reason and must not disable detection of a known user-visible failure.

Dashboards And Domain Signals

Public V1 uses two Grafana dashboards and separate Sentry issue and release views.

Launch creates two Grafana dashboards:

1. Operational Overview: public synthetics, HTTP rate/error/latency, critical dependency health, PostgreSQL/Redis pressure, runtime restarts, deployment version, backup result, and top bounded error events.
2. Async And Integration Health: queues, outbox, offline sync, uploads/media processing, notifications, provider outcomes, deletion/purge jobs, and oldest actionable ages.

Sentry project and release views remain the web/mobile/backend crash dashboard. Do not reproduce them in Grafana.

The domain-signal matrix uses these fields:

• operational area;
• user-visible symptom;
• starting diagnostic reference;
• required log/metric/error signal;
• alert or dashboard;
• first developer lookup;
• recovery authority.

Required rows are authentication, invitation abuse/failure, API, PostgreSQL, Redis/workers, queue/outbox/dead letters, offline sync, uploads/storage/media processing, push/missing notification, registered provider outage, web/mobile crash, privacy deletion/purge jobs, and backup/restore evidence.

The matrix may summarize required signals but must link to canonical behavior. It must not restate invitation state, offline conflict behavior, outbox retry/replay/discard rules, provider fallback, privacy deletion, or backup/restore steps.

Developer Diagnosis Playbooks

Each playbook starts with safe lookup and stops at the authority that owns recovery.

Call these developer diagnosis playbooks, not support runbooks. Each playbook contains only:

• starting request, receipt, operation, event, job, device, or provider reference;
• exact Grafana/Sentry lookup sequence;
• expected state and signal;
• link to the authority that owns recovery;
• permitted safe action under the separately authorized recovery mechanism;
• stop/escalation condition.

Required playbooks:

Developer Diagnosis Playbooks table 1

Symptom	Starting lookup	Recovery hand-off
Failed upload	requestId, clientOperationId, upload/session id, or receiptId	API Contract, Offline Sync, Integration Register, Privacy.
Duplicate client/account/relationship	requestId, internal user id, workspace id, relationship id	Workflow State Map, Roles And Permissions, Privacy; no silent merge or direct data repair.
Broken invitation	requestId, invitation id, bounded error code	Workflow State Map and API Contract.
Stuck job/outbox/dead letter	jobId, eventId, correlationId	Event And Outbox for retry, replay, or discard; all mutations remain auditable.
Missing notification	event id, notification id, normalized provider ticket/receipt	Integration Register and Event And Outbox; never claim device delivery from a handoff receipt.
Provider outage	provider, bounded operation, correlation id	Integration Register fallback/degradation behavior.
Backup/restore signal failure	backup job/reference and environment	Environment Runbook; Observability supplies detection, not restore steps.

Public V1 does not require a BullMQ/admin web UI. Event And Outbox, Roles And Permissions, and Toolchain must jointly define a narrow authorized operator mechanism for any required inspect, retry, replay, or discard action. The Observability Plan does not choose whether that mechanism is a script, internal command, or protected endpoint; its playbooks link to the approved mechanism.

Direct database edits, unreviewed queue mutation, unaudited replay/discard, raw user-data browsing, and production shell changes are never diagnosis steps.

Safety, Retention, Access, And Incident Hand-Offs

This page summarizes approved controls but does not grant access or set retention.

This section summarizes companion decisions and links to them; it does not create a second privacy or permissions policy.

Privacy And Retention owns the approved operational data classes summarized here:

• minimized logs, Sentry error events, and diagnostic identifiers that can be linked to a user, workspace, relationship, device, operation, or receipt: maximum 30 days;
• strictly aggregate operational and synthetic metrics with no subject identifiers, free text, or user-controlled dimensions: maximum 13 months, with the actual provider-plan window recorded in Privacy And Retention and Integration Register;
• synthetic logs and any diagnostic request samples follow the 30-day identifiable-evidence maximum;
• the selected plan must retain aggregate availability metrics for at least 30 days or the 30-day objective must be changed through authority review;
• no copied production data in local, test, or staging telemetry;
• audit, legal, and security evidence remains separate and legal-review-dependent;
• deletion/anonymization interaction with short-lived identifiers is explicit in Privacy And Retention;
• a provider whose fixed retention exceeds an approved maximum remains Conditional or Blocked.

The Environment Runbook selects one conditional Grafana stack for staging and production. Every dashboard and alert filters the Environment-owned deployment.environment.name field, production alert queries are isolated, and separate staging/production write credentials prevent cross-environment confusion. Integration remains responsible for provider approval and evidence.

The required access model is owned outside Observability:

• Roles And Permissions adds a named production-developer telemetry context for minimized operational logs/error events and approved internal ids; it requires MFA and least privilege but grants no product-record or raw-payload access.
• Integration Register and Environment own Grafana/Sentry organization roles, provider-admin controls, access/activity logging, credentials, and environment/data-source isolation.
• The existing time-limited, target-scoped, reason-coded, audited support/admin grant continues to gate raw sync receipts, raw diagnostic records, product data, payloads, media, messages, and repair actions.
• Roles And Permissions permits searching approved minimized request, correlation, event, job, client-operation, device, receipt, user, workspace, or relationship identifiers in restricted telemetry and explicitly states that this does not authorize access to raw payloads or product records.
• Provider admin/activity logs supplement but never replace CoachMe's required audit record.

This Observability page only summarizes and links these approved rules. It does not grant access by itself.

The Observability Plan owns operational incident detection, a technical timeline, diagnostic references, telemetry-preservation decisions within the approved window, and a sanitized engineering follow-up. Any suspected exposure of sensitive data, unauthorized access, or privacy impact hands off immediately to the Privacy And Retention incident workflow. This authority does not introduce a competing legal, regulator, or user-notification process.

Public V1 uses a sanitized private project issue or equivalent private engineering note for operational incidents. Required fields are impact, detection time, affected services/surfaces, diagnostic references, deployment version, containment, resolution, and follow-up. Sensitive payloads and personal data never enter the incident note. A dedicated incident-management platform is Maturity.

Environment Expectations

Environment remains authoritative for topology, configuration, secrets, health, backup, and deployment.

The Observability page summarizes the expected signal posture while Environment Runbook remains authoritative for topology, configuration, secrets, data, and deployment.

Environment Expectations table 1

Environment	Observability posture
Local	Human-readable Pino formatting allowed; external Grafana/Sentry export off by default; sample data only.
Test/CI	Machine environment value test; no production provider export; instrumentation, client-summary, and redaction behavior use test exporters/fakes.
Staging	Production-equivalent Pino, metrics, safe client diagnostics, backend/web/mobile Sentry symbolication, dashboards, synthetics, and alert drill using safe staging data.
Production	Pino, bounded metrics, safe client diagnostics, Sentry errors/crashes, two dashboards, production-scoped email alerts, and public synthetics enabled only through approved provider/topology decisions.

Feature-Page Rules

Feature pages add only feature-specific signal mappings and link back here.

Every feature page adds only feature-specific observability detail and links to this authority. It must identify:

• stable operation and failure names;
• user-visible failure outcomes already owned by the feature/workflow authority;
• request, correlation, event, job, operation, device, receipt, workspace, or relationship identifiers that apply;
• required completion/failure logs;
• required canonical metric names, instrument types, units, emitting surfaces, and bounded dimensions;
• web/mobile connectivity, app-version, and Offline Sync client-summary fields where applicable;
• unexpected failures sent to Sentry;
• sensitive fields explicitly excluded from telemetry;
• whether an existing dashboard and alert covers the failure;
• starting diagnostic reference and recovery authority;
• observability evidence required for implementation.

Feature pages must not copy the global schema, metric-label policy, provider rows, retention classes, access rules, alert-delivery policy, or full diagnosis playbooks.

Event-emitting features continue to follow the Event And Outbox checklist. Offline-capable features continue to follow the Offline Sync checklist. Data-handling features continue to follow the Privacy checklist. Observability adds signal coverage; it does not absorb those checklists.

Required Observability Evidence

Observability defines evidence shape; Toolchain owns runners and the Test And Release Gate owns final approval.

This authority defines evidence shape. Toolchain owns runners and stable commands; the Test And Release Gate owns execution cadence, merge/release blockers, and final approval.

Launch evidence must prove:

• representative sensitive request, event, sync, client-diagnostic, media, message, nutrition/wellness, and provider payloads are redacted before export;
• request ids and API-issued whole-workflow correlation ids propagate across the full invitation-acceptance example plus accepted outbox and BullMQ work;
• event, job, operation, device, receipt, workspace, relationship, and approved internal user ids are searchable only in the minimized telemetry contexts approved by Roles;
• authenticated session ids are not exported;
• the API/Offline Sync client diagnostic summary safely reports web/mobile connectivity, app version/build, local queue depth/oldest age, restart recovery, and cache-purge outcomes without content;
• every canonical metric has its exact instrument, unit, producer, fixed allowed attributes, server-side client-dimension normalization, and pinned OpenTelemetry semantic-convention version, and high-cardinality metric/index labels are rejected;
• required outbox transition and Offline Sync metric families are emitted, including oldest local queued-operation age;
• health behavior, environment enum, backup cadence/grace, Docker log rotation, collector permissions, and disk-pressure signals pass the Environment-owned contract;
• Sentry symbolicates deliberate staging failures for API, worker, coach web, coach mobile, and client mobile using the approved server/source-map runtime path;
• synthetic checks fire and resolve a test alert at the configured email destination;
• alert queries prove their evaluation interval, lookback, pending, recovery, and No Data/query-error behavior;
• Grafana/Sentry/Alloy outage does not block ordinary accepted requests or jobs;
• missing Alloy heartbeat, exporter drops, dead-letter, oldest-runnable-job, oldest-unpublished-event, backup-failure/staleness, auth/invitation abuse, sync, connectivity, upload, and provider signals are observable;
• both Grafana dashboards and their alert queries load successfully against staging evidence;
• minimized telemetry lookup versus raw diagnostic/product-record access follows the Roles-owned split;
• raw receipt/payload inspection and any retry/replay/discard action use the separately authorized, audited mechanism;
• provider retention preserves aggregate availability metrics for at least 30 days and configures every signal no longer than its Privacy-owned maximum;
• no Launch test claims distributed trace linkage, session replay, profiling, paging, or other Maturity behavior.

Maturity Targets And Revisit Triggers

These items do not block Public V1 and require a named trigger before promotion.

This authority includes one concise matrix with a revisit trigger for each Maturity target:

• OpenTelemetry distributed tracing and Grafana Tempo: revisit after repeated causal diagnosis failures across API/worker boundaries.
• Browser/mobile OpenTelemetry: revisit only after the JavaScript/browser posture is stable and the safe client summary plus Sentry are demonstrably insufficient.
• Sentry performance tracing: revisit only if it becomes the chosen single trace backend for a defined need.
• Session replay, screenshots, and profiling: revisit only after explicit privacy review and a demonstrated issue that safe logs/errors cannot diagnose.
• Dynamic production debug toggles: revisit after a repeated need and after scoped, expiring, audited controls exist.
• Error budgets, burn-rate alerts, and formal SLO review: revisit after stable production traffic and an operating history.
• Paging/on-call tooling: revisit when more than one engineer participates or an explicit response-time commitment exists.
• Anomaly detection and advanced browser/user-journey synthetics: revisit after baseline alerts and real incident history exist.
• Dedicated incident platform: revisit when incident coordination exceeds the private issue template.
• BullMQ/admin control UI: revisit when the separately authorized operator mechanism is insufficient.
• Long-term telemetry warehouse or business analytics: outside the observability launch problem and subject to separate product/privacy authority.

Change Control

These triggers require coordinated authority review.

The Observability Plan requires review when:

• a new product surface, worker type, queue, provider, or sensitive data class is added;
• an alert repeatedly fires without action or misses a user-visible failure;
• a metric introduces an unbounded dimension or changes a canonical name/unit;
• a provider region, retention, pricing, quota, data-processing term, or SDK changes;
• a new telemetry signal, session replay, profiling, screenshot, attachment, or trace backend is proposed;
• diagnosis requires raw payload access, direct data repair, or a new support/admin permission;
• either operational retention class changes;
• the client diagnostic summary adds a field or durable storage;
• incident history shows that logs, metrics, errors, or existing identifiers are insufficient;
• a Maturity item is promoted to Launch.

Provider changes update Integration Register evidence. Package/build changes update Toolchain. Topology/config changes update Environment. Retention or privacy changes update Privacy And Retention. Access changes update Roles And Permissions. Correlation, client-diagnostic, event/job, or recovery-interface changes update API, Offline Sync, and Event And Outbox as applicable. The Observability Plan records only the resulting signal contract.

Related Docs

Each linked document remains authoritative for its own decision boundary.

Workflow State Map

Workflow states, starts, terminal outcomes, and recovery.

Roles And Permissions

Telemetry and raw diagnostic access, grants, and audit.

Privacy And Retention

Classification, retention, deletion, and privacy incidents.

API Contract Standard

Identifiers, envelopes, diagnostic summaries, receipts, and errors.

Event And Outbox Standard

Envelope, lifecycle, jobs, retry, replay, discard, and dead letters.

Offline Sync Protocol

Device, operation, receipt, local diagnostics, replay, and recovery.

Environment Runbook

Topology, health, hardening, secrets, backup, and restore.

Toolchain Contract

Packages, builds, source maps, versions, commands, and updates.

Integration Register

Providers, terms, regions, retention capability, and evidence.

Public V1 Scope Contract

Launch scope and documentation/evidence gates.

Development Readiness Review

Current implementation and launch blockers.

Technical Architecture Plan

System topology and authority hand-offs.