Communication / Voice Notes / Media Processing

Voice notes let coaches and clients send short audio guidance without leaving the conversation thread.

The V1 baseline voice-note contract covers recording, upload intent creation, object-storage handoff, audio validation, waveform metadata, processing state, playback URLs, optional transcript hooks, and offline mobile retry. Voice notes are conversation messages with a dedicated media asset lifecycle, so the messaging timeline remains canonical while audio binaries stay in MinIO.

Tags: Communication, Signed uploads, Media processing, Offline queue

Boundary decision

Communication owns voice-note message intent, asset metadata, waveform, processing state, playback authorization, and notification preview. Shared storage owns signed upload/download mechanics. Transcription can be added later as a separate processing output without changing the conversation message contract.

V1 baseline constraint

voice notes are short asynchronous messages, not live calls. Cap duration and file size aggressively so mobile upload, moderation, retention, and offline retry behavior stay predictable.

Note

Voice-note message and media payloads stay in this feature contract. Offline send queues, upload-session refresh, local draft retention, receipts, and conflict handling follow the Offline Sync Protocol.

Voice Note Flows

The flow separates local recording from canonical message acceptance so users can recover from weak network conditions.

Record And Queue

1. User presses and holds or taps the composer microphone in an active conversation.
2. Mobile requests microphone permission, starts a local recording, and shows duration, cancel, lock, and stop states.
3. Client enforces max duration locally and stores the draft file in app-private storage with a stable client_message_id.
4. When the user sends, mobile creates a pending voice-note message in SQLite before requesting upload instructions.
5. If offline, the draft remains queued with local duration, byte size, checksum, and retry metadata.

Upload And Accept

1. Client calls the voice-note upload-intent endpoint with conversation id, file metadata, duration, MIME type, checksum, and source device id.
2. Backend verifies conversation access, creates a pending VOICE_NOTE message and asset row, and returns a signed upload URL.
3. Client uploads the audio bytes directly to object storage and completes the upload session.
4. Media processing validates MIME type, duration, size, checksum, and decodability before marking the asset playable.
5. The accepted message fans out through realtime and push using a private preview such as "Voice note".

Playback

1. Conversation message DTO includes voice-note state, duration, waveform summary, and playable/error status.
2. Client requests a short-lived playback URL only when the user opens or plays the message.
3. Playback UI supports seek, speed control, background interruption recovery, and unread-to-read cursor advancement.
4. If processing fails, the message stays in the thread with a failed asset state and a retry or support path for the sender.
5. Playback URLs are never stored in SQLite, push payloads, analytics events, or logs.

Retry And Recovery

1. Mobile retry reuses the same client_message_id, device id, checksum, and local file path.
2. Server deduplicates upload intents by conversation, sender, device, and client message id.
3. Expired upload URLs can be refreshed while preserving the same pending message and asset id.
4. If the local file is missing, the client marks the message failed and asks the user to record again.
5. Queued operations reconcile to canonical ids through the shared Communication sync endpoint.

Domain Model

Voice notes extend Communication messages with a purpose-specific media asset and processing lifecycle.

Aggregates & Entities

• ConversationMessage: canonical timeline item with message_kind = VOICE_NOTE, sender, status, reply reference, source device, and sent time.
• VoiceNoteAsset: audio object metadata, storage key, duration, MIME type, checksum, byte size, waveform, processing status, and playback readiness.
• VoiceNoteUploadSession: signed upload intent, expected metadata, expiration, completion state, retry token, and object key.
• VoiceNoteProcessingResult: validation output, normalized metadata, optional transcoded object key, error code, and moderation flags.
• VoiceNotePlaybackGrant: short-lived authorization record or computed DTO for a participant requesting playback.
• VoiceNoteTranscript: optional future entity for generated transcript text, language, confidence, review status, and retention policy.

Value Objects & Rules

• VoiceNoteDuration: positive duration in milliseconds capped by workspace policy, with a stricter mobile composer cap for V1 baseline.
• AudioMimeType: allow-list such as audio/m4a, audio/mp4, audio/aac, or audio/webm according to supported clients.
• AudioChecksum: SHA-256 supplied by the client and verified by processing before the asset becomes playable.
• WaveformSummary: compact normalized amplitude buckets used for UI rendering, not a source-of-truth audio representation.
• VoiceNoteProcessingStatus: PENDING_UPLOAD, UPLOADED, PROCESSING, READY, FAILED, REDACTED.
• Voice notes can be replies and can receive reactions through the base messaging contract.
• A voice-note message is visible immediately as pending to the sender, but recipient playback starts only after processing reaches READY.
• Push notifications never include signed URLs, transcript text, waveform data, or raw audio metadata beyond a safe generic preview.

Model	Owned By	Important Methods	Emits
ConversationMessage	Communication	createVoiceNote(), attachVoiceAsset(), markAccepted(), redact()	VoiceNoteMessageCreated, MessageRedacted
VoiceNoteAsset	Communication	reserveUpload(), markUploaded(), startProcessing(), markReady(), markFailed()	VoiceNoteUploadReserved, VoiceNoteReady, VoiceNoteFailed
VoiceNoteUploadSession	Communication	create(), refresh(), complete(), expire()	VoiceNoteUploadCompleted
VoiceNotePlaybackGrant	Communication	issueForParticipant(), assertNotExpired(), toSignedUrl()	VoiceNotePlaybackRequested

Table Structure

Keep message ordering in the base Communication tables and store audio-specific metadata in dedicated voice-note tables.

Table	Purpose	Important Columns	Constraints & Indexes
communication_messages	Canonical conversation message.	id, client_message_id, conversation_id, sender_user_id, message_kind, reply_to_message_id, reply_snapshot jsonb, status, source_device_id, sent_at, redacted_at	message_kind = VOICE_NOTE requires one active voice-note asset; unique nullable (conversation_id, sender_user_id, client_message_id).
communication_voice_note_assets	Audio asset metadata and processing state.	id, message_id, workspace_id, conversation_id, sender_user_id, storage_bucket, storage_key, original_filename, mime_type, byte_size, duration_ms, checksum_sha256, waveform jsonb, processing_status, processing_error_code, uploaded_at, ready_at, redacted_at	Unique (message_id); index (processing_status, uploaded_at); index (conversation_id, ready_at desc).
communication_voice_note_upload_sessions	Signed upload intent and retry state.	id, voice_note_asset_id, conversation_id, sender_user_id, source_device_id, client_message_id, expected_mime_type, expected_byte_size, expected_duration_ms, expected_checksum_sha256, upload_url_expires_at, completed_at, expired_at	Unique active (conversation_id, sender_user_id, source_device_id, client_message_id); index (upload_url_expires_at).
communication_voice_note_processing_jobs	Durable processing attempts for scans, metadata extraction, and waveform generation.	id, voice_note_asset_id, job_status, attempt_count, worker_version, started_at, completed_at, error_code, error_detail jsonb	Index (job_status, started_at); retain failed attempts for support diagnostics.
communication_voice_note_playback_events	Audit and diagnostics for playback URL issuance.	id, voice_note_asset_id, conversation_id, actor_user_id, issued_at, expires_at, surface, request_context jsonb	Index (actor_user_id, issued_at desc); prune after diagnostics window.
communication_voice_note_transcripts	Future optional transcription output.	id, voice_note_asset_id, language, transcript_text, confidence, status, generated_at, reviewed_at, redacted_at	Unique active (voice_note_asset_id, language); keep disabled until transcription policy is approved.

Offline storage

SQLite stores local draft file references, pending upload sessions, canonical asset state, duration, waveform, and retry receipts. It must not store signed playback URLs, long-lived copies of received audio, or transcript text beyond the offline cache policy.

Lifecycle Policy

Voice notes are sensitive message content. Keep the timeline stable while controlling media availability through asset state.

Message And Asset Lifecycle

• Upload intent creation reserves both a message id and a voice-note asset id so retries reconcile to the same timeline item.
• Pending uploads expire after a short window; expired sessions can be refreshed if the conversation remains writable and the same actor still owns the pending message.
• Uploaded assets become recipient-playable only after processing validates type, checksum, byte size, duration, and decodability.
• Failed processing keeps the message visible to the sender with a failure reason and hides playback from recipients.
• Redaction keeps the message id, sender, kind, and sent time but removes playback grants, waveform data, transcript output, and preview metadata from normal clients.

Retention And Privacy

• Voice-note binaries follow Communication message retention by default unless workspace privacy policy requires shorter audio retention.
• Playback events, processing attempts, and upload sessions can be pruned after the support diagnostics window.
• Transcripts, when enabled, require separate retention and redaction rules because they make audio content searchable text.
• Privacy export should include voice-note metadata and a controlled download path for authorized users, not raw object-storage keys.
• Hard purge removes or tombstones object-storage assets and anonymizes residual metadata according to account and relationship retention policy.

Caution

Do not make received voice notes permanently downloadable on mobile by default. Cache only what the product explicitly supports for offline playback, and make cache eviction visible to the playback layer.

API Contracts

Voice-note endpoints wrap signed storage access with conversation permission checks and idempotent mobile retries.

Endpoint	Purpose	Request	Response
POST /api/v1/communication/conversations/{conversationId}/voice-notes/upload-intents	Create or return a voice-note upload session.	Client message id, source device id, MIME type, byte size, duration ms, checksum, optional reply target.	Pending message dto, voice-note asset id, signed upload URL, headers, expiration, constraints, sync receipt.
POST /api/v1/communication/voice-notes/{assetId}/complete-upload	Confirm object upload and enqueue processing.	Upload session id, checksum confirmation, uploaded byte size, client completed timestamp.	Voice-note asset dto with processing status and conversation revision.
POST /api/v1/communication/voice-notes/{assetId}/refresh-upload	Refresh an expired upload URL for the owner of a pending asset.	Upload session id, client message id, source device id.	New signed upload URL, headers, expiration, unchanged message and asset ids.
GET /api/v1/communication/voice-notes/{assetId}/playback	Issue a short-lived playback URL.	Actor session, optional surface and client cache metadata.	Signed playback URL, expiration, duration, MIME type, waveform, cache policy.
GET /api/v1/communication/conversations/{conversationId}/messages	Load messages with voice-note summaries.	Cursor, limit, include reaction summary flag.	Message DTOs with voice-note processing state, duration, waveform, and playable flag, but no signed URLs.
POST /api/v1/communication/sync	Reconcile queued voice-note send and upload operations.	Device id, ordered operations, client operation ids, expected conversation revisions, upload session references.	Accepted operations, rejected operations, canonical id map, upload refresh hints, updated conversations.
POST /api/v1/internal/communication/voice-notes/{assetId}/processing-result	Worker callback for media processing results.	Worker token, status, normalized duration, waveform, checksum result, error code, moderation flags.	Updated asset dto, outbox event ids.

API note

Conversation message DTOs should use a nested voiceNote object with assetId, durationMs, waveform, processingStatus, playable, and failureReason. Playback URL issuance stays separate so ordinary message fetches do not leak media access.

Key Code Snippets

Implementation sketches show the voice-note schema, upload intent use case, and playback grant policy.

Drizzle schema shape

apps/api/src/communication/db/voice-notes.schema.ts

export const communicationVoiceNoteAssets = pgTable("communication_voice_note_assets", {
  id: uuid("id").primaryKey().defaultRandom(),
  messageId: uuid("message_id").notNull(),
  workspaceId: uuid("workspace_id").notNull(),
  conversationId: uuid("conversation_id").notNull(),
  senderUserId: uuid("sender_user_id").notNull(),
  storageBucket: text("storage_bucket").notNull(),
  storageKey: text("storage_key").notNull(),
  mimeType: text("mime_type").notNull(),
  byteSize: integer("byte_size").notNull(),
  durationMs: integer("duration_ms").notNull(),
  checksumSha256: text("checksum_sha256").notNull(),
  waveform: jsonb("waveform").$type<WaveformSummary>(),
  processingStatus: text("processing_status")
    .$type<VoiceNoteProcessingStatus>()
    .notNull()
    .default("PENDING_UPLOAD"),
  processingErrorCode: text("processing_error_code"),
  uploadedAt: timestamp("uploaded_at", { withTimezone: true }),
  readyAt: timestamp("ready_at", { withTimezone: true }),
  redactedAt: timestamp("redacted_at", { withTimezone: true }),
}, (table) => ({
  messageUnique: uniqueIndex("voice_note_assets_message_unique").on(table.messageId),
  processingIdx: index("voice_note_assets_processing_idx")
    .on(table.processingStatus, table.uploadedAt),
  conversationReadyIdx: index("voice_note_assets_conversation_ready_idx")
    .on(table.conversationId, table.readyAt),
}));

Create upload intent

apps/api/src/communication/use-cases/create-voice-note-upload-intent.ts

@Injectable()
export class CreateVoiceNoteUploadIntentUseCase {
  constructor(
    private readonly conversations: ConversationRepository,
    private readonly voiceNotes: VoiceNoteRepository,
    private readonly storage: MediaStorageGateway,
    private readonly access: CommunicationAccessPolicy,
    private readonly policy: VoiceNotePolicy,
    private readonly tx: TransactionRunner,
  ) {}

  async execute(input: CreateVoiceNoteUploadIntentInput, actor: Actor): Promise<VoiceNoteUploadIntentDto> {
    return this.tx.run(async () => {
      const conversation = await this.conversations.lockById(input.conversationId);
      await this.access.assertCanSend(actor, conversation);
      this.policy.assertAllowedUpload(input);

      const existing = await this.voiceNotes.findUploadSession({
        conversationId: conversation.id,
        senderUserId: actor.userId,
        sourceDeviceId: input.sourceDeviceId,
        clientMessageId: input.clientMessageId,
      });
      if (existing) return this.voiceNotes.refreshIntent(existing);

      const message = conversation.recordVoiceNote({
        senderUserId: actor.userId,
        clientMessageId: input.clientMessageId,
        replyToMessageId: input.replyToMessageId,
        sourceDeviceId: input.sourceDeviceId,
      });

      const asset = VoiceNoteAsset.reserveUpload(message, input);
      const upload = await this.storage.createSignedUpload({
        purpose: "COMMUNICATION_VOICE_NOTE",
        contentType: input.mimeType,
        byteSize: input.byteSize,
        checksumSha256: input.checksumSha256,
        ownerUserId: actor.userId,
      });

      await this.voiceNotes.saveReserved(message, asset, upload);
      await this.conversations.save(conversation);
      return VoiceNoteUploadIntentDto.from(message, asset, upload);
    });
  }
}

Playback grant policy

apps/api/src/communication/use-cases/get-voice-note-playback.ts

@Injectable()
export class GetVoiceNotePlaybackUseCase {
  constructor(
    private readonly voiceNotes: VoiceNoteRepository,
    private readonly access: CommunicationAccessPolicy,
    private readonly storage: MediaStorageGateway,
  ) {}

  async execute(assetId: string, actor: Actor): Promise<VoiceNotePlaybackDto> {
    const asset = await this.voiceNotes.getReadyAsset(assetId);
    await this.access.assertCanReadConversation(actor, asset.conversationId);
    asset.assertPlayable();

    const signed = await this.storage.createSignedDownload({
      storageKey: asset.playbackStorageKey,
      contentType: asset.mimeType,
      expiresInSeconds: 300,
      disposition: "inline",
    });

    return VoiceNotePlaybackDto.from(asset, signed);
  }
}

Events & Background Jobs

Voice-note delivery uses the same message outbox while media validation runs asynchronously.

Domain And Integration Events

• VoiceNoteUploadReserved: records pending message state and upload expiration for retry and cleanup.
• VoiceNoteUploadCompleted: schedules media scan, metadata extraction, checksum verification, and waveform generation.
• VoiceNoteReady: marks message accepted, updates conversation summary, fans out realtime event, and schedules push notification.
• VoiceNoteFailed: updates sender-visible failure state and support diagnostics.
• VoiceNotePlaybackRequested: audit event for playback grant issuance and support diagnostics.
• MessageRedacted: removes playback grants, hides waveform, and schedules object-storage redaction or purge.

Jobs

• Voice-note processing worker validates MIME type from bytes, file size, checksum, duration, channel count, and decodability.
• Waveform worker creates a compact amplitude summary and optional normalized playback derivative when source format support differs by client.
• Upload cleanup job expires abandoned sessions and removes orphaned object-storage uploads.
• Notification worker sends generic voice-note previews without audio URLs or transcript text.
• Retention worker purges redacted or expired binaries according to workspace policy and records tombstone metadata.

Permissions & Access Rules

Every voice-note operation must be scoped to the relationship conversation and the concrete media asset state.

Actor Rules

• Coach can record, upload, play, reply to, react to, and redact according to workspace role permissions for active relationships they can access.
• Client can record, upload, play, reply to, and react to voice notes only in their own active coach relationship conversation.
• Only the sender can refresh or complete a pending upload session, and only before the pending message is failed, redacted, or expired.
• Support/admin playback access is disabled by default and must be audited when enabled for troubleshooting or legal workflows.

Media Rules

• Signed upload URLs are single-purpose, short-lived, size-limited, content-type constrained, and bound to the reserved asset id.
• Signed playback URLs are issued only for READY assets visible to the actor and expire quickly.
• Object-storage keys, upload URLs, and playback URLs must not appear in push payloads, realtime public events, analytics events, or client logs.
• Processing failures are exposed as typed user-safe reasons such as unsupported format, too long, too large, upload expired, or processing failed.

Web, Coach Mobile, Client Mobile

Mobile owns recording. Web focuses on playback, review, and future desktop recording only if browser support is approved.

Coach Web

• Displays voice-note bubbles with duration, waveform, processing state, playback controls, speed control, reply, reaction, and redaction actions.
• Requests playback URLs on demand and releases player state when the coach switches clients or conversation threads.
• Desktop recording can remain disabled in Public V1 if browser permission, codec, and support policy are not finalized.

Coach Mobile

• Composer supports tap-to-record, hold-to-record, lock recording, cancel gesture, preview, send, and failed-upload retry.
• SQLite tracks local draft file path, pending upload session, retry count, canonical message id, asset id, and processing state.
• Playback integrates with system audio interruptions and avoids uploading or playing while the app lacks microphone or network permission state.

Client Mobile

• Simple recording controls support quick clarification, food-plan questions, workout feedback, and check-in context from Today or messages.
• Offline queue makes the pending voice note visible immediately and retries upload without duplicating the conversation item.
• Client can delete a local unsent draft, but accepted message removal follows Communication redaction policy instead of local-only deletion.

Test Scenarios & V1 Baseline Decisions

Voice notes need coverage across domain rules, signed storage, mobile retry, processing workers, permissions, and privacy.

Tests

• Domain: duplicate upload-intent calls with the same client message id return the same message, asset, and refreshed upload session.
• Domain: asset cannot become READY unless checksum, MIME type, byte size, duration, and decoder validation pass.
• API: coach and client cannot create, complete, refresh, or play voice notes outside their relationship conversation.
• API: playback endpoint never returns a URL for pending, failed, redacted, expired, or unauthorized assets.
• Worker: processing failure emits a typed failure event and does not update conversation summary as a playable message.
• Mobile E2E: offline recording queues locally, uploads after reconnect, and reconciles to one canonical message.
• Mobile E2E: expired upload URL refresh keeps the same local pending bubble and does not duplicate unread counts.
• Web E2E: ready voice note plays through a short-lived URL and refetches playback authorization after expiry.

V1 Baseline Decisions

• Public V1 recording is mobile-only and uses AAC in an MPEG-4 .m4a container from Expo on iOS and Android. The backend accepts audio/m4a, audio/mp4, and audio/aac; browser audio/webm remains disabled for public recording until desktop recording is explicitly approved.
• Cap each voice note at 120 seconds and 10 MB in Public V1. Workspace policy may lower those limits but cannot raise them for launch; longer consultations belong in future calls or external session notes.
• Received voice-note audio always streams through fresh short-lived playback URLs in Public V1. Mobile may cache duration, waveform, processing state, and the sender's unsent local draft for retry, but not received audio binaries or signed playback URLs.
• Automatic transcription is out of Public V1. Keep transcript hooks and tables disabled so a later workspace opt-in transcription policy can be added without changing the message contract.
• Support/admin playback access is disabled by default and requires elevated audited permission, a reason code, and explicit troubleshooting or legal workflow. Playback grant events are always written and retained for the 30-day diagnostics window before purge or anonymization.