Skip to main content

Real-Time Streaming

Coming soon

Real-time streaming works internally today and powers our production iOS app, but we haven't opened it up as a supported integration yet. We're launching it soon with a proper integration guide. The design below reflects how it works today — treat it as a preview, and contact your account team if you want early access.

send-message runs generation asynchronously — the HTTP call returns a runId immediately, before the assistant reply exists. To render tokens as they are produced (instead of polling conversation-state), subscribe to the platform's real-time stream over Firebase Realtime Database (RTDB).

This is the recommended path for any interactive client (mobile or web) that needs live chat.

Requires Firebase for your tenant

Real-time streaming is delivered through a Firebase Realtime Database that the platform mirrors conversation events into. Your tenant must have the Firebase bridge enabled and be given a Firebase config for your client apps. Contact your account team to turn it on. Clients without Firebase can still use polling.

How It Works​

During generation, the platform emits token chunks, run lifecycle events, and finalized messages onto internal Kafka topics. A platform component (the Firebase bridge) subscribes to those topics and mirrors each event into RTDB, keyed by conversation. Your client attaches a Firebase listener to the relevant node and receives every write in order.

send-message (POST)  ──▶  runId
        │
        ▼
  generation runs ──▶ Kafka ──▶ Firebase bridge ──▶ RTDB nodes
        │                                               │
        │                             client listener ◀─┘  (real-time)
        ▼
  poll conversation-state  (alternative, no Firebase)

You authenticate to RTDB with the same Firebase ID token you send as the Authorization: Bearer header on REST calls (see Building a Mobile App). Database security rules scope each user to their own conversations.

RTDB Node Layout​

All nodes are keyed by the conversation. {conversationId} is the threadId returned by create-thread (the same value you pass as conversation_key).

NodePurpose
streaming/{conversationId}/{messageSequence}Incremental token chunks for the in-flight assistant message
streaming/{conversationId}/{messageSequence}/metaStream status for that message (streaming → complete/error)
conversations/{conversationId}/statusAuthoritative conversation activity state (busy/idle)
events/{eventName}/{conversationId}/messagesFinalized messages once a turn settles
events/{eventName}/{conversationId}/runsGeneration started / completed markers

{messageSequence} is the sequence number of the assistant message being generated. Read the current sequence from the seq field on the conversations/{conversationId}/status node.

{eventName} is a sanitized event type, e.g. llm_v1_common_v1_conversationmessagepublishedevent for messages.

Streaming Chunks​

Listen on streaming/{conversationId}/{messageSequence}. Each write replaces the whole node, so you receive the accumulated chunks array plus a meta object every time.

{
  "meta": {
    "run_id": "64403669-5989-4ec3-ad9c-d84223f9679f",
    "status": "streaming",              // "streaming" | "complete" | "error"
    "started_at": "2026-04-23T22:43:44.700Z",
    "finished_at": null,
    "error_message": null
  },
  "chunks": [
    { "content": "I can ",  "type": "STREAM_CHUNK_TYPE_TEXT", "timestamp": "…", "is_final": false },
    { "content": "help ",   "type": "STREAM_CHUNK_TYPE_TEXT", "timestamp": "…", "is_final": false },
    { "content": "with…",   "type": "STREAM_CHUNK_TYPE_TEXT", "timestamp": "…", "is_final": true, "finish_reason": "stop" }
  ]
}

Chunk fields​

FieldMeaning
contentThe token text (or reasoning/tool-call fragment) for this chunk
typeSTREAM_CHUNK_TYPE_TEXT, STREAM_CHUNK_TYPE_REASONING, or STREAM_CHUNK_TYPE_TOOL_CALL
is_finaltrue on the last chunk of the message
finish_reasonstop, tool_calls, length, or error — present on the final chunk

The chunks array is indexed by position and may contain gaps (nulls) mid-stream as writes arrive out of order — always render by accumulating non-null entries in order, don't assume the array is dense.

Filter chunk types for display: render STREAM_CHUNK_TYPE_TEXT into the message bubble; treat STREAM_CHUNK_TYPE_REASONING and STREAM_CHUNK_TYPE_TOOL_CALL separately (or hide them). When reasoning is enabled, reasoning chunks arrive first, then text chunks.

Detecting completion​

A stream is done when any of these is true — check meta.status first, it is authoritative:

  1. meta.status == "complete" (or "error"), or
  2. the last chunk has is_final: true with a finish_reason, or
  3. conversations/{conversationId}/status returns to a non-generating state.

Conversation Status​

Listen on conversations/{conversationId}/status for a single last-write-wins node that tracks what the conversation is doing. Use it to drive a busy indicator and to de-duplicate out-of-order writes via seq.

{
  "state": "CONVERSATION_ACTIVITY_STATE_GENERATING",
  "run_id": "64403669-5989-4ec3-ad9c-d84223f9679f",
  "seq": "7",                       // int64 as string
  "since": "2026-04-23T22:43:44.712Z",
  "error": null                     // { "code": "...", "message": "..." } on failure
}
state valueMeaning
CONVERSATION_ACTIVITY_STATE_IDLENothing running — safe to send the next message
CONVERSATION_ACTIVITY_STATE_QUEUEDGeneration accepted, not started yet
CONVERSATION_ACTIVITY_STATE_GENERATINGModel is producing tokens
CONVERSATION_ACTIVITY_STATE_EXECUTING_TOOLSRunning tool calls
CONVERSATION_ACTIVITY_STATE_COMPACTINGCompacting context
CONVERSATION_ACTIVITY_STATE_ERRORLast run failed — see error

Finalized Messages​

Once a turn settles, the full message is written under events/{eventName}/{conversationId}/messages (event name llm_v1_common_v1_conversationmessagepublishedevent). Fields here use snake_case:

{
  "message": {
    "role": 2,
    "content": [ { "type": "CONTENT_PART_TYPE_TEXT", "content": "I can assist you with…" } ],
    "generated_by": "64403669-5989-4ec3-ad9c-d84223f9679f",
    "sequence": 8,
    "tool_calls": [
      { "id": "call_abc123", "name": "get_weather", "status": "…",
        "is_client_tool": false, "arguments_json": "{\"city\":\"NYC\"}" }
    ]
  },
  "eventContext": { "correlation_id": "…", "event_id": "…", "event_name": "…", "emitted_at": "…", "version": "1" }
}

Run lifecycle markers ({ "runId": "...", "status": "..." }) appear under events/{eventName}/{conversationId}/runs for the generation-started and generation-completed events.

Subscribing (example)​

Any Firebase Realtime Database SDK works — iOS, Android, or web. This is the shape our iOS client uses:

// 1. Send the message (async) — capture nothing beyond the runId if you like.
try await restate.sendMessage(conversationKey: conversationId, text: userText)

// 2. Observe the streaming node for the in-flight assistant message.
let seq = currentAssistantSequence            // from conversations/{id}/status → seq
let ref = database.reference()
    .child("streaming").child(conversationId).child(String(seq))

let handle = ref.observe(.value) { snapshot in
    guard let json = snapshot.value,
          let data = decode(StreamingData.self, from: json) else { return }

    let text = data.chunks
        .compactMap { $0 }
        .filter { $0.type == "STREAM_CHUNK_TYPE_TEXT" || $0.type == nil }
        .map { $0.content }
        .joined()
    render(text)                               // update the bubble

    if data.meta?.status == "complete" || data.meta?.status == "error" {
        ref.removeObserver(withHandle: handle) // stop listening
    }
}

Web (modular Firebase JS SDK) follows the same pattern — call onValue() on the streaming/{conversationId}/{seq} node.

When to Use Polling Instead​

Streaming needs a Firebase client and RTDB access. If you only need the final answer — a backend job, a webhook handler, a simple script — polling conversation-state is simpler and requires nothing beyond your API key. Streaming and polling read the same underlying run; you can mix them.