Streaming Protocol

This page documents the wire protocol so you can implement a client on a platform we do not ship an SDK for. If you use the Web or iOS SDK, all of this is handled for you.

Protocol version is 1. Control messages and events are text JSON frames; measurement data is binary. The maximum size of any message is 8 KB.

Connect to the wsUrl returned by POST /sdk/v1/cloud/sessions.

Handshake

The token goes in the first message, never in the query string — query strings end up in proxy and access logs.

// client → server, immediately after the socket opens
{ "t": "hello", "protocol": 1, "token": "<sessionToken>" }

// server → client
{ "t": "ready", "sessionId": "cj_…", "acceptedFeatures": ["hrv", "bp", "spo2"] }

acceptedFeatures is the intersection of what you asked for and what the session token allows. Start sending frames only after ready.

Client → server: control

{ "t": "hello", "protocol": 1, "token": "<sessionToken>" }
{ "t": "stop" }                               // finish normally, then wait for "final"
{ "t": "abort", "reason": "user_cancelled" }  // give up, no "final" will arrive
{ "t": "ping" }

Client → server: the four binary packets

Every binary message starts with the same two bytes: type, then protocol version. There are four types, the server understands all four, and an unknown type is a protocol_error that ends the session.

Only type 1 is required. Types 2 and 3 overlap on purpose — slots 6–8 of the telemetry packet carry exactly what the quality packet carries — and both forms are live: send either, or both, or neither. Type 2 is not deprecated.

Every packet is self-delimiting: types 1, 2 and 3 have a fixed length, and type 4 declares its payload length in its header. That is what lets the same packets be concatenated into a batch file with no message boundaries to rely on.

Type 1 — the pulse crop

One frame is one binary message, up to 30 per second.

3 900 bytes per frame, about 117 KB/s at 30 fps.

Timestamps are explicit and a uniform frame rate is not required. The inter-frame interval is an input to the model, and the sampling rate used for the blood-pressure window is derived from the median interval. When the face is lost, simply do not send a frame — the gap widens and the pipeline copes. Degradation on a poor network is gradual rather than a hard failure.

Type 2 — the optional quality packet

Up to twice per second. In v1 the server keeps the most recent one and reports it with the session's job statistics, but does not gate the measurement on it.

Exactly 18 bytes. A packet of any other length is a protocol_error.

Type 3 — telemetry

The same quality signals plus eye, jaw and head-pose state, quantized to one byte each so they can ride along at the full frame rate. 62 bytes, 1.9 KB/s at 30 Hz.

Nine slots are assigned. The remaining 47 are reserved and must be zero:

Quantization is fixed: values in [0, 1] are sent as round(v * 255), and angles as round((clamp(deg, -90, 90) + 90) / 180 * 255). The server inverts exactly that. Sending float32 would cost three times the bytes and buy nothing — quantized bytes do not compress either.

Type 4 — auxiliary JPEG chunk

A 448×448 JPEG is 21–37 KB and cannot fit in the 8 KB message limit, so it is cut into chunks that the server reassembles by tMicros.

byteLength is mandatory and is checked: a packet whose actual payload is not byteLength bytes long is a protocol_error. It exists because in batch_frames the packets are concatenated into one file with no message boundaries — without an explicit length the parser cannot find the end of a chunk. chunkIndex must be below chunkCount, which must be non-zero.

An incomplete set — chunks still missing after 2 seconds — is dropped silently. That is not a protocol error: the optional emotion and gaze module degrades, core vitals are untouched.

When the session ordered emotion or gaze, the node runs its emotion and gaze models on every assembled frame and answers with live events (see below); the aggregated values land in the final measurement. A node without those model weights strips the features from acceptedFeatures in ready and the job is not billed for them. The web SDK switches aux to "stream" automatically when a live session ordered either feature.

Server → client: events

{ "t": "ready", "sessionId": "cj_…", "acceptedFeatures": ["hrv", "bp", "spo2"] }
{ "t": "hr",    "hr": 72.4, "sqi": 0.61, "tMs": 12000 }
{ "t": "hrv",   "rmssdMs": 34.2, "sdnnMs": 48.1, "meanRrMs": 828 }
{ "t": "bp",    "systolicMmHg": 118, "diastolicMmHg": 76, "confidence": 0.62 }
{ "t": "spo2",  "value": 97, "confidence": 0.62 }
{ "t": "emotion", "probs": [0.01, 0.02, 0.01, 0.01, 0.6, 0.3, 0.03, 0.02], "tMs": 12000 }
{ "t": "gaze",  "yaw": 0.12, "pitch": -0.04, "tMs": 12000 }
{ "t": "stats", "received": 900, "dropped": 0, "backlogMs": 40 }
{ "t": "final", "measurementId": "…", "measurement": { /* MeasurementResult */ } }
{ "t": "error", "code": "…", "message": "…", "fatal": true }
{ "t": "pong" }

• hr is emitted only while the signal-quality index is above 0.38 — the same threshold the on-device pipeline uses. Below it, no heart rate is sent at all, rather than a noisy one.
• emotion carries eight probabilities in the fixed order Anger, Contempt, Disgust, Fear, Happiness, Neutral, Sadness, Surprise. gaze angles are radians; low-confidence and closed-eye samples are not sent. Both arrive at up to 2 Hz, following the aux frame rate.
• stats tells you how far behind the server is. A growing backlogMs or a rising dropped means the network or the node cannot keep up.
• final carries the stored measurement id and the full measurement result, with origin: "cloud" and attested: true.

Compression

The pulse signal is a fraction of a percent of the amplitude in the green channel, and inter-frame prediction destroys it — published measurements put the degradation of patch-based methods under MPEG-4 at roughly 5.8×. This is also why recompressed video is not supported for batch.

permessage-deflate is off because it does not help: on binary crops and quantized telemetry it inflates traffic by about 2 % — pixels and mantissas are uncorrelated at the byte level — while costing memory per connection.

Ending a session

• Send { "t": "stop" } and wait for final.
• Send { "t": "abort" } to discard the session; no final is produced.
• If the socket drops, the session ends as aborted. There is no resume in v1 — recurrent state dies with the connection. This is what delivery: 'auto' exists to cover: the SDK re-sends its local buffer as a frame batch and still produces a result.

Next: Batch processing.