Skip to article
HB CodeDocsGet the app

HandbookRelay & accounts

Device routing, reconnect, and recovery

Registration answers and refusals, the two watchdogs of a dead link, the reconnect grace, how timeline events reach only interested phones through two bounded buffers, the other relay events, push registration, and why the relay runs as one process.

Checked against the source on 5 October 2026

On this page

Registration and discovery

ClientWebSocket headersCross-checked against
DesktopX-Client-Type: desktopA desktop device row the same account registered over HTTP.
PhoneX-Client-Type: mobile, plus X-Target-Desktop-Device-ID for the selected desktopA mobile device row. The relay reads the target desktop from the register payload, else from the header, on every registration, never from a stored session.

The first frame on the socket is the register message. The relay answers registered with sessionId, resumeToken, expiresAt, and relayInstanceId. A register message that carries a valid resume token gets resumed with the same sessionId, expiresAt, and relayInstanceId, and no new token. When a desktop registers, the relay also sends it the current interest leases of its phones.

A refused registration gets an error frame with one of these codes.

CodeCause
invalidClientTypeThe client type is neither desktop nor mobile.
invalidDeviceIdThe device ID is not a UUID.
deviceNotFoundNo device row exists. The device must register over HTTP first.
deviceOwnershipFailedThe device row belongs to another account.
deviceTypeMismatchThe client type differs from the type of the device row.
desktopRelayDisabledThe desktop row is not eligible for the relay.
desktopDeviceForgottenThe device was forgotten in the account.
protocolUpdateRequiredThe relay protocol version is missing or below the floor.
invalidRelayProtocolVersionThe relay protocol version cannot be parsed.
invalidResumeThe resume token is unknown or expired. The client registers again without it.
relayDrainingThe process is draining for a deploy.
relayProtocolPolicyUnavailable, invalidRelayProtocolPolicy, deviceValidationUnavailable, validation_db_errorThe relay could not read its protocol policy or the device row. The fault is on the server.

Reconnect grace and the reconnect snapshot

The desktop closes a silent link after one minute, and the relay closes it after more than three minutes
The desktop closes a silent link after one minute, and the relay closes it after more than three minutesOne time axis in seconds. The desktop link dies silently at 0. The desktop sends heartbeats at 15, 30, and 45 seconds and gets no pong. Its check at 60 seconds finds more than 45 seconds without an inbound frame, so it drops the connection and retries with growing waits. The relay pings every 30 seconds and gets no answer. It allows 180 seconds of silence and closes the socket at its check at 200 seconds. Then the presence grace runs for 90 seconds, and phones see the desktop as offline at 290 seconds.
Desktoplink dies at 0 s
Relaysame socket
Phonerelay presence
45 s
heartbeat every 15 s, no pong
check at 60 s: reconnect
retry waits grow: up to 0.5, 1, 2, 4, 8, 16, then 30 s
for 140 s the desktop retries and the relay still waits
180 s of silence allowed
presence grace 90 s
ping every 30 s, no answer
socket closed at 200 s
offline at 290 s
desktop reported as connected
reconnecting
offline
0 s
60 s
120 s
180 s
240 s
300 s
  • Desktop watchdogINBOUND_ACTIVITY_TIMEOUT_SECS = 45

    Every 15 seconds the desktop checks the time since the last inbound frame and then sends a heartbeat, which the relay answers with a pong. A gap of more than 45 seconds ends the connection. The check runs on the 15 second tick, so the desktop gives up a dead link at most 60 seconds after its last frame.

  • Reconnect backoffreconnect_backoff

    Each wait is a random time between 100 milliseconds and a cap. The cap starts at 500 milliseconds, doubles per failed attempt, and stops at 30 seconds. The connect and the registration answer each have a 10 second timeout. The failure count resets only after a connection stayed ready for 30 seconds.

  • Relay watchdogCLIENT_TIMEOUT = 180 s

    Every 30 seconds the relay checks the time since the socket’s last frame and then sends a ping. More than 180 seconds of silence stops the socket, so the close comes 180 to 210 seconds after the last frame. A text, ping, or pong frame resets the clock.

  • Presence graceDESKTOP_RECONNECT_GRACE_SECS = 90

    Starts when the relay closes the desktop socket. Phones see reconnecting during the grace. At its end the relay broadcasts device-status disconnected with the reason reconnect_timeout, unless the desktop registered again first.

A socket that closes with a close frame is known to both sides at once. A link that dies silently is found by two independent watchdogs, and the desktop’s is the faster one. The desktop reconnects long before the relay would close the old socket, so in the normal case phones never see the desktop as offline.

A closed desktop socket ends its requests, while a closed phone socket ends its lease
A closed desktop socket ends its requests, while a closed phone socket ends its leaseThe relay treats the two sides differently. When the desktop socket closes, requests recorded for that connection are answered with -32011 at once, new requests get -32011 while the desktop is reconnecting, and only its presence waits 90 seconds before it counts as offline and requests get -32010. When a phone socket closes, its interest lease ends at once and the desktop stops sending timeline frames, while the phone’s pending request stays recorded. The phone registers again, gets the reply, sends its lease again, and catches up from the latest window.
When the desktop socket closes
When a phone socket closes
Desktop
Relay
Phone
socket closes
reconnecting
offline
-32011 at once
-32011
-32010
0 s
30 s
60 s
90 s
120 s
none sent
frames sent
lease ends at once
request kept
socket closes
registers, sends its lease
reply
latest window
not to scale
  • Desktop requests-32011 · -32010

    Requests recorded for the closed connection get -32011, “Desktop is reconnecting”, at once, and so does every new request during the wait. After it, new requests get -32010. The relay counts a silent socket as closed only after 180 seconds.

  • Reconnect waitreconnect_timeout

    It holds presence only, and phones see the desktop as reconnecting. At 90 seconds the relay marks it offline and broadcasts device-status disconnected, unless a new connection registered first.

  • Phone requests(user, desktop, id)

    The record outlives the phone’s socket, so a reply that arrives after the phone registers again is delivered. A final reply that lands during the gap is dropped, and the phone’s call ends at its own timeout.

  • Phone leasechat:timeline-interest

    Bound to the phone’s connection, so it ends when the socket closes and nothing from the gap is kept. The phone sends it again once the relay answers registered or resumed, pings the desktop, and loads the latest window.

A client that registers with its resume token still passes the protocol and device-ownership checks. The relay session record behind the token lives 24 hours, is touched on every RPC and every heartbeat, and is invalidated on logout. A sweep removes expired records every 5 minutes. iOS keeps the token in the keychain.

Offline, reconnecting, and timeout errors stay distinct, and only errors marked retryable permit a transport retry. A timeout describes the reply the caller did not get. It says nothing about whether the mutation ran, so the client follows the operation’s recovery path before it issues new work.

StateMeaning and next step
ReconnectingThe phone is restoring its server connection and then re-verifying the desktop. Give it a moment.
Offline or disconnectedCheck the desktop app, the computer’s sleep state, the account, the region, and the network.
Request timed outThe reply did not arrive in time. The request may still have run, so read the session state before you repeat a mutation.
Connected, but history unavailableThe relay works and the history read failed. Use the reload action in the timeline.

Timeline events reach only interested phones

The lease deadline stops a session’s updates when a phone disconnects without a message
The lease deadline stops a session’s updates when a phone disconnects without a messageA phone renews its interest lease every 20 seconds, and each renewal keeps the lease alive for 60 seconds more. The relay stores the lease and sends the desktop its deadline, so both sides know when it ends. If the phone disappears without closing its socket, updates sent in the meantime are lost, and at the deadline the desktop and the relay both stop without exchanging a message. A returning phone takes a new lease and loads a snapshot.
Desktopsession X
Relaylease per phone
Phoneshows session X
+60 s
+60 s
offline, socket still open
renews every 20 s
deadline to the desktop
lost
session X updates
deadline passes: both sides stop, no message
not serialized
new lease + snapshot
0 s
20 s
40 s
60 s
80 s
100 s
120 s
  • Interest leasechat:timeline-interest { appSessionId, active }

    Sent by the phone for the session on its screen and renewed every 20 seconds. The relay stores it per phone and never forwards it. It carries no timeline content.

  • Lease endTIMELINE_INTEREST_TTL = 60 s

    A lease ends 60 seconds after its last renewal, or at once when the phone sends active: false, opens another session, or its socket closes or is replaced. Queued updates for that phone go with it. The relay drops a silent socket only after 180 seconds, so the lease ends first.

  • Desktop deadlinechat:timeline-interest-updated { expiresAt }

    On every lease change the relay tells the desktop per session whether a lease exists and when it ends, never which phone holds it. The desktop keeps that deadline itself and stops serializing the session’s timeline once it passes.

  • Relay filterauthorizationGeneration

    Issued when a session goes from no lease to one. Every frame must carry the current value, and the relay strips it and sends the frame only to phones whose lease is still live at the final send. Nothing is kept for the others, so a returning phone loads a snapshot.

The lease is keyed by user, phone route, target desktop, and base session. appSessionId is trimmed, capped at 1,024 bytes, and stripped of a trailing skills-agent alias. Updates to the desktop also carry a revision and the relayInstanceId, so the desktop ignores stale ones and clears its state when they come from another instance.

Seven revisions of one session reach the phone as three frames
Seven revisions of one session reach the phone as three framesThe upper part is a time axis in milliseconds. Seven timeline revisions of one session enter the desktop queue, where a newer revision replaces the waiting one. The desktop sends one event per 350 millisecond tick, so revisions 4, 6, and 7 leave and the others are never sent. The relay holds each frame in the phone’s buffer and sends it at 600, 850, and 1,100 milliseconds. The lower part draws the pending-byte limits to scale: 4 MiB in the desktop queue and 1 MiB per phone connection in the relay, where one standard event is at most 512 KiB.
Desktopevent queue
Relaybuffer per phone
Phone
revision 4
revision 6
revision 7
replaced while waiting
one event per 350 ms tick
waits 250 ms
pause after a send
0 ms
350 ms
700 ms
1050 ms
1400 ms
Pending bytes, to scale
Desktop queue
4 MiB, 64 timeline keys, 64 other events
Relay buffer
512 KiB
1 MiB, 16 keys per phone. One standard event is at most 512 KiB.
One frame may reach 32 MiB, 8 times the desktop bar. A snapshot above a buffer’s limit waits alone in one oversized slot.
  • Desktop queueMAX_PENDING_EVENT_BYTES = 4 MiB

    One queue per connection. It holds at most 64 timeline keys, 64 other events, and 4 MiB. A key is the base session plus the thread. A newer snapshot replaces the waiting one and keeps its place in the order. An older or identical one is ignored. When space runs out, the oldest waiting snapshot is dropped.

  • Send tickRELAY_EVENT_SEND_INTERVAL = 350 ms

    The socket writer takes one event from the queue per tick. Control messages are checked before every event and do not wait for the tick. Each frame send has a 10 second timeout.

  • Relay bufferTIMELINE_EVENT_FLUSH_INTERVAL = 250 ms

    One buffer per phone connection with at most 16 keys and 1 MiB. A newer revision replaces the waiting one, and the oldest key is evicted when a limit is passed. The relay sends one event per flush. A frame waits 250 milliseconds, or the rest of the pause after the previous send. That pause is 250 milliseconds per started 512 KiB of the event sent.

  • Oversized slot32 MiB − 16 KiB

    A snapshot above 512 KiB in the relay, or above 4 MiB on the desktop, waits alone in one extra slot outside the byte limit, and a newer oversized one replaces it. The desktop refuses an event whose frame would pass 32 MiB minus 16 KiB.

A live timeline update is a complete snapshot of the live overlay with a settlementEpoch and a revision. That makes every older snapshot of the same key redundant, and both buffers use it. The two pacing steps set the floor of the live latency: up to 350 milliseconds on the desktop and up to 250 milliseconds in the relay.

FailureResult
Desktop: no current lease for the sessionThe event is refused before it is serialized. Nothing is queued.
Desktop: snapshot frame above 32 MiB minus 16 KiBThe snapshot is dropped. The next one can still pass.
Desktop: other event above 32 MiB minus 16 KiB, or no room for a regular eventThe desktop ends the connection and reconnects, and phones reconcile.
Desktop: no room for an ordered timeline event, or two different snapshots with one revisionThe queue is cleared and closed, and the desktop disconnects.
Relay: two different snapshots with one revisionThe phone’s buffer is cleared and its socket is closed. The phone reconnects and reconciles.
Relay: more than 16 keys or 1 MiB pendingThe oldest keys are evicted. The phone gets their state with the next snapshot of those keys.

Events besides RPC

An event frame carries an eventType and a payload. The relay treats the types in the table specially and broadcasts every other type to the account’s routes. A signed-in user token may always send one. An API key needs the rpc, read, and write scopes, and gets forbidden without them. Until registration is complete the relay answers registrationRequired.

EventWhat the relay does
chat:timeline-updatedAccepted from the desktop only. The relay sends it to phones with a live lease for the session. A phone that sends it gets forbidden.
chat:timeline-interestSent by a phone. The relay stores the lease and never forwards it.
chat:timeline-interest-updatedWritten by the relay. It tells the desktop the lease state. A client that sends it gets reservedEventType.
device-visibility-updatedSent by a desktop or a phone. The relay saves the flag on the device row, then broadcasts it. A desktop that reports visible: false is closed after the broadcast.
project-directory-updatedSent by a desktop or a phone. The relay saves the directory on the device row, then broadcasts it.
agent-notification-requestedSent by the desktop. The relay broadcasts it and sends a visible push. It ignores the push part when a phone sends it.
device-statusWritten by the relay. It reports presence changes of a desktop, such as disconnected with the reason reconnect_timeout.
device-unlinkedWritten by the relay. It goes to every route of the account when a device is forgotten. Then the relay closes the routes of that device. A client that sends it gets reservedEventType.

Push registration and alerts

Both phones register a push token with PUT api/devices/push-token while signed in. The server sends exactly two kinds of push. The first is the silent desktop_online data message, sent when your desktop registers on the relay. It carries the relayInstanceId and the connection generation, and both apps use it to restore their connection before you open them. The second is the visible alert written by the agent’s notification tool. The desktop sends it as the event agent-notification-requested with sessionId, title, and body, and the relay passes it to APNs and FCM.

Why the relay runs as one process

A new relay process has no saved state, so every device registers again
A new relay process has no saved state, so every device registers againRoutes, pending request records, interest leases, reconnect presence, and resume sessions exist only in the relay process’s memory. A deploy starts a new process with empty memory and drains the old one, which clears all of it without answering open requests, closes every device socket, and refuses registrations from then on. Devices reconnect to the new process, where their resume tokens are unknown, so each registers again and phones send their leases again. PostgreSQL survives the switch.
Old process
New process
Desktop
Phone
routing state in memory
refuses registrations
memory cleared, sockets closed
empty memory
resume token unknown
registers, no leases yet
open requests never answered
registers, sends its lease
new process starts
drain
devices reconnect
not to scale
PostgreSQL survives the switch with device rows, their status, push tokens, and presence history
  • Process memoryDeviceConnectionManager

    Holds every connection route, pending request record, reconnect presence, and interest lease. RelaySessionStore holds the resume sessions. A second process would see none of it, which is why the relay runs as one.

  • DrainrelayDraining

    One-way. From its start the process refuses registrations. It waits at most 60 seconds for routing work in progress, then clears its memory without answering open requests and closes every device socket.

  • Instance IDrelayInstanceId

    A new UUID per process, sent with every registration answer. Phones ignore presence events from another instance, and the desktop clears its interest state when updates come from a different one.

With two processes behind a load balancer, a phone whose socket landed on the process without its desktop’s socket would see that desktop as offline.

Production therefore has exactly one routing authority per region. A blue-green deploy starts the new process, drains the old one, and then routes traffic to the new one. Per connection the relay caps frames at 32 MiB, rate-limits with a token bucket of 300 burst and 150 per second, pings every 30 seconds, and drops a client after 180 seconds of silence.

A text frame over the rate limit is dropped before the relay validates it. When the frame is an rpc.request with a request ID and a valid traceparent, the relay answers with the error -32028, so the call fails at once instead of waiting for the client’s timeout. Every other dropped frame gets no answer. The answers have their own token bucket of the same size. Each dropped frame takes one token from it, and without a token the relay reads nothing from the frame and sends nothing.