Device routing, reconnect, and recovery
Registration answers and refusals, the two watchdogs of a dead link, the reconnect grace, how timeline events reach only interested phones through two bounded buffers, the other relay events, push registration, and why the relay runs as one process.
Checked against the source on 5 October 2026
On this page
Registration and discovery
| Client | WebSocket headers | Cross-checked against |
|---|---|---|
| Desktop | X-Client-Type: desktop | A desktop device row the same account registered over HTTP. |
| Phone | X-Client-Type: mobile, plus X-Target-Desktop-Device-ID for the selected desktop | A mobile device row. The relay reads the target desktop from the register payload, else from the header, on every registration, never from a stored session. |
The first frame on the socket is the register message. The relay answers registered with sessionId, resumeToken, expiresAt, and relayInstanceId. A register message that carries a valid resume token gets resumed with the same sessionId, expiresAt, and relayInstanceId, and no new token. When a desktop registers, the relay also sends it the current interest leases of its phones.
A refused registration gets an error frame with one of these codes.
| Code | Cause |
|---|---|
| invalid | The client type is neither desktop nor mobile. |
| invalid | The device ID is not a UUID. |
| device | No device row exists. The device must register over HTTP first. |
| device | The device row belongs to another account. |
| device | The client type differs from the type of the device row. |
| desktop | The desktop row is not eligible for the relay. |
| desktop | The device was forgotten in the account. |
| protocol | The relay protocol version is missing or below the floor. |
| invalid | The relay protocol version cannot be parsed. |
| invalid | The resume token is unknown or expired. The client registers again without it. |
| relay | The process is draining for a deploy. |
| relay | The relay could not read its protocol policy or the device row. The fault is on the server. |
Reconnect grace and the reconnect snapshot
- Desktop watchdogINBOUND_ACTIVITY_TIMEOUT_SECS = 45
Every 15 seconds the desktop checks the time since the last inbound frame and then sends a heartbeat, which the relay answers with a pong. A gap of more than 45 seconds ends the connection. The check runs on the 15 second tick, so the desktop gives up a dead link at most 60 seconds after its last frame.
- Reconnect backoffreconnect_backoff
Each wait is a random time between 100 milliseconds and a cap. The cap starts at 500 milliseconds, doubles per failed attempt, and stops at 30 seconds. The connect and the registration answer each have a 10 second timeout. The failure count resets only after a connection stayed ready for 30 seconds.
- Relay watchdogCLIENT_TIMEOUT = 180 s
Every 30 seconds the relay checks the time since the socket’s last frame and then sends a ping. More than 180 seconds of silence stops the socket, so the close comes 180 to 210 seconds after the last frame. A text, ping, or pong frame resets the clock.
- Presence graceDESKTOP_RECONNECT_GRACE_SECS = 90
Starts when the relay closes the desktop socket. Phones see reconnecting during the grace. At its end the relay broadcasts device-status disconnected with the reason reconnect_timeout, unless the desktop registered again first.
A socket that closes with a close frame is known to both sides at once. A link that dies silently is found by two independent watchdogs, and the desktop’s is the faster one. The desktop reconnects long before the relay would close the old socket, so in the normal case phones never see the desktop as offline.
- Desktop requests-32011 · -32010
Requests recorded for the closed connection get -32011, “Desktop is reconnecting”, at once, and so does every new request during the wait. After it, new requests get -32010. The relay counts a silent socket as closed only after 180 seconds.
- Reconnect waitreconnect_timeout
It holds presence only, and phones see the desktop as reconnecting. At 90 seconds the relay marks it offline and broadcasts device-status disconnected, unless a new connection registered first.
- Phone requests(user, desktop, id)
The record outlives the phone’s socket, so a reply that arrives after the phone registers again is delivered. A final reply that lands during the gap is dropped, and the phone’s call ends at its own timeout.
- Phone leasechat:timeline-interest
Bound to the phone’s connection, so it ends when the socket closes and nothing from the gap is kept. The phone sends it again once the relay answers registered or resumed, pings the desktop, and loads the latest window.
A client that registers with its resume token still passes the protocol and device-ownership checks. The relay session record behind the token lives 24 hours, is touched on every RPC and every heartbeat, and is invalidated on logout. A sweep removes expired records every 5 minutes. iOS keeps the token in the keychain.
Offline, reconnecting, and timeout errors stay distinct, and only errors marked retryable permit a transport retry. A timeout describes the reply the caller did not get. It says nothing about whether the mutation ran, so the client follows the operation’s recovery path before it issues new work.
| State | Meaning and next step |
|---|---|
| Reconnecting | The phone is restoring its server connection and then re-verifying the desktop. Give it a moment. |
| Offline or disconnected | Check the desktop app, the computer’s sleep state, the account, the region, and the network. |
| Request timed out | The reply did not arrive in time. The request may still have run, so read the session state before you repeat a mutation. |
| Connected, but history unavailable | The relay works and the history read failed. Use the reload action in the timeline. |
Timeline events reach only interested phones
- Interest leasechat:timeline-interest { appSessionId, active }
Sent by the phone for the session on its screen and renewed every 20 seconds. The relay stores it per phone and never forwards it. It carries no timeline content.
- Lease endTIMELINE_INTEREST_TTL = 60 s
A lease ends 60 seconds after its last renewal, or at once when the phone sends active: false, opens another session, or its socket closes or is replaced. Queued updates for that phone go with it. The relay drops a silent socket only after 180 seconds, so the lease ends first.
- Desktop deadlinechat:timeline-interest-updated { expiresAt }
On every lease change the relay tells the desktop per session whether a lease exists and when it ends, never which phone holds it. The desktop keeps that deadline itself and stops serializing the session’s timeline once it passes.
- Relay filterauthorizationGeneration
Issued when a session goes from no lease to one. Every frame must carry the current value, and the relay strips it and sends the frame only to phones whose lease is still live at the final send. Nothing is kept for the others, so a returning phone loads a snapshot.
The lease is keyed by user, phone route, target desktop, and base session. appSessionId is trimmed, capped at 1,024 bytes, and stripped of a trailing skills-agent alias. Updates to the desktop also carry a revision and the relayInstanceId, so the desktop ignores stale ones and clears its state when they come from another instance.
- Desktop queueMAX_PENDING_EVENT_BYTES = 4 MiB
One queue per connection. It holds at most 64 timeline keys, 64 other events, and 4 MiB. A key is the base session plus the thread. A newer snapshot replaces the waiting one and keeps its place in the order. An older or identical one is ignored. When space runs out, the oldest waiting snapshot is dropped.
- Send tickRELAY_EVENT_SEND_INTERVAL = 350 ms
The socket writer takes one event from the queue per tick. Control messages are checked before every event and do not wait for the tick. Each frame send has a 10 second timeout.
- Relay bufferTIMELINE_EVENT_FLUSH_INTERVAL = 250 ms
One buffer per phone connection with at most 16 keys and 1 MiB. A newer revision replaces the waiting one, and the oldest key is evicted when a limit is passed. The relay sends one event per flush. A frame waits 250 milliseconds, or the rest of the pause after the previous send. That pause is 250 milliseconds per started 512 KiB of the event sent.
- Oversized slot32 MiB − 16 KiB
A snapshot above 512 KiB in the relay, or above 4 MiB on the desktop, waits alone in one extra slot outside the byte limit, and a newer oversized one replaces it. The desktop refuses an event whose frame would pass 32 MiB minus 16 KiB.
A live timeline update is a complete snapshot of the live overlay with a settlementEpoch and a revision. That makes every older snapshot of the same key redundant, and both buffers use it. The two pacing steps set the floor of the live latency: up to 350 milliseconds on the desktop and up to 250 milliseconds in the relay.
| Failure | Result |
|---|---|
| Desktop: no current lease for the session | The event is refused before it is serialized. Nothing is queued. |
| Desktop: snapshot frame above 32 Mi | The snapshot is dropped. The next one can still pass. |
| Desktop: other event above 32 Mi | The desktop ends the connection and reconnects, and phones reconcile. |
| Desktop: no room for an ordered timeline event, or two different snapshots with one revision | The queue is cleared and closed, and the desktop disconnects. |
| Relay: two different snapshots with one revision | The phone’s buffer is cleared and its socket is closed. The phone reconnects and reconciles. |
| Relay: more than 16 keys or 1 Mi | The oldest keys are evicted. The phone gets their state with the next snapshot of those keys. |
Events besides RPC
An event frame carries an eventType and a payload. The relay treats the types in the table specially and broadcasts every other type to the account’s routes. A signed-in user token may always send one. An API key needs the rpc, read, and write scopes, and gets forbidden without them. Until registration is complete the relay answers registrationRequired.
| Event | What the relay does |
|---|---|
| chat:timeline-updated | Accepted from the desktop only. The relay sends it to phones with a live lease for the session. A phone that sends it gets forbidden. |
| chat:timeline-interest | Sent by a phone. The relay stores the lease and never forwards it. |
| chat:timeline-interest-updated | Written by the relay. It tells the desktop the lease state. A client that sends it gets reserved |
| device-visibility-updated | Sent by a desktop or a phone. The relay saves the flag on the device row, then broadcasts it. A desktop that reports visible: false is closed after the broadcast. |
| project-directory-updated | Sent by a desktop or a phone. The relay saves the directory on the device row, then broadcasts it. |
| agent-notification-requested | Sent by the desktop. The relay broadcasts it and sends a visible push. It ignores the push part when a phone sends it. |
| device-status | Written by the relay. It reports presence changes of a desktop, such as disconnected with the reason reconnect_ |
| device-unlinked | Written by the relay. It goes to every route of the account when a device is forgotten. Then the relay closes the routes of that device. A client that sends it gets reserved |
Push registration and alerts
Both phones register a push token with PUT api/devices/push-token while signed in. The server sends exactly two kinds of push. The first is the silent desktop_online data message, sent when your desktop registers on the relay. It carries the relayInstanceId and the connection generation, and both apps use it to restore their connection before you open them. The second is the visible alert written by the agent’s notification tool. The desktop sends it as the event agent-notification-requested with sessionId, title, and body, and the relay passes it to APNs and FCM.
Why the relay runs as one process
- Process memoryDeviceConnectionManager
Holds every connection route, pending request record, reconnect presence, and interest lease. RelaySessionStore holds the resume sessions. A second process would see none of it, which is why the relay runs as one.
- DrainrelayDraining
One-way. From its start the process refuses registrations. It waits at most 60 seconds for routing work in progress, then clears its memory without answering open requests and closes every device socket.
- Instance IDrelayInstanceId
A new UUID per process, sent with every registration answer. Phones ignore presence events from another instance, and the desktop clears its interest state when updates come from a different one.
With two processes behind a load balancer, a phone whose socket landed on the process without its desktop’s socket would see that desktop as offline.
Production therefore has exactly one routing authority per region. A blue-green deploy starts the new process, drains the old one, and then routes traffic to the new one. Per connection the relay caps frames at 32 MiB, rate-limits with a token bucket of 300 burst and 150 per second, pings every 30 seconds, and drops a client after 180 seconds of silence.
A text frame over the rate limit is dropped before the relay validates it. When the frame is an rpc.request with a request ID and a valid traceparent, the relay answers with the error -32028, so the call fails at once instead of waiting for the client’s timeout. Every other dropped frame gets no answer. The answers have their own token bucket of the same size. Each dropped frame takes one token from it, and without a token the relay reads nothing from the frame and sends nothing.