Documents, speech, video, alerts, sessions, and session messages
The rest of the hbcode namespace: narrated explainer videos, speech and video analysis with your Gemini key, push alerts through the relay, session management, and messages between sessions.
Checked against the source on 5 October 2026
On this page
The tools that the agent can call
- One catalogbuild_app_capabilities
Four tools are always in the list: use_user_browser, maintain_sessions, manage_sessions, and send_session_message. A saved Gemini key adds synthesize_speech, create_explainer_video, and extract_video_context. Desktop and phone notifications add send_mobile_notification.
- Codex listdynamicTools
The desktop sends the list when a thread starts. The request thread/resume has no field for it, so a resumed session keeps the tools it began with.
- Claude listappCapabilities
The desktop builds the list again for every run and sends it in the open request. The agent host registers it as the SDK MCP server hbcode.
- Check on every callhas_gemini_api_key · show_notifications
A handler reads the key or the setting again on each call. A tool that a Codex thread still lists fails at once after you delete the key or turn notifications off.
Codex and Claude share one tool catalog and one set of desktop handlers. The function build_app_capabilities makes the list. Four tools are always in it: use_user_browser, maintain_sessions, manage_sessions, and send_session_message. Desktop and phone notifications add send_mobile_notification. A saved Gemini key adds synthesize_speech, create_explainer_video, and extract_video_context. The full list has 8 tools. The agents differ in when they get the list and in how a call reaches the desktop.
| Step | Codex | Claude |
|---|---|---|
| The list arrives | As dynamic | As app |
| A setting changes later | The thread keeps its first list. Only a new Codex session sees the change. | The next run of the same session sees the change. |
| A call arrives | The request item/ | The host event tool |
| Session and run of the call | The desktop reads thread | The desktop binds the session, the run, and the project directory of the run that owns the host. It rejects an event that names session |
| Identity of the call | call | tool |
The model’s arguments never choose the session or the run. A Codex thread without a session link gets the tool error “is not linked to a HB Code session yet. Try the call again.” Codex threads from older versions still send sessionId, runId, and projectDirectory as arguments. The browser tool ignores them.
The desktop runs Claude tool calls as separate tasks beside the reader of host events. A slow tool does not block Stop, approvals, timeline updates, or other tool results. When Claude aborts a call, the host sends the event toolCancel, the desktop aborts the task, and Claude gets the result “Run stopped.” The end of a run aborts every call that is still open.
A handler checks its setting or its key again on every call. A tool that a Codex thread still lists stops at once when you delete the Gemini key or turn notifications off.
Who approves a tool call
Codex does not ask you before a call of an HB Code tool, at any access level. Codex sends the request item/tool/call, and the desktop starts the handler at once. Claude asks before a tool call, as the access level of the session says. The desktop gives each tool one of three approval classes for Claude, and the function tool_approval holds the rule. The class travels to the Claude host in the field approval of each tool.
- Class read-only: maintain_sessions. Both agents run it at once. For Claude the tool is in allowedTools and has the annotation readOnlyHint.
- Class desktop: send_session_message. Both agents run it at once, and for Claude the tool is in allowedTools. The desktop rule inside the tool can hold the message in the recipient’s queue.
- Class agent: use_user_browser, manage_sessions, send_mobile_notification, synthesize_speech, create_explainer_video, and extract_video_context. Codex runs them at once. Claude asks you, as the access level says.
The host builds allowedTools from every tool whose class is not agent. The SDK names a tool mcp__hbcode__ plus the tool name. A tool that is not in allowedTools goes through the same approval path as Claude’s own tools.
HTML documents
A requested HTML document can be written with the native agent tools, including a subagent. The author chooses the file location, writes the styles and scripts, and checks the result in a browser. The final message links to the saved file.
Document links open in the desktop, iOS, and Android file viewers. The preview supplies the app theme through --hb-* CSS variables. Native authoring has no automatic document stylesheet, interaction runtime, finish gate, or managed revision history.
Reply style and text rewrite
Settings → Agents → Reply style selects one style for the desktop and both phones. Simplified Technical English is the initial choice. Default lets each agent use its own style. The desktop owns this setting and its instructions.
Claude reads a style change with your next message. Codex reads it in new sessions; an older session can change after context compaction. Simplified Technical English adds a reply text to the instructions of both agents. It gives the ASD-STE100 rules. Default adds no reply text, so these rules apply only in Simplified Technical English.
Each new video worker also uses the current style. It gets the Narration section of the style: short spoken sentences, and no words about the page, the screen, or the layout.
The composer action Rewrite in STE rewrites your draft in Simplified Technical English. It preserves the meaning, facts, code, paths, and Markdown. The desktop uses one separate Codex request for the rewrite.
Both agents also receive the directions rule in each reply style. agents/session_rules.rs owns this text. The desktop puts it before the reply text of the style.
Directions. When the user or another agent gives you an explicit direction, do one of these three things:
1. Do it. Do all of it. Report each part as done. If you cannot do a part, say which part and why.
2. Ask. If the direction has two or more meanings that give different results, ask one short question. If a normal default is clear, use that default. Say which default you used.
3. Object. A direction is wrong when it causes a fault, when it conflicts with a fact or a rule, or when a better way exists. If the direction is wrong, say why. Give the evidence. Then wait for a decision. If the user already confirmed the direction, do it.
Do not do a different thing without saying so. Do not stop after only a part of the direction. Do not move a part to "later" or "out of scope" without asking. Permission and safety rules always apply. An unsafe or unpermitted direction gets an objection.Narrated explainer videos
Ask the agent for a short explainer video. Give it the question, the facts, and the sources. The create_explainer_video tool starts a private worker on the agent and model of the session. The worker creates animated SVG scenes and reviews rendered frames from the start and end of each scene. Its private drafts stay out of the main chat.
When the worker finishes, the desktop creates narration with your Gemini key and adjusts each scene to the spoken text. The narrated video must last 30–120 seconds. If it does not, the finish fails with the measured length, and the worker revises the narration and finishes again. Then the desktop renders a 1080p MP4 at 30 frames per second. SVG rendering and video encoding run on your computer. The worker uses the session account, and the narration text goes to Gemini.
The timeline shows the phase, revision, scene count, duration, and rendered frames. A finished row opens the saved video. Its review lists the checks and limitations. Pass the saved videoPath in a later call to revise the same file. A failed or cancelled revision leaves the previous video available.
Speech and video with your Gemini key
Save a Gemini API key in Settings → API Keys. Three more tools join the list. A Claude session gets them on its next run. A Codex session gets them only when its thread starts. Each tool reads the key on every call, so deleting it stops the tool even in sessions that still list it. Without a key, synthesize_speech and create_explainer_video fail with “Add a Gemini API key in Settings > API Keys.” The key stays on the desktop. Speech and video analysis requests go from your computer straight to Gemini.
- Text-to-speech. The desktop splits the text at sentence boundaries into chunks of at most 1,800 characters, synthesizes three at a time with gemini-3.1-flash-tts-preview, retries each chunk twice with a growing backoff, and stitches the 24 kHz mono 16-bit PCM parts with 180 ms of silence between them into one WAV file under agent-speech in the cache directory. Director notes stay outside the spoken transcript, and a multi-part prompt tells the model not to read the part number aloud. Thirty prebuilt voices are available and Kore is the default.
- Video context. The source is a public YouTube URL, a public https video URL, or a local recording, with a tutorial or meeting intent. A YouTube ID must be 11 characters and a playlist parameter is rejected. gemini-3.1-pro-preview runs at temperature 0.2 against a JSON schema that is rendered to Markdown. Tutorial mode extracts concepts, procedures, commands, code, on-screen text, and caveats; meeting mode extracts decisions, requirements, blockers, action items, and open questions. The result is timestamped notes scoped to your goal, with gaps called out.
Video sources are exactly one of youtube_url, local_file, or video_url. A remote URL needs one of twelve supported video MIME types such as video/mp4, video/webm, video/quicktime, or video/mpeg, and Gemini must be able to fetch it directly. There are no fallbacks: a private, inaccessible, or ambiguous source fails explicitly.
Alerts written by the agent
- Requestedagent-notification-requested
The call returns as soon as the event is queued on a connected device link. The server broadcasts it and sends the pushes afterwards, on its own.
- Visible failureinitialize_connection
If the link is not ready, the desktop restarts it and tries 8 more times. With Allow Remote Access off or while signed out it never comes up, so the error arrives after about 2 seconds, or at once if the link never started since the app launched.
- Silent dropsend_agent_notification_push
After the relay nothing comes back. A server without APNs or FCM, an account with no registered phone, and a rejected token are only logged. A phone without notification permission shows nothing.
Turn on Desktop and phone notifications in Settings → General and the list gets send_mobile_notification, a tool for one push alert about the current session per call. A Claude session gets it on its next run. A Codex session gets it only when its thread starts. The desktop checks the setting again on every call, so turning it off stops alerts at once, including from sessions that still have the tool. Each alert names the session and the run that is active when the agent calls the tool. The Agent notifications row below it selects the guidance in the tool description: Important limits it to results and blocking questions, Frequent adds meaningful milestones and failures but still tells the agent to avoid routine progress updates. Both policies expose the same tool. The title takes at most 60 characters and the body at most 180, which forces the outcome first.
Session lookup
maintain_sessions is a read-only view of appdata.db with three actions: guide, listSessions, and resolveSession. It joins sessions with the project workspace state, which marks the active session, with the Codex session links, and with the table claude_sessions. It prints the agent of each session, orders by last update, returns 25 rows by default and at most 200, and filters by the hash of the project directory. resolveSession gives the Codex thread link or the Claude session ID of one session. The tool cannot update session metadata or active-session state, and it is not a SQL tool.
Messages to another session
send_session_message lets the agent of one session write to another HB Code session. The recipient can run Codex or Claude. The tool takes two arguments: recipientSessionId and message. The sender session, its run, and the ID of the call come from the calling agent. The agent finds a recipient ID with maintain_sessions.
The call does not wait for the other agent. The desktop puts the message into the recipient’s outbox and returns a receipt. A recipient that runs gets the message in its running turn, after the agent’s current step. A recipient that does not run starts a turn for it. An answer comes later as a new message in the sender session, and the tool tells the agent not to wait or poll for it.
| Receipt status | Meaning |
|---|---|
| queued | The message is in the recipient’s outbox and goes out without you. |
| waiting-for-approval | The message is in the recipient’s queue and stays there until you send it from that session. The receipt gives the reason and tells the agent to tell you. |
| already-sent | An earlier attempt of the same tool call queued the message. The desktop stores one row per sender session and call ID in chat_ |
The desktop rejects a call before it queues anything in these cases.
| Limit | Value |
|---|---|
| Message text | An empty message fails. |
| Sender | The sender session must have an active run. |
| Recipient | An existing session that is not the sender. The same call ID cannot name a second recipient. |
A message may start work only with the access that its sender had. The desktop holds a message for your approval when one of two rules applies. It checks the rules when the call arrives and again just before delivery, because the recipient’s settings can change while the message waits. When the recipient runs a turn, the desktop compares the access of that turn. The turn runs with the recipient’s access level at that moment.
| Hold rule | Detail |
|---|---|
| The recipient has more access | The four levels form one scale for both agents: read-only, full auto, full access, unrestricted. A recipient above the sender’s level waits. |
| Another project | The project directories differ and the recipient is not read-only. |
The recipient’s agent gets the message as a user message in an envelope. The first line is plain text, and the tag names the sender. To answer, the agent sends a message to from_session_id. The operation ID of a session message starts with session-message-. A client cannot submit such an ID, so a tag that you type names no sender.
Message from another HB Code session. Answer this message in one of three ways: do it, ask one question, or object with the reason.
<session_message from_session_id="…" from_title="…" from_agent="codex">
…message…
</session_message>The desktop gives Claude one rule in the text that Claude appends to its system prompt: use send_session_message for another HB Code session, because Claude’s own SendMessage and ListAgents cannot reach HB Code sessions.
The timeline of the sender shows the call as a message row, “Message to <recipient title>”, with the signal Sent, Needs approval, or Not sent. While the delivery is Sent, the name of the recipient in the row is a link to the user row of the message in the recipient session. That user row has the link back.
Where the data goes
Browser control operates your Chrome, speech and video send text or video to Gemini under your key, and alerts travel through the regional server.