# Documents, speech, video, alerts, sessions, and session messages

> The rest of the hbcode namespace: narrated explainer videos, speech and video analysis with your Gemini key, push alerts through the relay, session management, and messages between sessions.

Handbook / Bundled tools · Checked against the source on 2026-10-05 · https://www.hbcode.ai/docs/tools/agent-tools

## The tools that the agent can call

**A Codex thread keeps its first tool list, and Claude gets a new list on every run**

Time runs left to right over three runs. A Gemini key is saved after run 1 and deleted after run 2. Codex session A starts in run 1 and keeps 4 tools in all three runs. Codex session B starts in run 2 and keeps 7 tools. In run 3 its Gemini calls fail because the key is gone. The Claude session gets a new list with each run: 4 tools, then 7 tools, then 4 tools again.

- **One catalog** (build_app_capabilities): Four tools are always in the list: use_user_browser, maintain_sessions, manage_sessions, and send_session_message. A saved Gemini key adds synthesize_speech, create_explainer_video, and extract_video_context. Desktop and phone notifications add send_mobile_notification.
- **Codex list** (dynamicTools): The desktop sends the list when a thread starts. The request thread/resume has no field for it, so a resumed session keeps the tools it began with.
- **Claude list** (appCapabilities): The desktop builds the list again for every run and sends it in the open request. The agent host registers it as the SDK MCP server hbcode.
- **Check on every call** (has_gemini_api_key · show_notifications): A handler reads the key or the setting again on each call. A tool that a Codex thread still lists fails at once after you delete the key or turn notifications off.

Codex and Claude share one tool catalog and one set of desktop handlers. The function build_app_capabilities makes the list. Four tools are always in it: use_user_browser, maintain_sessions, manage_sessions, and send_session_message. Desktop and phone notifications add send_mobile_notification. A saved Gemini key adds synthesize_speech, create_explainer_video, and extract_video_context. The full list has 8 tools. The agents differ in when they get the list and in how a call reaches the desktop.

| Step | Codex | Claude |
| --- | --- | --- |
| The list arrives | As dynamicTools in the namespace hbcode when a thread starts. The request thread/resume has no field for it. | As appCapabilities in the open request of every run. The agent host registers it as an SDK MCP server named hbcode. |
| A setting changes later | The thread keeps its first list. Only a new Codex session sees the change. | The next run of the same session sees the change. |
| A call arrives | The request item/tool/call with threadId, turnId, and callId. | The host event toolCall with requestId, tool, arguments, and Claude’s toolUseId. The desktop answers with the request toolResult. |
| Session and run of the call | The desktop reads threadId and looks up the durable session link. The link also covers a forked sub-agent. The run is the active run of that session. | The desktop binds the session, the run, and the project directory of the run that owns the host. It rejects an event that names sessionId, runId, or projectDirectory. |
| Identity of the call | callId. | toolUseId. A call without one uses requestId. |

The model’s arguments never choose the session or the run. A Codex thread without a session link gets the tool error “is not linked to a HB Code session yet. Try the call again.” Codex threads from older versions still send sessionId, runId, and projectDirectory as arguments. The browser tool ignores them.

The desktop runs Claude tool calls as separate tasks beside the reader of host events. A slow tool does not block Stop, approvals, timeline updates, or other tool results. When Claude aborts a call, the host sends the event toolCancel, the desktop aborts the task, and Claude gets the result “Run stopped.” The end of a run aborts every call that is still open.

A handler checks its setting or its key again on every call. A tool that a Codex thread still lists stops at once when you delete the Gemini key or turn notifications off.

## Who approves a tool call

Codex does not ask you before a call of an HB Code tool, at any access level. Codex sends the request item/tool/call, and the desktop starts the handler at once. Claude asks before a tool call, as the access level of the session says. The desktop gives each tool one of three approval classes for Claude, and the function tool_approval holds the rule. The class travels to the Claude host in the field approval of each tool.

- Class read-only: maintain_sessions. Both agents run it at once. For Claude the tool is in allowedTools and has the annotation readOnlyHint.
- Class desktop: send_session_message. Both agents run it at once, and for Claude the tool is in allowedTools. The desktop rule inside the tool can hold the message in the recipient’s queue.
- Class agent: use_user_browser, manage_sessions, send_mobile_notification, synthesize_speech, create_explainer_video, and extract_video_context. Codex runs them at once. Claude asks you, as the access level says.

The host builds allowedTools from every tool whose class is not agent. The SDK names a tool mcp__hbcode__ plus the tool name. A tool that is not in allowedTools goes through the same approval path as Claude’s own tools.

- [Access levels](https://www.hbcode.ai/docs/agents.md#access-levels): The four levels and what each agent does with them.
- [Claude approvals and tools](https://www.hbcode.ai/docs/claude-agent-sdk.md#app-tools): How the agent host asks the desktop and waits for your answer.
- [Codex approvals](https://www.hbcode.ai/docs/codex-turns.md#approvals): What Codex asks about, and what it does not.

## HTML documents

A requested HTML document can be written with the native agent tools, including a subagent. The author chooses the file location, writes the styles and scripts, and checks the result in a browser. The final message links to the saved file.

Document links open in the desktop, iOS, and Android file viewers. The preview supplies the app theme through --hb-* CSS variables. Native authoring has no automatic document stylesheet, interaction runtime, finish gate, or managed revision history.

## Reply style and text rewrite

Settings → Agents → Reply style selects one style for the desktop and both phones. Simplified Technical English is the initial choice. Default lets each agent use its own style. The desktop owns this setting and its instructions.

Claude reads a style change with your next message. Codex reads it in new sessions; an older session can change after context compaction. Simplified Technical English adds a reply text to the instructions of both agents. It gives the ASD-STE100 rules. Default adds no reply text, so these rules apply only in Simplified Technical English.

Each new video worker also uses the current style. It gets the Narration section of the style: short spoken sentences, and no words about the page, the screen, or the layout.

The composer action Rewrite in STE rewrites your draft in Simplified Technical English. It preserves the meaning, facts, code, paths, and Markdown. The desktop uses one separate Codex request for the rewrite.

Both agents also receive the directions rule in each reply style. agents/session_rules.rs owns this text. The desktop puts it before the reply text of the style.

The directions rule, exactly as each agent receives it:

```
Directions. When the user or another agent gives you an explicit direction, do one of these three things:
1. Do it. Do all of it. Report each part as done. If you cannot do a part, say which part and why.
2. Ask. If the direction has two or more meanings that give different results, ask one short question. If a normal default is clear, use that default. Say which default you used.
3. Object. A direction is wrong when it causes a fault, when it conflicts with a fact or a rule, or when a better way exists. If the direction is wrong, say why. Give the evidence. Then wait for a decision. If the user already confirmed the direction, do it.
Do not do a different thing without saying so. Do not stop after only a part of the direction. Do not move a part to "later" or "out of scope" without asking. Permission and safety rules always apply. An unsafe or unpermitted direction gets an objection.
```

> **Directions apply in both styles** A message from another session carries a short form of the rule in its first line: “Answer this message in one of three ways: do it, ask one question, or object with the reason.” These rules are instructions to the model. The desktop does not check the answer.

## Narrated explainer videos

Ask the agent for a short explainer video. Give it the question, the facts, and the sources. The create_explainer_video tool starts a private worker on the agent and model of the session. The worker creates animated SVG scenes and reviews rendered frames from the start and end of each scene. Its private drafts stay out of the main chat.

When the worker finishes, the desktop creates narration with your Gemini key and adjusts each scene to the spoken text. The narrated video must last 30–120 seconds. If it does not, the finish fails with the measured length, and the worker revises the narration and finishes again. Then the desktop renders a 1080p MP4 at 30 frames per second. SVG rendering and video encoding run on your computer. The worker uses the session account, and the narration text goes to Gemini.

The timeline shows the phase, revision, scene count, duration, and rendered frames. A finished row opens the saved video. Its review lists the checks and limitations. Pass the saved videoPath in a later call to revise the same file. A failed or cancelled revision leaves the previous video available.

## Speech and video with your Gemini key

Save a Gemini API key in Settings → API Keys. Three more tools join the list. A Claude session gets them on its next run. A Codex session gets them only when its thread starts. Each tool reads the key on every call, so deleting it stops the tool even in sessions that still list it. Without a key, synthesize_speech and create_explainer_video fail with “Add a Gemini API key in Settings > API Keys.” The key stays on the desktop. Speech and video analysis requests go from your computer straight to Gemini.

- Text-to-speech. The desktop splits the text at sentence boundaries into chunks of at most 1,800 characters, synthesizes three at a time with gemini-3.1-flash-tts-preview, retries each chunk twice with a growing backoff, and stitches the 24 kHz mono 16-bit PCM parts with 180 ms of silence between them into one WAV file under agent-speech in the cache directory. Director notes stay outside the spoken transcript, and a multi-part prompt tells the model not to read the part number aloud. Thirty prebuilt voices are available and Kore is the default.
- Video context. The source is a public YouTube URL, a public https video URL, or a local recording, with a tutorial or meeting intent. A YouTube ID must be 11 characters and a playlist parameter is rejected. gemini-3.1-pro-preview runs at temperature 0.2 against a JSON schema that is rendered to Markdown. Tutorial mode extracts concepts, procedures, commands, code, on-screen text, and caveats; meeting mode extracts decisions, requirements, blockers, action items, and open questions. The result is timestamped notes scoped to your goal, with gaps called out.

Video sources are exactly one of youtube_url, local_file, or video_url. A remote URL needs one of twelve supported video MIME types such as video/mp4, video/webm, video/quicktime, or video/mpeg, and Gemini must be able to fetch it directly. There are no fallbacks: a private, inaccessible, or ambiguous source fails explicitly.

> **Local recordings rely on the agent’s consent flag** The tool sends a local video file only when the call sets localFileConsent, which the instructions tell the agent to do only for a file you provided, selected, dropped, or recorded for the task. The desktop checks that the path is absolute; it does not prompt you, so this is an instruction, not an enforcement.

## Alerts written by the agent

**A successful alert call only means the desktop queued it for the relay**

The agent’s send_mobile_notification call returns as soon as the desktop queues the event on its device link. If a link that has started is not ready, the desktop restarts it and tries 8 more times 250 ms apart, so the agent gets an error after about 2 seconds. Past the relay nothing reports back: a server without APNs or FCM, an account without a registered phone, and a rejected token are only logged, and a phone without notification permission shows nothing. The tool is in the list only while Desktop and phone notifications are on. A Codex thread fixes the list when it starts, and Claude gets the list on each run. The tool sends nothing once the setting is turned off.

- **Requested** (agent-notification-requested): The call returns as soon as the event is queued on a connected device link. The server broadcasts it and sends the pushes afterwards, on its own.
- **Visible failure** (initialize_connection): If the link is not ready, the desktop restarts it and tries 8 more times. With Allow Remote Access off or while signed out it never comes up, so the error arrives after about 2 seconds, or at once if the link never started since the app launched.
- **Silent drop** (send_agent_notification_push): After the relay nothing comes back. A server without APNs or FCM, an account with no registered phone, and a rejected token are only logged. A phone without notification permission shows nothing.

Turn on Desktop and phone notifications in Settings → General and the list gets send_mobile_notification, a tool for one push alert about the current session per call. A Claude session gets it on its next run. A Codex session gets it only when its thread starts. The desktop checks the setting again on every call, so turning it off stops alerts at once, including from sessions that still have the tool. Each alert names the session and the run that is active when the agent calls the tool. The Agent notifications row below it selects the guidance in the tool description: Important limits it to results and blocking questions, Frequent adds meaningful milestones and failures but still tells the agent to avoid routine progress updates. Both policies expose the same tool. The title takes at most 60 characters and the body at most 180, which forces the outcome first.

- [When a tool list changes](https://www.hbcode.ai/docs/tools/agent-tools.md#overview): The list lifetime for Codex and for Claude, in one figure.

## Session lookup

maintain_sessions is a read-only view of appdata.db with three actions: guide, listSessions, and resolveSession. It joins sessions with the project workspace state, which marks the active session, with the Codex session links, and with the table claude_sessions. It prints the agent of each session, orders by last update, returns 25 rows by default and at most 200, and filters by the hash of the project directory. resolveSession gives the Codex thread link or the Claude session ID of one session. The tool cannot update session metadata or active-session state, and it is not a SQL tool.

## Messages to another session

send_session_message lets the agent of one session write to another HB Code session. The recipient can run Codex or Claude. The tool takes two arguments: recipientSessionId and message. The sender session, its run, and the ID of the call come from the calling agent. The agent finds a recipient ID with maintain_sessions.

The call does not wait for the other agent. The desktop puts the message into the recipient’s outbox and returns a receipt. A recipient that runs gets the message in its running turn, after the agent’s current step. A recipient that does not run starts a turn for it. An answer comes later as a new message in the sender session, and the tool tells the agent not to wait or poll for it.

| Receipt status | Meaning |
| --- | --- |
| queued | The message is in the recipient’s outbox and goes out without you. |
| waiting-for-approval | The message is in the recipient’s queue and stays there until you send it from that session. The receipt gives the reason and tells the agent to tell you. |
| already-sent | An earlier attempt of the same tool call queued the message. The desktop stores one row per sender session and call ID in chat_session_messages, so one call never sends twice. |

The desktop rejects a call before it queues anything in these cases.

| Limit | Value |
| --- | --- |
| Message text | An empty message fails. |
| Sender | The sender session must have an active run. |
| Recipient | An existing session that is not the sender. The same call ID cannot name a second recipient. |

A message may start work only with the access that its sender had. The desktop holds a message for your approval when one of two rules applies. It checks the rules when the call arrives and again just before delivery, because the recipient’s settings can change while the message waits. When the recipient runs a turn, the desktop compares the access of that turn. The turn runs with the recipient’s access level at that moment.

| Hold rule | Detail |
| --- | --- |
| The recipient has more access | The four levels form one scale for both agents: read-only, full auto, full access, unrestricted. A recipient above the sender’s level waits. |
| Another project | The project directories differ and the recipient is not read-only. |

The recipient’s agent gets the message as a user message in an envelope. The first line is plain text, and the tag names the sender. To answer, the agent sends a message to from_session_id. The operation ID of a session message starts with session-message-. A client cannot submit such an ID, so a tag that you type names no sender.

Envelope that the recipient’s agent reads:

```
Message from another HB Code session. Answer this message in one of three ways: do it, ask one question, or object with the reason.
<session_message from_session_id="…" from_title="…" from_agent="codex">
…message…
</session_message>
```

The desktop gives Claude one rule in the text that Claude appends to its system prompt: use send_session_message for another HB Code session, because Claude’s own SendMessage and ListAgents cannot reach HB Code sessions.

The timeline of the sender shows the call as a message row, “Message to <recipient title>”, with the signal Sent, Needs approval, or Not sent. While the delivery is Sent, the name of the recipient in the row is a link to the user row of the message in the recipient session. That user row has the link back.

- [How a session message is delivered](https://www.hbcode.ai/docs/message-delivery.md#session-messages): The outbox entry, the hold, your approval, and the turn in the recipient session.
- [Who approves the call](https://www.hbcode.ai/docs/tools/agent-tools.md#approval): Neither agent asks you before this tool. The hold is the control.
- [Message rows and their links](https://www.hbcode.ai/docs/timeline.md#message-rows): The message fact, the row title, the delivery, and the link to the other end.

## Where the data goes

Browser control operates your Chrome, speech and video send text or video to Gemini under your key, and alerts travel through the regional server.

- [Local execution still uses remote services](https://www.hbcode.ai/docs/accounts-and-auth.md#data-flow): Every destination that receives data, and what desktop-owned does and does not mean.