What it is
The ComfyTV Bot is a sidebar chat panel (the ✨ icon) backed by a local agent CLI. Every message you send spawns an agent turn that can use the full ComfyTV MCP toolset — and only that toolset: it can read and edit your canvas, run stages, inspect images, and manage your library, but it has no shell, no file system access, and no other tools. Typical asks:- “Add an image stage with Z-Image Turbo, prompt a neon cat at night, 16:9, and run it.”
- “Use that image as the reference for a 5-second image-to-video, wait for it, and QC the first frame.”
- “Here’s my song and its timed lyrics — trim it into sections and build an audio-driven MV section by section.”
- “Look at my canvas and tell me why the video stage failed.”
- “Open the Director timeline and re-take clip 3 with a slower camera.”
- “I just linked a new workflow — bind seed, width and height for me.”
No API keys, by design
The bot does not talk to any cloud model API directly and ComfyTV never stores a key. Instead it drives an agent CLI already installed on your machine using that CLI’s own login — or, with the Local LLM and ComfyUI LLM providers, a model running on your own hardware. Five providers ship today:
Prerequisites:
- Install at least one agent CLI and sign in once — or run a local model server and set its URL in Settings.
- In ComfyTV Settings → Agent & MCP, enable MCP server and then ComfyTV Bot (the bot requires MCP — it’s how the agent reaches your canvas).
codex exec sandboxed to the bot’s working directory with shell and web search disabled, every MCP server except ComfyTV’s turned off, and its localhost canvas-tool approvals routed through Codex’s automatic reviewer (headless runs cannot prompt); Qwen Code runs against a project-scoped .qwen/settings.json inside the bot’s working directory (ComfyTV MCP server only, built-in shell/file tools excluded) — your global CLI configuration is never touched.
Local LLM provider
The Local LLM provider needs no agent CLI at all: ComfyTV runs the agent loop itself against any OpenAI-compatible endpoint — LM Studio, llama.cpp’sllama-server, vLLM, Ollama and friends. Point Settings → Agent & MCP → Local LLM endpoint at the server’s base URL (e.g. http://127.0.0.1:1234/v1); the model dropdown suggestions come straight from the endpoint’s /models list. Keyless local endpoints only — consistent with the no-stored-keys rule (a COMFYTV_LOCAL_LLM_API_KEY environment variable is honoured for LAN servers that insist on a token, but nothing is ever stored).
Details worth knowing:
- Conversations replay from ComfyTV’s own transcript (the endpoint holds no session), so history survives server restarts.
- The bot exposes the core canvas toolset (build / run / wait / look) rather than all tools — small local models drown in the full catalog.
wait_stageis looped provider-side, so the model is only consulted when the render actually finishes.- If the LM Studio
lmsCLI is present, the driver model is unloaded from VRAM while a render runs and reloaded afterwards — on a single-GPU machine the canvas gets the whole card during generation.
ComfyUI LLM provider
The ComfyUI LLM provider goes one step further: no external server either — inference runs inside ComfyUI itself, on the same text-encoder stack the coreTextGenerate node uses. Drop a generation-capable Qwen3- or Gemma-family checkpoint (e.g. Qwen3 8B, or the Gemma 3/4 encoders LTX2 already uses) into models/text_encoders and the provider appears; pick the checkpoint under Settings → Agent & MCP → ComfyUI LLM model (blank = first found). Qwen3 has native tool-calling training and is the best driver; Gemma follows the same convention by instruction.
Compared to the Local LLM provider:
- Nothing to install or configure — no endpoint URL, no server process.
- VRAM is arbitrated by ComfyUI’s model management: while a render runs, the LLM is offloaded automatically and reloaded on the next turn. No
lms-style juggling. - Tool calls use the Hermes convention Qwen3 was trained on (
<tool_call>blocks), rendered and parsed by ComfyTV. - Turns are served one at a time from an OpenAI-compatible shim at
/comfytv/llm/v1(non-streaming) — other local apps on your machine may point at it too while the bot is enabled.
Using the panel
- Conversations are persistent: the list supports pin, rename and delete; each chat remembers its full context across turns (the CLI resumes the same session).
- Streaming: replies stream in live; tool activity collapses into a drawer (closed by default) with chips per call (e.g.
add_stage,wait_stage) so you can watch it work the canvas in real time — nodes appear and run on your canvas as it goes. - Attachments: on providers that support them, attach images / video / audio via the 📎 buttons, by dragging files in, picking from the asset library, or pasting. Videos are summarized with a middle frame, audio with a waveform, so the agent can actually see what you sent.
- Skills: type
/in the input to open the skill palette — pick an installed Agent Skill and it becomes a chip on your message; the agent reads that skill first and follows its instructions for the task. - Stop aborts the current turn; partial output is kept.
- Switching sidebar tabs (or closing the panel) does not interrupt a running turn — the turn continues server-side and the transcript catches up when you return.
How it works, briefly
Each turn spawns a fresh CLI process in headless mode, locked down to the ComfyTV MCP server (--strict-mcp-config, tools whitelisted to mcp__comfytv__*), resuming the chat’s session for continuity. The conversation state lives with the CLI; ComfyTV’s database keeps a display mirror of the transcript. Canvas writes still follow MCP rules — an open ComfyTV page executes them, whether it is in Comfy Desktop or a browser.
Troubleshooting
See also
- Agent access (MCP) — the toolset the bot uses, and how to connect external agents
- Agent Skills — instruction packs the bot (and external agents) can invoke
- Sidebar — the Settings panel with both switches