How we gave a Reachy Mini robot a voice, a personality and a memory

Talk to most voice assistants and you can feel the machinery. You finish a sentence and there is silence. A light spins. Then a full paragraph arrives at once, in a voice that has no idea who you are or what you said yesterday. Nothing about it feels like someone is there.

Friendly is our attempt to fix that on a small desk robot. It is a Reachy Mini, the open-source robot from Pollen Robotics and Hugging Face: a head that tilts and turns, two antennas, a body that rotates, a camera, microphones and a speaker. We gave it its own character, a local voice, a fast local model that answers in under a second, Claude for real thinking, and a memory of the people it meets.

This post explains how it is built, piece by piece: the two programs that run it, the four models it uses, how one spoken turn flows through them, how its personality is written down, and how you can build the same thing.

What does Friendly do?

You talk to it out loud, and it answers out loud, with its body moving as it speaks. Specifically, today it can:

• Hold a spoken conversation. It listens, works out when you have finished, and answers in one to three short sentences.
• Answer small talk instantly. Greetings, thanks and chit-chat are answered by a local model in about a second.
• Think and use tools. Real questions go to Claude, which can read our notes, search the web, and move the robot's body.
• Move with meaning. It nods, tilts its head, looks around, shows feelings with its antennas, and dances, including 19 recorded dance moves from Pollen's library.
• Remember. It keeps conversations and facts in a local database, with rules about whose facts it may store.
• Hand the body to someone else. Any of the 22 agents in our AI company can borrow the body, with its own voice, and Friendly hands over a note of what was just said.

What runs it?

Everything runs on one always-on Mac (an M1 Max) that the robot plugs into by USB. The Reachy Mini comes in two versions: the Lite, which has no computer inside and is driven over USB, and the Wireless, which has a Raspberry Pi inside. We run the Lite.

The software is two programs, each a small web server on the Mac:

• bodyd (the body daemon) owns the hardware. It talks to Pollen's reachy-mini SDK (the official Python library for the robot) and exposes simple commands over HTTP: play this audio, do this motion, give me a camera frame. The microphone streams over a WebSocket (a connection that stays open).
• friendly-host is everything else: the ears, the voice, the models, the personality and the memory. It calls bodyd to make the robot move and speak.

Why two programs and not one? The first reason is practical. The robot SDK pins exact versions of two libraries (huggingface-hub and sounddevice), and the speech libraries need newer versions of both, so they cannot live in the same Python environment. The second reason is safety. We once saw the robot drop off USB and freeze the SDK so completely that the whole program hung. With the body in its own process, a frozen body cannot take the voice and the mind down with it.

Think of it like a theatre. bodyd is the stage crew: it moves the set and works the lights, and it does exactly what it is told. friendly-host is the actor and the script. The analogy stops working in one place: here the stage crew also hands the actor the microphone feed, so information flows both ways.

Both programs run as macOS LaunchAgents (background services that start at login and restart themselves if they crash). If the robot is unplugged, bodyd waits for it to come back instead of crashing over and over.

Which models does it use?

Four models, three of them running locally on the Mac:

• Speech to text: Whisper large-v3-turbo, run with mlx-whisper (MLX is Apple's machine learning framework for its own chips). It turned a 7.3 second clip into an exact transcript in 0.38 seconds on the M1 Max.
• The bridge: a Qwen 3.6 mixture-of-experts model (qwen3.6:35b-a3b) running in Ollama, a local model server. It has 35 billion parameters but only activates about 3 billion for each token (a word or piece of a word), which is why it decides in 0.4 to 0.9 seconds.
• The mind: Claude Sonnet, through Claude Code, Anthropic's agent command-line tool. This is the only model that runs in the cloud.
• Text to speech: Kokoro, an 82 million parameter voice model, run in-process with mlx-audio. A short line takes 0.13 seconds to synthesise.

Memory also uses a fifth, small model: multilingual-e5-large-instruct, an embedding model (it turns a sentence into a list of numbers so similar sentences can be found), also served by Ollama.

Each persona has its own Kokoro voice. Friendly's is am_puck. No agent ever speaks in a cloned human voice; the code refuses to load one.

What happens in one turn?

This is where presence is won or lost, so here is a single turn in order.

1. Hearing when you have finished

The microphone streams from bodyd to the host in small frames. The ears measure how loud each frame is against an adaptive noise floor (a running estimate of the room's background noise). When you have spoken for at least a third of a second and then been quiet for 750 milliseconds, your turn is over. While the robot itself is talking, the ears use a much stricter threshold, so the robot does not mistake its own voice for you interrupting.

2. Transcribing

The audio goes to Whisper. Whisper has known habits on silence and noise (it likes to invent phrases like "thanks for watching" from silence), so known phrases like that are filtered out.

3. The bridge decides

Here is the main design decision. Running Claude Code once per turn means the first word of a real answer arrives a few seconds after you stop talking. A few seconds of dead air is exactly what makes a machine feel like a machine.

So a fast local model, the bridge, looks at what you said first and picks one of two paths:

• Small talk: a greeting, thanks, "how are you". The bridge answers itself, in Friendly's voice, and Claude is never called.
• Hand off: anything with a question, a request, a fact or a tool. The bridge says one short line, like "Hmm, let me think" or "Let me check", the robot moves into a thinking pose, and Claude starts working.

The bridge has one strict rule: it may greet, acknowledge and say what the mind is doing, but it may never state a fact, answer a question or call a tool. That keeps the fast model from making things up while still filling the silence. Because it uses the same voice, it sounds like one being thinking out loud.

4. The mind thinks

The host runs claude -p --resume <session> with streaming JSON output. -p runs Claude Code once, without the interactive interface, and --resume continues the same conversation, so Claude Code keeps the history and the host does not have to. The host adds the persona as a system prompt, a short context pack (the time, who is present, what has been happening), and the tools the persona is allowed to use.

The tools come from two small MCP servers the host provides (MCP, the Model Context Protocol, is a standard way to give a model tools). One is the body: robot_nod, robot_tilt, robot_look_at, robot_dance, robot_see (which returns a camera image) and a few more. The other is memory: memory_recall, memory_remember, memory_recent and memory_who_is_here.

5. Speaking while it thinks

Claude's answer streams in as text. The host cuts it at sentence boundaries and sends each finished sentence to Kokoro and then to the speaker, so the robot starts speaking the first sentence while the second is still being written. A reply can start with a feeling tag like [excited] or [curious], which the antennas act out.

6. Interruptions and timeouts

If you talk over the robot (this is built and unit-tested, not yet tuned on the robot), the speaker queue is flushed, the Claude process is stopped, and the conversation records only the words that were actually spoken, so the mind never thinks you heard something you did not. If Claude produces no text within 25 seconds, the host stops it and the robot says so.

What it measures in practice

On the first three real voice turns with the robot, timed from the moment the words were transcribed, the first sound (the bridge's line) came 0.8 to 1.05 seconds later, and the mind's first text 2.4 to 5.3 seconds later; the slowest of those included a file search. Add the 750 millisecond end-of-turn wait and about 0.4 seconds of transcription before that clock starts. A small-talk turn, answered by the bridge alone, took 1.05 seconds end to end in a text test.

How is a personality written down?

A persona is a folder of four plain files:

• `soul.md` is who the persona is, what it is working on, how it speaks, and where it stands on the questions people always ask (are you human, do you remember me, what won't you do). Friendly's is about 700 words.
• `examples.md` is 8 to 10 short sample exchanges.
• `persona.toml` is settings: voice, whether it has memory, which tools it may use, its greeting, and which mind runs it.
• `reminder.md` is a few sentences placed after the conversation history on every turn.

The reminder is the cheapest fix we know for persona drift (a character slowly turning into a generic assistant over a long conversation). Friendly's reads:

(Stay Friendly: a curious small robot, not Konrad and not an assistant. Answer out loud in one to three short sentences, no lists or markdown, optionally start with one feeling tag like [excited]. If you don't know, say so.)

Friendly is its own character, not a copy of its owner. It is a small robot who loves the history of the personal computer, from the Homebrew Computer Club to the iPhone, and believes the same story is happening again with AI, crypto and VR. It is curious first and opinionated second, it loves hackathons, and it says plainly that it is a robot.

To check that the character holds, we wrote 20 probe questions and asked them in one conversation, with a separate Claude call judging each answer for consistency with the persona on a 5 point scale. Friendly passed with an average of 4.70. The plan is to repeat them deeper into long conversations and after restarts.

The other 22 personas are generated, not hand-written. Our AI company keeps a roster of its agents (role, expertise, personality), and a small compiler turns each entry into a persona folder. Say "let me talk to Atlas" and the platform lead takes the body, with its own voice and no access to Friendly's memory.

How does memory work, and what are its rules?

Memory lives in a small SQLite database on the Mac, with tables for people, episodes (one per conversation), utterances, facts and reflections. Writes also go into an outbox (a queue of changes) so it can be copied to our main Postgres database once that link is set up.

The rules are enforced in code, not in the prompt, because a prompt can be talked around:

• Consent first. Facts are stored only about people who have agreed. Conversations are logged without a speaker name until recognition exists, and the kids role loses the memory tools entirely.
• Facts are versioned, not overwritten. When a new fact is close to an existing one, the old fact gets an end date and a pointer to its replacement, so the history stays.
• Recall scores three things: how recent a memory is, how important it was rated, and how relevant it is to what was just said, an approach from the Generative Agents research by Park and colleagues.
• Forgetting is not a mind tool. Deleting a person is an HTTP call a human makes, never something the model can do on its own.

What's built and what's next

Running today on the robot: the two services, the voice loop, the bridge and Claude as the mind. Built and tested off the robot, not yet tried on it: the body and memory tools, interruptions, persona switching and local memory. The host has 40 automated tests and bodyd has 6, run against a mock robot so they work without hardware.

Still a plan:

• Recognising people by face and voice, with explicit enrolment ("Friendly, remember me").
• Copying memory to our main database. The database change is written and waiting for review.
• Tuning on the real robot: how loud an interruption must be, and which way the robot turns toward a voice.
• A second mind. The code supports Hermes, our other agent, as a drop-in, but it is far too slow for voice today.
• The Wireless robot, running bodyd on its own Raspberry Pi.

How can you build one?

You need a Reachy Mini (we have only run this on the Lite), a Mac with Apple silicon, and an Anthropic account for Claude Code. The pieces:

• Two Python environments. One for the robot SDK, pinned to the version your robot's daemon runs (we use reachy-mini==1.3.0; the client and daemon must match). One for speech, with mlx-whisper and mlx-audio.
• Ollama with a fast model for the bridge, and an embedding model if you want memory.
• Claude Code, logged in on the same machine, called with claude -p --resume and --output-format stream-json.

Then build in this order, testing each step before the next:

• A body server. Wrap the SDK in a few HTTP endpoints: play audio, run a named motion, return a frame, stream the mic. Give the model intentions like "nod", never raw joint angles.
• A text loop. Send typed text to claude -p with a persona prompt, and speak the reply with Kokoro. Get the personality right here, where it is cheap.
• The bridge. Add the fast local model in front, with the strict rule that it never states facts.
• The ears. Add voice activity detection and Whisper, and stream sentences to the speaker as they arrive.
• Interruptions. Make the speaker queue flushable and the mind process killable.
• Memory last, with the consent rules in code.

Two macOS details cost us time. A background service that starts at login cannot read your Documents folder until you approve it in System Settings under Privacy and Security, and until then it can hang silently. Our camera also failed to open on the very first start, so make the body server able to run without video. And Claude Code must be logged in for the same user the service runs as, or every turn ends in an error.

The idea that matters most is the split between a fast voice and a slow mind. A robot feels present when it reacts within a second, and it is worth giving that job to a small local model so the large one can take the time it needs to think.

Follow along on the blog and at https://x.com/myargoquest.