ElatoAI: Realtime Speech-to-Speech on an ESP32, With Three Interchangeable Server Backends

The problem ElatoAI targets: a toy that talks, without a laptop in the room

A speech-to-speech agent on a desktop is a solved problem: open a WebSocket to a realtime API, pipe microphone frames in, play audio frames out. Put the same agent inside a plush toy and the constraints change. The device has a single-core-ish budget for audio capture, Opus encoding, TLS and Wi-Fi, and it has no screen for configuration. ElatoAI's README frames the goal as realtime AI speech on an Arduino ESP32 with secure WebSockets and edge functions, and it claims more than 20-minute uninterrupted conversations globally. The feature list is written for that hardware context: a captive portal for Wi-Fi setup, OTA firmware updates, factory reset from a web app, button or touch sensor input, and a note that no PSRAM is required. That last item matters more than it looks. Many ESP32 audio projects assume PSRAM for buffering, and dropping that requirement widens the set of boards you can use. The intended user is someone shipping a consumer device or a one-off companion build, not someone writing a Python script that talks to GPT.

The architecture: ESP32 firmware, a device-management webapp, and a server you choose

The repository is organised around three cooperating pieces. On the device, Arduino firmware runs on the ESP32, captures audio, compresses it with Opus, and holds a secure WebSocket to a server. On the phone or browser, a Next.js webapp handles OAuth login, creates custom agents with a personality and a voice, registers devices, controls speaker volume, triggers factory reset, and shows conversation history and realtime transcripts. In the middle, the server terminates the WebSocket and bridges to a speech-to-speech model. The README lists three distinct server implementations. Under server/deno there are per-provider adapters: openai, gemini, grok, elevenlabs, hume, and a single boson.ts for Boson Higgs Realtime. Under Cloudflare Workers the split is by task rather than by vendor: LLM (80+ models), TTS (10+ models, Deepgram and MeloTTS among them), and STT (5 models, Whisper and Deepgram among them). Under server/fastapi there is a Pipecat-based pipeline where you assemble STT, LLM and TTS yourself. The data flow is consistent across all three: audio frames up the WebSocket, model audio frames back down, and transcripts written into Supabase. Note that the Deno path is speech-to-speech (the model consumes and produces audio directly), while the Cloudflare and FastAPI paths are pipelines (separate STT, LLM and TTS stages). That is a real architectural difference, not a deployment preference.

Choosing between the Deno, Cloudflare and FastAPI backends

These three paths are not interchangeable, and the README does not present them as such. The Deno path gives you six named realtime providers, each in its own directory, which means adding a seventh means writing an adapter that matches the existing shape. It is the shortest route to a working demo because the model handles turn-taking internally. The Cloudflare path moves the STT and TTS stages into Workers AI, so according to the April 2026 news entry, Deepgram STT and TTS run natively and you bring only an LLM API key. That is a meaningful reduction in the number of vendor accounts you manage, at the cost of accepting Cloudflare's model catalogue and its per-model limits. The FastAPI path is the most open and the most work: the README describes it as a Pipecat server where you select from the 100+ models listed above and build your own pipeline. You get control over each stage and a Python codebase that is easier to instrument than an edge worker, but you also own latency tuning across three network hops. There is also a separate repository, akdeb/local-ai-toys, announced on Pi Day, for running local models with MLX. The README links to it rather than folding it in, so treat it as a sibling project with its own setup.

Getting it running: docs-first, with the firmware built in PlatformIO or the Arduino IDE

The README does not inline installation commands. It points to a docs site at elatoai.com/docs with separate pages for Quick Start, PlatformIO, Arduino IDE, deploying globally, and deploying multiple devices. So the honest description of the setup path is: read the quickstart, then pick a toolchain. Two firmware build paths are documented, PlatformIO and the Arduino IDE, which is a deliberate choice for hobbyist hardware where Arduino IDE familiarity is common. The repository layout tells you where things live: server/deno/models/<provider> for each realtime adapter, server/fastapi for the Pipecat pipeline, and a Next.js app for device management. Because there are no releases retrieved for this repository, there is no pinned version to install and no changelog to diff against. You are tracking the main branch. The configuration surface implied by the README is provider API keys (OpenAI, Gemini, xAI, ElevenLabs, Hume, or a Cloudflare account plus an LLM key) and Supabase credentials for auth and transcript storage. I cannot give you the exact environment variable names because the README does not list them; they will be in the quickstart page or in the server directories.

Where ElatoAI is the wrong tool

The first limitation is documentation distribution. Everything operational lives on an external docs site, and the README is a feature list with links. If elatoai.com/docs moves or the project is abandoned, you are reading source code to find out how authentication is wired. The second is the absence of releases. With no tagged versions, there is no way to pin a known-good firmware and server pair, and no upgrade path other than re-reading the diff on main. For a hobby build that is fine. For a product you have already shipped to customers, it is a problem, because OTA updates are a listed feature and you will be pushing firmware to devices you cannot physically reach. The third is the licence. The repository metadata says NOASSERTION, which means GitHub could not map the licence file to a known identifier. That is not the same as saying there is no licence, but it does mean you cannot assume MIT or Apache-2.0, and anyone shipping hardware with this firmware should read the actual licence file before building a business on it. The fourth is scope: ElatoAI assumes you want the whole stack. If you already have a device management backend and only need the ESP32 audio client, you will be adopting a Next.js app, a Supabase schema and a server you may not want.

Alternatives and the actual difference in approach

The closest comparison in the material is Pipecat, which ElatoAI itself uses for the FastAPI backend. Pipecat is a Python framework for composing voice pipelines: you wire STT, LLM and TTS components together and it handles the audio transport and interruption logic. The difference is where the abstraction sits. Pipecat gives you a pipeline library and expects you to supply the device, the transport and the user management. ElatoAI gives you a finished device: firmware that already speaks Opus over a secure WebSocket, a webapp that already registers devices and stores transcripts, and adapters that already talk to six realtime providers. If you are starting from nothing and your target is a physical toy, ElatoAI has done the boring parts. If you already have a device platform and want to swap voice models freely, Pipecat is the lower-level and more portable choice, because your pipeline code does not depend on ElatoAI's device protocol. A second alternative is to use a realtime API directly from the ESP32 with no intermediate server, which removes a hop and a hosting bill but puts your provider API key on a device you ship to customers. ElatoAI's server-mediated design exists precisely to avoid that.

Maintenance cost and the licence question

Three server backends is three things to keep working. The Deno adapters track six vendors' realtime APIs, and those APIs change: model names get retired, event schemas get revised, authentication headers get reissued. Each adapter is a maintenance surface. The Cloudflare path shifts that burden partly onto Cloudflare's model catalogue, which is a trade you may or may not want. The FastAPI path depends on Pipecat, so you inherit Pipecat's release cadence on top of your own. Firmware adds a second axis: OTA is listed as a feature, so a server-side protocol change can strand devices in the field unless the firmware handles version negotiation, which the README does not describe. On licensing, the repository is marked NOASSERTION. I am not a lawyer and this is not legal advice, but the practical consequence is that you cannot treat ElatoAI as a drop-in permissive dependency. Read the licence file in the repository, and if you are shipping hardware commercially, have someone who understands the terms confirm what obligations attach to the firmware, the server code and the webapp separately, since a repository can carry more than one licence across directories.

Editorial conclusion

Adopt ElatoAI if you are building a hardware voice companion and you want the ESP32 side, the device management webapp and the model adapters already wired together, and you are willing to run your own Supabase project and your own model API keys. Do not adopt it if you need a permissively licensed, self-contained library you can vendor into a commercial firmware without legal review, because the repository is marked NOASSERTION and the README points at a hosted elatoai.com docs site and a Kickstarter campaign rather than a versioned release. Before committing, verify three things yourself: which server backend you will run (the Deno, Cloudflare and FastAPI paths have different dependencies and different model coverage), how the Supabase schema is created and migrated, and what the actual licence text in the repository says.

Official sources

1. akdeb/ElatoAI on GitHub
2. Issues
3. Project website
4. README