Patter SDK: a self-hosted voice agent stack with swappable providers

The gap Patter fills: carrier plumbing between your agent and the PSTN

A voice agent is not one program. It is an agent loop, a language model, a speech-to-text step, a text-to-speech step, audio handling for interruptions, and a telephony carrier that terminates a real phone number. Most teams start with a hosted platform that bundles all of it and pay per minute. Patter takes the other position: the README describes it as the full voice stack between your application and the phone network, and explicitly frames it as the option for builders who want to own the stack.

The intended user is a developer who already has opinions about providers. If you want OpenAI Realtime for one agent and a Deepgram plus ElevenLabs pipeline for another, Patter exposes both from the same SDK surface. The README claims Python and TypeScript are at parity, same hooks and same events, which matters if your backend is Python but your tooling is TypeScript. The MIT licence removes the redistribution question that comes with a proprietary voice platform, though that is a licensing fact and not legal advice about your own deployment.

Three voice modes and a provider per layer

Patter does not pick a model for you. The README lists the layers it abstracts: LLM, STT, TTS, realtime engine, telephony carrier, and audio processing for VAD and noise suppression. Each layer has named options. LLM covers OpenAI, Anthropic, Google Gemini, Groq and Cerebras. STT covers Deepgram, AssemblyAI, Cartesia, Soniox, Speechmatics, Whisper and Fish Audio. TTS covers ElevenLabs, OpenAI, Cartesia, LMNT, Rime, Telnyx and Fish Audio. Realtime covers OpenAI Realtime, Gemini Live, Ultravox and ElevenLabs ConvAI. Carriers are Twilio, Telnyx and Plivo. Audio is Silero VAD, Krisp and DeepFilterNet.

The composition model is the part worth understanding before you adopt. Realtime mode routes the whole conversation through one all-in-one engine such as OpenAIRealtime, which is the shortest path in the quickstart. Pipeline mode chains separate STT, LLM and TTS providers, which is what the Custom Voice template demonstrates with Deepgram plus ElevenLabs. Hybrid mode is documented as a third option in the README but the supplied material does not describe how it splits work between the two, so treat that as something to read up on rather than assume. Above the layers, the README lists an automatic LLM fallback chain for provider failover mid-call, tools and call transfer and guardrails that behave the same on every carrier, and an OpenTelemetry trace per call.

Getting a number answering: install, credentials, serve

Both SDKs install from their public registries. TypeScript is npm install getpatter. Python is pip install getpatter, and the badge in the README states Python 3.11+ and TypeScript 5.0+.

The README's TypeScript example constructs a Patter instance with a carrier and a phone number, builds an agent with an engine and a system prompt, and calls serve with tunnel enabled. The Python version is the same shape with snake_case keys: Patter(carrier=Twilio(), phone_number="+15550001234"), then phone.agent(engine=OpenAIRealtime(), system_prompt=..., first_message=...), then await phone.serve(agent, tunnel=True).

Credentials come from environment variables. The README names TWILIO_ACCOUNT_SID and OPENAI_API_KEY as examples and points at the docs for the full catalog. Switching carrier means replacing the Twilio class with Telnyx or Plivo. The tunnel flag spawns a Cloudflare quick tunnel and points your number at it, which the README calls ideal for local dev; production is expected to use a static webhook_url or ngrok. That distinction is the first operational decision you make, and it is not optional: a quick tunnel is a development convenience, not a webhook target you want behind a live number.

Telemetry is on by default and you should decide about it early

The README is unusually direct here. Patter collects anonymous, opt-out usage data covering SDK version, bucketed provider and model, and call facts. It states that call content, prompts, phone numbers, keys and free text are never collected. Whether you accept that description is a judgement you make, but the controls are concrete: Patter(telemetry=False) in Python, new Patter({ telemetry: false }) in TypeScript, the getpatter telemetry disable command, or the PATTER_TELEMETRY_DISABLED=1 environment variable. The SDK also honours DO_NOT_TRACK=1 and turns telemetry off automatically in CI and tests. PATTER_TELEMETRY_DEBUG=1 lets you inspect what would be sent without sending it.

For a regulated deployment, the default matters. A voice agent that handles health or financial calls will usually be reviewed by someone who wants the data flow written down, and "opt-out anonymous usage data" is a line they will ask about. Setting the environment variable in your container definition is cheaper than retrofitting it after a review.

Where the bring-your-own-provider model costs you

The main limitation follows directly from the design. Patter hands you the agent loop and the carrier integration, but every provider you compose is a separate account, a separate key, a separate rate limit and a separate failure mode. The README advertises an automatic LLM fallback chain for mid-call failover, which addresses one class of outage, but it does not claim the same for STT, TTS or the carrier. If your Deepgram key hits a quota mid-call, the material does not describe a fallback for that layer.

Latency is the other honest unknown. Pipeline mode chains three network hops (STT, LLM, TTS) where Realtime mode uses one, and the README does not publish latency figures for either. Any team evaluating this for a latency-sensitive use case should measure it on their own providers rather than infer it from the architecture. There is also a version-cadence cost: releases v0.6.5, v0.6.8 and v0.7.0 landed between early June and late June 2026, with v0.7.0 described as adding an Inworld integration, security hardening and a pipeline barge-in fix. A barge-in fix in a 0.x release is a reminder that interruption handling is where these systems break, and that you are tracking a young API surface.

Finally, the tunnel path. Running locally with tunnel=True is genuinely convenient, but it means your laptop or dev container is in the call path during testing. Teams that skip the move to a static webhook_url will discover this the first time a call arrives while their machine is asleep.

The alternative: a hosted platform, and what actually differs

The README positions Patter against Vapi and Retell, which are hosted voice-agent platforms. The difference is not feature parity, it is where the runtime lives. With a hosted platform you configure an agent in their console, they run the agent loop, they hold the provider relationships, and you pay per minute. You get an operations team you did not hire and you give up the ability to choose, say, a specific Whisper deployment for STT.

With Patter you write the agent in your own Python or TypeScript process, run phone.serve, and point a Twilio, Telnyx or Plivo number at your endpoint. The provider relationships are yours, the process is yours to scale, and the OpenTelemetry trace described in the README goes to whatever collector you already run. The cost model shifts from per-minute to per-provider-plus-your-compute, which is cheaper at volume and more work at any volume.

There is a middle option the README also addresses: Patter ships skills for coding agents (npx skills add patterai/skills), which the README says works across roughly 55 agent harnesses consuming the Anthropic Agent Skills standard. That is a documentation-distribution choice rather than an architecture choice, but it tells you the project expects you to be writing this code with an AI assistant in the loop.

Maintenance cost and what to verify before you commit

The maintenance surface is the provider matrix. Twenty-seven or more integrations across six layers, per the README, means that when any provider changes an API or a model name, someone has to update the SDK. The release notes show active work, including security hardening in v0.7.0, but the cadence also means you should pin a version rather than track main. The MIT licence means you can fork and patch if a provider breaks and upstream is slow, which is the practical reason to prefer MIT here over a source-available licence.

Two things are not answerable from the supplied material. First, the Hybrid mode: the README names it but does not explain how it mixes Realtime and Pipeline behaviour. Second, the full environment variable catalog, which the README defers to docs.getpatter.com. Before you build on this, read those two pages, then run the quickstart against a real Twilio or Telnyx number with tunnel=True and confirm that the audio path and barge-in behave as the v0.7.0 release notes claim. The templates (patter-inbound-agent, patter-outbound-calls, patter-tool-calling, patter-custom-voice, patter-dynamic-variables, patter-custom-llm) are self-contained repos with a .env and a run command, and the outbound one is the only place the README mentions answering-machine detection and voicemail drop, so start there if your use case is outbound.

Editorial conclusion

Patter fits teams that already run their own infrastructure and want the LLM, STT, TTS and carrier choices to be theirs, in Python or TypeScript, under MIT terms. It does not fit teams that want a managed dashboard, per-minute billing and no servers to operate, because the SDK hands you the agent loop and the tunnel and expects you to keep them running. Before committing, verify the current provider matrix against the docs at docs.getpatter.com, confirm the telemetry defaults in your environment (Patter(telemetry=False) or PATTER_TELEMETRY_DISABLED=1), and test one real call on your chosen carrier rather than trusting the four-line example.

Official sources

1. License: MIT
2. PatterAI/Patter on GitHub
3. Project website
4. README
5. Releases