whisper.api: self-hosted Whisper ASR behind a Deepgram shaped interface

Whisper transcription with a familiar request shape

whisper.api is an open source, self-hosted API for speech to text transcription powered by whisper.cpp. The repository description frames it as an API with user level access support that transcribes speech using a finetuned and processed Whisper ASR model.

The decision that defines the project is the interface. It exposes a Deepgram compatible interface over both REST and WebSocket, and the feature list calls it drop-in compatible with the /v1/listen endpoints. The stated benefit is easy integration into existing workflows while maintaining full data ownership, and for a compliance sensitive deployment that combination is the whole proposition: the audio never leaves your infrastructure, and the client code barely changes.

The transcription features go beyond the compatible baseline. There is custom vocabulary through prompting, so you can steer the model towards names and jargon. There is audio cropping with start and duration, so a long recording can be cut into the segment you care about without processing the rest. And there is speaker diarization, which is the feature that turns a flat transcript into something you can attribute to people.

Output formats are native rather than converted, with JSON, SRT and VTT all supported directly.

The implementation is Python on FastAPI with uvicorn, the model runs as a separate whisper.cpp binary, and the licence is MIT. The homepage is whisper.vedgupta.in, and the last push was on 2026-04-28.

Installation ends with a script that builds the binary

The quick start is four steps, and the third one is the unusual part.

pip install -r requirements.txt
cp .env.example .env
chmod +x setup_whisper.sh
./setup_whisper.sh

The Python requirements are conventional: FastAPI, uvicorn with the standard extras, python-dotenv, SQLAlchemy with psycopg2 for the database, pydantic-settings, websockets and aiofiles for the streaming path, python-multipart for uploads, and gdown for fetching models. The test tooling is there too, with pytest, pytest-cov, faker, requests-mock and httpx, and black for formatting.

Notice what is not in the list. There is no Whisper package to install, because whisper.cpp is a C++ program, not a Python library. The setup script is what downloads or builds that binary and places it where the configuration expects it, and the environment file points at the result with WHISPER_BINARY_PATH set to ./binary/whisper-cli.

That is the whole reason the setup step is a script rather than a pip command. The Python side is a wrapper around a native binary, and the binary is what does the work.

The Dockerfile takes the same approach. It builds on python:3.10-slim-bookworm, installs ffmpeg, curl, git, cmake and build-essential, copies the project, and then runs the same script during the image build.

RUN chmod +x ./setup_whisper.sh && ./setup_whisper.sh

Keys, the database and a test endpoint you should switch off

The second step of the quick start creates the database and the API keys, and it is done with a module CLI rather than a web page.

python -m app.cli init
python -m app.cli create --name "MyAdminKey"

The init command sets up the schema, which given the SQLAlchemy and psycopg2 dependencies is a database the application manages itself, and the create command mints a key with a name attached so you can tell them apart later. The default configuration points at a local SQLite file, DATABASE_URL is set to sqlite:///./whisper.db and the test database to sqlite:///./test.db, so a first run does not need a database server, while the Postgres driver is present for a deployment that wants one.

Then there is the flag that deserves the closest reading. The README says that for local testing only you can enable POST /v1/auth/test-token in Swagger by setting ENABLE_TEST_TOKEN_ENDPOINT=true, that it defaults to off, and that it should never be enabled in production.

That is a well written warning. It is also worth noticing what the published container image does.

# Add test token endpoint flag explicitly, Disable this flag for production environments!
ENV ENABLE_TEST_TOKEN_ENDPOINT=true

The Dockerfile sets it to true, with a comment asking you to disable it. If you deploy the image as published, that endpoint is live until you override the environment variable, and it exists precisely to hand out tokens without a login. Setting the variable at run time is the difference.

Two ways to send audio, one endpoint

The listening endpoint takes audio in the body, either as the raw bytes or as a JSON document with a URL.

curl -X POST 'http://localhost:7860/v1/listen' \
  -H "Authorization: Token <YOUR_KEY>" \
  -H "Content-Type: audio/wav" \
  --data-binary @audio.wav

curl -X POST 'http://localhost:7860/v1/listen' \
  -H "Authorization: Token <YOUR_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/audio.mp3"}'

Authentication is a token in the Authorization header, which is the Deepgram convention rather than a bearer token, and the content type decides which of the two paths you are on.

The features that make it more than a thin wrapper are parameters on the same call. The custom vocabulary prompting lets you bias the recogniser towards your domain terms. The start and duration cropping lets you point at the part of the recording you want, which turns a long meeting recording into a cheap query. And the diarization gives you speaker labels on the result.

For live audio there is a WebSocket path, described as real-time 16kHz PCM transcription. The chunk duration is configured by STREAM_CHUNK_DURATION_MS and set to 5000 by default, so the streaming granularity is five seconds of audio at a time.

The example files in the repository reflect the same three modes: a microphone transcription script, a speech to text demo directory, and a separate test for streaming.

URL ingestion is guarded, because it has to be

The second call mode, where the server fetches the audio itself, is the one with a security problem, and the project treats it accordingly.

The README states that the server fetches the URL for you with SSRF protections: public hosts only, size limits, and redirects off by default. The environment file documents the knobs behind that, with MAX_AUDIO_DOWNLOAD_BYTES for the size cap and AUDIO_URL_FOLLOW_REDIRECTS for the redirect behaviour, and it ships the safer default.

# Safer default: no redirects on URL ingest (set true only if you trust your callers)
AUDIO_URL_FOLLOW_REDIRECTS=false

The reasoning is straightforward once you state it. An endpoint that takes a URL from an authenticated caller and fetches it will happily reach internal addresses unless it is told not to, and a redirect is the easiest way around a host allowlist, so the redirect switch is off until someone has a reason to turn it on.

The size limit matters for the same reason the concurrency limit exists. A transcription service that downloads arbitrary content is a service that can be asked to fill its disk or its memory, so the download cap is set to the same 50 MB as the upload cap.

Both of these are visible in .env.example, which is the right place for them, and both are documented in the README rather than being behaviour you discover by reading the code.

The configuration file is where the operational limits live

The .env.example file is the most operationally useful document in the repository, and it is organised into labelled sections.

The security section holds DEBUG, which defaults to true, SECRET_KEY with a placeholder and a warning to generate a strong random value for production and never commit real secrets, and the test token flag. The server section has SERVER_NAME, SERVER_HOST and BACKEND_CORS_ORIGINS, which defaults to an empty list, meaning no cross origin access is allowed until you name the origins you want.

The database section is the SQLite by default configuration described earlier.

The Whisper section is the one to read before you size the machine.

WHISPER_BINARY_PATH=./binary/whisper-cli
MODELS_DIR=./models
MAX_CONCURRENT_TRANSCRIPTIONS=2
MAX_AUDIO_UPLOAD_BYTES=52428800
WHISPER_CLI_TIMEOUT_SEC=3600
FFMPEG_TIMEOUT_SEC=600

The concurrency cap of two simultaneous transcriptions is the number that determines how many clients the service really supports, since each transcription shells out to the whisper binary. The two timeouts are separate on purpose, an hour for the transcription itself and ten minutes for the ffmpeg conversion that precedes it, so a corrupt or pathological file cannot hold a worker for an hour in the conversion step.

The documentation is an Astro Starlight site in the docs directory, run locally with Bun, and it covers getting started and local setup, authentication and API keys, the REST and WebSocket reference, code examples, models and deployment guides, and the contributing workflow.

What the project is and is not

It is worth being clear about what this repository is, because the name suggests more than it delivers in one direction and less in another.

It is not a Whisper implementation. The transcription is done by whisper.cpp, a separate project, and this repository is the service around it: the HTTP surface, the key management, the queueing of work onto the binary, the streaming path, and the model handling. The project references both whisper.cpp and OpenAI Whisper, and the distinction is between the engine and the API, not between two competing engines.

It is also not a multi-tenant platform in the sense of a hosted speech service. The keys are issued by a local CLI and the users are the people who can reach your instance, which is what the user level access support in the description refers to. For an internal tool or a service on a private network, that is the right shape. For a public multi-tenant product, the key administration and the isolation model would need work that the repository does not claim to have done.

What it does well is the unglamorous part. There is a test suite with pytest and coverage configured, there is a Dockerfile that builds the native dependency rather than assuming it, there is an examples directory with a microphone script and a streaming test, and the configuration is explicit about limits rather than leaving them implicit.

The one thing to fix before production is the test token endpoint that the published Dockerfile switches on.

Editorial conclusion

Adopt whisper.api when you need speech to text without sending audio to a third party and when your existing client already speaks the Deepgram API, since the compatibility layer means the migration is a base URL change. Do not adopt it if you need a guaranteed low latency stream under load without tuning, because the default concurrency is two simultaneous transcriptions and the streaming chunk duration is five seconds. Before the first deploy, generate a real secret key and set ENABLE_TEST_TOKEN_ENDPOINT to false, which is the opposite of what the published Dockerfile does, and confirm your audio size against MAX_AUDIO_UPLOAD_BYTES, which is set to 50 MB by default.

Frequently asked questions

Official sources

1. innovatorved/whisper.api on GitHub
2. Issues
3. License: MIT
4. Project website
5. README