WhisperPlus for Python Speech, Summarization, and Video Queries

What WhisperPlus Covers

WhisperPlus targets Python developers who want to combine speech recognition with downstream language tasks. The library can download YouTube audio or video, run Whisper-based transcription, summarize transcripts, identify speakers, answer questions about video content, synthesize speech, and create automatic captions. That makes it more than a wrapper around one transcription call. It presents several pipelines around audio and video files, with model selection left to the caller.

The primary transcription example uses `SpeechToTextPipeline` with `distil-whisper/distil-large-v3`. Other examples use `openai/whisper-large-v3`, an MLX Whisper model, and `LightningWhisperMLX`. The summarization pipelines use `facebook/bart-large-cnn`, while the RAG examples use embedding and language models selected by the application.

This scope suits experiments, internal tools, and applications that need multiple AI stages. It is less suitable as a lightweight transcription dependency. A project that only needs a speech-to-text API may carry unnecessary model, video, and summarization dependencies. The README does not document a rollback procedure, deployment configuration, or a single command for running the complete set of pipelines.

The Transcription Data Flow

The documented flow begins with `download_youtube_to_mp3`, which creates an audio file at a chosen output location. That path is passed to `SpeechToTextPipeline`, together with the model identifier and optional quantization configuration. The pipeline is then called with audio segmentation and decoding parameters, including `chunk_length_s=30`, `stride_length_s=5`, `max_new_tokens=128`, and `batch_size=100` in the README example. The returned transcript is printed directly.

The example also demonstrates two model-efficiency paths. `HqqConfig` configures four-bit quantization, while `BitsAndBytesConfig` requests a four-bit bitsandbytes model with `nf4`, `bfloat16` computation, and double quantization. The call enables `flash_attention_2=True`. These settings are not merely presentation options. They alter how the model is loaded and executed, and they create hardware and dependency constraints.

For Apple systems, the README provides an MLX path through `mlx_whisper.transcribe`, using `mlx-community/whisper-large-v3-mlx`. A separate `LightningWhisperMLX` example accepts a model name, batch size, and quantization value. The repository therefore exposes several execution paths rather than one fixed transcription implementation. The exact internals of each pipeline are not documented in the README, so performance and memory behavior cannot be inferred from the project description alone.

Installation and First Transcription

The README lists the base installation as a pip command. It also requires a Transformers Git URL and `flash-attn`, so the documented setup is broader than installing a single wheel from PyPI. After installation, the first real use is the YouTube audio path shown below. The reader should see the downloaded audio path followed by a transcript printed by the script.

pip install whisperplus git+https://github.com/huggingface/transformers

This installs WhisperPlus and the Transformers source dependency named by the README. The second command installs Flash Attention, which the transcription example enables through `flash_attention_2=True`.

pip install flash-attn --no-build-isolation

The first-use script downloads the video audio, creates a quantized Whisper pipeline, transcribes it, and prints the result:

from whisperplus import SpeechToTextPipeline, download_youtube_to_mp3
import torch

url = "https://www.youtube.com/watch?v=di3rHkEZuUw"
audio_path = download_youtube_to_mp3(url, output_dir="downloads", filename="test")

pipeline = SpeechToTextPipeline(
    model_id="distil-whisper/distil-large-v3",
    flash_attention_2=True,
)
transcript = pipeline(
    audio_path=audio_path,
    chunk_length_s=30,
    stride_length_s=5,
    max_new_tokens=128,
    batch_size=100,
    language="english",
    return_timestamps=False,
)
print(transcript)

The README's fuller example also constructs `HqqConfig` and `BitsAndBytesConfig` before creating the pipeline. The reduced block above follows the same documented pipeline shape while avoiding the larger configuration block. The README does not state supported operating systems or a minimum GPU, so those points must be checked against the selected model and local environment.

Summarization, Diarization, and Video Question Answering

WhisperPlus extends a transcript beyond transcription. `TextSummarizationPipeline` takes the transcript and returns summary output. For longer input, `LongTextSummarizationPipeline` is offered as a separate class. Both README examples select `facebook/bart-large-cnn`, but the library exposes different interfaces: the standard example reads `summary[0]["summary_text"]`, while the long-text example assigns `summary_text` directly.

Speaker diarization uses `ASRDiarizationPipeline` with an ASR model and `pyannote/speaker-diarization-3.1`. The README requires confirmation of licensing permissions for two pyannote models, then installs additional requirements and authenticates through `huggingface-cli login`. The example passes speaker bounds such as `num_speakers=2`, `min_speaker=1`, and `max_speaker=2`, then formats the result with `format_speech_to_dialogue`.

The RAG paths read an input text file or an input directory, create an embedding index, and query it with `run_query`. One example uses LanceDB-related dependencies and local model names. The AutoLLM example can use `gpt-3.5-turbo`, `YOUR_OPENAI_KEY`, and `YOUR_HUGGINGFACE_KEY`, while relying on a document context supplied by the prompt. This is a broad application layer, not a single optimized video chat implementation. Each extra stage adds model downloads, configuration, and licensing review.

Hardware, Dependency, and License Boundaries

The base `requirements.txt` lists `torch>=2.0.0`, `torchvision>=0.15.0`, `moviepy>=1.0.3`, `yt_dlp`, `ffmpeg`, `ffmpeg-python`, `accelerate`, `bitsandbytes`, `hqq`, `transformers`, `Requests`, `fire`, and `pre-commit`. Optional features add more packages, including `sentence-transformers`, `ctransformers`, `langchain`, `autollm>=0.1.9`, and `moviepy` is already part of the base list. Speaker diarization has its own requirements file.

This dependency surface creates upgrade work. Changes to Torch, Transformers, bitsandbytes, MLX integrations, and model repositories can affect several pipelines. The README does not pin complete versions for most dependencies, and the listed release history ends with `v0.3.0` on 2024-05-04. The last push was on 2026-10-05, so the repository has recent repository activity, but the README does not describe a compatibility policy between the package release and the current `main` branch.

The project is distributed under `Apache-2.0`. That applies to the repository's own code, not automatically to downloaded models, YouTube media, or external services. The README specifically calls out licensing permissions for the two pyannote models. License obligations for `facebook/bart-large-cnn`, Whisper variants, local GGUF models, and hosted APIs are not explained, so model terms require separate review.

Choosing WhisperPlus Instead of a Narrow Whisper Integration

A direct Transformers Whisper pipeline is the clearest alternative when the requirement is only transcription. The direct approach keeps the model loading and decoding code in the application, while WhisperPlus adds helpers for YouTube downloads, quantization configuration, MLX support, Lightning MLX, summarization, diarization, captioning, text-to-speech, and RAG. The difference is breadth and convenience, not a documented change to the underlying Whisper model family.

For a video question-answering product, a separate transcription and indexing stack may also be easier to reason about. The WhisperPlus examples combine transcript files, embedding models, vector storage or AutoLLM, language models, and prompt wrappers. That flexibility is useful when several related tasks share the same Python codebase. It also means that diagnosing a problem requires identifying whether the failure occurred during download, transcription, summarization, indexing, retrieval, or generation.

The project is therefore a poor fit when one operation, minimal dependencies, or predictable deployment behavior matters more than pipeline variety. The README does not publish a compatibility table, resource requirements, or a single reference configuration, so an adoption decision depends on validating the exact pipeline and model combination.

Editorial conclusion

Adopt WhisperPlus when you need several local AI workflows behind one Python API and can manage model, hardware, and dependency requirements. Do not choose it for a minimal transcription-only setup without reviewing its Torch, Transformers, FFmpeg, and model-specific requirements. Verify first whether your hardware supports the selected backend and whether the required model licenses permit your intended use.

Frequently asked questions

Official sources

1. Issues
2. kadirnar/whisper-plus on GitHub
3. License: Apache-2.0
4. README
5. Releases