ThreatIngestor: a pipeline that turns feeds into IOCs

Extraction separated from where the data lands

ThreatIngestor is described as an extendable tool to extract and aggregate IOCs from threat feeds, and the wording is worth unpacking. It is not a feed reader and not a threat intelligence platform. It watches configured sources such as Twitter, RSS feeds, sitemap feeds or generic web pages, pulls meaningful information out of them, including malicious IPs and domains and YARA signatures, and hands that material to another system for analysis. The audience is a security team that already has somewhere to put indicators and lacks the plumbing to fill it automatically. The README notes it integrates out of the box with ThreatKB and MISP, and that it can sit inside an existing workflow through SQS, Beanstalk or custom plugins, and that it is the engine behind the InQuest Labs IOC-DB service.

Source plugins feed operator plugins

The architecture is a plugin model with two roles. A source plugin is an input, and the supported sources include Git repositories, GitHub repository search, gists by username, RSS feeds, sitemap feeds, image extraction, Amazon SQS queues, Twitter and generic web pages. An operator plugin is an output, and the supported destinations are CSV files, MISP, a MySQL table, a SQLite database, Amazon SQS queues, ThreatKB and Twitter. The value of the split is that the same extraction logic can feed several systems in one run, and the same destination can receive from several sources. The documentation also covers writing your own plugins, so a source for an internal feed or an operator for an internal ticketing system is a supported extension point rather than a fork. The repository carries a scripts/ directory and a tests/ directory next to the threatingestor/ package, which is where that plugin surface lives.

Where the indicators end up

The operator side determines what ThreatIngestor is for in your stack. Pushing to MISP or ThreatKB keeps your existing platform as the system of record and lets the tool act purely as a collection agent, which is the arrangement most teams want. The SQS and Beanstalk operators are for pipelines: a queue between the extractor and whatever consumes the indicators next, so that processing does not block the poll loop. The database operators write to MySQL or SQLite, and CSV output suits a small team that wants a file to import somewhere else. The Twitter operator can also publish, which is how a shared indicator list gets built, and the README points to an InQuest Twitter List that posts C2 domains and IPs as a ready made source. Using the Twitter source requires a Twitter developer account, which is the first credential hurdle you hit when setting this up.

Installing from PyPI and choosing the extras

ThreatIngestor requires Python 3.6 or later with development headers, and the base install is a single pip command:

pip install threatingestor

Most integrations are optional and shipped as extras, so the install for a working setup is usually the complete one:

pip install threatingestor[all]

The package metadata shows how the extras map to plugins. There are extras named twitter, rss, misp, threatkb, beanstalk, sqs and mysql, a catch-all extras group holding hug, boto3, greenstalk and watchdog, and an all group built from the testing requirements file. The console entry point is registered as threatingestor = threatingestor:main, so the tool is driven from the shell once installed. For the image extraction source, or to add image extraction to other sources, the README asks for Python 3.7 or later because of the dependencies, which are installed separately:

pip install opencv-python pytesseract numpy

One config.yml drives every source and operator

Configuration is a single YAML file. You create a new config.yml and configure each source and operator module you want to use, with config.example.yml in the repository showing the layout, and the run command takes that file as its only argument:

threatingestor config.yml

By default the process runs forever, polling each configured source every 15 minutes. That default is the behaviour to be aware of when you deploy it: it is a long running poller, not a one-shot script, so it belongs in a service unit or a container that is expected to stay up. For a quick start the README suggests RSS feeds and points at rss.example.yml, which holds a few pre-configured security blogs, and mentions the quick walkthrough in the documentation for trying it out. There is also a config.yml copied into the Docker image by the Dockerfile, which is where you would mount your own file.

Image extraction pulls in OCR and OpenCV

The image extraction source is the heavy end of the plugin set, and it is the one feature that changes the deployment picture. It needs Python 3.7 or later, and the extras are opencv-python, pytesseract and numpy, which together mean an OCR stack rather than a text parser. The Docker image installs tesseract-ocr and python3-lxml alongside the Python packages for the same reason. The practical limit is that image extraction is only worth its cost if your feeds actually publish indicators as images, which is common for some C2 and IOC channels and useless for the many that publish text. Since the same dependencies are also pulled in by the Dockerfile for everyone, a container based deployment pays for OCR even if the source is never enabled, which is a reasonable argument for installing from PyPI in a virtual environment instead when you do not need the image source.

The Docker image starts from ubuntu:18.04

A Dockerfile is provided for running ThreatIngestor in a container, and the base image is old enough to matter. The build starts from ubuntu:18.04, installs python3-pip, sqlite3, tesseract-ocr and python3-lxml, copies requirements.txt, installs the requirements plus the image extraction packages, installs threatingestor itself with pip3 install threatingestor -U, and copies config.yml into the image. The documented build and run commands are:

docker build . -t threatingestor
docker run -it --mount type=bind,source=/,target=/dock threatingestor /bin/bash

The run command opens a shell inside the container with the host filesystem mounted at /dock, and from there you start the tool the same way as anywhere else, with threatingestor config.yml. There is a separate Dockerfile.dev for development, which expects a wheel built by python3 -m build. Note that the bind mount uses source=/ , so the container can see the whole host filesystem, which is broader than a configuration file needs and worth narrowing if you adopt this pattern.

Version drift between the last release and the last push

The two dates to keep in view are far apart. The last push to the master branch was on 2026-05-26, so the code in the tree has seen work within the last few months. The releases are older: v1.4.0 from 2023-11-02, v1.3.3 from 2023-09-07 and v1.2.0 from 2023-06-14, and the setup.py metadata still declares version 1.4.0. That means the PyPI package and the repository head are not the same thing, and the documentation at the ReadTheDocs site is built from the branch rather than from the wheel you install. The licence field on the repository says GPL-2.0, and setup.py declares the license as GPL, which matters if you intend to modify the tool and keep it internal: a pull request also counts as agreeing to release your contribution under those terms. The runtime dependencies listed in requirements.txt are modest and include iocextract, requests, beautifulsoup4, PyYAML and jsonpath_rw.

Editorial conclusion

Adopt ThreatIngestor when your team already keeps indicators in MISP or ThreatKB and needs a repeatable way to pull IOCs out of a mixed set of feeds, because the source and operator split means you can change where data comes from without touching where it lands. Do not adopt it as a ticketing system, a full platform with alerting, or a replacement for a feed parser you only need occasionally, since the extraction library it depends on can be used on its own. Verify first that the plugins you need are covered by the extras you install, that the image extraction source has the OCR dependencies available, and that the version you pin is the one you have read the documentation for, since the newest release is v1.4.0 from 2023-11-02.

Frequently asked questions

Official sources

1. License: GPL-2.0
2. pedramamini/ThreatIngestor on GitHub
3. Project website
4. README
5. Releases