IntelMQ: a message queue for threat intelligence feeds, one bot at a time

Feed processing as a queue of small independent programs

IntelMQ is a solution for IT security teams, and the project names its audience directly: CERTs and CSIRTs, SOCs, abuse departments. It collects and processes security feeds such as log files through a message queuing protocol, and it started as a community driven initiative called IHAP, the Incident Handling Automation Project, conceptually designed by European CERTs and CSIRTs during several InfoSec events.

The design goal was to give incident responders an easy way to collect and process threat intelligence. The listed uses are automated incident handling, situational awareness, automated notifications, use as a data collector for other tools, and more. The project describes itself as a pipeline rather than a library, and the difference is worth internalising: the unit of work is a message moving between stages, not a function you call.

The components are called bots. Each bot does one thing, such as parse a feed from a particular source, harmonise the fields, or write the result to a database, and bots are wired together through the queue. A bot is a normal Python process with its own configuration file, so adding a source means adding a bot rather than changing a central program.

That design has a direct consequence for operations. A pipeline that must survive crashes needs its state somewhere durable, and IntelMQ's answer is the queue itself, backed by Redis, with persistence handled as a first-class concern rather than as an afterthought.

The goals the project states, including what it refuses to grow into

The project's own list of aims is unusually concrete, and reading it tells you what to expect. It aims at reducing the complexity of system administration, reducing the complexity of writing new bots for new data feeds, and reducing the probability of events being lost across the process with persistence functionality, even on a system crash.

The remaining aims are about interoperability. IntelMQ uses and improves an existing Data Harmonization Ontology, so the fields in a message have a shared definition rather than being whatever a particular feed happened to provide. It uses JSON format for all messages, which means a bot written for one stage can be inspected and tested without a parser. It provides an easy way to store data into databases and log collectors such as PostgreSQL, Elasticsearch and Splunk, and an easy way to create blacklists of your own. Communication with other systems goes through an HTTP RESTful API.

Then there are the meta-guidelines, which read like a constitution. Do not break simplicity, KISS. Keep it open source, forever. Strive for perfection while keeping a deadline. Reduce complexity and avoid feature bloat. Embrace unit testing. Code readability is tested with inexperienced programmers. Communicate clearly.

Those guidelines are the honest signal about this project. It is explicitly not trying to become a platform with a plugin marketplace and a policy engine. A team that needs those things should look elsewhere, and a team that wants a boring pipeline that a new joiner can read in an afternoon is in the right place.

Redis carries the messages, and every bot is configured in YAML

The runtime dependencies list in the packaging metadata is short and tells you how little machinery is involved. dnspython at 2.0.0 or newer handles name resolution, psutil at 1.2.1 or newer provides process information, python-dateutil at 2.5 or newer handles timestamps, python-termstyle at 0.1.10 or newer adds colour to the console output, redis at 2.10 or newer is the queue itself, requests at 2.2.0 or newer handles HTTP fetching, ruamel.yaml handles the configuration files, and packaging is used for version handling.

That redis dependency is the centre of the architecture. Bots do not call each other. They publish and consume messages on Redis, and the queue is what provides the decoupling, the retry behaviour and the persistence. Redis being a separate service is the one real infrastructure requirement, and it is a small one.

Configuration is YAML and lives in the package. The Makefile shows the two files that define the system's behaviour: intelmq/etc/feeds.yaml, which holds the feed definitions, and intelmq/etc/harmonization.conf, the harmonization configuration. The docs target depends on both, alongside the harmonization module itself, which tells you the documentation is generated from the same data the runtime uses rather than maintained separately and drifting.

The setup script reflects the same principle in code. The bots are discovered by walking the intelmq/bots directory for Python files, skipping the private modules whose names begin with a leading underscore, and each one is registered as an entry point mapping the module name to its run function. A new bot becomes available by being dropped into the tree, with no manual registry edit.

Installing the package and the developer tooling around it

IntelMQ is a Python package published under the name intelmq, and the metadata requires Python 3.9 or newer. The project homepage points at the documentation site, and the packaging metadata declares a development extra that combines the test requirements with the docs requirements, so a contributor installs the published package and then asks for that extra to get the tooling in place.

The test dependencies are Cerberus with the 1.3.0 version excluded, and requests_mock, which tells you the test suite validates configuration with a schema library and stubs HTTP calls rather than reaching the network. The documentation set is a full mkdocs-material setup with mkdocstrings for API reference, mike for versioned docs, lunr and pygments for search and highlighting, plus redirects, a glightbox extension and a minify plugin.

For a contributor, the Makefile is the working surface. The test target runs pytest with coverage disabled, the coverage target runs pytest with the intelmq package measured, and the codestyle target runs pycodestyle over the bots, lib and bin directories. There is a codespell target with an exclusion file for the GitHub workflows, a licenses target that runs reuse lint, and a clean target that removes the build, cache and coverage artefacts.

The docs target is the interesting one, since it regenerates two documentation sets before building:

python3 scripts/generate-feeds-docs.py
python3 scripts/generate-event-docs.py
mkdocs build

Both generators run before the site is built, so the feed and event reference pages in the documentation are derived from the runtime configuration. That is why the docs make target depends on feeds.yaml, harmonization.conf and the harmonization module.

Where the scope stops, and what is missing from the documentation

The first limitation is that IntelMQ is a pipeline framework, not a data store you query. The outputs go to PostgreSQL, Elasticsearch, Splunk, log collectors, or out over an HTTP RESTful API, so if you want to search historical indicators, you depend on one of those systems. The project gives you the collection and processing, not the analysis, and that is a reasonable division but worth being clear about before you install it.

The second is that the bot catalogue is the real product, and it is a community asset. IntelMQ is used with feeds such as log files, and the value of the framework for a given source depends entirely on whether a bot for that source exists or whether you write one. The project states the aim of reducing the complexity of writing new bots for new data feeds, which confirms that writing them is an expected activity rather than an exceptional one. Nothing in the project description promises a complete feed coverage, so an adopter should assume writing or maintaining at least some bots.

The third limitation is legal and administrative rather than technical. The project is licensed AGPL-3.0, with the SPDX identifier AGPL-3.0-or-later, and the repository has a separate LICENSES directory alongside the LICENSE file, a COPYRIGHT file and a .reuse directory for the reuse compliance tooling. Network copyleft is a genuine consideration for a service that processes data on behalf of others, and the answer is not something this article can give you.

On maintenance, the repository is not archived and the last push was on 2026-04-28. The published releases are 3.5.0 from 2025-11-01, 3.4.0 from 2025-03-14 and 3.3.1 from 2024-09-03, so release tags lag the working tree, and the default branch is develop rather than a release branch.

Editorial conclusion

IntelMQ fits a CERT, CSIRT, SOC or abuse team that ingests feeds from many sources and needs a durable, harmonised record of indicators, and it fits teams that already have a feed-processing script they want to run continuously rather than by hand. It is a poor fit for a single analyst who needs to check one source occasionally, because the operational surface, the bots configuration and the harmonization ontology are all sized for a pipeline rather than a query. Before adopting, read the Data Harmonization Ontology that the project reuses, check that your feed has a matching parser in the bots tree, and confirm the AGPL-3.0 obligations suit how you intend to run it, since the project was rewritten from scratch and carries network copyleft terms.

Frequently asked questions

Official sources

1. certtools/intelmq on GitHub
2. License: AGPL-3.0
3. Project website
4. README
5. Releases