Machinae, the OSINT collector that queries many feeds at once

What the tool actually collects

Machinae is a tool for collecting intelligence from public sites and feeds about various security-related pieces of data: IP addresses, domain names, URLs, email addresses, file hashes and SSL fingerprints. It was inspired by Automater, and the project was born from a wish to improve Automater in four areas: bringing it to Python 3 compatibility while making the code more pythonic, using a more human readable configuration format in YAML, supporting JSON parsing out of the box without needing regular expressions while still supporting regex scraping when needed, and supporting additional output types including JSON while making extraneous output optional.

Those four goals are the design. A single command takes one or more observables and queries a configured set of services about each one, normalising the results into a common output.

The out-of-the-box data source list is long and worth scanning because it shows the shape of the tool. IPVoid, URLVoid, the URL Unshortener, Malc0de, SANS, FreeGeoIP, Fortinet Category and VirusTotal pDNS appear both as a web scrape, commented out, and through the JSON API. Then there are reputation services: Reputation Authority, ThreatExpert, VxVault, ProjectHoneypot, McAfee Threat Intelligence, StopForumSpam and Cymru MHR, plus ICSI Certificate Notary for SSL fingerprints and TotalHash, which is disabled by default. The WHOIS side is covered by DomainTools Parsed Whois, DomainTools Reverse Whois, DomainTools Reputation, IP WHOIS using the RIR REST interfaces, and Hacked IP. Several of these require an API key: DomainTools, Metadefender Cloud, GreyNoise and IBM XForce, the last of which the documentation says requires an API key outright.

The primary language is Python, the licence is MIT, and the author is Steve McMaster of Hurricane Labs. The recent release tags are 1.4.8 from 2019-05-09, v1.4.7 from 2019-02-07 and 1.4.6 from 2019-02-05, while the last push was on 2026-06-03.

The audience is an analyst who wants breadth quickly, and who is comfortable with a tool whose source of truth is a YAML file listing sites.

The configuration merging system

Machinae supports a simple configuration merging system so you can adjust the configuration without modifying the file that ships with the tool, which makes configuration updates painless.

The precedence order is worth internalising, because it determines where a change actually takes effect. It finds a system-wide default configuration, by default at /etc/machinae.yml, merges into that a system-wide local configuration at /etc/machinae.local.yml, and finally a per-user local configuration at ~/.machinae.yml. The system-wide configuration can also be located in the current working directory, can be set with the MACHINAE_CONFIG environment variable, or of course be given with the -c or --config command line option. Configuration merging can be disabled with the --nomerge option, which causes Machinae to load only the default system-wide configuration or the one passed on the command line.

The practical pattern is that you never edit /etc/machinae.yml. The documentation makes that explicit: if you want to enable the Fortinet Category site, which is disabled by default, you could modify /etc/machinae.yml, but those changes would be overwritten by an update, so instead you put the following in either /etc/machinae.local.yml or ~/.machinae.yml:

    fortinet_classify:
      default: true

And the reverse, to disable a site such as VirusTotal pDNS:

    vt_ip:
      default: false
    vt_domain:
      default: false

So the site name is the top level key and default is a boolean saying whether the site runs when it applies to the observable's type.

That is a small design with real consequences. It means an upgrade that adds a new site does not turn it on in your deployment, because a new site is not in your local file and therefore not enabled, unless the site defaults to on in the shipped configuration. It also means a site you deliberately turned off stays off across upgrades. For a tool that queries third-party services on your behalf, that stability is more than a convenience.

Running it: observables, output formats and site selection

The usage is close to Automater's. The options are given as:

    usage: machinae [-h] [-c CONFIG] [--nomerge] [-d DELAY] [-f FILE] [-i INFILE] [-v]
                [-o {D,J,N,S}] [-O {ipv4,ipv6,fqdn,email,sslfp,hash,url}] [-q]
                [-s SITES] [-a AUTH] [-H HTTP_PROXY]
                [--dump-config | --detect-otype]

The last line of that usage is the important operational point: all arguments other than the listed options are interpreted as targets, which Machinae calls observables, with the type referred to as the otype.

The output format is chosen with -o followed by a single character. N is the default output, called Normal, D is the default output with dot characters replaced, and J is JSON output. The -f or --file option specifies where output goes, with the default being - for stdout.

Type detection is automatic, and the -O or --otype option overrides it, with the available choices being ipv4, ipv6, fqdn, email, sslfp, hash and url. The --detect-otype option forces detection on its own, which is useful for checking what Machinae thinks it is looking at before you trust a query.

By default Machinae runs in verbose mode, printing status information about the services it is querying to the console as it queries them, and that status output always goes to stdout regardless of the output format setting. The -q flag turns verbose mode off. The reason for the split is that the status stream is diagnostic rather than data, so mixing it into a JSON file would corrupt the document.

Site selection has three parts. By default Machinae runs through all services in the configuration that apply to the target's otype and are not marked default: false. You can pass a comma separated list of sites to run, using the top level key from the configuration, and you can pass the special keyword all to run through all services including those marked default: false. The documentation notes that in both cases otype validation is still applied, which is the sensible behaviour: asking for a site that cannot answer your observable type is not something the tool should do just because you named it.

A delay option is supported like Automater's, with one difference, since Machinae uses 0 by default. The -H or --http-proxy argument sets an HTTP proxy, and if none is specified the tool searches the standard HTTP_PROXY and HTTPS_PROXY environment variables as well as the less standard lowercase variants.

Installation, the Docker path and the dependencies

Installation is a pip3 install, and there is a second route for the adventurous:

    pip3 install machinae

    pip3 install git+https://github.com/HurricaneLabs/machinae.git

The system requirements are the ones people usually forget. You need whatever dependencies are required on your system for compiling Python modules, which on Debian based systems is python3-dev, as well as the libyaml development package, libyaml-dev. And you will want to grab the latest configuration file and place it in /etc/machinae.yml.

The dependency list in setup.py explains the shape of the tool. There is dnspython3 and ipwhois for the name resolution and WHOIS side, requests and beautifulsoup4 with html5lib for the HTTP and HTML scraping, feedparser for the feed sources, python-magic for file type identification, which is how a hash gets turned into a file type, defang for the defanging of URLs, tzlocal and relatime for time handling, stopit for timeouts, and pyyaml for the configuration. The console script entry point is machinae = machinae.cmd:main.

The Dockerfile is the shortest possible packaging:

FROM python:3

RUN pip3 install machinae

#make sure you have a machinae.yml file to build with
COPY machinae.yml /etc

ENTRYPOINT ["/usr/local/bin/machinae"]

Copying machinae.yml to /etc is what lines up with the default configuration path, so the container finds it without extra arguments.

HTTP Basic Authentication is supported for sites that need it through the --auth or -a flag, and the credentials go in a YAML file with a key naming the site and a list of two items, the username and the password or API key. The documentation gives the PassiveTotal site as the worked example.

The project carries a CII Best Practices badge, a SonarCloud quality gate badge and a Travis CI build badge, and the repository has a Dockerfile, a pylintrc, a Code of Conduct and a .travis.yml, which describes a project with institutional habits rather than a script collection.

Where Machinae is the wrong choice

Three limits follow from what the tool is.

The first is the age of the release line. The tagged releases are all from 2019, with 1.4.8 from May 2019 being the most recent, while the last push was on 2026-06-03 and the setup.py carries a version of 1.4.11. The project is therefore in the awkward position of a codebase that has moved well past its last tag. The most recent GitHub release, 1.4.8, is from 2019, so the PyPI version tells you nothing about how current the code is. The repository is not archived, so work is continuing, but you would be installing from a branch rather than a release, and that is a different maintenance conversation from consuming a published version.

The second is the dependency on the outside world. The value of the tool is entirely in the quality and availability of third-party services, and several of those are free tiers, several are disabled by default, and several require an API key you have to apply for. A source that changes its response format, or a free tier that starts rate limiting, breaks the corresponding site without any change on your side. The tool also scrapes, which is more fragile than using an API, and the list contains both approaches for the same service, which is a hint that the API path is the more reliable one.

The third is that this is a querying tool, not a data platform. It produces output for the observables you name, on demand. It does not retain history, schedule recurring collection, index results, or give you a query interface over what it found. If your requirement is a searchable archive of indicator sightings over time, you need something with storage and a query layer, and Machinae is the wrong shape for it.

The alternative worth comparing against is a commercial threat intelligence platform with a single API and a credit-based pricing model. The difference in approach is that Machinae gives you breadth across many sources for free and no vendor relationship, while a commercial platform gives you one reliable, contracted, consistently formatted source at a per-query cost. Which is better depends entirely on whether you can absorb the flakiness and the key management.

Editorial conclusion

Machinae is a good fit for an analyst who wants a single command that fans out across reputation, threat intelligence and WHOIS sources for an observable, because the site list is broad and the output formats include JSON for feeding into other tools. It is a poor fit if you need a current, supported release, since the last tagged version is from 2019 while the repository has been pushed to as recently as 2026-06-03, and a poor fit if your sources of interest require API keys you cannot obtain. Before deploying, check the credentials each site needs, since several of them including DomainTools, Metadefender Cloud, GreyNoise and IBM XForce require an API key, and read the configuration merging rules, because a local override file that is out of step with the shipped one will silently change which sites run.

Frequently asked questions

Official sources

1. HurricaneLabs/machinae on GitHub
2. Issues
3. License: MIT
4. README
5. Releases