pydruid: a Python connector that returns Druid results as DataFrames

A Python connector for Apache Druid with a pandas-shaped result

pydruid exposes a simple API to create, execute and analyze Druid queries, and it can parse query results into Pandas DataFrame objects for subsequent data analysis. The README's framing of the value is worth reading carefully, because it says the tight integration is between Druid, the SciPy stack for scientific computing and scikit-learn for machine learning. In practice that means the analytical loop stays in one language. A query goes out, a DataFrame comes back, and you can plot it, feed it to a model or join it against other frames without a translation step. The library also exports query results into TSV or JSON for further processing with tools such as R, Julia, Matlab or Excel, which is the escape hatch when the analysis leaves Python. The repository is the druid-io/pydruid project with a custom licence classification, the last push was on 2026-05-04, and there are no GitHub releases.

Installation is a pip install with optional extras

The install section is one block, and the extras are the interesting part because they map onto the features:

pip install pydruid
# or, if you intend to use asynchronous client
pip install pydruid[async]
# or, if you intend to export query results into pandas
pip install pydruid[pandas]
# or, if you intend to do both
pip install pydruid[async, pandas]
# or, if you want to use the SQLAlchemy engine
pip install pydruid[sqlalchemy]
# or, if you want to use the CLI
pip install pydruid[cli]

The setup.py behind it confirms the mapping. The base install requires only `requests`, and the extras are `pandas`, `async` which pulls in `tornado`, `sqlalchemy`, and `cli` which pulls in `pygments`, `prompt_toolkit` at 2.0.0 or newer and `tabulate`. So the default install is deliberately small, and you only pay for what you use. The console script is `pydruid = pydruid.console:main`, and the SQLAlchemy dialects register three names, `druid`, `druid.http` and `druid.https`, all pointing at `pydruid.db.sqlalchemy:DruidHTTPDialect` or its HTTPS variant.

extras_require = {
    "pandas": ["pandas"],
    "async": ["tornado"],
    "sqlalchemy": ["sqlalchemy"],
    "cli": ["pygments", "prompt_toolkit>=2.0.0", "tabulate"],
}

Timeseries queries build aggregations and post-aggregations in Python

The examples section works through three query types against Twitter's public data set, and the timeseries one is the clearest. The question is what the average tweet length was, per day, surrounding the 2014 Sochi olympics:

from pydruid.client import *
from pylab import plt

query = PyDruid(druid_url_goes_here, 'druid/v2')

ts = query.timeseries(
    datasource='twitterstream',
    granularity='day',
    intervals='2014-02-02/p4w',
    aggregations={'length': doublesum('tweet_length'), 'count': doublesum('count')},
    post_aggregations={'avg_tweet_length': (Field('length') / Field('count'))},
    filter=Dimension('first_hashtag') == 'sochi2014'
)
df = query.export_pandas()
df['timestamp'] = df['timestamp'].map(lambda x: x.split('T')[0])
df.plot(x='timestamp', y='avg_tweet_length', ylim=(80, 140), rot=20,
        title='Sochi 2014')
plt.ylabel('avg tweet length (chars)')
plt.show()

The API reads as a builder. `PyDruid(druid_url_goes_here, 'druid/v2')` takes the broker URL and the endpoint prefix, then `query.timeseries(...)` takes the datasource, the granularity, the intervals, the aggregations dictionary, the post-aggregations dictionary and a filter. The post-aggregation is a `Field('length') / Field('count')` division, which is how a computed ratio is expressed. The filter is a `Dimension` comparison, and the results come back through `query.export_pandas()` rather than being parsed by hand. The trailing plot call is the payoff the README is selling, and the code truncates the timestamp with a `split('T')` before plotting.

Filters compose with comparison operators and boolean logic

The filter argument is the piece that appears in every example, and the composition rules are worth understanding because they are not Python's own. A `Dimension('first_hashtag') == 'sochi2014'` produces a filter object rather than a boolean, and those objects combine with `&` and `|` and `~` to build the filter tree Druid expects. The topN example stacks four of them, including a negation:

top = query.topn(
    datasource='twitterstream',
    granularity='all',
    intervals='2014-03-03/p1d',  # utc time of 2014 oscars
    aggregations={'count': doublesum('count')},
    dimension='user_mention_name',
    filter=(Dimension('user_lang') == 'en') & (Dimension('first_hashtag') == 'oscars') &
           (Dimension('user_time_zone') == 'Pacific Time (US & Canada)') &
           ~(Dimension('user_mention_name') == 'No Mention'),
    metric='count',
    threshold=10
)

df = query.export_pandas()
print df

The `&` and `~` here are overloaded on the filter objects, so the expression reads like boolean algebra but produces a Druid filter. The `metric='count'` and `threshold=10` parameters choose which aggregation ranks the results and how many come back. The README notes that a bare `print df` is Python 2 syntax in that example, which is a small sign of the age of the code in the documentation.

Groupby returns dimensions you can feed to a graph library

The groupby example asks what the social network of users replying to other users looks like, and it is the most interesting one because it shows what a DataFrame is for:

group = query.groupby(
    datasource='twitterstream',
    granularity='hour',
    intervals='2013-10-04/pt12h',
    dimensions=["user_name", "reply_to_name"],
    filter=(~(Dimension("reply_to_name") == "Not A Reply")) &
           (Dimension("user_location") == "California"),
    aggregations={"count": doublesum("count")}
)

df = query.export_pandas()

The query groups by two dimensions, `user_name` and `reply_to_name`, filtered to California and excluding the placeholder for a non-reply. The code that follows turns the two name columns into integer vertex indices with a lookup table, builds an undirected igraph graph from the edges, computes a Fruchterman-Reingold layout and plots it. That is the integration claim in concrete form. The data came out of Druid as a DataFrame and became a graph without leaving Python, with igraph and cairo doing the drawing. The same pattern would work with networkx or with a machine learning model trained on the resulting frame.

An asynchronous client built on Tornado

The asynchronous client is `pydruid.async_client.AsyncPyDruid`, and the README explains the design. It utilizes an asynchronous HTTP client from the Tornado framework, and it is suitable for use with async frameworks such as Tornado, providing much better performance at scale. The point it makes is serving multiple requests at the same time without blocking on Druid executing your queries. The example uses the `@gen.coroutine` decorator:

from tornado import gen
from pydruid.async_client import AsyncPyDruid
from pydruid.utils.aggregators import longsum
from pydruid.utils.filters import Dimension

client = AsyncPyDruid(url_to_druid_broker, 'druid/v2')

@gen.coroutine
def your_asynchronous_method_serving_top10_mentions_for_day(day
    top_mentions = yield client.topn(
        datasource='twitterstream',
        granularity='all',
        intervals="%s/p1d" % (day, ),
        aggregations={'count': doublesum('count')},
        dimension='user_mention_name',
        filter=(Dimension('user_lang') == 'en') & (Dimension('first_hashtag') == 'oscars') &
               (Dimension('user_time_zone') == 'Pacific Time (US & Canada)') &
               ~(Dimension('user_mention_name') == 'No Mention'),
        metric='count',
        threshold=10)

The `raise gen.Return(top_mentions)` at the end, with the comment that it can be a plain `return` in Python 3, shows the code was written for a Python 2 era and has been kept compatible. The `install_requires` of the async extra is `tornado`, and the pinned version in the requirements file is 6.0.4, which is the version where the coroutine style is still fully supported.

Theta sketch post-aggregators need a Druid extension

One corner of the API is specialised and the README flags it. Theta sketch post-aggregators are built slightly differently to normal post-aggregators, as they have different operators, and you must have the `druid-datasketches` extension loaded into your Druid cluster in order to use them. The README points to the Druid datasketches documentation for details. The example imports the three utility modules separately, `aggregators`, `filters` and `postaggregator`, rather than the star import used in the earlier examples, which is a hint that the sketch post-aggregators live in a different namespace from the ordinary ones. This is the kind of feature that makes the library worth having, since theta sketches are how Druid does distinct counting at scale, and it is also the kind that will fail with a confusing error if the extension is missing on the cluster.

Beyond the query builder, pydruid implements the Python DB API 2.0, a SQLAlchemy dialect and a command line interface. The SQLAlchemy route is the one to know if your team already speaks SQL. The dialect registers under three names, and the HTTPS variant exists because a Druid broker behind TLS is the normal case in production, so the engine URL carries the scheme. The CLI comes with the `cli` extra, which brings in `pygments` for highlighting, `prompt_toolkit` for the interactive prompt and `tabulate` for the output formatting, and it is invoked as the `pydruid` console script. The DB API implementation is what makes the library usable from a generic data access tool rather than only from Python code you write by hand.

Where pydruid is the wrong tool, and how the native HTTP API differs

Three limits are concrete. The first is the version of Druid it talks to. The client targets the `druid/v2` endpoint, which is the older HTTP query API, and the repository has had no GitHub releases with the last push in May 2026, so this is a connector that is not obviously tracking the current Druid surface. If your cluster is on a newer query layer, the endpoints pydruid posts to are not the ones you would use directly with `requests`. The second is the dependency pinning. The requirements file is autogenerated by pip-compile and pins `requests==2.23.0`, `pandas==1.0.4`, `sqlalchemy==1.3.17` and `numpy==1.18.5`, all of which are several years old, and the setup.py classifiers stop at Python 3.8. Installing pydruid into a modern environment means either resolving those old pins or accepting newer versions the code was not written against. The third is that this is a query client, not a data platform. It sends a query, waits for the broker and parses the result. It does not manage a cluster, it does not ingest data, and it has no notion of a result cache beyond what Druid itself does.

The realistic alternative is to post JSON to the Druid HTTP API yourself with `requests` and parse the response, or to use a tool that speaks SQL at Druid. The difference in approach is where the work goes. With pydruid, the query construction is typed, the aggregations and filters are objects, and the result parsing into a DataFrame is handled, so the code you write is about the analysis rather than about JSON shapes. Hand-rolling the HTTP call gives you full control over the request and lets you use Druid features pydruid has not wrapped, and it costs you the parsing layer and the post-aggregation expression building, which is the fiddly part. A SQL-over-HTTP engine is the third option, and it changes the shape of the work rather than the amount of it, since you write SQL instead of Python and lose the direct access to Druid-specific aggregators. The honest summary is that pydruid is worth reaching for when your work is analysis in Python, and worth replacing with a direct API call when your work is exploring what Druid itself can do.

Editorial conclusion

pydruid fits an analyst who wants Druid results as a DataFrame inside a SciPy or scikit-learn workflow, and who prefers typed filter and aggregation objects to hand-built JSON. It is the wrong choice if you need features the client has not wrapped, or if you are on a Druid deployment whose query API has moved past the `druid/v2` endpoint this client targets. Before you adopt it, check the pinned versions in the requirements file against your environment, since the pins are old enough to conflict with a modern pandas or SQLAlchemy install, and confirm your cluster's query API matches what the client posts to.

Frequently asked questions

Official sources

1. druid-io/pydruid on GitHub
2. Issues
3. README