# DungBeetle: a job server for the reports that would drown your database

> DungBeetle is a single binary distributed job server for queuing heavy SQL read queries asynchronously. Queries live in .sql files, results land in their own table in an ephemeral cache database, and the same job ID can be used as a throttle so a user cannot queue the same report twice.

**zerodha/dungbeetle** — A distributed job server built specifically for queuing and executing heavy SQL read jobs asynchronously. Separate out reporting layer from apps. MySQL, Postgres, ClickHouse.

- Repository: https://github.com/zerodha/dungbeetle
- Stars: 1,282 · Forks: 176
- Language: Go
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/zerodha-dungbeetle

## A reporting layer separated from the application

DungBeetle is a lightweight, single binary distributed job server designed for queuing and asynchronously executing large numbers of SQL read jobs, for example reports, against SQL databases. The repository description puts the motivation in one line: separate out the reporting layer from the apps, with MySQL, Postgres and ClickHouse as the source databases.

The design idea is that a reporting query has a different cost profile from an application query, and a different failure profile too. An application query is small, interactive and has to answer in milliseconds. A report is large, minutes long, and if a thousand users trigger one at the same time the database does not degrade gracefully, it collapses. Moving the report out of the request path is the fix, and the job server is the mechanism.

The feature list gives the shape of the thing. It supports MySQL, PostgreSQL and ClickHouse as source databases, and MySQL and PostgreSQL as the result or cache databases for job results. It is a standalone server that exposes HTTP APIs for managing jobs and groups of jobs, with list, post, status check and cancel. It reads SQL queries from `.sql` files and registers them as jobs ready to be queued. And it supports multi-process, multi-threaded, asynchronous distributed job queueing through a common broker backend, with Redis and AMQP given as the examples.

That last point is the design decision that makes it a distributed server rather than a library. The broker is the seam, and the fact that it is a common one means the queueing is not special to this project.

The project is MIT licensed, written in Go, on the `master` branch, with the last push on 2026-09-29. The releases are v2.0.1 from 2024-04-10, v2.0.0 from 2024-03-14 and v1.0.0 from 2023-06-05, and the module path carries the major version, `github.com/zerodha/dungbeetle/v2`.

## The problem: reports are a denial of service in slow motion

The use case section is written in terms of an application with a very large SQL database, and the failure it describes is worth reading closely.

When there are several thousand concurrent users requesting reports from an application connected to it, every second of I/O delay during query execution can bottleneck the application and the database, causing a snowball effect. The I/O delay is the interesting part. A report does not have to be wrong to hurt; it only has to occupy the disk queue long enough that the interactive queries behind it start waiting, and then the interactive queries take longer, and more reports pile up.

The fix is to defer the report generation to a job queue in the backend, which immediately frees up the frontend application. The reports are presented to users as they are executed, with the frontend polling the job's status and preventing the user from sending any more queries. And the server and worker instances act as traffic control, preventing the primary database from being inundated with requests.

The second half of the design is about what happens after the report is done. It is natural for users to further transform the results by slicing, sorting and filtering, which generates additional queries against the primary database. To offset that load, those subsequent queries can be sent to the smaller and much faster results cache database. And these results are of course ephemeral, and can be thrown away or expired.

So the shape is a two-database arrangement where one is authoritative and enormous and the other is small, fast and disposable, and the job server is what moves data between them. That is why the results backend and the source database are configured separately, and why the result tables are named after jobs.

## Tasks are SQL files with tags that decide where the query runs

The unit of definition is a task, and it is a named SQL query in a `.sql` file, in the goyesql format, loaded into the server on startup.

```sql
-- queries.sql

-- name: get_profit_summary
SELECT SUM(amount) AS total, entry_date FROM entries WHERE user_id = ? GROUP BY entry_date;

-- name: get_profit_entries
-- db: db0, other_db
-- queue: myqueue
-- results: my_res_db
SELECT * FROM entries WHERE user_id = ?;
```

The queries are self-contained and produce the desired final output with neatly named columns, and they can take arbitrary positional arguments for execution. The tags are the routing.

The `db` tag attaches a task to one or more specific databases defined in the configuration, and in case of multiple databases the query is executed against a random one from the list, unless a specific database is specified in the API request. The `queue` tag always routes the task to a particular queue, unless it is overridden by the `queue` parameter when making a job request. The `results` tag specifies the results backend to which the results of a task will be written, and if there are multiple result backends the results are written to a random one.

There is a fourth tag, `raw`, which the example shows as `-- raw: 1` with the comment that the query will not be prepared. So a task is either a parameterised statement that the server validates and prepares on startup, or a raw one that is passed through.

The placeholders follow the source database. The `?` are MySQL value placeholders, and for Postgres they are `$1, $2` and so on, which is the kind of detail that decides whether a task file is portable across the two databases or whether it is quietly MySQL only.

## Job IDs that are not unique, on purpose

A job is an instance of a task that has been queued to run, and each job has an ID that can be used to track its status. If an ID is not passed explicitly it is generated internally and returned.

The interesting part is what comes next. These IDs need not be unique, but only a single job with a certain ID can run at any given point, and for the next job with the same ID to be scheduled, the previous job has to finish execution.

That is a throttle, and the documentation is explicit about why it exists: using non-unique IDs like this is useful in cases where users can be prevented from sending multiple requests for the same reports, like in the usecases. So a client that passes a stable identifier, a user id and a report name perhaps, gets the guarantee that the same report for the same user runs once at a time, and a second request for the same identifier waits rather than piling a second heavy query onto the database.

It is a small design decision with a large effect on the shape of the client, because the client has to think in terms of stable identifiers rather than request identifiers. And the cost is that a second request for the same identifier is delayed rather than rejected, which is the right behaviour for a report and the wrong behaviour for a job that is supposed to run twice.

The client polls with the job ID to check if results are ready for consumption, and the HTTP surface has both a single job status route and a queue listing route, so a client can either wait for one job or watch a queue.

## Results: a table per job, in a database you can throw away

The results design is the part that makes the whole thing work, and it is simple.

The results from an SQL query job are written to a result backend, MySQL or Postgres, from where they can be further read or queried. The results from a job are written to a new table named after that job, and the schema of the results table is automatically generated from the results of the original SQL query. All fields are transformed into one of these types: `BIGINT, DECIMAL, TIMESTAMP, DATE, BOOLEAN, TEXT`.

So every job materialises its output as a real table, with a schema derived from the query's columns. There is no generic result format to decode and no JSON blob to parse on the client, and the type list is the contract between what the source database returns and what the result database can hold. A report with a float column becomes a `DECIMAL`, which is a deliberate choice: a floating point result that is copied into a result table has to survive the copy, and rounding to a decimal type is a decision about what the numbers mean.

The table is named after the job, which means the results of a given job are addressable as a table, which is what makes the follow-up queries, the slicing and sorting and filtering, expressible as ordinary SQL against a small database rather than as transformations in the client.

The model is a cache, and it is designed as one. The results are of course ephemeral and can be thrown away or expired, and a system built this way has to decide separately what happens when a user asks for a report that has expired.

## The control interface, the broker and the build

The HTTP surface is small, and it is the whole of the control plane.

| Method | URI | |
|---|---|---|
| GET | `/tasks` | Returns the list of registered SQL tasks |
| POST | `/tasks/{taskName}/jobs` | Schedules a job for a given task |
| GET | `/jobs/{jobID}` | Returns the status of a given job |
| GET | `/jobs/queue/{queue}` | Returns the list of all pending jobs in a queue |
| POST | `/groups` | Schedule a group of jobs |
| GET | `/groups/{groupID}` | Get the status of a job group and its jobs |
| DELETE | `/jobs/{jobID}` | Deletes a pending job from the queue |

Groups are the addition to the per-job model, and they exist because a report is often several queries, and the group is what a client polls when it wants to know whether the whole report is ready.

The server starts a set of workers listening on a default job queue and an HTTP service on `http://127.0.0.1:6060`, which is the control interface, and it can be run without the HTTP interface by passing the `--worker-only` flag, which is the arrangement for a worker that should not be reachable at all.

Underneath, the dependency list shows the seams. The queueing library is `tasqueue`, and the documented broker backends are Redis and AMQP, the SQL layer is the standard drivers for MySQL and Postgres plus the ClickHouse client, the router is chi, the configuration is koanf with TOML, environment and flag providers, and the SQL file format is parsed by goyesql, the same library the task format comes from.

The build is a Go build with the version and build string stamped from git, CGO disabled, and goreleaser for the releases, and the `Dockerfile` is a two stage build with a Go builder and an Alpine runtime that carries the binary, the `sql/` directory and a sample configuration.

## Where DungBeetle is the wrong tool

The limits follow from the design, and there is one that is worth checking before anything else.

The first is the version skew in the container. The module declares Go 1.25.0, and the `Dockerfile` builds with a `golang:1.21-alpine` base image. The image therefore ships an older toolchain than the module asks for. Recent Go versions fetch a newer toolchain automatically, so the build may still succeed, but the mismatch is worth checking before relying on the image.

The second is that the tool is for reads. The whole design assumes that the expensive queries are read queries, that the results are disposable, and that the source database must be protected from a flood of them. A write workload does not fit, because a write cannot be thrown away and cannot be cached, and because a job whose result is a mutation does not belong in a result table in a different database.

The third is the result cache. The design treats the result database as a cache that can expire, which is right for reports and wrong for anything that becomes a record. A team that finds itself joining against those result tables is using a cache as a database, and the failure mode arrives later, when the cache is cleared.

The alternative approach, for a team that does not want to run a job server at all, is a general purpose queue with a worker that a developer writes, and a fixed schema in the application database instead of a table per job. The difference in approach is that DungBeetle trades flexibility for a small amount of operational surface, a single binary, a declarative task format and automatic result schemas, and the alternative trades that for a job definition in Go and a migration for every new report. A team with a handful of reports and an existing queue should keep the queue; a team with dozens of reports across several databases, and analysts rather than developers, should look at this.

## Conclusion

DungBeetle fits an application whose reporting queries are heavy enough to slow the primary database, whose users can wait for a report rather than block on it, and whose team is prepared to maintain SQL files and a result cache. It is the wrong choice for write workloads, since the tool is built for read jobs and its results tables are ephemeral, and it is the wrong choice for a team that needs a single database of record, since the results are designed to be thrown away. Before deploying it, check the version skew between the `go.mod` requirement and the `golang:1.21-alpine` base image in the `Dockerfile`, decide whether the results cache is a cache or a store in your design, and read the broker configuration, because the queue implementation is the part you will have to operate.

## FAQ

### What is DungBeetle?

It is a lightweight, single binary distributed job server for queuing and asynchronously executing large numbers of SQL read jobs against MySQL, PostgreSQL or ClickHouse, with the results written to separate ephemeral result databases so that follow-up queries hit a smaller and faster cache instead of the primary database.

### How do I define a job in DungBeetle?

Tasks are named SQL queries in .sql files in the goyesql format, loaded when the server starts, with tags that decide the routing. The db tag attaches the task to one or more databases, the queue tag routes it to a queue, the results tag picks the results backend, and raw: 1 marks a query that is not prepared.

### Where do the results of a DungBeetle job go?

To a MySQL or PostgreSQL result backend, in a new table named after the job, with the schema generated automatically from the columns of the original query and the fields mapped to BIGINT, DECIMAL, TIMESTAMP, DATE, BOOLEAN or TEXT. The results are ephemeral by design.

### How do I start the DungBeetle server?

With `dungbeetle --config /path/to/config.toml --sql-directory /path/to/your/sql/queries`. It starts workers listening on a default job queue and an HTTP control interface on http://127.0.0.1:6060, and the --worker-only flag runs the workers without that interface.

## Sources

- [Issues](https://github.com/zerodha/dungbeetle/issues)
- [License: MIT](https://github.com/zerodha/dungbeetle/blob/master/LICENSE)
- [README](https://github.com/zerodha/dungbeetle/blob/master/README.md)
- [Releases](https://github.com/zerodha/dungbeetle/releases)
- [zerodha/dungbeetle on GitHub](https://github.com/zerodha/dungbeetle)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/zerodha-dungbeetle