# LemonGraph, a log-based graph engine built for seed set expansion > LemonGraph is a transactional graph database engine from the National Security Agency, backed by a single file, with a C core, friendly Python bindings, a query language for long node-edge-node chains and a REST service. It is built on LMDB and inherits LMDB's caveats, including a single writer, and the interesting design decision is the log position bookmark that lets a client consume a growing graph incrementally. **NationalSecurityAgency/lemongraph** — Log-based transactional graph engine - Repository: https://github.com/NationalSecurityAgency/lemongraph - Stars: 1,236 · Forks: 168 - Language: Python - License: NOASSERTION - Published: 2026-10-19 · Updated: 2026-10-19 · Language: en - Canonical page: https://hysenlabs.com/projects/nationalsecurityagency-lemongraph ## A client that remembers where it stopped reading LemonGraph is a log-based transactional graph engine for nodes, edges and properties, backed by a single file, and the README states its primary use case first: supporting streaming seed set expansion. The workflow is written out as an eight-step loop, and it is the clearest explanation of the design. Create a new graph. In client code, initialise a log position bookmark, with pos equal to zero. Inject seed data into the graph. Starting at that log position, query the graph for the desired patterns. Process the returned results. Update the bookmark to the next identifier, either from the transaction or through the HTTP header X-lg-maxID plus one. Inject new data into the graph. And go back to step four. The important idea is that the graph is a log and the client is a cursor over it. A query returns matches, but a match can also contain new facts, and those facts are what the next round of expansion is built from. The bookmark is what makes the loop incremental: the client does not rescan, it continues. The two ways of reading the next position, from the transaction object and from an HTTP header, tell you that the same model is exposed both in-process and over the network, and that the REST service is not a thin wrapper but a first-class client of the same log semantics. The core of the graph library is written in C, and the Python layer on top adds the bindings, the query language and the REST service. The engine rides on Symas LMDB, the transactional key/value store that the OpenLDAP project developed to replace BerkeleyDB. ## LMDB underneath, and the caveats that come with it The features section is written as two lists: what LMDB provides, and what the graph layer adds on top. Reading the first list explains the second. LMDB provides transactions, multi-process MVCC with a single writer and multiple non-blocking readers, nested write transactions, and rapid non-blocking binary snapshots. That is a specific and unusually good set of properties for a graph engine that has to be read by several processes while one of them writes. The readme also lists the caveats it inherits, and inherits is the right word. The maximum database mapsize has to be maintained manually, though on 64-bit platforms it is cheap to overestimate. Transactions should only be used from the thread that created them. And the database file should be opened only once within a process. Those are not complaints about LMDB, they are the price of its design, and the graph layer makes two adjustments to them. The mapsize caveat is partly answered by the automatic mapsize growth the graph layer adds, as long as a single transaction's updates stay under one gigabyte. The thread rule and the open-once rule are not answered, because they are properties of the storage layer, and they propagate straight into the Python API where the transaction object is used inside a with block. The single-writer rule is the one to plan around. It is what makes the readers non-blocking and the snapshots cheap, and it is also what stops two processes from writing at once, so a deployment that wants concurrent ingestion from several processes has to queue the writes. ## The graph model and what deletion takes with it The graph layer's own feature list is short and precise about identity, which is the part that decides whether a query returns what you meant. Nodes are uniqued within the graph by type and value. Directed edges are uniqued by source node, target node, type and value. Property key/value pairs can be attached to the graph, to nodes, to edges, or to properties, and they are uniqued by key and parent object. Nodes, edges and properties can all be deleted. The uniqueness rules are what make the log semantics work. If a node is identified by its type and value, then injecting the same fact twice does not create a second node, it advances the log position past an existing entry, and the bookmark still moves. That is the property that lets a client re-inject overlapping data without corrupting the graph, and it is why the example code can delete a node and cascade, without worrying that a stale reference will dangle. Two features are worth singling out. Deletion cascades: the example deletes a node and the comment says it cascades to edges and properties. And historical views let you examine the graph as of a log position X, which is the read-side counterpart of the bookmark on the write side. Together they mean a client can ask what the graph looked like at the moment it last processed something, which is a property that a graph database with mutable state cannot offer. The catch of the historical view is the same catch as the bookmark: it is positional, not temporal. Position X is an identifier in the log, so it survives only as long as the log does, and the readme's own tooling makes that explicit by shipping a snapshot command that copies a database to a file. ## The Python layer, and the query language underneath it The Python wrapper is designed to look like Python, and the example is the best evidence. Transactions are managed with with blocks, and a write transaction and a read transaction are both entered the same way, with the write flag distinguishing them. Properties are manipulated with standard dictionary-style operations, so setting a property is an assignment and removing one is a del on the key. The example also shows the deletion cascade in one line, with a call to delete on a node. Custom serializer objects may be used to support complex data structures, with json and msgpack given as the examples, which is how a property holds something other than a string. And ad-hoc or streaming queries are performed with the custom graph query language. The query language is the part designed for the actual use case. It is described as designed to query for arbitrarily long node-edge-node chains, and the example calls show the two forms. An ad-hoc query uses the query function with a pattern string, and it accepts a stop position, which is how the example reads the graph as it was before the deletions by passing the last identifier it recorded. The streaming form uses mquery with a list of patterns and a start position, and it yields the pattern alongside each chain, so a caller can process several patterns in one pass. The example also shows the log itself. A call to dump on the transaction prints the internal graph log, which is the same idea the tools section uses: the interactive database tool queries a potentially active graph, and the snapshot tool copies one. ## The benchmarks, and what they actually measure The benchmark section is worth reading carefully, because the numbers are really about the interpreter rather than the engine. The setup is stated precisely: PyPy 6.0 and stock CentOS 7 Python 2.7.5, under a single transaction, on a 3.4 gigahertz Core i7-4770 with plenty of RAM. Three tests: T1 inserts one million nodes, T2 adds a property on each node, and T3 adds one million random edges. The disk usage is given as 108 megabytes for T1, 157 for T2 and 292 for T3, which is a useful data point on its own, since the whole graph is one file. The timings run the same three tests across six interpreters. Python 2.6.9 takes 23.915, 8.622 and 16.787 seconds. Python 2.7.5 takes 17.238, 5.443 and 18.164. Python 3.5.5 takes 12.159, 5.322 and 15.434, and Python 3.7.0 takes 11.740, 5.119 and 15.235. PyPy2 6.0 takes 3.730, 2.145 and 6.863, and PyPy3.5 6.0 takes 5.117, 2.294 and 8.815. The same C engine, the same machine, the same transaction, and the spread is dominated by the interpreter. PyPy2 is about 4.6 times faster than Python 2.7.5 on the insert test and about 2.5 times faster on the property test, and the newer CPython versions are faster than the older ones mainly because the property access got faster. So the benchmark is really two results. The graph engine is not the bottleneck at this scale, and the choice of Python implementation is a larger factor than anything in the engine. For a project deciding how to deploy this, that is a more useful conclusion than the raw numbers. ## The REST service, the tools, and the build that fetches its own dependencies The service and tool layer is small and conventional. There are two command line tools. The database tool queries a potentially active graph interactively, and the snapshot tool copies one, which is how you get a consistent copy of a database that is being written to. The REST service maintains an indexed cache of basic graph information and, every N milliseconds, using the -d option, checks for and flushes updated graphs to disk. The -s option trades durability for throughput, with the readme being explicit that this is at the expense of possible data loss in the event of an operating system, hardware or power related failure. The default process model is one master process responsible for respawning and killing child threads, one sync process responsible for syncing graphs to disk, and N worker processes, set with -w, to handle HTTP requests. The build is where the age of the project shows most. The setup script does not just build the extension, it fetches the LMDB source, cloning the repository into the deps directory if it is not already a git checkout, and it downloads three JavaScript files with curl, two versions of d3 and a crowbar script, for the browser front end. The Makefile compiles the C sources into a static or shared library, with the LMDB include path added to the compiler flags, and its snapshot target clones the repository into a dated directory and zips it. The dependency list is equally dated, and it is the reason to be careful about adopting this today. The requirements file pins cffi below 1.12, lazy below 1.4, msgpack-python below 0.6, pysigset below 0.4 and python-dateutil below 2.7, plus six. The install instructions are written for CentOS and Ubuntu of the mid-2010s, with a compiler toolchain and the C bindings installed first: ```bash yum install -y gcc gcc-c++ make libffi-devel zlib-devel python-devel python-setuptools easy_install 'cffi>=1.0' python setup.py install ``` On Ubuntu the equivalent line installs the development headers, and on a compiled CPython or on PyPy the only extra step is bootstrapping setuptools with ensurepip, since cffi is bundled with PyPy. To run without a proper installation, the dependencies are installed by hand with easy_install. ## Where LemonGraph is the wrong tool The first limit is the writer. LMDB gives a single writer with non-blocking readers, and that is a deliberate trade for the read performance. A workload with several processes ingesting simultaneously will serialise at the write, and a design that assumed a follower could write its own findings will need a queue in front of the graph. The second is the Python version. The readme says the REST service cannot run on CentOS 6's Python 2.6, because it relies on the new memoryview behaviour, and that Python 3 should work now with the interpreter and package names replaced with their Python 3 variants. The example code in the readme is Python 2, with print statements, so the code as printed will not run unmodified on a modern interpreter. The third is the dependency pins, which come from the Python 2 era. cffi below 1.12 and msgpack-python below 0.6 are the two that will be awkward, because both have moved on considerably, and a project that wants current versions has to test rather than assume. The fourth is the durability option. The -s flag on the REST service is explicitly documented as trading data loss on a power or hardware failure for throughput, and a graph engine used for intelligence analysis is exactly the kind of system where an unacknowledged loss is a serious problem rather than a footnote. The fifth is the thread rule, which the readme inherits from LMDB and does not paper over: a transaction should only be used from the thread that created it. That is a sharp constraint for a Python program that uses threads, and it means the transaction object cannot be handed between workers. The last is the maintenance state. The repository has no tagged releases, the last push was on 2026-04-30, and the licence is one that GitHub cannot classify, so anyone building on it is depending on a snapshot of a codebase whose upstream dependency, LMDB, is separately maintained while this layer is not. ## Conclusion LemonGraph fits a team doing graph expansion where the graph grows as a consequence of the queries, since the log position bookmark is exactly the primitive that workflow needs, and the historical views let a client re-read the graph as it was at a given position. It does not fit a project that needs concurrent writers, because LMDB allows one, and it does not fit a modern Python stack without work, since the dependency pins are from the Python 2 era. Before building on it, read the LMDB caveat list in the README, particularly the rule that a transaction must only be used from the thread that created it, and note that the REST service can lose data if you enable the faster mode. The repository publishes no releases, the licence is a custom one, and the last push was on 2026-04-30. ## FAQ ### What is LemonGraph? A log-based transactional graph database engine for nodes, edges and properties, backed by a single file. The core is written in C, and the Python layer adds friendly bindings, a query language for long node-edge-node chains and a REST service, all on top of Symas LMDB, the transactional key/value store developed by the OpenLDAP project. ### How does the streaming seed set expansion work in LemonGraph? The client keeps a log position bookmark, starting at zero, queries the graph from that position, processes the results, then updates the bookmark to the next identifier, from the transaction or from the X-lg-maxID header plus one, injects the new data and repeats. The graph is therefore consumed as a log rather than scanned as mutable state. ### Which LMDB caveats does LemonGraph inherit? The maximum database mapsize has to be maintained manually, though it is cheap to overestimate on 64-bit platforms, transactions should only be used from the thread that created them, and the database file should be opened only once within a process. The graph layer adds automatic mapsize growth, as long as a single transaction's updates stay under one gigabyte. ### How do I install LemonGraph and its dependencies? With `python setup.py install`, which also fetches the LMDB source into the deps directory. The Python dependencies are cffi, lazy, msgpack, pysigset, python-dateutil and six, and the requirements file pins cffi below 1.12 and msgpack-python below 0.6. The REST service additionally requires a Python with the memoryview behaviour, so it does not run on CentOS 6's Python 2.6. ### What does the LemonGraph REST service do, and what do its options mean? It maintains an indexed cache of basic graph information and, every N milliseconds set with -d, checks for and flushes updated graphs to disk. The -s option gives greater throughput at the cost of possible data loss on a power or hardware failure, and -w sets the number of worker processes, alongside a default master process and a sync process. ## Sources - [Issues](https://github.com/NationalSecurityAgency/lemongraph/issues) - [NationalSecurityAgency/lemongraph on GitHub](https://github.com/NationalSecurityAgency/lemongraph) - [README](https://github.com/NationalSecurityAgency/lemongraph/blob/master/README.md) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/nationalsecurityagency-lemongraph