mWater/minimongo: a client-side MongoDB clone with local storage and a remote sync

A fork of Meteor's minimongo, kept for a decade

minimongo is a client-side in-memory MongoDB clone with multiple storage backends and hybrid remote and local synchronization. It uses code from Meteor.js's minimongo package, reworked to support more geospatial queries and made npm friendly, and it was forked in January 2014.

That origin explains the shape of the library better than anything else in the documentation. Meteor needed a client-side database that could hold documents, run MongoDB selectors against them, cache results locally and reconcile with a server, all inside a browser, and it built exactly that inside its own framework. This fork took that engine out of the framework and made it installable, which is the moment it became useful to everyone else.

The API is the familiar one. The features list describes a MongoDB-like API with familiar CRUD operations and query syntax, and the usage example is the kind of code you would recognise from any MongoDB client.

var minimongo = require("minimongo");

var LocalDb = minimongo.MemoryDb;

// Create local db (in memory database with no backing)
db = new LocalDb();

// Add a collection to the database
db.addCollection("animals");

doc = { species: "dog", name: "Bingo" };

// Always use upsert for both inserts and modifies
db.animals.upsert(doc, function() {
	// Success:

	// Query dog (with no query options beyond a selector)
	db.animals.findOne({ species:"dog" }, {}, function(res) {
		console.log("Dog's name is: " + res.name);
	});
});

// Access collections via collection for Typescript
await db.collection["animals"].upsert(doc)

Two conventions in that snippet are worth naming. `upsert` is used for both inserts and modifications, and the `db.collection["animals"]` form exists for TypeScript users who want the collection accessed as a property of a typed object rather than by dynamic name.

Four storage engines, and an autoselect

The storage layer is where this library does more than a naive cache. It is either IndexedDB backed, WebSQL backed, Local storage backed, or in memory only, and the four are exposed as separate classes.

import { MemoryDb, IndexedDb } from 'minimongo';

const memoryDb = new MemoryDb();
const indexedDb = new IndexedDb({namespace: 'mydb'}, successCallback, errorCallback);

The engines differ in more than persistence. Memory is the default and is synchronous, which is what makes it useful in tests. LocalStorage is persistent browser storage with batch updates for writes. IndexedDB is asynchronous browser storage with bulk operations, and it takes a namespace so several databases can share one origin. WebSQL is the legacy SQL based storage, included for the browsers that had it.

The autoselect helper exists because picking one per browser is a nuisance, and `utils.autoselectLocalDb(options, success, error)` calls the success callback with the selected database. A Cordova sqlite plugin is also supported when it is available, activated with the option `{storage: 'sqlite'}` in `utils.autoselectLocalDb`.

Every engine exposes the same operations, which is the point. The list is `find(selector, options)`, `findOne(selector, options)`, `upsert(docs, bases, success, error)`, `remove(id, success, error)`, `cache(docs, selector, options, success, error)`, and `pendingUpserts()` and `pendingRemoves()`.

Those last two are the interesting ones. They report what has been written locally but not yet pushed, which is the state a sync layer has to be able to ask about, and their presence is a hint that the library was designed for a lazy client that batches its writes.

The query language is MongoDB's, minus a documented list

The selector syntax is MongoDB's, and the coverage is described in a feature list that is worth reading as a spec of what the library does and does not do.

The logical operators `$and`, `$or` and `$nor` are supported, as are the comparison operators `$lt`, `$lte`, `$gt` and `$gte`, the element operators `$exists` and `$type`, and the array operators `$in`, `$nin`, `$all` and `$elemMatch`. Regex support is there, and so are sorting and limiting.

The example combines several of them with a geospatial query:

collection.find({
  name: { $regex: /^A/ },
  age: { $gt: 18 },
  location: {
    $near: {
      $geometry: { type: "Point", coordinates: [-73.9667, 40.78] },
      $maxDistance: 1000
    }
  }
}, {
  sort: { age: -1 },
  limit: 10
}).fetch();

Then the limitations section, which is short and blunt. There is no aggregation pipeline, `$where` clause support is limited, update operator support is partial, and there are no transactions across collections.

Each of those exclusions removes a category of use. An aggregation pipeline is how you compute derived values and roll up counts on the client, and without it that work has to happen somewhere else. Partial update operator support means some MongoDB update operators are not available. And no transactions across collections means a write that must touch two collections atomically has no home here, which is the constraint that pushes a design towards a server side of its own.

HybridDb queries locally, then reconciles

The hybrid database is the feature that gives the library its name and its purpose, and it is constructed from a local and a remote database.

import { HybridDb } from 'minimongo';

const hybridDb = new HybridDb(localDb, remoteDb);
hybridDb.addCollection('items');

The key features listed are that it queries the local database first, automatically syncs with the remote source, handles pending upserts and removes, does batched uploads of changes, and resolves conflicts using base documents.

The synchronisation process is spelled out as six steps. Check local pending changes, query the local database, simultaneously query the remote database, merge the results from local data and remote updates, cache the remote results locally, and return the combined results.

That order is the design. The local read is what makes the interface feel instant, and the remote query runs in parallel rather than after, so a slow network does not block the local result. The merging step is where the conflict handling lives, and the caching step is what keeps the next read from paying the network cost again.

The remote side is a separate class, `RemoteDb`, constructed with a path, a name and an options object that carries an HTTP client and a `useQuickFind` flag. The protocols it supports are regular REST operations, the quickfind protocol for efficient data syncing, and a custom HTTP client, which is what lets you plug the library into an application that already has a request layer.

Conflict resolution is base document versioning

The conflict model is worth understanding, because it is the part of a client database that determines whether two devices can work at the same time.

Resolution uses base document versioning, and the mechanism is visible in the upsert signature. `db.collection["sometable"].upsert(docs, bases, success, error)` takes either a single document or multiple documents as an array for the first and second parameters. `docs` is the document or documents to upsert. If `bases` is present, it is the base version on which the update is based. It can be omitted to use the current cached value as the base, or put as `null` to force an overwrite, a true upsert and not a patch.

collection.upsert(
  { _id: '1', name: 'updated' },
  { _id: '1', name: 'original' }, // base document
  successCallback
);

So the caller states which version of the document it believes it is editing, and the library can detect that the server has moved on. That is optimistic concurrency control with the client state, and it is the same principle the `cache` and seeding operations are built on. Caching takes the documents that a find produced, together with the same selector and options that produced them, and the local database adds, removes and updates its copy accordingly. Seeding does not overwrite an existing row, it only makes sure that a row with that `_id` exists.

There is a second class, `ReplicatingDb`, constructed from a master and a replica database, where all operations are applied to both databases, with atomic transaction support and cache synchronisation. That is the multi device case, and it is a different problem from the local and remote case that HybridDb solves.

Geospatial queries, custom types, and a build stack from another era

Three smaller areas are worth a look, and one of them is a warning.

Geospatial support covers `$near` for proximity searches and `$geoIntersects` for polygon intersections, with Turf.js integration for the spatial operations, and the package dependencies are individual Turf modules, boolean crosses, boolean point in polygon, boolean within, distance, intersect and nearest point on line. Extended JSON support handles dates, binary data as a `Uint8Array`, and custom objects that implement clone, equals and toJSONValue methods. And on performance, the library compiles queries to JavaScript functions, uses bulk operations in IndexedDB, batch updates in LocalStorage, and uses the quickfind protocol for efficient diffs.

Then the build. The `package.json` puts the package at version 7.1.1, with a build script that runs a build-library tool and gulp, and a test script of `gulp test`. The dev dependencies are from the previous decade, webpack 3, browserify 3, karma 0.13, mocha 3, gulp-uglify 0.3, chai 1.9 and testem 0.6. The repository has a `gulpfile.js`, a `webpack.config.js` and a `webpack.config.tests.js`, a `karma.conf.js` and a `testem.json`, plus `jquery-shim.js` and `lodash-shim.js` in the root.

The library still supports IE11 with polyfills and modern browsers including Chrome, Firefox, Safari and Edge, plus mobile browsers on iOS Safari and Chrome for Android, which is consistent with a codebase that has to keep running in old places.

The licence is where the documentation is not consistent, and it deserves a look. The package manifest records `"license": "LGPLv3"`, while the README's licence section says MIT License, based on Meteor's minimongo implementation. Two different statements about the terms of a library you may be embedding in a product is worth resolving before you ship, and it is a question for the upstream project and for whoever owns your compliance obligations.

Editorial conclusion

Adopt minimongo if you need a MongoDB-shaped API in the browser, with local persistence and a server to sync with, because the hybrid database, the base document conflict resolution and the four storage engines are real engineering rather than a thin shim. Do not adopt it for anything that needs aggregation, cross collection transactions or full update operators, since the project's own limitations list rules all three out. Verify first which storage backend fits your browsers, that the remote API speaks the REST or quickfind protocol the RemoteDb expects, and which licence applies, since the package manifest records LGPLv3 while the README's licence section says MIT, which is a question for whoever owns your compliance obligations.

Frequently asked questions

Official sources

1. Issues
2. License: LGPL-3.0
3. mWater/minimongo on GitHub
4. README