Falcon crossfilters hundreds of millions of records in the browser

Brushing and linking over data too big to just load

Falcon is an interactive visual analysis system for big data. The one line summary in the project documentation is blunt: crossfilter millions of records without latencies.

The problem behind that is a familiar one. Cross filtering, where brushing a histogram or a map narrows every other view, is the core interaction of analytical tools. The obstacle is that the usual implementation sends a query to a server for every filter change and waits for the answer, so the latency of the round trip sets how fluid the exploration feels. In a small dataset this is invisible. In a large one, brushing becomes a series of stalls.

Falcon's approach is to keep the interaction on the client side. The paper behind it is titled Falcon: Balancing Interactive Latency and Resolution Sensitivity for Scalable Linked Visualizations, published at CHI 2019 by Dominik Moritz, Bill Howe and Jeffrey Heer, and the project asks you to cite it if you use Falcon in a publication.

@inproceedings{moritz2019falcon,
  doi = {10.1145/3290605},
  year  = {2019},
  publisher = {{ACM} Press},
  author = {Dominik Moritz and Bill Howe and Jeffrey Heer},
  title = {Falcon: Balancing Interactive Latency and Resolution Sensitivity for Scalable Linked Visualizations},

The scale figures are stated as the largest experiments the authors have done, not as a general guarantee: 10M flights in the browser, 33M flights in the browser with DuckDB, and roughly 180M flights or about 1.7 billion stars when connected to OmniSciDB. The topic list of the repository is a single tag, scalable-visualization, which tells you the intended audience.

Two query engines: ArrowDB in the browser and MapDDB on a server

The package is installed from npm under the name falcon-vis.

yarn add falcon-vis

The library works with two query engines, and the choice determines where the work happens. The first is ArrowDB, which reads data from Apache Arrow. The documentation says this engine works completely in the browser and scales up to ten million rows.

The second is MapDDB, which connects to OmniSci Core, the database formerly known as MapD. That route moves the heavy lifting to a database server and is the one behind the largest figures, the 180 million flights and the GAIA star catalogue.

The indexes are created as ndarrays, so the index structure is the same in both cases, and the project points at the examples directory for how to set up an app with your own data.

The dependency list backs this up. The package pulls in apache-arrow for the Arrow path, @duckdb/connector for the OmniSci path, @duckdb/duckdb-wasm for the browser database route, d3 for the rendering, ndarray for the indexes, and two ndarray helper packages for image display and linear interpolation.

A caveat that matters for planning: the documentation says more documentation will follow. The engine choice, the ArrowDB browser limit and the ndarray index structure are the facts you get, and the rest of the integration path is left to the examples.

Zoom re-bins instead of redrawing the same pixels

Falcon's visual features are small and specific rather than a catalogue of chart types.

Zoom is the first one. You can zoom histograms, and Falcon automatically re-bins the data. That is the detail that carries the feature. A naive zoom would magnify the existing bins, which shows you fewer bars with wider meaning, or redraw every bar with an enormous number of thin bins and lose the aggregation that made the histogram readable in the first place. Re-binning keeps the bin count sensible while the range narrows, which is the same trade-off a static histogram makes between resolution and legibility, applied live.

The second feature is showing and hiding unfiltered data. The original counts without filters can be displayed behind the filtered counts to provide context, and hiding the unfiltered data shows the relative distribution of the data.

That second one is a small piece of interface design with a large effect on interpretation. When a filter is applied, the filtered bars are what you read, and the ghosted background shows what the distribution looked like before. Remove the background and the same bar means something different, because you lose the denominator. Tools that always show only filtered counts quietly make a filtered subset look like the whole dataset.

Neither of these features is documented with an API reference in the text available, so the exact option names for toggling the unfiltered layer and for configuring the re-binning are things to find in the source or the examples rather than in a documented flag.

Marks as circles, heatmap cells, bars or text

How the filtered data is drawn is configurable in ways that matter for dense regions.

The default is a heatmap with circles, and it can also show the data without filters. The alternative is a heatmap with colored cells, which reads as a grid rather than a scatter.

Counts can be drawn as a vertical bar, a horizontal bar, or text only. Those three options are really about orientation and density. Bars fit when the categories are few enough to be named. Text only is the option for a dense crossfilter view where a bar per cell would be unreadable, and it is the reason the library depends on ndarray-imshow, a package for showing images of ndarray data, which is how a matrix of counts gets drawn without an SVG node per cell.

There is also a timeline visualization of the brush interactions in Falcon, which shows the sequence of brushes as they happened over time.

That timeline is more than a log. Brushing in a crossfilter is not commutative, the order in which you apply filters can change what you end up looking at, and a tool that shows you the sequence of your own interactions is acknowledging that. It also lets you go back and see which brush caused a result, which is the kind of provenance that analytical interfaces usually leave out.

None of these choices are described as configuration options in the documentation text, so the practical route to them is the example applications in the flights, weather and gaia-mapd directories.

Running the demos with yarn and parcel

The developer workflow is a yarn project. Install the dependencies with yarn, then run yarn start to start the flight demo with in memory data.

yarn
yarn start

The package.json shows what is behind that. The start script runs yarn start:flights, which is a parcel invocation for the flights entry point.

parcel flights/index.html

There is a start script per demo, one for each directory in the repository: flights, flights-mapd, flights-duckdb, flights-http, weather and gaia-mapd. The duckdb one adds the --https flag to the parcel invocation, which matters because DuckDB-WASM needs a secure context to work in the browser.

The build script bundles the flights, flights-duckdb and weather entry points with a public URL of /falcon/, and a second build script produces the flights-mapd bundle with its own dist directory and no optimisation step. The deploy script cleans, builds and publishes to gh-pages, which is how the demo pages end up on the project's own pages.

Tests run under jest, and the script sets the NODE_OPTIONS environment variable to enable experimental VM modules, which is what lets jest run the ES module sources.

NODE_OPTIONS=--experimental-vm-modules npx jest

The published demos are the 1M flights page, the 10M flights page with DuckDB-WASM, the 7M flights page against OmniSci Core, and a weather page with 500k records.

The research code has partly moved into falcon-vis, VegaFusion and Mosaic

This is the most important sentence in the project documentation for anyone deciding whether to depend on it: this project was developed for the paper, and a lot of the functionality is now in the falcon-vis library, in VegaFusion and in Mosaic.

Read that carefully. The repository you are looking at is the research artefact and its demo collection. The maintained implementations live elsewhere, in the separately released falcon-vis library on npm, in VegaFusion for the data computation layer, and in Mosaic.

That split is normal for academic work and it matters commercially. If you are building a product, the questions to ask are about falcon-vis, VegaFusion and Mosaic, and about whether the ideas here have moved on. The paper is from 2019, the paper itself notes that functionality has since migrated, and the repository shows the shape of a project that has handed off its implementation and kept the experiments and the examples.

The repository layout supports that reading. The top level is a set of demo directories, flights, flights-duckdb, flights-http, flights-mapd, gaia-mapd and weather, alongside data, images, src and test. There is a CITATION.cff, which is the citation metadata file, and a tsconfig.json and jest.config.js, so the code is TypeScript under test.

The documentation also records two earlier attempts. A first version that turned out to be too complicated is kept on a branch named complex, and a client-server version is on a branch named client-server. That is a useful signal: the current design is the third shape of the idea, and the abandoned branches are where the reasoning about what did not work is kept.

Licensing needs reading, and the citation is part of the deal

The licence situation here needs a direct look, because the repository and the package disagree in a way that a scanner would not make obvious.

The repository records its licence as other, a custom licence that GitHub cannot classify, with a note to see the LICENSE file. The package.json for the npm package falcon-vis, at version 0.16.2, lists the licence as BSD-3-Clause and names Dominik Moritz as the author. The package.json repository field points at uwdata/falcon, while the project also appears under the vega organisation, which tells you the code moved between organisations at some point.

Which of the two statements governs your use depends on what you consume. If you depend on the npm package, the BSD-3-Clause declaration in its metadata is what you get. If you take code from this repository directly, the LICENSE file is what governs, and the classification GitHub could not apply suggests it contains terms that are not one of the standard licences. That is a reading task for your legal team, and it is not legal advice.

The citation request is not a formality either. The documentation asks you to cite the paper if you use Falcon in a publication, and provides the BibTeX entry for it, with a DOI of 10.1145/3290605 and the CHI 2019 proceedings as the venue.

On maintenance, the repository is not archived and the last push was on 2026-08-02. There are no GitHub releases, so version history is the npm package versions and the commit log rather than a tag list. The package.json version of 0.16.2 and the CHI 2019 paper date together describe a project that has been in the open long enough to have a large body of academic and industrial interest, with the current code being the second generation of the implementation rather than the first.

Editorial conclusion

Falcon suits researchers and engineers who need to brush and filter a dataset that is too large to load naively into a page, and who want the interaction to happen on the client rather than behind a server. It is the wrong tool for a production dashboard, because the documentation says a lot of the functionality now lives in falcon-vis, VegaFusion and Mosaic, and the falcon repository itself is primarily the research artefact and demo collection. Before adopting it, read the LICENSE file, since the package.json declares BSD-3-Clause while the repository licence is a custom one that GitHub cannot classify, and decide whether the DuckDB-WASM or the OmniSci Core route matches the data you actually have.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. vega/falcon on GitHub