DVF-app: exploring French property sale data at parcel level

What the DVF data actually contains

The DVF database, Demandes de Valeur Foncière, records mutations à titre onéreux, meaning property transfers with a price, across the last five years. The README lists the transaction types it covers, and the list is specific: sale, sale in a future state of completion, sale of land for construction, exchange, adjudication and expropriation. The application at app.dvf.etalab.gouv.fr builds on this data and proposes to visualise the transactions at the granularity of the cadastral parcel. It was developed by the Etalab mission team, and the README says in French that the team will read improvement suggestions gladly. Two facts about coverage matter before anything else. First, the five-year window means the dataset is a rolling archive, not a historical record, so anything long-run has to come from somewhere else. Second, the data for Mayotte and for Alsace and Moselle is missing, because those areas are managed by the Livre Foncier rather than the DGFiP, which is a real gap for anyone doing regional analysis in eastern France.

Two halves that can be run separately

The installation instructions split into two options, and that split is the most useful thing in the README. The minimal installation is for developing the user interface alone, and the README states plainly that to take part in the development of the user interface it is not necessary to install Python and PostgreSQL. The full stack option is for developing the whole application, back end and front end together. Architecturally the two halves are joined by an HTTP call rather than by shared code. The front end is a Node.js server using Express, with http-proxy to forward to an API and morgan for request logging, and it reads its API location from configuration. The back end is a Python application using Flask, with psycopg2 for PostgreSQL, pandas and SQLAlchemy in the dependency list, and it reads the same data from a database instead of an API. The .env.sample file shows the two settings that matter for the front end, API_URL pointing at the production API and PORT, which defaults to 3000.

Installing the front end alone

The front end path needs a Linux server, with Ubuntu 18.04 given as the example, and it starts by fetching the repository:

$ git clone https://github.com/etalab/DVF-app
$ cd DVF

Then Node.js 8 or later, installed and checked:

$ sudo apt-get update && sudo apt install nodejs
$ nodejs -v
v8.10.0 

Yarn is added through its own apt repository:

$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
$ sudo apt-get update && sudo apt-get install yarn
$ yarn --version
1.15.2

The dependencies and the dev server are then two commands, with the web server on port 3000 by default:

$ yarn

$ yarn start

A Node.js script serves the interface and proxies to the production API, which is why a front end contributor needs no database at all. The package scripts reflect that arrangement, with start bound to node server and a second script, build-communes-mapping, that regenerates the communes-mapping.json file the interface depends on.

Building the database for the full stack

The full stack path adds PostgreSQL and then the data itself. The server package is installed and checked:

$ sudo apt-get update && sudo apt install postgresql-10
$ psql -V
psql (PostgreSQL) 10.7 (Ubuntu 10.7-0ubuntu0.18.04.1)

The database creation and import is a single script, and the README describes what it does. The script starts by creating a PostgreSQL database and a table, then downloads the DVF data reprocessed by Etalab, which is available in a separate repository at github.com/etalab/dvf, and finally some post-processing is carried out, a treatment of a few minutes. The command is:

$ sh db/build_db.sh

So the volume of the dataset is handled by a script rather than by you, but the download still happens on your machine, which is worth knowing before you plan a rebuild. The data source is the processed DVF files published by Etalab, not the raw government files.

Database access and the Python packages

Access to the database is configured by setting a password for the postgres user and writing the connection details to a config.csv file. The README says to replace <YOUR PASSWORD> with your password, and the sequence is a psql session followed by a write:

$ sudo -u postgres psql
postgres=# \password postgres
Enter new password: <YOUR PASSWORD>
Enter it again: <YOUR PASSWORD>
postgres=# \q

$ echo -e "postgres\n<YOUR PASSWORD>\nlocalhost" > config.csv

The Python side then needs its packages installed:

$ sudo apt-get update && sudo apt install python3
$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10
$ sudo apt install libpq-dev python-dev
$ sudo apt install python3-psycopg2 python3-flask python3-pandas python3-sqlalchemy python3-psycopg2

The alternative call pins the default python to Python 3 with priority 10, and the second apt line installs the PostgreSQL client headers needed to build psycopg2 from source. The web server for the back end then runs on port 5000 by default:

$ python app.py

That port difference is the practical detail that catches people. The front end serves on 3000 and the back end on 5000, and the front end's API_URL decides which one it talks to.

Configuration and the communes mapping

Some parameters can be overridden with a .env file, and the .env.sample file is provided as an example. The sample contains two lines, API_URL pointing to the production API endpoint and PORT set to 3000, which is the whole configuration surface of the front end. Changing API_URL to a local back end is how a full stack developer sees their own Python changes, and it is the step that connects the two halves of the installation. There is also a separate generated artefact, the communes-mapping.json file, which maps French commune names and is regenerated with a script rather than by hand:

yarn build-communes-mapping

That script is listed in the package manifest as build-communes-mapping, pointing at lib/build-communes-mapping, and the development dependencies include @etalab/decoupage-administratif, which is the administrative division dataset the mapping is built from, plus fs-extra and lodash. So the mapping is not a hand-maintained list of place names but something derived from an official dataset, which is the right way to do it and also explains why the script exists.

A public API, and where the licence sits

The README points out that an API developed by Christian Quest is available, with its own documentation, and it is worth separating the two things a reader might want. The application at app.dvf.etalab.gouv.fr is the interface, and the project is Apache 2.0 licensed with the LICENSE file at the repository root and the package manifest carrying the same licence. The application repository is private false and public in the manifest sense, meaning it is a published package rather than an internal one. The data itself is open data, published by Etalab under the DVF programme, and the topics attached to the repository name the domain precisely: cadastre, cartography, dvf, open-data, ventes-immobilieres and visualisation. The homepage of the deployed application is app.dvf.etalab.gouv.fr, and the repository also carries an app.json, which is the Heroku-style manifest that tells you how the service was originally expected to be deployed.

Where this application is the wrong tool

Three limits are worth stating plainly. The first is that this is a visualisation application, not a data product, so if you want the transactions themselves in a machine-readable form you should go to the DVF dataset published by Etalab rather than scrape the interface, and the README itself points there. The second is coverage, which comes back in two ways: the five-year rolling window and the missing Mayotte, Alsace and Moselle data, both of which are properties of the source rather than of the application, but which determine what questions the application can answer. The third is the age of the documented toolchain. The instructions are written for Ubuntu 18.04, Node.js 8 with a version check that prints 8.10.0, Yarn 1.15.2, PostgreSQL 10.7 and the python-dev and python3-psycopg2 packages, which is a stack from the end of the 2010s. That does not mean the code no longer runs, since the package manifest pins modest dependency versions, but a new contributor should expect to adjust the setup before the documented commands work verbatim.

Editorial conclusion

DVF-app suits a developer who wants to work on the interface of a real open data application, since the front end runs against the production API with no database and no Python required, and the whole thing is Apache 2.0 licensed with the data itself public. It is the wrong tool if you want a current national dataset in one file, because the DVF base itself only covers the last five years and omits Mayotte plus Alsace and Moselle, which are handled by the Livre Foncier rather than the DGFiP. Verify first which data you actually need, since the README points to a separate Etalab repository for the processed DVF files, and check the API configuration, because the .env sample points the front end at the production API and your local changes will only be visible once the back end is running on port 5000 with your own database.

Frequently asked questions

Official sources

1. etalab/DVF-app on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README