karlicoss/HPI: A Python Interface for Personal Data

The data-silo problem HPI targets

HPI addresses a specific problem: useful personal records are split across cloud services, applications, and local devices, while each source exposes them through a different interface. The README points to posts and comments from social networks, browser history, messages, books, annotations, location records, photographs, videos, health measurements, notes, and todos. Even when a service offers an API, the README argues that access has limited value unless someone is prepared to write integration code and operate the supporting infrastructure.

The intended user is therefore a programmer with a personal archive or data-access problem, not someone looking for a consumer dashboard. HPI's approach is to expose those records through Python modules, allowing existing libraries and frameworks to process them. Its repository topics, including `data-liberation`, `lifelogging`, `personal-api`, and `quantified-self`, align with that goal.

The scope extends beyond passive import. The project presents personal data as an object that can be queried and combined with other records. That makes cross-service work possible in principle, but HPI does not claim that every module uses the same schema. Consumers must still understand the shape and provenance of each returned type. The package reduces repetitive integration work; it does not erase differences between source systems.

Python modules and HPI's data flow

HPI is a Python package published under the distribution name `HPI`, while its familiar import name is `my`. The `pyproject.toml` explains that `my` was unavailable as a PyPI project name, so `HPI` is used for package discovery. The code is organized under `my/`, with supporting code in `src/`, documentation in `doc/`, examples in `demo.py`, and additional development utilities elsewhere in the repository.

According to the README, the package hides the mechanics of locating data, parsing formats, handling errors, and caching. That abstraction sits between external personal-data sources and ordinary Python consumers. Callers import a module and call functions that return familiar Python types and data structures. The README's Reddit example imports `my.reddit.all`, obtains saved items, and counts their subreddits.

This design favors composability over a single universal personal-data database. Each domain can preserve semantics appropriate to its source, such as social posts, health measurements, or reading activity. The trade-off is that HPI is not documented as providing one normalized identity model across all sources. A module may also need credentials, local exports, or service-specific configuration. The package metadata supports Python 3.12 and later and lists dependencies including `platformdirs`, `kompress`, `click`, `more-itertools`, and `typing-extensions`.

Setup starts with source-specific modules

The repository directs installation and configuration to `doc/SETUP.org`, while `doc/modules.md` contains module-specific setup information. The README and the referenced setup files do not show a complete installation command, authentication procedure, or universal first-run sequence, so this article does not invent one. A prospective user must follow `doc/SETUP.org` and then consult the entry for each data source in `doc/modules.md`.

This is an important distinction between the package and a plug-and-play application. The core project requires Python 3.12 or newer according to `pyproject.toml`, but personal-data modules can have separate needs. Social services, browsers, messaging systems, local archives, and compressed files do not all provide data through the same access method. The presence of configuration checks and optional dependencies also suggests that installation alone does not make every module operational.

For a first real use, the clearest documented example is Reddit analysis. It counts the subreddits represented among saved Reddit items and returns the four most common values. Install and configure HPI according to `doc/SETUP.org` and `doc/modules.md` before attempting the import. The expected result is a `Counter` result ordered by frequency; exact values depend entirely on the user's saved Reddit data.

Running the documented Reddit query

The README's first useful example is deliberately short because HPI's main job is to prepare access rather than prescribe an analysis workflow. The example uses Python's standard `Counter` after importing the relevant HPI module. No visualization or database layer is involved.

The reader should see a list of subreddit names paired with occurrence counts. The numbers in the README are illustrative output tied to that user's own archive and should not be expected from another account. This example also demonstrates the intended extension pattern: call a source module, receive ordinary Python values, then use the standard library or another analysis package.

Because the setup instructions are distributed across `doc/SETUP.org` and `doc/modules.md`, a failure at this stage may come from configuration or missing source data rather than from the counting expression itself. HPI says it handles data location, parsing, errors, and caching, but the README does not define a uniform error policy or recovery interface for every module. Teams that require deterministic ingestion and centralized observability would need to establish those expectations during evaluation.

Cross-service analysis without a universal schema

The repository's use cases show why a programmable personal-data layer can be valuable. The README asks whether reading progress, browser history, messages, exercise data, commits, maps, and other records could be searched or compared across service boundaries. HPI aims to make such questions executable in Python instead of restricting analysis to the interface offered by each vendor.

That capability is real at the architectural level, but it does not imply that any two records can be joined directly. HPI modules cover domains with different time formats, identifiers, update semantics, and privacy controls. The README does not document a shared schema, entity-resolution system, or cross-module query language. The practical result is a set of Python interfaces that can be combined in code, with integration decisions retained by the programmer.

A conventional data warehouse is the clearest alternative for a different case. A warehouse centers ingestion, transformation, schema management, and querying around a shared database. HPI begins with domain modules and leaves composition to Python code. A warehouse may be a better fit when many users need the same governed analytical model. HPI is more suitable when one programmer already understands their sources and wants direct access without adopting a centralized database design first.

Privacy, permissions, and operational boundaries

HPI handles categories of data that can reveal identity, relationships, health, movement, interests, and private communications. The README includes privacy as a discussion topic, but it does not specify encryption, secret storage, retention defaults, redaction features, or a complete threat model. Those omissions should not be interpreted as guarantees. Users must inspect the relevant module documentation and configuration files before placing sensitive exports or credentials into HPI's data paths.

The project is not documented as a hosted personal-data service. Its package-based model suggests that processing occurs through the user's Python environment, but module behavior can depend on external services and local exports. Users should verify those data paths rather than assume that every integration remains local.

HPI is also the wrong tool when a finished interface matters more than programmability. It is a collection of modules, not a documented application for browsing, reviewing, or deleting personal records. It is a poor fit for users who cannot maintain Python dependencies or diagnose missing credentials, malformed exports, cache behavior, and changing service interfaces. The README does not provide a rollback guide for bulk data operations, which further limits its suitability as an administration console.

Maintenance state and upgrade work

The repository is not archived, and its last push was on 2026-10-10. Recent rolling releases include `v0.8.20260912`, `v0.7.20250911`, and `v0.6.20250816`. These dates indicate recent development activity, although release frequency does not guarantee that every external data source remains compatible.

Maintenance work extends beyond upgrading HPI itself. Personal-data modules may depend on APIs, authentication methods, local export formats, third-party libraries, or service behavior outside this repository. The `CHANGELOG.md`, `doc/DEVELOPMENT.org`, and module documentation are the relevant places to inspect before changing versions. The package metadata declares Python `>=3.12`, so adopting it can also require a Python runtime upgrade.

HPI is distributed under the MIT license. That permits use and modification under the license terms, but operators remain responsible for reviewing dependencies, protecting private data, and complying with source-service terms and applicable law. The project does not remove those obligations. License coverage for personal records, credentials, and third-party exports is separate from the license of the HPI code.

Editorial conclusion

HPI is appropriate for Python programmers who already understand where their personal data resides and can manage credentials, caches, parsing errors, and sensitive exports. It is a poor fit for users seeking a ready-made dashboard, a uniform schema across every source, or a service that removes privacy and maintenance work. Before adoption, inspect `doc/SETUP.org`, choose the relevant modules from `doc/modules.md`, and verify that the target data can be imported without uploading it to a third-party system.

Frequently asked questions

Official sources

1. karlicoss/HPI on GitHub
2. License: MIT
3. Project website
4. README
5. Releases