HTTPDirFS: mounting a website as a filesystem

A FUSE mount over a plain web server

HTTPDirFS is a filesystem that mounts arbitrary websites using the FUSE framework. The basic invocation takes a URL, a mount point and the cache flag:

./httpdirfs -f --cache $URL $MOUNT_POINT

The -f flag keeps the program in the foreground, which the README describes as useful for monitoring which URL the filesystem is visiting. That is the single most useful operational detail in the whole document, because a FUSE mount that fails silently looks like an empty directory.

The example URL in the README is the Debian CD image server, which is a good illustration of the target use case: a directory of large files, browsable as a tree, that you want available to a program that only understands files and directories.

Help is available three ways. The program has its own help output:

./httpdirfs --help

There is a man page, though the README notes it only works if HTTPDirFS has been installed properly rather than run from a build directory:

man httpdirfs

And the full set of flags is documented in the usage page in the repository, at docs/usage.md, which is also where the Airsonic and Subsonic options are described.

The project is written in C and builds with Meson, with the repository carrying meson.build, a subprojects/ directory for vendored dependencies, .clang-format and .clang-tidy for the C style and analysis rules, and a Doxyfile for documentation generation. The topic list names libcurl and libfuse as the underlying libraries, which is what you would expect from the job it does.

HTML listings become filenames, and collisions are resolved by path

The mechanism that makes a plain web server usable as a filesystem is the HTML parsing. HTTPDirFS parses directory listings using the Gumbo HTML5 parser and extracts descriptive filenames from anchor text, meaning the text inside the a elements rather than the raw URL.

That choice matters more than it might appear. Directory listings on the web are inconsistent. Some servers generate them with link text equal to the filename, some show a date and size instead, and some show something entirely different. Using the anchor text gives the best available description of what the entry actually is, and using a real HTML5 parser rather than regular expressions means malformed markup does not derail the listing.

The second half of the problem is collisions. Two different URLs can easily produce the same anchor text, for example two directories both called index or two files with the same name in different subdirectories flattened into one view. HTTPDirFS resolves this with progressive URL path escalation, so where a name is already taken, the next component of the path is used to disambiguate it rather than appending an arbitrary counter.

The consequence for a user is that the tree you see reflects the server's listing, and the names are the ones a human would read. Where the server's own naming is poor, the filesystem inherits that, because there is no better information available.

The cache stores segments, and 1.3.3 broke its format

The cache is the feature that makes repeated use bearable, and it works on segments rather than whole files. The README describes it as caching the file segments you have accessed, so you do not download those segments again if you access them later. The --cache flag turns it on, and the comparison the project draws is with the --vfs-cache-mode full feature of rclone mount.

The validity rule is precise. Once a segment has been downloaded, it will not be downloaded again as long as the server still reports the same Last-Modified timestamp and content length. Subsequent reads are then served from local storage at local speed. Directory listings, and files whose remote metadata cannot be verified, are treated differently: they are considered stale and refetched once they are older than --refresh-timeout seconds, which defaults to 3600.

So the cache is trustworthy for immutable media and deliberately unreliable for listings, which is the right trade-off for a filesystem that has to reflect a live server.

There is one sharp edge. The README carries a warning that versions of HTTPDirFS above 1.3.3 contain a breaking change to the cache format, introduced in a specific commit. Before using any version above 1.3.3 you must delete your existing cache with --cache-clear, or remove the ~/.cache/httpdirfs directory, which is the default location. A stale cache from an older format is not something the tool will migrate.

Cache files are stored under ${XDG_CACHE_HOME}/httpdirfs, ${HOME}/.cache/httpdirfs or ./.cache/httpdirfs in the current directory, whichever is found first, and a custom root can be set with --cache-location.

Sparse allocation is a hard requirement of the cache

The README states plainly that the cache system relies on sparse allocation and that you must make sure your filesystem supports it. Otherwise, local storage takes heavy I/O from cache file creation.

This is the kind of requirement that is easy to miss and expensive to diagnose later. The cache writes segments as they arrive into files that will grow as more of the remote file is read. On a filesystem that does not support sparse files, each write allocates blocks in a way that fills the disk with data you have not actually read yet, which shows up as unexplained disk usage and slow IO rather than as an error message.

The README links a comparison of filesystem allocation and layout policies on Wikipedia for readers who need to check whether their filesystem is on the list.

Two other cache operations are documented. You can clear the whole cache with --cache-clear, or clear a single server with --cache-clear-host and a URL or host. That granular option is the one to reach for when a server has moved a file and you want to drop that server's cached state without losing everything else.

The design overall is closer to a read-through cache for immutable media than to a synchronised mirror. Nothing is pushed, nothing is written back to the server, and the local copy is a by-product of reading rather than a separately managed replica.

Website mode turns HTML pages into browsable directories

Most of the web is not a directory listing, and the README addresses that directly with the --website-mode flag.

By default, resources whose URLs do not end with a trailing slash are treated as regular files. With --website-mode enabled, HTTPDirFS inspects the HTTP Content-Type response header of linked resources during link initialisation. Any resource returning Content-Type: text/html, with a size within --max-html-size, is promoted to a virtual directory, so you can browse into it as a subdirectory. Non-HTML resources stay regular files.

That is a neat trick and it has an obvious limit. It relies on the server telling the truth about the type of each resource, and it depends on the page's links pointing at resources that are individually fetchable. A page whose content is assembled by a script, or whose links all point back to the same page with query parameters, will not turn into a useful tree.

The recommended combination with the flag matters for practical use:

./httpdirfs -f --website-mode --progressive-directory-preload $URL $MOUNT_POINT

The README's tip explains why. If you mount a generic website that has not been cached before and plan to browse it with a graphical file browser, you probably want the progressive preloading mode, because the file browser will otherwise issue an excessive number of read requests while trying to calculate the number of files in each subdirectory.

Progressive preloading hides entries until they are ready

The progressive directory preloading feature addresses a specific pathology of FUSE mounts over slow remote trees.

When you browse a large directory listing, a file manager will stat every entry, and for each subdirectory it will try to read the whole listing to count the files inside. Against a remote HTTP server, that turns one directory view into a flood of read requests, and the interface appears to hang.

The --progressive-directory-preload flag changes the behaviour. HTTPDirFS hides subdirectory entries from the listing until their contents have been preloaded in the background by a background worker. The listing therefore starts small and fills in as the worker catches up, rather than blocking on every subdirectory at once.

The invocation is straightforward:

./httpdirfs -f --progressive-directory-preload https://example.com/files/ mnt

The README frames the feature as avoiding the flood of read requests, and in the website mode section adds that moving the burst of subdirectory listing requests to a background worker largely mitigates the slowdown described above.

This is one of those features that exists because the underlying problem is real and specific. Filesystems are pull based, HTTP is request and response, and the two do not match when the tree is deep and the server is slow. The fix is not clever, it is just deferral, and deferral is the right answer here.

Single file mode and Subsonic music collections

Not every server presents a directory listing. When you know the URL of one file and want it treated as a file rather than as a listing, the --single-file-mode flag creates a virtual directory containing exactly that one file.

./httpdirfs -f --cache --single-file-mode https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.0.0-amd64-netinst.iso mnt

The README describes this mode as similar to the unmaintained httpfs project, which is a useful signal about the level of expectation to set. It is a thin convenience over one URL, and the point is that a file browser or a media player sees an ordinary file path rather than something it has to be told to fetch.

The third shape is the Airsonic and Subsonic support, which the README dedicates to the project's Debian package maintainer, Jerome Charaoui. It lets you mount the music collection held on an Airsonic or Subsonic server locally and browse it with your usual file browser, which in practice means a media player can treat a remote music library as a directory tree.

The repository's topic list reflects this spread: alongside fuse-filesystem, http-client and libcurl, it carries airsonic, subsonic, subsonic-client and funkwhale, which points at the music server side of the project.

The options themselves are documented in the usage page under the Airsonic and Subsonic mounting options, and the README does not claim more than that.

Where HTTPDirFS is the wrong tool

The first and simplest case is a single file. If you want one ISO or one archive, mounting a filesystem to fetch it is more machinery than the job needs, and the single file mode still leaves you with a FUSE mount to manage.

The second case is a server that will not talk to a parser. The listing must be HTML with links the tool can read, and in website mode the server must send an accurate Content-Type for each resource. A site behind a login, an API that returns JSON, or a server that requires a session cookie will not work, and the README documents no authentication options for the HTTP side.

The third limitation is that this is a read-only view. The README describes mounting and caching, and nothing about writing back, uploading or synchronising. If your requirement is a local mirror that stays in step with the origin, you want a synchronisation tool instead.

The alternative approach for that case is rclone, which the README itself uses as the comparison point for the cache behaviour. The difference in approach is that HTTPDirFS is a single purpose HTTP mount that understands one thing well: parsing a web server's HTML listing and presenting it as files. HTTPDirFS gives you one protocol and one cache, and handles the awkward part of that protocol for you, which is turning an anchor tag into a filename.

For the Subsonic case the comparison is different again, since a dedicated client for that protocol can speak it directly, while HTTPDirFS exposes it through a filesystem so that tools which only understand files can use it.

Editorial conclusion

Adopt HTTPDirFS when you want a remote HTTP server to appear as a directory tree that a file browser, a media player or a copy command can work with, and when the server presents a directory listing or responds with usable Content-Type headers. Skip it when you need a one-off file, where a plain download is simpler, or when you need to write, since this is a read-only view of a remote server. Verify three things first: that your filesystem supports sparse allocation, because the cache relies on it and otherwise every cache file creation causes heavy I/O, that you clear any cache created by a version at or below 1.3.3, since versions above 1.3.3 change the cache format and the old cache must be removed, and what the custom LICENSE file actually permits, since GitHub cannot classify it and the terms are not summarised anywhere in the README. The last push to master was on 2026-10-09 and version 1.3.3 was published on 2026-06-11.

Frequently asked questions

Official sources

1. fangfufu/httpdirfs on GitHub
2. Issues
3. README
4. Releases