goop: rebuilding a git repository from a web server

The mistake this tool exists for

The description says what goop does in one line: it is yet another tool to dump a git repository from a website, and it focuses on as-complete-as-possible dumps and on handling edge cases, compared to other tools which seem to focus on bare minimum dumps.

The situation it addresses is a deployment mistake that is still common. A web application is built, the source tree contains a .git directory, and the deployment copies the whole tree to the server, including the repository metadata. The application works, the site is live, and the server will happily hand out the entire history of the project to anyone who asks for the right path.

What that exposes is usually more than the current source. The object store contains previous commits, so a deleted file, a reverted change, or a credential that was committed and later removed is often still there. The refs show branch and tag names. And the working tree, once reconstructed, is the project as it was on the server, which may not be what the repository looks like today.

So the value of a tool like this is the completeness of the recovery. A dumper that fetches the obvious files gets you a snapshot of some of the project. A dumper that walks the object graph gets you the history, which is where the interesting material usually is.

The codebase is described as heavily inspired by another well known project of the same kind, and the original author is credited in the readme.

The command line, and the flags that change the behaviour

The usage is a single URL and an optional directory, with five flags.

Usage:
  goop [flags] url [DIR]

Flags:
  -f, --force   overrides DIR if it already exists
  -h, --help    help for goop
  -k, --keep    keeps already downloaded files in DIR, useful if you keep being ratelimited by server
  -l, --list    allows you to supply the name of a file containing a list of domain names instead of just one domain

The force flag overwrites the output directory if it is already there, which is the default behaviour of a tool that expects to start from a clean directory.

The keep flag is the one with a story behind it. It keeps the files that have already been downloaded, and the readme gives the reason: it is useful if you keep being rate limited by the server. A reconstruction of a large repository is many requests, and a server that is slow or that rate limits will make a naive run start over from nothing. With the flag, the already fetched files stay and the run resumes.

The list flag turns a single target into a batch. Instead of one domain you give the name of a file containing a list of domain names, which is how a researcher sweeps a list of hosts, and the rate limiting note is the reason the keep flag matters in that mode.

The example is the shortest possible invocation, one domain and no directory, and the install is the standard Go install of the module.

go install github.com/deletescape/goop@latest

One detail in that line is worth noticing. The module path in the manifest is the original author's path, while the repository being analysed is a fork under a different account, so the install command points at the upstream module rather than at the fork. That is what a fork with a diverged history usually looks like when the fork has become the working copy.

How the reconstruction works when there is no listing

The tool has two modes, and the second one is where the work is.

First it checks whether a directory listing is available. If it is, the tool recursively downloads the .git directory, which is what you would do with a downloader such as wget, and the problem is solved.

If there is no listing, goop has to guess the paths, and the readme describes the sequence it steps through.

It fetches all the common files, and the examples given are the ignore file, the head pointer, and the index. It then finds as many refs as possible, and the examples are a branch ref, a remote head ref, and so on, and the sources it reads to find them are the head file, the head log, the config, and the packed refs.

Next it finds as many objects as possible, identified by their SHA-1, and again the sources are the packed refs, the index, the refs directory, and the logs directory.

From objects to a working tree, and the two second pass

Then it fetches all objects recursively, analysing each commit to find its parents, which is how it discovers more objects to fetch and how it builds the history.

After that it runs a checkout of the current tree, which is what produces the working copy you actually want to read.

And then it goes back for the things a plain checkout would miss: it attempts to fetch the files listed in the index that are still missing, it attempts to create objects for the files it fetched by hand, and it attempts to fetch the files listed in the ignore file.

That last step is the interesting one, and it is the clearest statement of the tool's philosophy. A file that is in the ignore list was never committed, so it is not in the object store, but the server may still be serving it from the working tree, and a file that was never committed is often the one that matters, because that is where local configuration and local secrets live.

That last step is the interesting one, and it is the clearest statement of the tool's philosophy. A file that is in the ignore list was never committed, so it is not in the object store, but the server may still be serving it from the working tree, and a file that was never committed is often the one that matters, because that is where local configuration and local secrets live.

The same reasoning explains the checkout step. A plain checkout gives you the files the server has, which is the point, and the second pass then fills in the two categories a checkout cannot reach: the files the index mentions but the object store no longer holds, and the files the ignore list mentions that were never in the object store at all.

The dependencies, and what they say about the implementation

The module file is short, and every entry corresponds to a part of the job.

The HTTP client is fasthttp rather than the standard library client, which is the usual choice for a tool that makes a very large number of small requests and wants the connection handling to be cheap. The command line is cobra, and the configuration file support is the ini library, which is what the tool reads its settings from.

The HTML and URL handling is goquery, which is the query library built on the standard HTML parser, and it is there for the case where the tool has to look at a page rather than at a file.

The git machinery is go-git, and it is the go-git library in two forms: the core library and the billy filesystem abstraction. The abstraction is the interesting choice, because it lets the same git code read from a memory filesystem rather than from disk, which is what you want when the objects are being fetched one at a time from a server rather than read from a local clone.

The job tracker is a small library by the same original author, and it is how the tool keeps track of the work it still has to do across the recursive fetches, which is the part of this design that has to be right for the completeness claim to hold.

The logger is phuslu/log, which is a drop-in structured logging library rather than the standard library logger, and the standard library logger would have been a poor fit for a tool whose output is a sequence of progress states.

The Go version in the manifest is 1.14, which is old enough to say something about the project's age, and it is worth noting that a version that old predates the generics work that later versions of the same ecosystem adopted.

Editorial conclusion

goop fits a security researcher or a bug bounty hunter checking a scope for the classic mistake of deploying a web server with the repository directory still reachable, and it fits a developer who has found such a deployment and wants to see exactly what was exposed. It does not fit anyone who needs to reach a repository that is not already being served, because the tool has no notion of credentials or of gaining access. Verify first that the target is inside your scope, that the server really is publishing the directory rather than only a subset, and that you understand the result is a reconstruction rather than the original, since the recovered object set may be incomplete and the checkout reflects the server's state rather than the author's.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. nyancrimew/goop on GitHub
4. README