# gitlabber: cloning or pulling a whole GitLab group tree by pattern

> gitlabber walks the GitLab API to build a tree of groups, subgroups and projects, then lets you select a subset of it with glob patterns or regular expressions before cloning or pulling. It is designed for backups of large instances, and its performance work is aimed at the API calls rather than at the git operations.

**ezbz/gitlabber** — Gitlabber - clones or pulls entire groups tree from gitlab

- Repository: https://github.com/ezbz/gitlabber
- Stars: 525 · Forks: 80
- Language: Python
- License: MIT
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/ezbz-gitlabber

## The problem it solves

gitlabber is a utility to clone and pull GitLab groups, subgroups, group projects or personal user projects based on path selection.

The stated purpose is that it clones or pulls all projects under a subset of groups or subgroups by building a tree from the GitLab API and allowing you to specify which subset of the tree you want to clone using glob patterns and regular expressions.

That is the whole design in two clauses, and both parts matter. Building the tree from the API is what makes the selection possible at all, because a GitLab instance can hold a deep hierarchy of groups inside groups, and the only way to select a meaningful subset is to see the structure first. And the selection by pattern is what makes the tool usable at scale, because nobody wants to enumerate four hundred project paths by hand.

The tree view is the centre of the workflow. Run the command with your includes and excludes and the -p flag and it prints the tree without cloning:

```bash
root [http://gitlab.my.com]
├── group1 [/group1]
│   └── subgroup1 [/group1/subgroup1]
│       └── project1 [/group1/subgroup1/project1]
└── group2 [/group2]
    ├── subgroup1 [/group2/subgroup1]
    │   └── project2 [/group2/subgroup1/project2]
    ├── subgroup2 [/group2/subgroup2]
    └── subgroup3 [/group2/subgroup3]
```

That tree view is also the debugging tool. If your pattern selects nothing, you can see why, because you can see what the actual paths look like. The print format is configurable between tree, json and yaml, so the same tree can be fed into another tool.

The project is Python, MIT licensed, with the author recorded as Erez Mazor and the version at 2.1.2. The recent releases are v2.1.2 from 2026-09-26, v2.1.0 and v2.0.0 both from 2025-11-19, so the 2.0 release was a deliberate reset and the 2.1 line is the settled one.

## Patterns, and the API cost they hide

The filtering behaviour has a consequence that the readme is upfront about, and it is the first thing to understand before using the tool on a large instance.

Include and exclude patterns do not work at the API level but work on the results returned from the API. For large GitLab installations this can take a lot of time, and if you need to reduce the amount of API calls for such projects, use the --group-search parameter to search only for the top level groups you are interested in, using the GitLab Group Search API, which allows a partial like query for a group's path or name.

So the architecture is that the API is queried broadly and the selection happens in memory afterwards. That is simple and predictable, and the cost is that the tool has to walk the whole instance to find the projects you actually wanted.

The --group-search parameter is the mitigation, and it is a good one because it moves the narrowing to the server side where it belongs. If you want one team out of fifty, you search for that team first and the rest of the walk is small.

The glob and regex syntax comes from the globre library, and the readme points at that project's page for how to use the patterns. Reusing an existing library for this rather than inventing a pattern language means the patterns behave the way people expect from other tools.

The argument list has both -i for include and -x for exclude, plus an -a flag with the values include, exclude or only, which selects which of the two lists is authoritative when they conflict. That third mode is what you want when a pattern set includes many paths and the exclusion is the exception.

The result is that selection is a filter over a fetched tree, and the only lever for efficiency is reducing the tree before the filter runs.

## Where the time actually goes, and the concurrency options

The readme quantifies the problem well, and the numbers make the argument for the concurrency flags.

Building a tree with 21 subgroups and 21 projects can be reduced from about 96 seconds sequentially to about 16 to 21 seconds with --api-concurrency 5 to 10. The readme puts this at a 4 to 6x speedup in real-world scenarios.

And the two concurrency flags control different things. The -c and --concurrency option controls parallel git operations, which is the cloning and pulling. The --api-concurrency option controls parallel API calls, which is the fetching of groups and projects. Both can be tuned independently based on your needs.

That separation is the useful insight. A tree of twenty-one subgroups is not twenty-one git clones, it is many more API calls, because listing subgroups, listing projects and resolving each project are separate round trips, and they can be issued concurrently because they do not depend on each other beyond the parent path. Git operations are bounded by disk and network bandwidth, while API calls are bounded by round trip latency, so parallelising them helps for different reasons and by different amounts.

The clone versus pull behaviour is also worth knowing precisely, because it is what makes the tool usable repeatedly rather than only once. When running gitlabber consecutively with the same parameters, it scans the local tree structure. If the project directory exists and is a valid git repository, meaning it has a .git folder in it, gitlabber will perform a git pull in the directory. Otherwise the project directory will be created and the GitLab project will be cloned into it.

So the decision is made by inspecting the filesystem rather than by consulting the remote. That is simple and it works, but it means the tool has no memory of what it cloned previously, and a directory that was created by something else with a .git folder in it will be pulled into rather than re-cloned.

Submodules are handled with the -r flag, which recurses git submodules, uses --recursive for cloning and relies on GitPython's smart update method for updating cloned repositories.

## Token handling, and why the keyring extra exists

The token story is the part of this tool that has been thought about most, and there is a separate design document for it in the repository, TOKEN_SECURITY_DESIGN.md.

The baseline requirement is an access token from GitLab with the API scopes read_repository and read_api, or api for GitLab versions below 12.0. Note that these are read scopes, which is the right requirement for a tool that only clones and pulls. A backup tool does not need write access to your instance.

Secure token storage uses the OS-native keyring, which is Keychain on macOS, Secret Service on Linux and Windows Credential Manager. The point is to avoid passing the token via the command line or an environment variable, both of which leak into shell history and process listings.

The resolution priority is explicit and ordered. The CLI argument has the highest priority, then a stored token from secure storage, then the environment variable. That ordering means the stored token is the default and the other two are overrides, which is the right way round for a tool you use often.

The workflow is a one-time store and then normal use:

```bash
# Install keyring (optional, for secure storage)
pip install gitlabber[keyring]

# Store token securely (one-time setup)
gitlabber --store-token -u https://gitlab.com
Enter token: [hidden input]
Token stored securely in keyring for https://gitlab.com ✓

# Use stored token automatically (no -t flag needed)
gitlabber -u https://gitlab.com .

# Override with CLI token if needed
gitlabber -t <token> -u https://gitlab.com .
```

The token is stored per URL, so a machine that talks to both gitlab.com and a self-hosted instance keeps them separate.

There is one more flag that addresses a related leak. The -T and --hide-token option uses an inline URL token, which avoids storing the GitLab personal access token in the .git/config. That is a separate leak from the shell history one, and it is the kind of detail that only surfaces once you have inspected a cloned repository's configuration files.

If keyring is not installed, gitlabber falls back to environment variables or CLI arguments, which the readme notes as the current behaviour rather than as a failure.

## How it is built, and how it is packaged

The dependency list is short and each entry has a job.

anytree is the tree structure, globre does the glob and regex matching, pyyaml and pydantic with pydantic-settings handle configuration, typer provides the command line interface and rich the terminal output. urllib3 is the HTTP layer, GitPython is the git implementation, and python-gitlab is the GitLab API client.

Splitting configuration between pydantic-settings and an explicit argument table is worth noting. The readme shows that every argument can be given on the command line or through an environment variable: the token as GITLAB_TOKEN, the url as GITLAB_URL, the method as GITLABBER_CLONE_METHOD, the naming as GITLABBER_FOLDER_NAMING, the include as GITLABBER_INCLUDE and the exclude as GITLABBER_EXCLUDE.

That pattern is the one to recognise from CI, where a token on the command line would end up in the job log.

The build is setuptools with a minimal setup.py that does nothing but call setup, with everything in pyproject.toml. The console script is gitlabber, mapped to gitlabber.cli:main. The package data is included, and the MANIFEST.in handles the rest.

The pytest configuration is unusually detailed. Coverage is on by default with the term-missing report and an XML report, the import mode is importlib, coverage is appended rather than replaced so repeated runs accumulate, and there is a --no-cov-on-fail flag. The without-slow-integration and integration-cover options separate the slow integration tests from the unit run, which is the arrangement you want when a test suite includes real GitLab interactions.

The requirements file pins every dependency exactly, including anytree, GitPython, python-gitlab, globre, PyYAML, rich, typer, pydantic, pydantic-settings and urllib3, while the pyproject lists them with ranges. That combination means the library metadata is loose and the development environment is reproducible.

The repository also has a Dockerfile.test and a docker-compose.test.yml for running the integration tests against a real GitLab, and the top level includes SECURITY.md, CONTRIBUTING.md, DEVELOPMENT.md, a pre-commit configuration and a .python-version pinning the interpreter.

## Conclusion

Adopt gitlabber if you need a mirror or backup of a large part of a GitLab instance and want to select projects by path pattern rather than listing them. Do not adopt it if you need incremental syncing of arbitrary projects, because the tool decides between clone and pull by looking for a .git folder rather than tracking remote state. Before you commit, create a token with the read scopes only, since the tool needs to read rather than write, and consider the keyring extra so the token does not sit in your shell history.

## FAQ

### What does gitlabber do?

It is a utility to clone and pull GitLab groups, subgroups, group projects or personal user projects based on path selection. It builds a tree of the instance from the GitLab API and lets you select which subset of that tree to clone using glob patterns and regular expressions.

### How do I install gitlabber and what does it need?

Python 3.11 or higher, Git 2.0 or higher and network access to the GitLab instance. Install it with pip install gitlabber from PyPI, or from source with git clone and pip install -e ., and add the keyring extra for secure token storage. You need a GitLab access token with the read_repository and read_api scopes.

### Why are my include and exclude patterns slow on a large GitLab instance?

Because the patterns do not work at the API level, they work on the results already returned from the API, so the tool has to walk the whole instance first. The readme advises using the --group-search parameter with the GitLab Group Search API to narrow to the top level groups you care about before the tree is built.

### What is the difference between --concurrency and --api-concurrency?

The -c and --concurrency option controls parallel git operations for cloning and pulling, while --api-concurrency controls parallel API calls for fetching groups and projects. The readme reports that a tree of 21 subgroups and 21 projects drops from about 96 seconds sequentially to about 16 to 21 seconds with an api concurrency of 5 to 10.

### How does gitlabber decide between cloning and pulling?

It scans the local tree structure. If the project directory exists and is a valid git repository with a .git folder in it, it performs a git pull in the directory, otherwise it creates the directory and clones the GitLab project into it. The -r flag recurses git submodules using the --recursive clone and GitPython's smart update method.

## Sources

- [ezbz/gitlabber on GitHub](https://github.com/ezbz/gitlabber)
- [Issues](https://github.com/ezbz/gitlabber/issues)
- [License: MIT](https://github.com/ezbz/gitlabber/blob/main/LICENSE)
- [README](https://github.com/ezbz/gitlabber/blob/main/README.md)
- [Releases](https://github.com/ezbz/gitlabber/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/ezbz-gitlabber