neutree

What Neutree manages

Neutree is an open-source large language model infrastructure management platform. The README summarizes it as an enterprise grade private Model as a Service platform. Its stated purpose is to let a team deploy and operate inference workloads without relying on a single external provider. The feature list names multi cluster management as the first capability: the system deploys and manages inference workloads across Kubernetes clusters and static node clusters built from Ray and Docker. A second capability is an OpenAI compatible API, described as a unified inference gateway with token based authentication and usage tracking, so existing client code that talks to the OpenAI interface can target a self hosted deployment instead. A third is multi tenancy, implemented as workspace based resource isolation with fine grained role based access control. A fourth is production ready observability, provided through integrated metrics collection and Grafana dashboards. A fifth is flexible model storage, with support for the HuggingFace Hub and file based model registries. The combination covers the parts a team needs to run models internally: scheduling onto hardware, a stable API boundary, isolation between groups, and a way to see what the system is doing. The project keeps its longer design documents in a docs directory, with pages covering architecture overview, cluster management, online inference, model registry, user management, RBAC and workspace, and cluster monitoring.

Building and running it

Neutree is written in Go and the contributor prerequisites listed in the README are Go 1.23 or newer, Docker, and Make. The common workflows are exposed as make targets. make build compiles all of the binaries. make test runs the unit tests. make lint runs the linter. make db-test runs the database tests. For faster iteration there are make docker-test-api and make docker-test-core, which rebuild and restart the local containers for the API and core services. A separate target, make deploy-remote, pushes a component to a container on a remote host given a host string and a component name, and the documentation notes it backs up and verifies the swap. The README points to docs.neutree.ai for installation guides, tutorials, and API references rather than embedding those details in the root file. Bug reports and feature requests go through GitHub issues, and community questions go through GitHub discussions. The split between a local quick loop and a remote deploy path suggests the project expects both laptop scale experimentation and a managed deployment on a controlled host. The use of Make as the entry point keeps the commands uniform across the different services instead of requiring developers to remember per component build invocations.

Where the project is headed

The README closes with a roadmap that lists the capabilities the maintainers want to add next. The first item is more accelerator support, with Intel XPU named as an example beyond the GPU paths the system already targets. The second is inference endpoint auto scaling, so a deployment can grow and shrink with demand instead of running at a fixed size. The third is external KV cache integration, which would let the system share prompt caches across requests or nodes. The fourth is quota and usage limits, a natural follow on to the usage tracking the API already records. The fifth is GPU memory hard isolation, a stronger guarantee than the workspace based isolation the platform provides today. The sixth is more inference engine adapters, widening the set of back ends the gateway can drive. The seventh is external endpoint support, described as a way to manage both local and external model services through one control plane. None of these are promised as shipped features, and the README presents them as direction rather than commitment. The project is licensed under the Apache License 2.0, and the community sections point contributors to the issue tracker and discussion board for coordination. The roadmap gives a clear picture of the gaps the team considers most important before the platform can be called complete for production private LLM serving.

Editorial conclusion

The repository is published under the Apache 2.0 license and its most recent commit was recorded on 2026-08-26. The source lives at https://github.com/neutree-ai/neutree.