awsome-distributed-ai: AWS reference architectures for distributed training and inference
Best practices, reference architectures, and examples for distributed AI training and inference on AWS.
At a glance
- What is it?
- The awslabs repository collects deployable cluster architectures, runnable training and inference examples, and validation tooling for SageMaker HyperPod, ParallelCluster, AWS PCS, and EKS. It is a starting-point catalogue for teams already committed to AWS, not a framework of its own.
- Who is it for?
- Adopt it if your GPU cluster already runs on SageMaker HyperPod, ParallelCluster, AWS PCS, or EKS and you want deployable CloudFormation or Terraform starting points plus Slurm and Kubernetes launch scripts rather than a framework to install. Skip it if you need a portable training stack, if your scheduler is not Slurm or Kubernetes, or if you expect a supported product: this is an MIT-0 example repository with no stated compatibility matrix.
- Can I use it commercially?
- Yes. MIT-0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Shell, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap this repository fills: cluster plumbing before the training script
Getting a multi-node GPU job to start is rarely a modelling problem. It is a VPC with the right subnets, an EFA-capable instance type, a scheduler that can see the nodes, a container image with matching CUDA and NCCL versions, and a launch command that wires the ranks together. The README frames the repository as reference architectures and examples for distributed AI training and inference on SageMaker HyperPod, AWS ParallelCluster, AWS PCS, and Amazon EKS, and the directory tree backs that up: architectures/ holds CloudFormation and Terraform cluster definitions, ami/ holds Packer and Ansible scripts for custom machine images, and examples/ holds runnable training and inference code. The audience is an infrastructure or ML platform engineer at an organisation that has already chosen AWS for GPU workloads. If you are still comparing clouds, this repository assumes the decision is made. It also assumes a specific set of schedulers: Slurm appears under architectures/sagemaker-hyperpod-slurm and architectures/aws-pcs, while Kubernetes manifests appear under architectures/amazon-eks and architectures/sagemaker-hyperpod-eks. There is no path here for a bare-metal or on-premises cluster.
Two axes of organisation inside examples/
The examples directory is split deliberately, and the README explains the split rather than leaving you to infer it. examples/training/ and examples/inference/ are framework-centric: the engine is the subject and the model variants underneath illustrate it, so training/fsdp/ and training/megatron-lm/ and training/nemo/ each stand on their own, and swapping the model gives what the README calls the same example with a different model. examples/use-cases/ is use-case-centric: a model or task is the subject and the framework is incidental, with detr-finetune and vjepa2 named as examples, so swapping the framework would still leave a recognizable demo. Each example follows a common shape: a Dockerfile and README at the top level, then slurm/ and kubernetes/ subdirectories holding the service-specific launch instructions. That layout is the most useful thing in the repository for planning purposes, because it tells you the cost of adapting an example. Changing the model inside a framework-centric example means editing configs. Changing the orchestrator means writing a new launch path, since the slurm/ and kubernetes/ directories are separate artefacts rather than one abstraction with two backends.
What the repository does not contain: an abstraction layer
There is no CLI, no Python package, and no scheduler-agnostic launcher described in the material. The primary language is Shell, and the components are directories of templates, scripts, and manifests. That is a real constraint, not a stylistic one. A tool that abstracts Slurm and Kubernetes would let you write one training definition and target either; this repository gives you two parallel sets of files and expects you to maintain whichever one your cluster uses. The same applies across cluster types. The README lists aws-parallelcluster, sagemaker-hyperpod-slurm, sagemaker-hyperpod-eks, amazon-eks, and aws-pcs as separate architecture directories, each with its own templates. Cross-cutting concerns are handled by shared building blocks instead: architectures/common for an S3 bucket and event notifications, architectures/vpc_network for the VPC and subnets, architectures/ldap_server for multi-user access, and architectures/accounting-database for job tracking. Composing those yourself is the intended workflow. The trade-off is that you get templates close to the underlying AWS services and you inherit the maintenance burden that comes with them.
Getting a cluster and a job running
The material does not spell out an end-to-end command sequence, so what follows is what the layout and README support rather than a verified procedure. Each subdirectory under architectures/ is described as a deployable cluster architecture or a shared building block, with CloudFormation and Terraform as the two formats, so provisioning starts by applying the templates for the building blocks you need (vpc_network, then the compute architecture such as aws-parallelcluster or sagemaker-hyperpod-slurm) in that directory. Custom images are built with Packer, driven by Ansible roles and playbooks under ami/, for ParallelCluster, Amazon EKS, and plain EC2. Once a cluster exists, you move into examples/, pick a framework such as training/fsdp/ or training/megatron-lm/, and use the Dockerfile plus the slurm/ or kubernetes/ launch scripts for that example. Two additional paths matter operationally. The README points to docs/efa-cheatsheet.md for EFA tuning and the recommended environment variables, which is the file to read before debugging a slow multi-node job. And validation/ holds environment and cluster health validation tools, with observability/ holding monitoring stacks, metrics exporters, and profiling guides. Treat the validation tools as the first thing to run on a new cluster, before any training job, since they exist precisely to catch configuration problems early.
The v2.0.0 reorganisation and what it costs you
The release history is unusually informative for a repository of this kind. v2.0.0 is labelled as a repository reorganisation, and it is accompanied by v2.0.0-pre-reorg, described as a pre-reorganization snapshot, and later by v2.0.1-pre-reorg, described as a pre-reorg snapshot and path migration map. That sequence tells you two things. First, paths moved between v2.0.0 and the current layout, which means any tutorial, internal runbook, or forked copy written against the pre-reorg tree may reference directories that no longer exist. Second, the maintainers anticipated that breakage by publishing a migration map rather than only a changelog. If you forked the repository before June 2026, or if your internal documentation quotes paths from it, check v2.0.1-pre-reorg before assuming a path is still valid. This is a maintenance cost that comes with the territory: a catalogue of AWS service templates has to track service changes, and restructuring the tree is how that tracking sometimes lands. The tag naming (pre-reorg snapshots published alongside the reorganisation itself) is a convention worth understanding before you pin a version.
Licence and contribution model: MIT-0 with a curated blog
The repository is MIT-0, which is the MIT licence with the attribution requirement removed. In practice that means you can copy templates and scripts into your own repository and modify them without carrying a notice, subject to the usual absence of warranty. This is not legal advice; read the licence text and your own organisation's policy before redistributing anything. The contribution model is split in a way that is easy to miss. The README states that blog content is editorially curated by AWS authors, with the Hugo source on the content branch, while code samples under architectures/, examples/, and the rest accept external contributions as usual through CONTRIBUTING.md. So the prose you find on the published site is not community-editable in the same way the templates are. If you plan to upstream a fix to a launch script, that path is open. If you disagree with a blog post, the route is different. The split also means the site and the code can drift: the blog is curated separately from the directories it describes.
Where this is the wrong tool, and what to use instead
The clearest failure mode is portability. If your team needs one training definition that runs on AWS today and on a different provider or an on-premises Slurm cluster next quarter, this repository does not help with the second half. A framework like Ray Train or Kubeflow Training Operator takes the opposite approach: you express the job in the framework's own API and it manages placement across whatever cluster backs it, at the cost of an extra layer between you and the hardware and of features that lag behind what a native Slurm or EKS setup exposes. The choice is between templates that stay close to AWS services and an abstraction that travels. A second case where this is the wrong tool: small-scale work. The repository is explicitly about distributed training and inference across HyperPod, ParallelCluster, PCS, and EKS, and the validation and observability components assume a cluster worth monitoring. A single-node fine-tune on one GPU gains nothing from the architectures/ templates. A third limitation is verification. The material describes what each component is for, not what has been tested against which instance types, driver versions, or framework releases. There is no compatibility matrix in the README, and the repository does not claim one. You are expected to validate against your own environment, which is presumably why validation/ exists as a first-class directory.
Who should adopt it, and what to check first
The repository is a reasonable fit for a platform team standing up a GPU cluster on one of the four supported services and wanting a deployable starting point plus runnable examples for PyTorch DDP/FSDP, Megatron-LM, NeMo, vLLM, SGLang, or NVIDIA Dynamo, as the README lists them. It is a poor fit if you need scheduler portability, if you are not on AWS, or if you want a supported product with a compatibility guarantee. Three things to verify before you build on it. Read docs/efa-cheatsheet.md and confirm the recommended environment variables match your instance types and driver stack, because EFA misconfiguration is the classic cause of a multi-node job that runs correctly and slowly. Check which release tag corresponds to the directory paths you intend to use, given the v2.0.0 reorganisation and the v2.0.1-pre-reorg migration map. And run the tools under validation/ against a freshly provisioned cluster before you port any training script, since that is the only part of this repository whose stated purpose is telling you whether the environment is actually healthy.
Editorial conclusion
Adopt it if your GPU cluster already runs on SageMaker HyperPod, ParallelCluster, AWS PCS, or EKS and you want deployable CloudFormation or Terraform starting points plus Slurm and Kubernetes launch scripts rather than a framework to install. Skip it if you need a portable training stack, if your scheduler is not Slurm or Kubernetes, or if you expect a supported product: this is an MIT-0 example repository with no stated compatibility matrix. Before committing, read docs/efa-cheatsheet.md for the EFA environment variables, confirm which release tag matches the current directory layout, and run the validation/ tools against your own cluster, because nothing here has been verified on your hardware.
Community notes