TrailScraper: Turning AWS CloudTrail Logs into IAM Policies from the Command Line

The Problem: CloudTrail Logs Are Too Noisy to Read Directly

AWS CloudTrail records supported account activity, but coverage depends on the event types and configuration enabled. Trails log management events by default; data events require separate configuration. The JSON logs can be verbose and dispersed across S3. Finding which actions a particular role actually performed, or turning a month of logs into a usable IAM policy, is tedious. TrailScraper addresses this by providing a command-line interface to filter, download, and transform CloudTrail data. It is aimed at security engineers, DevOps staff, and anyone who needs to understand IAM usage after the fact. The tool is not a monitoring solution; it is a batch analysis utility. You run it when you need answers, not continuously.

How It Works: Select, Download, Generate, and Guess

TrailScraper has four main commands. The select command queries the CloudTrail API or reads downloaded logs, applying filters like --filter-assumed-role-arn to narrow down events. The download command fetches logs from an S3 bucket, with options for account ID, region, and time range. The generate command reads CloudTrail records and outputs an IAM policy document, grouping actions by service and setting Resource to "*" by default. The guess command extends an existing policy by inferring related actions, such as adding DeleteObject and GetObject when your logs only show PutObject. The data flow is simple: events come in as JSON, filters reduce them, and the output is a policy you can paste into AWS. The README shows a pipeline: trailscraper select | trailscraper generate, which is the fastest way to get a policy from recent activity.

Getting It Running: Homebrew, pip, or Docker

Installation is straightforward. On macOS, you can use Homebrew: brew install trailscraper. For other environments, pip install trailscraper works with Python 3.5 or later. A Docker image is also published on ghcr.io, and you can run it with a command that passes your AWS environment variables and mounts your ~/.aws directory. The Docker invocation in the README uses --env-file <(env | grep AWS_) to avoid hardcoding credentials. Once installed, you need AWS credentials configured, either through environment variables or the shared config file. The basic usage requires specifying a time range with --from and --to, which accept natural language like 'one hour ago'. For downloading logs, you must know the S3 bucket name and the account ID.

A Real Limitation: Heuristic Action Mapping Can Be Wrong

The most significant caveat is that TrailScraper does not use an authoritative mapping from CloudTrail events to IAM actions. The README states plainly: 'there is no good, machine-readable documentation on how CloudTrail events map to IAM actions so TrailScraper is using heuristics.' This means the generated policy may include actions that do not exist in IAM, or miss actions that do. For example, a CloudTrail event named 'DescribeLaunchConfigurations' might map to 'ec2:DescribeLaunchConfigurations', but for some services the naming conventions differ. The tool's own FAQ warns about this and asks users to report special cases. If you are generating policies for production, you must manually review every action. This is not a set-and-forget tool; it is a starting point for policy refinement.

The Global Services Trap: Why You Need us-east-1

Global-service and regional events require careful coverage. AWS documents that STS calls to its global endpoint are logged to us-east-1, while calls to regional STS endpoints are logged in their respective Regions. Global-event delivery also depends on trail configuration. Consult the CloudTrail documentation and include the Regions and event types relevant to your environment before generating a policy; observed events may still omit permissions required by unobserved workflows.

Extending Policies with guess: A Clever But Imperfect Heuristic

The guess command attempts to solve a real problem: CloudTrail logs only show actions that were actually performed, not the full set of permissions you need. For example, a Terraform run might only create resources, so your logs contain only Create actions, but you will also need Delete and Update actions later. TrailScraper guesses these related actions by matching action names. The README shows an example where a policy with s3:PutObject gets expanded to include DeleteObject, GetObject, and ListObjects. This is useful for building a more complete policy, but the guess is purely based on naming conventions. It will not understand that s3:PutObject might require s3:ListBucket in some contexts. The --only flag limits the output to guessed actions, which is helpful when you want to see what was added. Still, treat the output as a suggestion, not a definitive authorization set.

Alternatives: Manual Policy Writing and CloudTrail Console

The obvious alternative is to write IAM policies by hand, using the AWS Console or a text editor. This gives you full control but requires you to know exactly which actions your workloads use. Another alternative is to use the AWS CloudTrail console itself, which lets you search events and view them, but it does not generate policies. There is also the AWS Access Analyzer, which can generate policies based on CloudTrail trails, but that is a managed service with its own interface and pricing. The difference is that TrailScraper is a command-line tool that fits into scripts and pipelines, and it outputs JSON that you can feed into other tools like cfn-flip or iam-policy-json-to-terraform. The README shows how to convert the JSON to CloudFormation YAML or Terraform HCL. If you want a fully automated policy generation pipeline, TrailScraper is more flexible than a console, but you trade accuracy for convenience.

Maintenance and License: Apache-2.0 with a Single Maintainer

The project is licensed under Apache-2.0, which means you can use, modify, and distribute it freely, even commercially, as long as you preserve the license notice. The repository shows a release history with version 0.10.0 pushed on 2025-12-31, so it is actively maintained. However, the project appears to be primarily maintained by one person (flosell), which is a risk if you depend on it for critical workflows. The README mentions a contribution guide and asks users to report special cases, which indicates the maintainer welcomes community input. Upgrading between versions should be straightforward via pip, but you should check the release notes for breaking changes. The Docker image is updated on GitHub Container Registry for versions 0.7.0 and later. If you fork the project, you take on the maintenance burden. For a tool like this, the Apache-2.0 license gives you the freedom to adapt it to your needs, but you must be prepared to patch it yourself if you rely on it heavily.

Editorial conclusion

Adopt TrailScraper if you are an AWS administrator or developer who needs to derive IAM policies from actual CloudTrail activity, especially for least-privilege cleanup or post-incident analysis. Do not use it as a sole source for security-critical policies, because the action mapping relies on heuristics and may miss special cases. Before relying on generated policies, verify each action against AWS documentation and test the policy in a staging environment. Also confirm that your CloudTrail logs include us-east-1 for global services, or you will miss events. The tool is Apache-2.0 licensed, so you can modify it, but you must handle maintenance yourself if you fork it.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. Release notes