ShadowClone: fanning a CLI job across thousands of serverless functions

The mechanism is chunk, dispatch, merge

ShadowClone distributes long running tasks dynamically across thousands of serverless functions and returns the results within seconds where the same work would have taken hours. The how it works section describes the pipeline. A container image is created during the initial setup and registered as a runtime for the function in AWS, GCP, Azure or whatever. When you run ShadowClone on your computer, instances of that container are activated automatically and are only active for the duration of the execution. The number of instances is decided dynamically at runtime from the size of the input file and the split factor. The input is then split into chunks and distributed equally between the instances to execute in parallel. The worked example in the README is concrete: an input file of 10,000 lines with a split factor of 100 lines becomes 100 chunks of 100 lines, and 100 instances run in parallel. Nothing about the design is clever; everything about it is about the fan-out.

The CLI is one file with four flags

The usage surface is a single Python script, shadowclone.py, invoked with a handful of arguments. The help output in the README shows the whole interface, with a lightning marker before the command:

python shadowclone.py -h
usage: shadowclone.py [-h] -i INPUT [-s SPLITNUM] [-o OUTPUT] -c COMMAND

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
  -s SPLITNUM, --split SPLITNUM
                        Number of lines per chunk of file
  -o OUTPUT, --output OUTPUT
  -c COMMAND, --command COMMAND
                        command to execute
  --no-split NOSPLIT    File to be used without splitting

The semantics are worth reading carefully. The split factor is expressed in lines per chunk, not in bytes or in chunks, so the number of invocations is derived from the file. The command is passed as a single string, which means your tool runs once per chunk with the chunk as input. The no-split option names a file to be used without splitting, which is the escape hatch for a job that cannot be parallelised and for a second file that should be available to every invocation. Output goes to a path you choose. It is a thin wrapper, and that thinness is the point: the tool you already use stays the tool you already use, and only the fan-out is new.

Lithops is the actual engine underneath

ShadowClone is not the layer that talks to your cloud provider. The References section links the Lithops documentation, and the dependency list in requirements.txt pins lithops at 3.4.1, which tells you the serverless execution comes from that project and ShadowClone is a front end for it. The rest of the requirements file is much larger, more than eighty pinned packages, and it is a heavier set than a script that splits files would need: boto3 and botocore for AWS, the IBM Cloud SDKs for Code Engine and COS, kubernetes 29.0.0, redis, pika and kafka-python for messaging, plus numpy, scipy, pandas, pillow, matplotlib and seaborn. The container build makes the same point. The Dockerfile installs awslambdaric, boto3, redis, httplib2, requests, numpy, scipy, pandas, pika, kafka-python, cloudpickle, ps-mem, tblib, delegator.py and invoke into a function directory, and it also installs nmap via apt in the build image, which tells you the author expected people to run network tools inside the function. There is a comment in the Dockerfile inviting you to put your own dependencies and tools there.

Cloud agnostic in name, provider setup in practice

The feature list claims cloud agnostic, with the same script working on AWS, GCP, Azure and so on, and the comparison table repeats the claim in the context of instance counts and free tiers. The repository layout shows how that is achieved in practice. There is a config.py and a config.py.example, and a lithops-config.example, which is the Lithops configuration file that names backends and credentials. There is a bins/ directory, a hasher.py, a Dockerfile for the current version and two older ones, Dockerfile_v0.0 and Dockerfile_v0.2, which shows the container build has been revised across versions. The wiki carries the installation and initial configuration instructions, and the README's Installation section is just a pointer to it rather than a set of commands. So the abstraction is real in the sense that one script drives several clouds, but the provider-specific part lives in configuration files you have to write, and the README does not enumerate them.

The free tier is the economic argument, with a ceiling

The pricing comparison in the README is the core of the pitch. Axiom and Fleex are listed at 10 to 100s of instances, per instance per minute cost, 4 to 5 minutes of startup time, unlimited max execution time and a nonzero idle cost. ShadowClone is listed at 1000s of instances, mostly free cost, 2 to 3 seconds of startup time, a 15 minute max execution time, free idle cost and unlimited on demand scalability. The footnotes explain the constraints. Most cloud providers do not allow spinning up too many instances by default, so the tools it compares itself to are limited to around 10 to 15 instances unless you ask support to raise the limit. And AWS and Azure allow 1 million invocations per month for free while Google allows 2 million, with charges only above those limits. The free tier table gives the detail: Google Functions at 2 million invocations and 400,000 GB-seconds per month, AWS Lambda at 1 million invocations and up to 3.2 million seconds of compute, Azure Functions at 1 million invocations. The README also points out that you can make any number of invocations, the table only shows how many are free.

Where the approach breaks down

There are two hard boundaries and one soft one. The hard boundary is the 15 minute maximum execution time, which is a property of the serverless platforms rather than of ShadowClone, so any chunk of your job that takes longer than a quarter of an hour cannot be run this way at all. The second is that the tool needs a container image built and registered before anything works, which means a Docker build, a push to a registry, and provider-side setup, so the first run is a setup project rather than a command. The soft one is concurrency. The feature list says up to 1000 parallel invocations, but the same README notes that providers cap how many instances you may spin up by default and that raising the cap means asking support. A job whose runtime is dominated by the time to start a thousand containers will not see the advertised seconds. There is also a data residency cost that the README does not discuss: your input file is uploaded to cloud storage and the results come back, which is the entire point and also the reason the tool is a poor fit for anything you are not allowed to send to a third party.

A proof of concept by its own description, and the demo output

The disclaimer at the end of the README is worth reading in full, because it sets expectations. The tool is described as designed as a proof-of-concept implementation, with usage intended for educational purposes only, and it states that usage for attacking targets without prior mutual consent is illegal, that it is the end user's responsibility to obey applicable laws, and that the developers assume no liability and are not responsible for misuse or damage. The demo section shows what that means in practice, with two asciinema recordings: DNS bruteforcing using a 43mb file in 34 seconds, and running httpx on 94K subdomains in one minute. Those are the two numbers the README offers, and they are the shape of the workload the tool is tuned for, which is reconnaissance over large wordlists against infrastructure you control. The Similar Tools section lists Axiom, Fleex and gopherblazer, and the comparison table is explicitly against Axiom and Fleex. The README says the project was inspired by both and goes beyond their VPS-based concept by using serverless functions and containers. The Support section asks for bugs and offers a coffee link, which places the project firmly in the hobbyist end of the range.

Editorial conclusion

Adopt ShadowClone if you have a CLI job that is embarrassingly parallel over a large file and you want to use cloud free tiers instead of maintaining a fleet of machines, and you are prepared to build and register a container image first. Do not adopt it if your job takes longer than 15 minutes per chunk, since the comparison table records a 15 minute maximum execution time against the unlimited runtime of the tools it compares itself to, or if your data cannot leave your machine. Before you commit, read the wiki for the initial configuration, confirm your provider's per-invocation duration limit and concurrency rules, and check the hasher.py file in the repository, since whatever it does to your input before dispatch is worth understanding before you point it at anything.

Frequently asked questions

Official sources

1. fyoorer/ShadowClone on GitHub
2. Issues
3. License: Apache-2.0
4. README