aws-glue-libs runs Glue PySpark code on your own machine

A local harness, not a runtime

The repository supports Python libraries for local development of Glue PySpark batch jobs, and the README is explicit that Glue streaming is supported in a separate repository, aws-glue-streaming-libs.

That division tells you what the project is. Inside this repository there are two things. The `awsglue` library is the Python library you use to author an AWS Glue ETL job, and the README describes it as extending Apache Spark with additional data types and operations for ETL workflows, and as the interface for the Glue ETL library in Python. The `bin` directory hosts several executables that allow you to run the Python library locally or open up a PySpark shell to run Glue Spark code interactively.

The important detail sits in the setup guide. The `awsglue` library provides only the Python interface to the Glue Spark runtime, and you need the Glue ETL jar to run it locally. The jar is available via the Maven build system in an S3 backed Maven repository, and the guide's first step is to install Apache Maven from a tarball hosted alongside those artifacts.

So the library is half a runtime. The Python side is here, the Java side is a jar you fetch, and Spark itself is a distribution you download per Glue version.

A branch per Glue version, and the version table

Different Glue versions support different Python versions, and the README gives a table that also tells you the branch to use for each one. Glue 2.0 pairs with Python 3.7 on the `glue-2.0` branch, Glue 3.0 pairs with Python 3.7 on `glue-3.0`, Glue 4.0 pairs with Python 3.10 on `glue-4.0`, Glue 5.0 and Glue 5.1 both pair with Python 3.11 on `glue-5.0` and `glue-5.1` respectively, and Glue 6.0 pairs with Python 3.13 on `main`.

That table is the operational centre of the project. Checking out the wrong branch gives you a library whose runtime assumptions do not match the version you deploy, and since the library talks to a specific Spark build, the mismatch is not always a clean error.

The release notes at the top of the README record the most recent move. The August 18 2026 entry says the `main` branch now represents Glue 6.0, which runs Apache Spark 4.1 on Python 3.13, and that Glue 5.1 has moved to the `glue-5.1` branch. The same notes record that in July 2023 branches for Glue 0.9 and 1.0 were removed as deprecated, and that in August 2021 the master branch was changed from representing Glue 0.9 to Glue 3.0, with a `glue-0.9` branch created to preserve the former state, and it gives the git commands for renaming a local clone.

The GitHub release tags are a separate line again, v4.0 from December 2022, v3.0 from August 2021 and a combined v1.0-and-v2.0 tag from the same day. The branches are what the README tells you to work from.

The Maven step that fetches the Glue jar

Getting the runtime is a three part operation, and the middle part is the one people miss.

You install Apache Maven from the location given in the guide, a tarball under `aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/`. You then use the `copy-dependencies` target in Apache Maven to download the jar from S3 to your local development environment. The reason for the S3 backed Maven repository is that the Glue ETL jar is not in a public artifact repository, so a normal dependency resolution would not find it.

That is a deliberate piece of friction, and it is there for a reason: the local runtime is the same jar AWS runs, which is the point of the whole exercise. A Glue job that works against a public Spark build may still behave differently on the Glue runtime, so the project asks you to use the matching one.

The `pom.xml` at the repository root is where the Maven configuration lives, alongside `base-requirements.txt` for the Python side, and the package build is declared in `pyproject.toml` and `setup.py`.

SPARK_HOME depends on the Glue version

Next you download and extract the Apache Spark distribution for the Glue version you are using, and the README lists one per version. Glue 2.0 uses a Spark 2.4.3 hadoop2.8 build, Glue 3.0 uses `spark-3.1.1-amzn-0-bin-3.2.1-amzn-3`, and Glue 4.0 uses `spark-3.3.0-amzn-1-bin-3.3.3-amzn-0`, all from the same S3 artifacts bucket. From Glue 5.0 onwards the README points at Apache Spark itself, version 3.5.4 for Glue 5.0, 3.5.6 for Glue 5.1 and 4.1.1 for Glue 6.0, from spark.apache.org.

Then you export `SPARK_HOME` to the extracted location, and the README gives the values per version:

    Glue version 2.0: export SPARK_HOME=/home/$USER/spark-2.4.3-bin-hadoop2.8
    Glue version 3.0: export SPARK_HOME=/home/$USER/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3
    Glue version 4.0: export SPARK_HOME=/home/$USER/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0
    Glue version 5.0: export SPARK_HOME=/home/$USER/spark-3.5.4-bin-hadoop3
    Glue version 5.1: export SPARK_HOME=/home/$USER/spark-3.5.6-bin-hadoop3
    Glue version 6.0: export SPARK_HOME=/home/$USER/spark-4.1.1-bin-hadoop3

Two details to note. The hadoop suffix changes between versions, so a stale `SPARK_HOME` pointing at a hadoop2 build breaks a job that expects hadoop3. And the URL in the README for the Glue 2.0 Spark distribution ends with `tgz1`, which looks like a typo in the documentation and will need fixing before that link works.

Three executables and one PATH assumption

With the environment in place, the README says you can run the executables in the `bin` directory to start a Glue shell or submit a Glue Spark application:

    Glue shell: ./bin/gluepyspark
    Glue submit: ./bin/gluesparksubmit
    pytest: ./bin/gluepytest

The first two are the interactive and batch entry points, and the third is how you run the tests. The README attaches one caveat to that last one: the `gluepytest` script assumes that the pytest module is installed and available in the `PATH` env variable. That is a small thing, and it is the kind of small thing that costs twenty minutes when the failure message is a missing file rather than a missing dependency.

The scripts are what make the library usable in a normal development loop. `gluepyspark` gives you a REPL with the Glue context available, which is where you check that a schema inference behaves as you expect, and `gluesparksubmit` runs a job the way the service would run it.

The `bin` directory is not a thin wrapper around a console script. The README describes it as hosting several executables that allow you to run the Python library locally, which is how a library that needs a specific jar and a specific Spark build can present a single command.

Glue 6.0 rejects abbreviated job arguments

The most recent release notes describe a breaking change worth understanding before you upgrade. `getResolvedOptions` parses the arguments your job was invoked with, and previously an abbreviated argument name was accepted as long as it was an unambiguous prefix of a declared option, so passing `--additional-p` would resolve to `--additional-python-modules`. As of Glue 6.0, abbreviations are rejected and raise `GlueArgumentError`.

The fix in your job is to use the full option name:

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'additional-python-modules'])

And if you depend on abbreviated names, you can opt back in explicitly:

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'additional-python-modules'], allow_abbrev=True)

This is a small change in the code and a large one in what it implies for a team. Any job, script or wrapper that was passing a prefix will now fail at argument resolution, and the failure appears in the job logs rather than at build time. The opt-in flag means you can stage the change, but only if you know it is there, which is the reason it is in the release notes rather than in a deprecation warning.

The other half of the same note is the branch move. If your team develops against Glue 5.1, the code you have been working on is now on the `glue-5.1` branch, and `main` has moved to Glue 6.0 with a newer Spark and a newer Python.

The Amazon Software License, not an open source licence

The licensing paragraph is short and decisive. The libraries in this repository are licensed under the Amazon Software License, and they may not be used except in compliance with the License, a copy of which is included in the LICENSE file. The repository metadata classifies the licence as one GitHub cannot categorise, and the root carries `LICENSE.txt`, `NOTICE.txt` and a `THIRD-PARTY-LICENSES` directory, plus a `SECURITY.md`.

That is the constraint that decides adoption for anyone outside AWS. The code is readable and the tooling is useful, but the grant is conditional, so a team that requires an OSI approved licence for its dependencies, or that needs to vendor the code into a product without a commercial agreement, has to settle that question before the rest of the setup matters. This is not legal advice, and the LICENSE file is the document to read.

Two packaging details are worth a look while you are in the repository. The `pyproject.toml` declares the project as `AWSGlueDataplanePython`, requires Python `>=3.13,<3.14` on the current main branch, and takes its version dynamically through setuptools-scm. The older `setup.py` names the distribution `aws-glue-libs` and sets version `6.0.0` with the licence `Amazon Software License 1.0`. Two different names for the same artefact, one modern and one legacy, is a small trap when you are working out what to install locally.

The maintenance picture is the last piece. The repository is not archived, the last push was on 2026-10-21, and the `main` branch is tracking the current Glue release while the older Glue versions stay on their own branches.

Editorial conclusion

Adopt aws-glue-libs when you write AWS Glue PySpark batch jobs and want to run and test them on a laptop against a Spark distribution that matches your production Glue version. Do not adopt it as a general Spark library, and do not assume it is open source, because the code is under the Amazon Software License and may be used only in compliance with it. Verify first that you are on the branch matching your Glue version, since the default `main` is now Glue 6.0 on Python 3.13 while older Glue versions live on their own branches, that `SPARK_HOME` points at the distribution for that version, and that your job arguments use full option names, because Glue 6.0 rejects the abbreviated names earlier releases accepted.

Frequently asked questions

Official sources

1. awslabs/aws-glue-libs on GitHub
2. Issues
3. README
4. Releases