amazon-kinesis-client: the Java client that leases, checkpoints and reshards

Why a consumer needs more than a GetRecords call

The Amazon Kinesis Client Library for Java exists because reading a Kinesis Data Stream correctly is more work than issuing API calls. A stream is split into shards, shards are leased to consumers, and the set of shards changes over time. The library's job is to hide that bookkeeping from the record processing code.

The features list spells out what it takes on. Scalability comes from distributing the processing load across multiple workers, so you can scale in or out, manually or with auto-scaling, without handling load redistribution yourself. Load balancing is automatic across the available workers. Checkpointing is managed, so an application resumes from its last successfully processed position. Fault tolerance is built in, so processing continues when individual workers fail, and the delivery guarantee is at-least-once.

That last point is the one to keep in mind while reading the rest. At-least-once means a record can be delivered again after a failure, so the processing logic has to be idempotent. A library that quietly gives you exactly-once would be a different kind of tool.

Monitoring comes through Amazon CloudWatch at the consumer level, and the repository layout shows a conventional Java library with a pom.xml at the root, separate amazon-kinesis-client and amazon-kinesis-client-multilang modules, a checkstyle/ directory, a formatter/ directory and a docs/ folder.

Ordering is preserved by never processing a child shard early

Stream-level change is the piece of KCL behaviour that most often surprises people, and the README describes it precisely.

Kinesis shards can be split and merged when the volume of data in a stream changes. The client adapts to those changes, and it maintains ordering by making sure that child shards are processed only after their parent shard has been completed and checkpointed.

Read that as a scheduling constraint, not a detail. While a parent shard is still being processed, its children wait. If your processor is slow, or if it crashes and has to resume from a checkpoint, the children of that shard do not start early. Ordering across the whole stream is a consequence of that rule, and the cost of it is that a lagging shard holds back the ones that came from it.

The combination of the three behaviours is what gives KCL its semantics. Leases decide which worker owns which shard. Checkpoints record how far that worker got. The parent before child rule keeps the resulting order sensible even as the shard topology changes underneath the application.

All of this happens below your record processing code, which is the point of the library, and also the reason it is not a thin wrapper.

KCL 1.x is out of support, so 3.x is the only starting point

The most consequential line in the README is a lifecycle notice, and it is dated.

Amazon Kinesis Client Library 1.x reached end-of-support on January 30, 2026, and entered maintenance mode on April 17, 2025. During maintenance mode, AWS provides updates only for critical bug fixes and security issues, and major versions in maintenance mode do not receive new features or enhancements. The README recommends migrating to the latest versions.

The migration is not free. When moving from KCL 1.x to 3.x you need to update interfaces and security credential providers in your application, and the README links both the AWS blog announcement of the end of support and the Kinesis documentation for migrating from 1.x to 3.x.

If you are starting fresh, this is a non-issue. If you are running a 1.x application in production, then the credential provider change is the part to plan around, because that is a behavioural change in how the library obtains credentials rather than a compile error, and it sits in a code path that usually has no unit tests of its own.

The releases visible in the repository are all on the 3.x line: v3.5.1 from 2026-07-22, v3.5.2 from 2026-09-04 and v3.5.3 from 2026-09-28.

Building from source with Maven, and the two flags you will need

The README documents a Maven build, and it names the two properties that change what the build does.

The first is GPG signing. The documented command disables it:

mvn clean install -Dgpg.skip=true

That command does not run integration tests, which the README states explicitly. The second property controls unit tests:

# add the property -DskipUTs=true to the build

The normal way to consume the library is also Maven, as the README says the recommended way to use the KCL for Java is to consume it from Maven, which means a dependency declaration in your pom.xml rather than a vendored copy of the source.

The repository is a multi-module Maven build: the amazon-kinesis-client module holds the library, amazon-kinesis-client-multilang holds the daemon used for non-Java processors, META-INF and NOTICE.txt are present for packaging metadata, and there is a CHANGELOG.md plus CODE_OF_CONDUCT.md and CONTRIBUTING.md for the project side. A Travis CI badge sits at the top of the README and a.travis.yml file is in the tree, alongside a.github/ directory.

The library targets Java 1.8 or later, which the README lists as the minimum requirement, and the Apache-2.0 licence means no copyleft obligation when you embed it in a commercial service.

Integration tests create real AWS resources and read your default profile

The integration test story is where a new contributor's first afternoon can go wrong, and the README is unusually clear about it.

Running integration tests creates AWS resources, and they require valid AWS credentials. The library looks for a default AWS profile specified in your local .aws/credentials file, so if you have no profile there, the tests fail before they start rather than halfway through.

To run everything:

mvn verify -DskipITs=false

To run a single integration test class, the README shows how to name it:

mvn -Dit.test="BasicStreamConsumerIntegrationTest" -DskipITs=false verify

And to select which IAM user or role to run the tests as, you pass the profile name:

mvn -DskipITs=false -DawsProfile="<PROFILE_NAME>" verify

The default build skips these, since the unit test and integration test behaviour are separate switches. The practical advice for a team without AWS credentials is to rely on the unit tests and to treat the integration suite as something you run in an account you are willing to have resources created in, because the test code creates streams and other resources as part of its work.

MultiLangDaemon runs one subprocess per shard over a pipe

KCL natively supports Java, and the README explains how other languages are supported as well, through a Java based daemon called MultiLangDaemon.

The design is worth understanding. The daemon does all the heavy lifting, and it spawns a sub-process that in turn runs the record processor, which can be written in any language. The daemon and the processor sub-process talk to each other over STDIN and STDOUT using a defined protocol. There is a one to one correspondence among record processors, child processes and shards, so the parallelism of the Java client maps directly onto the processor you write.

For Python developers specifically, the project abstracts those implementation details away and exposes an interface so you can focus on the record processing logic in Python, described in the README as a kclpy interface. The stated goal is to make KCL language agnostic while keeping identical features and a similar parallel processing model across languages.

There is a related convenience on the producer side. When a producer uses the Kinesis Producer Library, the KCL integrates without additional effort: when it retrieves an aggregated Kinesis record made up of multiple KPL user records, it automatically invokes the KPL to extract the individual user records before returning them to you. If you use KPL, that deaggregation happens for you.

Release cadence, and the AWS SDK version window to avoid

The maintenance picture for the 3.x line is healthy on the evidence of the release list. v3.5.1 was published on 2026-07-22, v3.5.2 on 2026-09-04 and v3.5.3 on 2026-09-28, and the last push to the repository was on 2026-10-19. That is a patch release roughly every few weeks, which is a normal cadence for a library that has to track the AWS SDK underneath it.

That last part is also where the sharp edge is. The README carries a warning that you should not use AWS SDK for Java versions 2.27.19 to 2.27.23 with KCL 3.x, because problems can be encountered with that combination. The note is truncated in the visible text, so the specific symptom is not documented in the excerpt, but the guidance is unambiguous: keep the AWS SDK for Java outside that version range when you run KCL 3.x.

This is the real upgrade cost of the library. It is not the KCL version that breaks you, it is the interaction between the KCL version and the SDK version underneath it, and the README's warning exists precisely because that combination matrix is not something a dependency resolver can reason about.

The licensing is Apache-2.0 with a LICENSE.txt and NOTICE.txt in the repository root, so embedding the client in a proprietary service is straightforward.

Editorial conclusion

amazon-kinesis-client suits Java teams running a long-lived consumer against a Kinesis Data Stream, where lease coordination, checkpointing and automatic handling of shard splits and merges are worth more than direct control over the stream. It does not suit a stateless pull or a simple fan-out job, since KCL assumes a consumer group that leases shards, and it does not suit a team unwilling to pin the AWS SDK for Java version, since the README warns against specific SDK releases with KCL 3.x. Check three things before you adopt it: that your producer side is compatible, because the KCL automatically unpacks aggregated KPL records, that you have an AWS profile in .aws/credentials ready for the integration tests, and that you have read the shard split behaviour, since child shards are processed only after the parent shard is completed and checkpointed. The most recent release is v3.5.3 from 2026-09-28 and the last push was on 2026-10-19.

Frequently asked questions

Official sources

1. awslabs/amazon-kinesis-client on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases