athena-query-federation: teaching Athena to read your data sources

What the SDK actually lets you do

The Amazon Athena Query Federation SDK allows you to customize Amazon Athena with your own code, and the readme spells out the three ways that is used: to integrate with new data sources, to work with proprietary data formats, and to build in new user defined functions.

That is the whole value proposition, and it is worth being concrete about why it matters. Athena is a serverless query engine over data in object storage, and it is very good at that. But most organisations do not have all their data in object storage in a format Athena can read. The order records are in a Redis instance, the customer records are in a document database, the log events are in a log service, the shipping status is in a table store, the payment records are in a cluster behind a secure network. Athena's answer has traditionally been to extract the data into a table, and extraction is a pipeline to build, a schedule to run, a place for the data to land, and a delay before the data is visible.

Federation replaces the extraction with a query.

The constraint, stated in the same paragraph, is the important one. Initially these customizations will be limited to the parts of a query that occur during a table scan operation, but will eventually be expanded to include other parts of the query lifecycle using the same easy to understand interface.

So today the extension point is the table scan, which is exactly where you need it for a connector: a federated table is read row by row, and the connector's code produces those rows. What you cannot yet do is customise the parts of the plan that come before or after the scan, which is the honest boundary of the current design.

The federated query feature itself is available wherever Athena is supported, which is the qualification the readme puts at the top of the document.

The project is under the awslabs organisation, which is the usual home for the experimental and reference parts of a cloud platform's tooling, and it is Apache 2.0 licensed. The release history is dated by year and week number, with three releases across the second half of 2026, and the last push was on 2026-10-15.

Getting started, which is a deployment exercise

The short form of the getting started section is five steps, and the fact that the first two are about permissions and a console rather than about code is the clearest signal about what this project is.

First, ensure you have the proper permissions and policies to deploy and use the federated queries. That link is to the service documentation for the identity and access configuration, and it is step one because everything else fails without it.

Second, navigate to the serverless application repository in the console and search for the federation entries, with the instruction to check the box that shows entries requiring custom IAM roles. Then look for entries published by the Athena Federation author specifically, which is a way of saying: there are other things in the repository that match the search, and these are the ones from the project.

Third, deploy the application. Fourth, run a query against it, and the example query is the shortest possible proof that the thing works: a show databases statement against a Lambda function by name.

That fourth step is worth pausing on, because it shows the shape of the addressing. A federated source is referred to by a scheme and a name, so the function you deployed becomes a database, and the metadata query that every SQL engine supports is how you confirm the connection is alive. The scheme in the example is the one for a Lambda function, and the name is the one you gave it at deployment.

The rest of the documentation is a list of wiki pages, and the list is a good map of what you will need. There is a page on the available connectors, one on the federation features, one on how to build a connector or a user defined function, one on predicate pushdown, and then a set of pages that are the standard ones any project of this size accumulates: installation prerequisites, known limitations and open issues, common problems, a frequently asked questions page, and a page on gathering diagnostic information for support.

That last one is a good sign. A project that has a documented way to collect the information its maintainers need in order to debug your deployment is a project that expects to be run by people other than its authors, which is the point of releasing it.

The connector catalogue

The repository is a monorepo with a directory per connector, and the directory listing is the catalogue. Counting them gives around forty modules, and the range of systems is the argument for the approach.

The databases and warehouses: PostgreSQL, MySQL, the Oracle family, the SQL Server family, the Synapse variant, DB2 and its AS 400 variant, the Teradata and Vertica and SAP HANA line, the Redshift connector, the Saphana module, the TPC-DS module, and the JDBC connector, which is the generic escape hatch for anything with a driver.

The NoSQL and key value stores: the DocumentDB module, the DynamoDB module, the Neptune graph module, the MongoDB-compatible access through the document store, and the Redis module.

The file and object formats: the data lake generation two module for the storage service's table format, the Google Cloud Storage module, and the Cloudera, Hortonworks, and generic Hive and Impala modules for the Hadoop ecosystem.

The streaming and time series sources: the Kafka and MSK modules for the log service, the Timestream module for the time series database, and the CloudWatch and CloudWatch metrics modules.

The search and analytics engines: the Elasticsearch module, the ClickHouse module, the Google BigQuery module, and the Snowflake module.

And the two that are not connectors at all: the user defined functions module, and the example module, which is the quick start project.

Around those sit the SDK itself, a tools module for the SDK, an integration test module, a testing module with a name of its own, a validation testing directory, a tools directory, and a documentation directory.

The readme explains why the count is what it is. The project has written integrations with more than twenty databases, storage formats, and live APIs in order to refine this interface and balance flexibility with ease of use, and it expresses the hope that making the SDK and the initial set of connectors open source will allow the work to continue.

That is a revealing sentence. The connectors were not written to be published; they were written to try the interface out, and publishing them is the consequence. Which means the interface has been tested against a wide range of systems, and the awkward cases have already been found by somebody, and the connectors are the evidence.

That also explains the presence of a separate user defined functions module. A function and a connector share the same deployment model, since both are code Athena invokes, but they sit at different points in the query, and the split means the same machinery serves two different purposes.

The worked example, and what a federated query looks like

The readme's second worked example is the best available description of what the project is for, and it is worth following in detail.

The scenario is a hypothetical e-commerce company with a nine-part architecture. Payment processing runs in a secure VPC with transaction records in HBase on EMR. Redis holds the active orders so the processing engine can get at them fast. A document database holds the customer account data, the email addresses and the shipping addresses. The site itself runs on auto-scaling with the product catalogue in a managed relational store. Log events go to the log service. There is a write-once-read-many warehouse. Shipment tracking data is in a table store. A fleet of drivers does the last-mile delivery on IoT enabled tablets. And there is advertising conversion data from a third party.

Then the incident: customer service agents get calls about orders stuck in a weird state, some showing as pending although they have been delivered, others showing as delivered although they have not shipped. The wish is to run one query across the whole architecture to find the affected orders and see what they have in common.

The query does six things, and each one names the connector it uses. It grabs the active orders from Redis. It joins against the orders with warning or error events in the logs, using regular expression matching and extraction to pull the order identifier out of the message. It joins against the compute inventory to get the hostnames and the state of the processors that logged those events. It joins the document database for the customer contact details. It joins the table store for the shipping status and the tracking details. And it joins the HBase cluster for the payment status.

The shape of the SQL is the interesting part, because the sources are named the way a federated source is named.

     active_orders 
     AS (SELECT * 
         FROM   redis.redis_db.redis_customer_orders), 
     order_processors 
     AS (SELECT instanceid, 
                publicipaddress, 
                state.NAME 
         FROM   awscmdb.ec2.ec2_instances), 

A source is a scheme, a connector instance, and a table, and the document store and the log service are named the same way. The common table expressions pull from six different systems, and the joins between them are ordinary joins.

So a single statement now spans a key value store, a log service, a compute inventory, a document database, a table store, and a cluster behind a VPC. No extraction, no intermediate tables, and no pipeline to maintain. The cost is that every one of those is a live call at query time, which is the trade the reader has to make.

Predicate pushdown, and what the wiki is for

Among the wiki pages listed in the readme, one stands out as the one that tells you the project is thinking about performance rather than only about connectivity: a how-to page on predicate pushdown.

Pushdown is the idea of pushing the filter into the data source instead of pulling everything across and filtering afterwards. It is the single most important performance property of a federated query engine, because without it a query that selects one row from a million-row table transfers a million rows.

The fact that there is a dedicated page for it tells you two things. It tells you the feature exists, and it tells you the project considers it a thing users need to understand and configure, which usually means it does not always happen automatically. A pushdown page is frequently also a troubleshooting page, because the common question is why my filter is being ignored.

The presence of a known limitations and open issues page is the second signal. A project with that page is a project that has hit cases its own model does not cover, and it is honest about them rather than leaving you to discover them. For a project that has been developed against twenty different systems in order to refine the interface, that is exactly the page you would expect.

The rest of the wiki follows the pattern of a project with a support burden: a common problems page, a frequently asked questions page, installation prerequisites, and a page on gathering diagnostic information for support.

The Java documentation is published separately, which is a small but useful detail for a Java SDK, because it means the API documentation is browsable without a build. And there is a video linked as an introduction, which for a project whose model is spatial rather than textual is a reasonable choice: the architecture diagram in the readme is a picture, and the way the parts fit together is easier to show than to describe.

The readme's own framing of the interface is worth repeating, because it is the design goal in one phrase: the same easy to understand interface, extended to more of the query lifecycle. Every connector that has been written is evidence that the interface survives contact with a new system, and the second half of that sentence is the roadmap.

The container image, and the RDS certificate problem

The Dockerfile in the repository is worth reading, because most of it is about one problem that has nothing to do with Athena.

The image is built on the managed Java runtime for Lambda, with the Java version as a build argument that defaults to a specific build of a specific distribution. The tool options are also an argument, and the environment variable is set to point the trust store at the runtime's own certificate store, with a comment explaining that this is for a specific Java version and that the default trust store is always included to avoid SSL handshake problems.

Then the package manager is updated and two tools are installed: a scripting language and the TLS library. The scripting language is there for the certificate processing, and the security library is there for the command that reads the certificates.

The core of the file is the certificate handling, and it is a well known problem. The trust store of a slim runtime image does not necessarily contain the certificate authorities that a database service uses, and the fix is to download the bundle, split it into individual certificates, and import each one into the trust store.

The download is a single command that fetches the global certificate bundle from the service's own trust store endpoint and writes it into the task root directory.

The split is done with a scripting language, and the one-liner is worth understanding: it scans for the end of a certificate marker, increments a counter, and writes each chunk to its own file.

Then the import loop runs for each certificate. It extracts an alias from the certificate's subject by parsing the human readable form, imports the file with the key tool under that alias with the store password, and removes the intermediate file. After the loop, the combined bundle is deleted, and a final step lists the trust store contents, with a comment saying that the list is there for verification.

That last step is the one that shows the author's habits. A build that imports a hundred certificates into a trust store and fails silently is a build that fails in production and nowhere else. Listing the aliases after the import, and keeping the command in the image so it can be run again, means the failure is visible at build time.

The file also shows the shape of the deployment. The image is the Lambda function's container image, the environment is set once in the build, and the whole thing is designed to be immutable: the trust store is modified in the image at build time rather than at run time.

Editorial conclusion

athena-query-federation fits an organisation whose data is spread across systems Athena cannot query, and whose analysts would rather not extract it first. It does not fit a workload that would be better served by a copy into a table, because every federation call is a network round trip to a live system. Verify first that the operations you need happen during the table scan phase, because that is where the customisations are currently supported, that the connector you need exists or that you can write one against the same interface, and that you have the permissions and the custom role the deployment expects, since the serverless application repository entries have to be enabled deliberately.

Frequently asked questions

Official sources

1. awslabs/aws-athena-query-federation on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases