# The PySpark style guide argues against dot syntax on dataframes

> Palantir's PySpark style guide collects the practices that came up most often across its Spark repositories, and its first rule is to reference columns by name rather than through the dataframe variable, because the dot syntax produces long expressions, discourages short variable names and makes renames error prone. The guide is opinionated and says so.

**palantir/pyspark-style-guide** — This is a guide to PySpark code style presenting common situations and the associated best practices based on the most frequent recurring topics across the PySpark repos we've encountered.

- Repository: https://github.com/palantir/pyspark-style-guide
- Stars: 1,270 · Forks: 170
- Language: Python
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/palantir-pyspark-style-guide

## Why a style guide for PySpark at all

The guide opens by naming the problem. PySpark is a wrapper language that allows users to interface with an Apache Spark backend to quickly process data, and Spark can operate on massive datasets across a distributed network of servers, providing major performance and reliability benefits when utilized correctly. It also presents challenges even for experienced Python developers, because the PySpark syntax draws on the JVM heritage of Spark and therefore implements code patterns that may be unfamiliar. That is the argument for the document: the awkwardness is a consequence of the platform, and once you are inside it, the same awkwardness appears in every repository. The guide describes itself as opinionated, presents common situations with the associated best practices, and is based on the most frequent recurring topics across the PySpark repositories the authors have encountered. Beyond the PySpark specifics, it points at the Google PyGuide as a strong starting point for the general practices of clean code.

## The column reference rule, with its exceptions

The first rule is to prefer implicit column selection to direct access, except for disambiguation. The three forms are shown side by side:

```python
# bad
df = df.select(F.lower(df1.colA), F.upper(df2.colB))

# good
df = df.select(F.lower(F.col('colA')), F.upper(F.col('colB')))

# better - since Spark 3.0
df = df.select(F.lower('colA'), F.upper('colB'))
```

The reasoning is longer than the rule. Spark 3.0 greatly expanded the cases where a plain string works, so the third form is the default where it is possible. The dot syntax has limitations that lead to the preference for the second: a long dataframe variable name makes expressions unwieldy; a column name with a space or other unsupported character forces the bracket operator, which creates inconsistency since `df1['colA']` is as hard to write as `F.col('colA')`; column expressions built on the dataframe are not reusable and cannot be used to define abstract functions; and renaming a dataframe variable is error prone because every column reference has to be updated in tandem. The dot syntax also encourages short, non-descriptive dataframe names, which the authors consider harmful for maintainability.

## The spooky interaction argument

There is a second half to the column rule that is more philosophical than practical. By contrast, `F.col('colA')` will always reference a column designated `colA` in the dataframe being operated on, named `df` in that case. It does not require keeping track of other dataframes' states at all, so the code becomes more local and less susceptible to what the guide calls spooky interaction at a distance, which is often challenging to debug. The stated principle underneath it is that dataframes are containers for data, and descriptive names are a helpful way to quickly set expectations about what is contained within. The caveat section is precise about where the rule does not apply: in some contexts there may be access to columns from more than one dataframe with overlapping names, a common example being a join like `df.join(df2, on=(df.key == df2.key), how='left')`, and in those cases it is fine to reference columns by their dataframe directly. Dataframe aliases can also be used to disambiguate joins.

## Three expressions per block, and the redundancy you stop seeing

The second rule is about refactoring complex logical operations, which often sit inside `.filter()` or `F.when()` and need to be readable. The guide applies the same limit it uses for chaining functions, keeping logic expressions inside the same code block to three expressions at most, and if they grow longer it is often a sign the code can be simplified or extracted. The example is a single dense `F.when` condition that the guide breaks into named variables:

```python
# better
has_operator = ((F.col('originalOperator') != '') | (F.col('currentOperator') != ''))
delivery_date_passed = (F.datediff('deliveryDate_actual', 'current_date') < 0)
has_registration = (F.col('currentRegistration').rlike('.+'))
is_delivered = (F.col('prod_status') == 'Delivered')

F.when(is_delivered | (delivery_date_passed & (has_registration | has_operator)), 'In Service')
```

The point is not only readability. PySpark requires expressions to be wrapped in parentheses, and mixed with the parentheses that group the logic, that hurts readability badly enough to hide a redundant sub-expression that the original author did not notice. The final version of the example reduces the number of operations further by pulling the disjunction into its own `is_active` variable, and the guide notes that a reviewer now only has to visit the individual expressions if they suspect an error, and that each chunk is easy to unit test and to abstract into a function. The last duplication in the example is left as an exercise for the reader.

## Select statements are a schema contract

The third rule is to use `select` statements to specify a schema contract. Doing a select at the beginning of a PySpark transform, or before returning, is considered good practice, because the select states the contract with both the reader and the code about the expected dataframe schema for inputs and outputs, and any select should be seen as a cleaning operation preparing the dataframe for consumption by the next step in the transform. The rules for keeping them simple are quantitative: allow only one function from `spark.sql.function` per selected column, plus an optional `.alias()` to give it a meaningful name, and if there are more than three such uses in the same select, refactor it into a separate function such as `clean_<dataframe name>()` to encapsulate the operation. Expressions involving more than one dataframe, or conditional operations like `.when()`, are excluded from that simplification, since they cannot be expressed as a plain per-column function.

## A document, not a library

The repository is deliberately small. The top level holds the LICENSE, the README.md and a `src/` directory, and there are no GitHub releases, so the guide is versioned with the branch rather than with tags. The default branch is `develop`, and the last push was on 2026-10-02. The project description says it is a guide to PySpark code style presenting common situations and the associated best practices based on the most frequent recurring topics across the PySpark repos the authors have encountered. That framing is worth keeping in mind when you read it: these are the patterns that came up again and again in one organisation's codebases, not a catalogue of everything possible in Spark, and the selection reflects what a large engineering organisation running Spark on a schedule actually got stuck on.

## Conclusion

Adopt this guide if you maintain PySpark code in a team and want shared conventions that are argued rather than asserted, since each rule comes with a stated reason and a before and after example. Do not adopt it as a general Python style guide, because it assumes you are already inside the Spark model and it points to the Google PyGuide for the general practices. Verify first which Spark version you run, because the implicit column selection rule depends on what Spark 3.0 enabled, and check whether your own pipelines join dataframes where column names overlap, which is the documented exception to the main rule.

## FAQ

### What is Palantir's PySpark style guide?

It is an opinionated guide to PySpark code style that presents common situations with the associated best practices, based on the most frequent recurring topics across the PySpark repositories the authors have encountered.

### Should I use df.colA or F.col('colA') in PySpark?

The guide prefers referencing the column by name, and since Spark 3.0 that can be a plain string such as `F.lower('colA')`. Where the string form is not possible it recommends `F.col('colA')`, and only allows `df.colA` where columns from more than one dataframe overlap, such as in a join.

### How long should a complex PySpark logical expression be?

The guide keeps logic expressions inside the same code block to three at most, and treats a longer expression as a sign the code should be simplified or extracted into named variables, which also makes the pieces easier to unit test.

### Why does the guide recommend select statements?

A select at the beginning of a transform or before returning specifies the schema contract for inputs and outputs, and is treated as a cleaning operation for the next step. The guide limits each selected column to one `spark.sql.function` call plus an optional alias, and says to extract the select into a `clean_<dataframe name>()` function when more than three columns use one.

### Is the PySpark style guide a general Python style guide?

No. It focuses on PySpark specifics and points at the Google PyGuide for the general practices of clean code in PySpark repositories. The repository is developed on the `develop` branch and has no GitHub releases.

## Sources

- [Issues](https://github.com/palantir/pyspark-style-guide/issues)
- [License: MIT](https://github.com/palantir/pyspark-style-guide/blob/develop/LICENSE)
- [palantir/pyspark-style-guide on GitHub](https://github.com/palantir/pyspark-style-guide)
- [README](https://github.com/palantir/pyspark-style-guide/blob/develop/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/palantir-pyspark-style-guide