tidylog: feedback for every dplyr and tidyr call

The idea, and the load order

The goal of tidylog is to provide feedback about dplyr and tidyr operations, and it does that with simple wrapper functions for almost all of the functions in both libraries: filter, mutate, select, full_join, group_by, and the rest.

The mechanism is the one you would guess, and it is worth being precise about because it explains the whole design. The library is loaded after the two it wraps, and its functions shadow them.

``` r library("dplyr") library("tidyr") library("tidylog", warn.conflicts = FALSE)

The order matters, and the third line has an argument that looks like boilerplate and is not. When you load a package whose function names collide with functions you already have, the loading library asks whether you want to replace the existing ones. The warning is suppressed here because replacing them is the entire point of the package, and being asked about it every session would be noise.

That is the whole trick, and it is why the package needs no changes to the code it wraps. The functions you call are the same functions, with the same arguments and the same return value. What has changed is that they now print a line describing what they did.

The result is a data analysis script that narrates itself.

r filtered <- filter(mtcars, cyl == 4) #> filter: removed 21 rows (66%), 11 rows remaining mutated <- mutate(mtcars, new_var = wt ** 2) #> mutate: new variable 'new_var' (double) with 29 unique values and 0% NA ```

The filter line says what fraction went and how many stayed, so a filter that removed two thirds of the data is visible at the moment it happens rather than in a bug report three steps later. The mutate line says the type of the new column, how many distinct values it has, and what proportion is missing.

Those two numbers, the unique values and the proportion missing, are the ones that catch errors. A column of type double with a single unique value is usually a mistake, and a column that is entirely missing is usually a join that did not do what you thought. Neither is obvious from the code.

The join report

Joins get more detail than the other operations, and the reason is obvious once you see what a join can do wrong.

``` r joined <- left_join(nycflights13::flights, nycflights13::weather, by = c("year", "month", "day", "origin", "hour", "time_hour")) #> left_join: added 9 columns (temp, dewp, humid, wind_dir, wind_speed, …) #> > rows only in nycflights13::flights 1,556 #> > rows only in nycflights13::weather ( 6,737) #> > matched rows 335,220 #> > ========= #> > rows total 336,776

The first line says how many columns were added. The block underneath is a small table: how many rows existed only on the left, how many only on the right, how many matched, and the total.

And the parentheses are the part that needs explaining, because they are the most useful thing in the report. Numbers in parentheses indicate rows that are not included in the result.

So in that example, the six thousand seven hundred and thirty-seven rows that exist only in the weather table are in parentheses because a left join drops them. They are still reported, because knowing that they exist is the whole point: the readme's own interpretation is that one thousand five hundred and fifty-six rows from the flights table do not have weather information, and that is a fact about the data that you would otherwise have to compute yourself.

The report also says whether any rows were duplicated, which the readme notes is often unintentional.

r x <- tibble(a = 1:2) y <- tibble(a = c(1, 1, 2), b = 1:3) # 1 is duplicated j <- left_join(x, y, by = "a") #> left_join: added one column (b) #> > rows only in x 0 #> > rows only in y (0) #> > matched rows 3 (includes duplicates) #> > === #> > rows total 3 ```

Two rows in the left table become three in the result, because the duplicated key matched twice. The report says so, in the matched rows line, and the fact that the matched count is greater than the number of rows on the left is itself the signal.

That is a class of bug that a plain join returns silently. The result looks plausible, the column count is right, and the row count is three instead of two, and the tidylog output is the only thing that says why.

The pipe example, where the point is a no-op

The long pipe example is the one the readme uses to make the case, and the interesting line is the last one.

``` r summary <- mtcars |> select(mpg, cyl, hp, am) |> filter(mpg > 15) |> mutate(mpg_round = round(mpg)) |> group_by(cyl, mpg_round, am) |> tally() |> filter(n >= 1) #> select: dropped 7 variables (disp, drat, wt, qsec, vs, …) #> filter: removed 6 rows (19%), 26 rows remaining #> mutate: new variable 'mpg_round' (double) with 15 unique values and 0% NA #> group_by: 3 grouping variables (cyl, mpg_round, am) #> tally: now 20 rows and 4 columns, 2 group variables remaining (cyl, mpg_round) #> filter (grouped): no rows removed ```

Every line of that report is useful, and two of them are the reason to read it.

The tally line says the result is now twenty rows and four columns, with two grouping variables remaining, which tells you the third grouping variable was consumed by the summary. That is a detail you would otherwise have to check in the documentation, and the report gives it to you at the moment it happens.

The last line is the one the readme points at. The final filter removed no rows, and the readme says it might have been accidental.

That is a perfect example of the bug this package exists to find. A filter that removes nothing is either correct, because the condition was already satisfied, or a typo, because the condition was never what you meant. The first case you do not want to think about again, and the second case you want to know about immediately. In a script with twenty steps, a line that says no rows removed is the one you scroll back up to.

The parenthetical in the label is also worth reading: it says the filter was grouped. The same function behaves differently on a grouped data frame, because the filter then applies within each group, and the report distinguishes the two cases so that a no-op in one is not confused with a no-op in the other.

So the report is not just a summary of counts. It is a description of what the operation did in the context it was applied, which is the thing that is actually hard to reconstruct by reading the code later.

The overhead, and the escape hatch

The benchmarks section is short and honest, and it is the part to read before you decide to leave the package loaded.

Tidylog adds a small overhead to each function call. The readme says this can be relevant for very large datasets, and especially for joins.

That is a reasonable place for the cost to be concentrated, because a join on a large table is the operation where the extra work is proportional to the size of the result, and the report itself has to summarise every row that was dropped, matched, or duplicated. The report is the cost.

And the escape hatch is one character of typing, which is the right design for it.

If you want to switch off tidylog for a single long-running command, you prefix the call with the namespace of the library you want, such as the namespaced form of the join function. The original function is still there, tidylog has not replaced it, it has shadowed it, and the namespaced call reaches past the shadow.

So the workflow is: leave the package loaded, read the reports, and when you hit the step that is taking too long, namespace that one call. There is a vignette in the package about the benchmarks, with the numbers in it.

What is not offered is a way to keep the reports and remove the cost, which is a reasonable thing to leave out, since the whole value of the package is the report.

The other thing worth noticing is that the overhead is per call rather than per data set, which means it hurts most in a script that calls a small function many times rather than in one that processes one very large table. A pipeline of fifty small transformations pays fifty times, and a single filter over ten million rows pays once. That is the opposite of the intuition that a big dataset is the expensive case, and it is worth knowing before you decide where to switch it off.

The catalogue of what gets reported

The more examples section in the readme is effectively the specification, and reading it tells you what the library knows about each operation.

The filtering group covers filter and its variant that takes a selection of variables, distinct, and the missing value remover. The outputs are consistently the same shape: how many rows went, as a percentage and a count, and how many remain. And the edge cases are reported distinctly. A filter that removes everything says so, a filter that removes nothing says that, and a filter applied to a selection of variables is labelled with the variant that was used.

The mutation group is where the library does the most work, and the outputs are correspondingly richer.

``` r a <- mutate(mtcars, new_var = 1) #> mutate: new variable 'new_var' (double) with one unique value and 0% NA b <- mutate(mtcars, new_var = runif(n())) #> mutate: new variable 'new_var' (double) with 32 unique values and 0% NA c <- mutate(mtcars, new_var = NA) #> mutate: new variable 'new_var' (logical) with one unique value and 100% NA

The type is reported, and it is the type that the data ended up with rather than the type you asked for. The third line is the good one: a column of missing values is a logical column, not a numeric one, and the report says so.

The variant that applies a function to a selection of variables reports each column separately, with the number of values changed and the number of new missing values.

r d <- mutate_at(mtcars, vars(mpg, gear, drat), round) #> mutate_at: changed 28 values (88%) of 'mpg' (0 new NAs) #> changed 31 values (97%) of 'drat' (0 new NAs)

And the conversions are reported as conversions, with the number of values that were lost in the process.

r f <- mutate(mtcars, am = as.ordered(am)) #> mutate: converted 'am' from double to ordered factor (0 new NA) ```

That last one catches a real and quiet bug. Converting a numeric column to a factor keeps the values but changes the type, and a downstream operation that expects a number now fails or, worse, silently coerces back. The report says the conversion happened, and how many values were lost if any were.

The package as a repository

The repository layout is a well organised R package, and one detail in it is worth a look.

The readme is generated from a file whose name says so in its first line: the readme is generated from the source document, and the instruction is to edit that file rather than the readme. The examples in the readme are therefore executed when the document is knitted, and their outputs are captured, which is why every example shows the printed report.

That is the right way to keep an example document honest, and it is a practice that is far more common in the R world than elsewhere, because the document that produces the readme is the document that runs the code.

The rest of the layout is conventional. There is a description file with the package metadata, a namespace file, the R source directory, the tests, the generated manual pages, the vignettes, a news file for the change log, a build ignore file, a licence, a contributing guide, and a coverage configuration. There is also a project file for the editor, a tools directory, and a file recording the comments from the archive check, which is the file a maintainer updates after each submission to the package archive.

The package is on the main R package repository, and the badges show the current version there, the download count, the result of the package check, and the coverage. The release history shows that most of the work happened in 2020, with a version in the first half of that year, and then a long gap and a version in 2024. So this is a project that found its shape quickly and has been maintained rather than developed since, which for a debugging tool is exactly the right pattern: the useful set of operations is fixed, and the work is keeping it working as the libraries it wraps change.

The one design decision that a reader should carry away is the namespace trick, because it is the reason the package is usable in a real script rather than only in a demonstration. The wrapper reports, and the original is one prefix away whenever the report is not what you want.

Editorial conclusion

tidylog fits a data analyst working in a script or a notebook who keeps being surprised by how many rows a filter removed, or who wants to see at a glance whether a join matched what they thought it would. It does not fit a production pipeline, because the overhead on every call is real and it is documented as such. Verify first which functions you want wrapped, since the wrappers are for almost all of the library but not all of it, that the report is not hiding a change you need, and that you use the namespaced call to bypass the wrapper in the one place in your script where the extra output would be noise.

Frequently asked questions

Official sources

1. elbersb/tidylog on GitHub
2. Issues
3. README
4. Releases