Often you have to deal with data that is fundamentally tree-like --- rather than a rectangular structure of rows and columns, you have items that with one or more children.
In this chapter, you'll learn the art of "rectangling", taking complex hierarchical data and turning it into a data frame that you can easily work with using the tools you learned earlier in the book.
We'll start by talking about lists, an new type of vector that makes hierarchical data possible.
Then you'll learn about three key functions for rectangling from tidyr: `tidyr::unnest_longer()`, `tidyr::unnest_wider()e`, and `tidyr::hoist()`.
Then see how these ideas apply to some real data from the repurrrsive package.
Finish off by talkign about JSON, source of many hierarchical dataset.
The viewer starts by showing just the top level of the list, but you can interactively expand any of the components to see more, as in @fig-view-expand-1.
You can do this as many times as needed and RStudio will also show you the subsetting code you need to access that element, as in @fig-view-expand-2.
We'll come back to how this code works in @sec-vector-subsetting.
This is a powerful idea because it allows you to store arbitrarily complex objects in a data frame; even things that wouldn't typically belong there.
This idea is used a lot in tidymodels, because it allows you to store things like models or resamples in a data frame.
And those things are carried along like any other column:
```{r}
df |>
filter(x == 1)
```
The default print method just displays a rough summary of the contents.
The list column could be arbitrarily complex, so there's no good way to print it.
If you want to see it, you'll need to pull the list-column out and apply of the techniques that you learned above:
```{r}
df |>
filter(x == 1) |>
pull(z) |>
str()
```
Similarly, if you `View()` a data frame in RStudio, you'll get the standard tabular view, which doesn't allow you to selectively expand list columns.
To explore those fields you'll need to `pull()` and view, e.g.
`View(pull(df, z))`
::: callout-note
## Base R
It's possible to put a list in a column of a `data.frame`, but it's a lot fiddlier.
List-columns are implicit in the definition of the data frame: a data frame is a named list of equal length vectors.
A list is a vector, so it's always been legitimate to use a list as a column of a data frame.
However, base R doesn't make it easy to create list-columns because `data.frame()`treats a list as a list of columns:
```{r}
data.frame(x = list(1:3, 3:5))
```
You can prevent `data.frame()` from doing this with `I()`, but the result doesn't print particularly well:
```{r}
data.frame(
x = I(list(1:3, 3:5)),
y = c("1, 2", "3, 4, 5")
)
```
Tibbles make it easier to work with list-columns because `tibble()` doesn't modify its inputs and the print method is designed with lists in mind.
:::
## Unnesting
Now that you've learned the basics of lists and how you can use them as a column of a data frame, lets start to see how you can turn them back into regular columns and rows so you can use them with the tidyverse functions you've already learned about.
We'll start with very simple sample data so you can get the idea of how things work, and then in the next section switch to more realistic examples.
Lists tend to come in two basic forms:
- A named list where every row has the same number of children with the same names.
- An unnamed list where the number of children varies from row to row.
The following code creates an example of each.
In `df1`, every element of list-column `y` has two elements named `a` and `b`.
If `df2`, the elements of list-column `y` are unnamed and vary in length.
These two cases correspond to two tools from tidyr: `unnest_wider()` and `unnest_longer()`.
Their suffixes have the same meaning as `pivot_wider()` and `pivot_longer()`: `_wider()` adds more columns and `_longer()` adds more rows.
If your situation isn't as clear cut as these cases, you'll still need to use one of `unnest_longer()` and `unnest_wider()`; you'll just need to do a bit more thinking and experimentation to figure out which one is best.
When each row has the same number of elements with the same names, like `df1`, it's natural to put each component into its own column with `unnest_wider()`:
By default, the names of the new columns come exclusively from the names of the list, but you can use the `names_sep` argument to request that they combine the original column with the new column.
As you'll learn in the next section, this is useful for disambiguating repeated names.
There are few other useful rectangling functions that we're not going to talk about here:
- `unnest_auto()` automatically picks between `unnest_longer()` and `unnest_wider()`based on the structure of the list-column. It's a great for rapid exploration, but I think it's ultimately a bad idea because it doesn't force you to understand how your data is structured, and makes your code harder to understand.
- `unnest()` modifies rows and columns simultaneously. It's useful when you have a list-column that contains a 2d structure like a data frame (which we often call a nested data frame), which we don't otherwise use in this book.
- `hoist()` allows you to reach into a deeply nested list and extract just the components that you need. It's mostly equivalent to repeated invocations of `unnest_wider()` + `select()` so you should read up on it if there's just a couple of important variables that you want to pull out, embedded in a bunch of data that you don't care about.
1. From time-to-time you encounter data frames with multiple list-columns with aligned values. For example, in the following data frame, the values of `y` and `z` are aligned (i.e. `y` and `z` will always have the same length within a row, and the first value of `y` corresponds to the first value of `z`). What happens if you apply two `unnest_longer()` calls to this data frame? How can you preserve the relationship between `x` and `y`? (Hint: carefully read the docs).
Now that you understand the basics of `unnest_wider()` and `unnest_longer()` lets use them to tackle some real rectangling challenges.
These challenges share the common feature that they're mostly just a sequence of multiple `unnest_wider()` and/or `unnest_longer()` calls, with a little dash of dplyr where needed.
In this chapter, we'll focus mostly on JSON data, since this is a common way that you'll encounter deeply nested hierarchical data.
JSON, short for javascript object notation, is a data format that grew out of the javascript programming language and has become an extremely common way of representing data.
(Fortunately, once you've learned the basic ideas with JSON, you'll be able to apply them to any hierarchical data structure that you might encounter in R, as long as it gets turned into a list. You can also use these skills to selectively extract parts of data structures created by other R packages.)
