Fix code language

This commit is contained in:
Hadley Wickham
2022-11-18 11:26:25 -06:00
parent 69b4597f3b
commit 868a35ca71
29 changed files with 912 additions and 907 deletions

View File

@@ -27,7 +27,7 @@ Prerequisites</h2>
<p>In this chapter, well focus on tools provided by dplyr and purrr, both core members of the tidyverse. Youve seen dplyr before, but <a href="http://purrr.tidyverse.org/">purrr</a> is new. Were going to use just a couple of purrr functions from in this chapter, but its a great package to explore as you improve your programming skills.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">library(tidyverse)</pre>
<pre data-type="programlisting" data-code-language="r">library(tidyverse)</pre>
</div>
</section>
</section>
@@ -37,7 +37,7 @@ Prerequisites</h2>
Modifying multiple columns</h1>
<p>Imagine you have this simple tibble and you want to count the number of observations and compute the median of every column.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df &lt;- tibble(
<pre data-type="programlisting" data-code-language="r">df &lt;- tibble(
a = rnorm(10),
b = rnorm(10),
c = rnorm(10),
@@ -46,7 +46,7 @@ Modifying multiple columns</h1>
</div>
<p>You could do it with copy-and-paste:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df |&gt; summarise(
<pre data-type="programlisting" data-code-language="r">df |&gt; summarise(
n = n(),
a = median(a),
b = median(b),
@@ -60,7 +60,7 @@ Modifying multiple columns</h1>
</div>
<p>That breaks our rule of thumb to never copy and paste more than twice, and you can imagine that this will get very tedious if you have tens or even hundreds of columns. Instead you can use <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df |&gt; summarise(
<pre data-type="programlisting" data-code-language="r">df |&gt; summarise(
n = n(),
across(a:d, median),
)
@@ -78,7 +78,7 @@ Selecting columns with<code>.cols</code>
<p>The first argument to <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code>, <code>.cols</code>, selects the columns to transform. This uses the same specifications as <code><a href="https://dplyr.tidyverse.org/reference/select.html">select()</a></code>, <a href="#sec-select" data-type="xref">#sec-select</a>, so you can use functions like <code><a href="https://tidyselect.r-lib.org/reference/starts_with.html">starts_with()</a></code> and <code><a href="https://tidyselect.r-lib.org/reference/starts_with.html">ends_with()</a></code> to select columns based on their name.</p>
<p>There are two additional selection techniques that are particularly useful for <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code>: <code><a href="https://tidyselect.r-lib.org/reference/everything.html">everything()</a></code> and <code>where()</code>. <code><a href="https://tidyselect.r-lib.org/reference/everything.html">everything()</a></code> is straightforward: it selects every (non-grouping) column:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df &lt;- tibble(
<pre data-type="programlisting" data-code-language="r">df &lt;- tibble(
grp = sample(2, 10, replace = TRUE),
a = rnorm(10),
b = rnorm(10),
@@ -108,7 +108,7 @@ df |&gt;
<li>
<code>where(is.logical)</code> selects all logical columns.</li>
</ul><div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_types &lt;- tibble(
<pre data-type="programlisting" data-code-language="r">df_types &lt;- tibble(
x1 = 1:3,
x2 = runif(3),
y1 = sample(letters, 3),
@@ -138,7 +138,7 @@ Calling a single function</h2>
<p>The second argument to <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> defines how each column will be transformed. In simple cases, as above, this will be a single existing function. This is a pretty special feature of R: were passing one function (<code>median</code>, <code>mean</code>, <code>str_flatten</code>, …) to another function (<code>across</code>). This is one of the features that makes R a function programming language.</p>
<p>Its important to note that were passing this function to <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code>, so <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> can call it, not calling it ourselves. That means the function name should never be followed by <code>()</code>. If you forget, youll get an error:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df |&gt;
<pre data-type="programlisting" data-code-language="r">df |&gt;
group_by(grp) |&gt;
summarise(across(everything(), median()))
#&gt; Error in vapply(.x, .f, .mold, ..., USE.NAMES = FALSE): values must be length 1,
@@ -146,7 +146,7 @@ Calling a single function</h2>
</div>
<p>This error arises because youre calling the function with no input, e.g.:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">median()
<pre data-type="programlisting" data-code-language="r">median()
#&gt; Error in is.factor(x): argument "x" is missing, with no default</pre>
</div>
</section>
@@ -156,7 +156,7 @@ Calling a single function</h2>
Calling multiple functions</h2>
<p>In more complex cases, you might want to supply additional arguments or perform multiple transformations. Lets motivate this problem with a simple example: what happens if we have some missing values in our data? <code><a href="https://rdrr.io/r/stats/median.html">median()</a></code> propagates those missing values, giving us a suboptimal output:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">rnorm_na &lt;- function(n, n_na, mean = 0, sd = 1) {
<pre data-type="programlisting" data-code-language="r">rnorm_na &lt;- function(n, n_na, mean = 0, sd = 1) {
sample(c(rnorm(n - n_na, mean = mean, sd = 1), rep(NA, n_na)))
}
@@ -178,7 +178,7 @@ df_miss |&gt;
</div>
<p>It would be nice if we could pass along <code>na.rm = TRUE</code> to <code><a href="https://rdrr.io/r/stats/median.html">median()</a></code> to remove these missing values. To do so, instead of calling <code><a href="https://rdrr.io/r/stats/median.html">median()</a></code> directly, we need to create a new function that calls <code><a href="https://rdrr.io/r/stats/median.html">median()</a></code> with the desired arguments:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_miss |&gt;
<pre data-type="programlisting" data-code-language="r">df_miss |&gt;
summarise(
across(a:d, function(x) median(x, na.rm = TRUE)),
n = n()
@@ -190,7 +190,7 @@ df_miss |&gt;
</div>
<p>This is a little verbose, so R comes with a handy shortcut: for this sort of throw away, or <strong>anonymous</strong><span data-type="footnote">Anonymous, because we never explicitly gave it a name with <code>&lt;-</code>. Another term programmers use for this is “lambda function”.</span>, function you can replace <code>function</code> with <code>\</code><span data-type="footnote">In older code you might see syntax that looks like <code>~ .x + 1</code>. This is another way to write anonymous functions but it only works inside tidyverse functions and always uses the variable name <code>.x</code>. We now recommend the base syntax, <code>\(x) x + 1</code>.</span>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_miss |&gt;
<pre data-type="programlisting" data-code-language="r">df_miss |&gt;
summarise(
across(a:d, \(x) median(x, na.rm = TRUE)),
n = n()
@@ -198,7 +198,7 @@ df_miss |&gt;
</div>
<p>In either case, <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> effectively expands to the following code:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_miss |&gt;
<pre data-type="programlisting" data-code-language="r">df_miss |&gt;
summarise(
a = median(a, na.rm = TRUE),
b = median(b, na.rm = TRUE),
@@ -209,7 +209,7 @@ df_miss |&gt;
</div>
<p>When we remove the missing values from the <code><a href="https://rdrr.io/r/stats/median.html">median()</a></code>, it would be nice to know just how many values we were removing. We can find that out by supplying two functions to <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code>: one to compute the median and the other to count the missing values. You supply multiple functions by using a named list to <code>.fns</code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_miss |&gt;
<pre data-type="programlisting" data-code-language="r">df_miss |&gt;
summarise(
across(a:d, list(
median = \(x) median(x, na.rm = TRUE),
@@ -231,7 +231,7 @@ df_miss |&gt;
Column names</h2>
<p>The result of <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> is named according to the specification provided in the <code>.names</code> argument. We could specify our own if we wanted the name of the function to come first<span data-type="footnote">You cant currently change the order of the columns, but you could reorder them after the fact using <code><a href="https://dplyr.tidyverse.org/reference/relocate.html">relocate()</a></code> or similar.</span>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_miss |&gt;
<pre data-type="programlisting" data-code-language="r">df_miss |&gt;
summarise(
across(
a:d,
@@ -251,7 +251,7 @@ Column names</h2>
</div>
<p>The <code>.names</code> argument is particularly important when you use <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> with <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code>. By default the output of <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> is given the same names as the inputs. This means that <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> inside of <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code> will replace existing columns. For example, here we use <code><a href="https://dplyr.tidyverse.org/reference/coalesce.html">coalesce()</a></code> to replace <code>NA</code>s with <code>0</code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_miss |&gt;
<pre data-type="programlisting" data-code-language="r">df_miss |&gt;
mutate(
across(a:d, \(x) coalesce(x, 0))
)
@@ -266,7 +266,7 @@ Column names</h2>
</div>
<p>If youd like to instead create new columns, you can use the <code>.names</code> argument to give the output new names:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_miss |&gt;
<pre data-type="programlisting" data-code-language="r">df_miss |&gt;
mutate(
across(a:d, \(x) abs(x), .names = "{.col}_abs")
)
@@ -286,7 +286,7 @@ Column names</h2>
Filtering</h2>
<p><code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> is a great match for <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarise()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code> but its more awkward to use with <code><a href="https://dplyr.tidyverse.org/reference/filter.html">filter()</a></code>, because you usually combine multiple conditions with either <code>|</code> or <code>&amp;</code>. Its clear that <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> can help to create multiple logical columns, but then what? So dplyr provides two variants of <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> called <code><a href="https://dplyr.tidyverse.org/reference/across.html">if_any()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/across.html">if_all()</a></code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_miss |&gt; filter(is.na(a) | is.na(b) | is.na(c) | is.na(d))
<pre data-type="programlisting" data-code-language="r">df_miss |&gt; filter(is.na(a) | is.na(b) | is.na(c) | is.na(d))
#&gt; # A tibble: 3 × 4
#&gt; a b c d
#&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
@@ -317,7 +317,7 @@ df_miss |&gt; filter(if_all(a:d, is.na))
<code>across()</code> in functions</h2>
<p><code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> is particularly useful to program with because it allows you to operate on multiple columns. For example, <a href="https://twitter.com/_wurli/status/1571836746899283969">Jacob Scott</a> uses this little helper which wraps a bunch of lubridate function to expand all date columns into year, month, and day columns:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">library(lubridate)
<pre data-type="programlisting" data-code-language="r">library(lubridate)
#&gt; Loading required package: timechange
#&gt;
#&gt; Attaching package: 'lubridate'
@@ -347,7 +347,7 @@ df_date |&gt;
</div>
<p><code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> also makes it easy to supply multiple columns in a single argument because the first argument uses tidy-select; you just need to remember to embrace that argument, as we discussed in <a href="#sec-embracing" data-type="xref">#sec-embracing</a>. For example, this function will compute the means of numeric columns by default. But by supplying the second argument you can choose to summarize just selected columns:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">summarise_means &lt;- function(df, summary_vars = where(is.numeric)) {
<pre data-type="programlisting" data-code-language="r">summarise_means &lt;- function(df, summary_vars = where(is.numeric)) {
df |&gt;
summarise(
across({{ summary_vars }}, \(x) mean(x, na.rm = TRUE)),
@@ -390,7 +390,7 @@ Vs<code>pivot_longer()</code>
</h2>
<p>Before we go on, its worth pointing out an interesting connection between <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> and <code><a href="https://tidyr.tidyverse.org/reference/pivot_longer.html">pivot_longer()</a></code> (<a href="#sec-pivoting" data-type="xref">#sec-pivoting</a>). In many cases, you perform the same calculations by first pivoting the data and then performing the operations by group rather than by column. For example, take this multi-function summary:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df |&gt;
<pre data-type="programlisting" data-code-language="r">df |&gt;
summarise(across(a:d, list(median = median, mean = mean)))
#&gt; # A tibble: 1 × 8
#&gt; a_median a_mean b_median b_mean c_median c_mean d_median d_mean
@@ -399,7 +399,7 @@ Vs<code>pivot_longer()</code>
</div>
<p>We could compute the same values by pivoting longer and then summarizing:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">long &lt;- df |&gt;
<pre data-type="programlisting" data-code-language="r">long &lt;- df |&gt;
pivot_longer(a:d) |&gt;
group_by(name) |&gt;
summarise(
@@ -417,7 +417,7 @@ long
</div>
<p>And if you wanted the same structure as <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code> you could pivot again:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">long |&gt;
<pre data-type="programlisting" data-code-language="r">long |&gt;
pivot_wider(
names_from = name,
values_from = c(median, mean),
@@ -431,7 +431,7 @@ long
</div>
<p>This is a useful technique to know about because sometimes youll hit a problem thats not currently possible to solve with <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code>: when you have groups of columns that you want to compute with simultaneously. For example, imagine that our data frame contains both values and weights and we want to compute a weighted mean:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_paired &lt;- tibble(
<pre data-type="programlisting" data-code-language="r">df_paired &lt;- tibble(
a_val = rnorm(10),
a_wts = runif(10),
b_val = rnorm(10),
@@ -444,7 +444,7 @@ long
</div>
<p>Theres currently no way to do this with <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code><span data-type="footnote">Maybe there will be one day, but currently we dont see how.</span>, but its relatively straightforward with <code><a href="https://tidyr.tidyverse.org/reference/pivot_longer.html">pivot_longer()</a></code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_long &lt;- df_paired |&gt;
<pre data-type="programlisting" data-code-language="r">df_long &lt;- df_paired |&gt;
pivot_longer(
everything(),
names_to = c("group", ".value"),
@@ -488,7 +488,7 @@ Exercises</h2>
<li>
<p>Explain what each step of the pipeline in this function does. What special feature of <code>where()</code> are we taking advantage of?</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">show_missing &lt;- function(df, group_vars, summary_vars = everything()) {
<pre data-type="programlisting" data-code-language="r">show_missing &lt;- function(df, group_vars, summary_vars = everything()) {
df |&gt;
group_by(pick({{ group_vars }})) |&gt;
summarise(
@@ -508,14 +508,14 @@ nycflights13::flights |&gt; show_missing(c(year, month, day))</pre>
Reading multiple files</h1>
<p>In the previous section, you learned how to use <code><a href="https://dplyr.tidyverse.org/reference/across.html">dplyr::across()</a></code> to repeat a transformation on multiple columns. In this section, youll learn how to use <code><a href="https://purrr.tidyverse.org/reference/map.html">purrr::map()</a></code> to do something to every file in a directory. Lets start with a little motivation: imagine you have a directory full of excel spreadsheets<span data-type="footnote">If you instead had a directory of csv files with the same format, you can use the technique from <a href="#sec-readr-directory" data-type="xref">#sec-readr-directory</a>.</span> you want to read. You could do it with copy and paste:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">data2019 &lt;- readxl::read_excel("data/y2019.xlsx")
<pre data-type="programlisting" data-code-language="r">data2019 &lt;- readxl::read_excel("data/y2019.xlsx")
data2020 &lt;- readxl::read_excel("data/y2020.xlsx")
data2021 &lt;- readxl::read_excel("data/y2021.xlsx")
data2022 &lt;- readxl::read_excel("data/y2022.xlsx")</pre>
</div>
<p>And then use <code><a href="https://dplyr.tidyverse.org/reference/bind_rows.html">dplyr::bind_rows()</a></code> to combine them all together:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">data &lt;- bind_rows(data2019, data2020, data2021, data2022)</pre>
<pre data-type="programlisting" data-code-language="r">data &lt;- bind_rows(data2019, data2020, data2021, data2022)</pre>
</div>
<p>You can imagine that this would get tedious quickly, especially if you had hundreds of files, not just four. The following sections show you how to automate this sort of task. There are three basic steps: use <code><a href="https://rdrr.io/r/base/list.files.html">list.files()</a></code> to list all the files in a directory, then use <code><a href="https://purrr.tidyverse.org/reference/map.html">purrr::map()</a></code> to read each of them into a list, then use <code><a href="https://purrr.tidyverse.org/reference/list_c.html">purrr::list_rbind()</a></code> to combine them into a single data frame. Well then discuss how you can handle situations of increasing heterogeneity, where you cant do exactly the same thing to every file.</p>
@@ -528,7 +528,7 @@ Listing files in a directory</h2>
<li><p><code>full.names</code> determines whether or not the directory name should be included in the output. You almost always want this to be <code>TRUE</code>.</p></li>
</ul><p>To make our motivating example concrete, this book contains a folder with 12 excel spreadsheets containing data from the gapminder package. Each file contains one years worth of data for 142 countries. We can list them all with the appropriate call to <code><a href="https://rdrr.io/r/base/list.files.html">list.files()</a></code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">paths &lt;- list.files("data/gapminder", pattern = "[.]xlsx$", full.names = TRUE)
<pre data-type="programlisting" data-code-language="r">paths &lt;- list.files("data/gapminder", pattern = "[.]xlsx$", full.names = TRUE)
paths
#&gt; [1] "data/gapminder/1952.xlsx" "data/gapminder/1957.xlsx"
#&gt; [3] "data/gapminder/1962.xlsx" "data/gapminder/1967.xlsx"
@@ -552,7 +552,7 @@ gapminder_2007 &lt;- readxl::read_excel("data/gapminder/2007.xlsx")</pre>
</div>
<p>But putting each sheet into its own variable is going to make it hard to work with them a few steps down the road. Instead, theyll be easier to work with if we put them into a single object. A list is the perfect tool for this job:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">files &lt;- list(
<pre data-type="programlisting" data-code-language="r">files &lt;- list(
readxl::read_excel("data/gapminder/1952.xlsx"),
readxl::read_excel("data/gapminder/1957.xlsx"),
readxl::read_excel("data/gapminder/1962.xlsx"),
@@ -562,7 +562,7 @@ gapminder_2007 &lt;- readxl::read_excel("data/gapminder/2007.xlsx")</pre>
</div>
<p>Now that you have these data frames in a list, how do you get one out? You can use <code>files[[i]]</code> to extract the i-th element:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">files[[3]]
<pre data-type="programlisting" data-code-language="r">files[[3]]
#&gt; # A tibble: 142 × 5
#&gt; country continent lifeExp pop gdpPercap
#&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
@@ -583,7 +583,7 @@ gapminder_2007 &lt;- readxl::read_excel("data/gapminder/2007.xlsx")</pre>
</h2>
<p>The code to collect those data frames in a list “by hand” is basically just as tedious to type as code that reads the files one-by-one. Happily, we can use <code><a href="https://purrr.tidyverse.org/reference/map.html">purrr::map()</a></code> to make even better use of our <code>paths</code> vector. <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> is similar to<code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code>, but instead of doing something to each column in a data frame, it does something to each element of a vector.<code>map(x, f)</code> is shorthand for:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">list(
<pre data-type="programlisting" data-code-language="r">list(
f(x[[1]]),
f(x[[2]]),
...,
@@ -592,7 +592,7 @@ gapminder_2007 &lt;- readxl::read_excel("data/gapminder/2007.xlsx")</pre>
</div>
<p>So we can use <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> get a list of 12 data frames:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">files &lt;- map(paths, readxl::read_excel)
<pre data-type="programlisting" data-code-language="r">files &lt;- map(paths, readxl::read_excel)
length(files)
#&gt; [1] 12
@@ -611,7 +611,7 @@ files[[1]]
<p>(This is another data structure that doesnt display particularly compactly with <code><a href="https://rdrr.io/r/utils/str.html">str()</a></code> so you might want to load into RStudio and inspect it with <code><a href="https://rdrr.io/r/utils/View.html">View()</a></code>).</p>
<p>Now we can use <code><a href="https://purrr.tidyverse.org/reference/list_c.html">purrr::list_rbind()</a></code> to combine that list of data frames into a single data frame:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">list_rbind(files)
<pre data-type="programlisting" data-code-language="r">list_rbind(files)
#&gt; # A tibble: 1,704 × 5
#&gt; country continent lifeExp pop gdpPercap
#&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
@@ -625,13 +625,13 @@ files[[1]]
</div>
<p>Or we could do both steps at once in pipeline:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">paths |&gt;
<pre data-type="programlisting" data-code-language="r">paths |&gt;
map(readxl::read_excel) |&gt;
list_rbind()</pre>
</div>
<p>What if we want to pass in extra arguments to <code>read_excel()</code>? We use the same technique that we used with <code><a href="https://dplyr.tidyverse.org/reference/across.html">across()</a></code>. For example, its often useful to peak at the first few row of the data with <code>n_max = 1</code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">paths |&gt;
<pre data-type="programlisting" data-code-language="r">paths |&gt;
map(\(path) readxl::read_excel(path, n_max = 1)) |&gt;
list_rbind()
#&gt; # A tibble: 12 × 5
@@ -654,7 +654,7 @@ Data in the path</h2>
<p>Sometimes the name of the file is itself data. In this example, the file name contains the year, which is not otherwise recorded in the individual files. To get that column into the final data frame, we need to do two things.</p>
<p>First, we name the vector of paths. The easiest way to do this is with the <code><a href="https://rlang.r-lib.org/reference/set_names.html">set_names()</a></code> function, which can take a function. Here we use <code><a href="https://rdrr.io/r/base/basename.html">basename()</a></code> to extract just the file name from the full path:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">paths |&gt; set_names(basename)
<pre data-type="programlisting" data-code-language="r">paths |&gt; set_names(basename)
#&gt; 1952.xlsx 1957.xlsx
#&gt; "data/gapminder/1952.xlsx" "data/gapminder/1957.xlsx"
#&gt; 1962.xlsx 1967.xlsx
@@ -670,13 +670,13 @@ Data in the path</h2>
</div>
<p>Those names are automatically carried along by all the map functions, so the list of data frames will have those same names:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">files &lt;- paths |&gt;
<pre data-type="programlisting" data-code-language="r">files &lt;- paths |&gt;
set_names(basename) |&gt;
map(readxl::read_excel)</pre>
</div>
<p>That makes this call to <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> shorthand for:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">files &lt;- list(
<pre data-type="programlisting" data-code-language="r">files &lt;- list(
"1952.xlsx" = readxl::read_excel("data/gapminder/1952.xlsx"),
"1957.xlsx" = readxl::read_excel("data/gapminder/1957.xlsx"),
"1962.xlsx" = readxl::read_excel("data/gapminder/1962.xlsx"),
@@ -686,7 +686,7 @@ Data in the path</h2>
</div>
<p>You can also use <code>[[</code> to extract elements by name:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">files[["1962.xlsx"]]
<pre data-type="programlisting" data-code-language="r">files[["1962.xlsx"]]
#&gt; # A tibble: 142 × 5
#&gt; country continent lifeExp pop gdpPercap
#&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
@@ -700,7 +700,7 @@ Data in the path</h2>
</div>
<p>Then we use the <code>names_to</code> argument to <code><a href="https://purrr.tidyverse.org/reference/list_c.html">list_rbind()</a></code> to tell it to save the names into a new column called <code>year</code> then use <code><a href="https://readr.tidyverse.org/reference/parse_number.html">readr::parse_number()</a></code> to extract the number from the string.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">paths |&gt;
<pre data-type="programlisting" data-code-language="r">paths |&gt;
set_names(basename) |&gt;
map(readxl::read_excel) |&gt;
list_rbind(names_to = "year") |&gt;
@@ -718,7 +718,7 @@ Data in the path</h2>
</div>
<p>In more complicated cases, there might be other variables stored in the directory name, or maybe the file name contains multiple bits of data. In that case, use <code><a href="https://rlang.r-lib.org/reference/set_names.html">set_names()</a></code> (without any arguments) to record the full path, and then use <code><a href="https://tidyr.tidyverse.org/reference/separate_wider_delim.html">tidyr::separate_wider_delim()</a></code> and friends to turn them into useful columns.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit"># NOTE: this chapter also depends on dev tidyr (in addition to dev purrr and dev dplyr)
<pre data-type="programlisting" data-code-language="r"># NOTE: this chapter also depends on dev tidyr (in addition to dev purrr and dev dplyr)
paths |&gt;
set_names() |&gt;
map(readxl::read_excel) |&gt;
@@ -743,7 +743,7 @@ paths |&gt;
Save your work</h2>
<p>Now that youve done all this hard work to get to a nice tidy data frame, its a great time to save your work:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">gapminder &lt;- paths |&gt;
<pre data-type="programlisting" data-code-language="r">gapminder &lt;- paths |&gt;
set_names(basename) |&gt;
map(readxl::read_excel) |&gt;
list_rbind(names_to = "year") |&gt;
@@ -762,7 +762,7 @@ Many simple iterations</h2>
<p>Here weve just loaded the data directly from disk, and were lucky enough to get a tidy dataset. In most cases, youll need to do some additional tidying, and you have two basic basic options: you can do one round of iteration with a complex function, or do a multiple rounds of iteration with simple functions. In our experience most folks reach first for one complex iteration, but youre often better by doing multiple simple iterations.</p>
<p>For example, imagine that you want to read in a bunch of files, filter out missing values, pivot, and then combine. One way to approach the problem is write a function that takes a file and does all those steps then call <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> once:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">process_file &lt;- function(path) {
<pre data-type="programlisting" data-code-language="r">process_file &lt;- function(path) {
df &lt;- read_csv(path)
df |&gt;
@@ -777,7 +777,7 @@ paths |&gt;
</div>
<p>Alternatively, you could perform each step of <code>process_file()</code> to every file:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">paths |&gt;
<pre data-type="programlisting" data-code-language="r">paths |&gt;
map(read_csv) |&gt;
map(\(df) df |&gt; filter(!is.na(id))) |&gt;
map(\(df) df |&gt; mutate(id = tolower(id))) |&gt;
@@ -787,7 +787,7 @@ paths |&gt;
<p>We recommend this approach because it stops you getting fixated on getting the first file right because moving on to the rest. By considering all of the data when doing tidying and cleaning, youre more likely to think holistically and end up with a higher quality result.</p>
<p>In this particular example, theres another optimization you could make, by binding all the data frames together earlier. Then you can rely on regular dplyr behavior:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">paths |&gt;
<pre data-type="programlisting" data-code-language="r">paths |&gt;
map(read_csv) |&gt;
list_rbind() |&gt;
filter(!is.na(id)) |&gt;
@@ -801,12 +801,12 @@ paths |&gt;
Heterogeneous data</h2>
<p>Unfortunately sometimes its not possible to go from <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> straight to <code><a href="https://purrr.tidyverse.org/reference/list_c.html">list_rbind()</a></code> because the data frames are so heterogeneous that <code><a href="https://purrr.tidyverse.org/reference/list_c.html">list_rbind()</a></code> either fails or yields a data frame thats not very useful. In that case, its still useful to start by loading all of the files:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">files &lt;- paths |&gt;
<pre data-type="programlisting" data-code-language="r">files &lt;- paths |&gt;
map(readxl::read_excel) </pre>
</div>
<p>Then a very useful strategy is to capture the structure of the data frames to data so that you can explore it using your data science skills. One way to do so is with this handy <code>df_types</code> function that returns a tibble with one row for each column:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df_types &lt;- function(df) {
<pre data-type="programlisting" data-code-language="r">df_types &lt;- function(df) {
tibble(
col_name = names(df),
col_type = map_chr(df, vctrs::vec_ptype_full),
@@ -839,7 +839,7 @@ df_types(nycflights13::flights)
</div>
<p>You can then apply this function all of the files, and maybe do some pivoting to make it easy to see where there are differences. For example, this makes it easy to verify that the gapminder spreadsheets that weve been working with are all quite homogeneous:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">files |&gt;
<pre data-type="programlisting" data-code-language="r">files |&gt;
map(df_types) |&gt;
list_rbind(names_to = "file_name") |&gt;
select(-n_miss) |&gt;
@@ -864,7 +864,7 @@ Handling failures</h2>
<p>Sometimes the structure of your data might be sufficiently wild that you cant even read all the files with a single command. And then youll encounter one of the downsides of map: it succeeds or fails as a whole. <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> will either successfully read all of the files in a directory or fail with an error, reading zero files. This is annoying: why does one failure prevent you from accessing all the other successes?</p>
<p>Luckily, purrr comes with a helper to tackle this problem: <code><a href="https://purrr.tidyverse.org/reference/possibly.html">possibly()</a></code>. <code><a href="https://purrr.tidyverse.org/reference/possibly.html">possibly()</a></code> is whats known as a function operator: it takes a function and returns a function with modified behavior. In particular, <code><a href="https://purrr.tidyverse.org/reference/possibly.html">possibly()</a></code> changes a function from erroring to returning a value that you specify:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">files &lt;- paths |&gt;
<pre data-type="programlisting" data-code-language="r">files &lt;- paths |&gt;
map(possibly(\(path) readxl::read_excel(path), NULL))
data &lt;- files |&gt; list_rbind()</pre>
@@ -872,7 +872,7 @@ data &lt;- files |&gt; list_rbind()</pre>
<p>This works particularly well here because <code><a href="https://purrr.tidyverse.org/reference/list_c.html">list_rbind()</a></code>, like many tidyverse functions, automatically ignores <code>NULL</code>s.</p>
<p>Now you have all the data that can be read easily, and its time to tackle the hard part of figuring out why some files failed load and what do to about it. Start by getting the paths that failed:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">failed &lt;- map_vec(files, is.null)
<pre data-type="programlisting" data-code-language="r">failed &lt;- map_vec(files, is.null)
paths[failed]
#&gt; character(0)</pre>
</div>
@@ -894,13 +894,13 @@ Writing to a database</h2>
<p>Sometimes when working with many files at once, its not possible to fit all your data into memory at once, and you cant do <code>map(files, read_csv)</code>. One approach to deal with this problem is to load your into a database so you can access just the bits you need with dbplyr.</p>
<p>If youre lucky, the database package youre using will provide a handy function that takes a vector of paths and loads them all into the database. This is the case with duckdbs <code>duckdb_read_csv()</code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">con &lt;- DBI::dbConnect(duckdb::duckdb())
<pre data-type="programlisting" data-code-language="r">con &lt;- DBI::dbConnect(duckdb::duckdb())
duckdb::duckdb_read_csv(con, "gapminder", paths)</pre>
</div>
<p>This would work well here, but we dont have csv files, instead we have excel spreadsheets. So were going to have to do it “by hand”. Learning to do it by hand will also help you when you have a bunch of csvs and the database that youre working with doesnt have one function that will load them all in.</p>
<p>We need to start by creating a table that will fill in with data. The easiest way to do this is by creating a template, a dummy data frame that contains all the columns we want, but only a sampling of the data. For the gapminder data, we can make that template by reading a single file and adding the year to it:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">template &lt;- readxl::read_excel(paths[[1]])
<pre data-type="programlisting" data-code-language="r">template &lt;- readxl::read_excel(paths[[1]])
template$year &lt;- 1952
template
#&gt; # A tibble: 142 × 6
@@ -916,12 +916,12 @@ template
</div>
<p>Now we can connect to the database, and use <code><a href="https://dbi.r-dbi.org/reference/dbCreateTable.html">DBI::dbCreateTable()</a></code> to turn our template into database table:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">con &lt;- DBI::dbConnect(duckdb::duckdb())
<pre data-type="programlisting" data-code-language="r">con &lt;- DBI::dbConnect(duckdb::duckdb())
DBI::dbCreateTable(con, "gapminder", template)</pre>
</div>
<p><code>dbCreateTable()</code> doesnt use the data in <code>template</code>, just the variable names and types. So if we inspect the <code>gapminder</code> table now youll see that its empty but it has the variables we need with the types we expect:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">con |&gt; tbl("gapminder")
<pre data-type="programlisting" data-code-language="r">con |&gt; tbl("gapminder")
#&gt; # Source: table&lt;gapminder&gt; [0 x 6]
#&gt; # Database: DuckDB 0.5.1 [root@Darwin 22.1.0:R 4.2.1/:memory:]
#&gt; # … with 6 variables: country &lt;chr&gt;, continent &lt;chr&gt;, lifeExp &lt;dbl&gt;,
@@ -929,7 +929,7 @@ DBI::dbCreateTable(con, "gapminder", template)</pre>
</div>
<p>Next, we need a function that takes a single file path, reads it into R, and adds the result to the <code>gapminder</code> table. We can do that by combining <code>read_excel()</code> with <code><a href="https://dbi.r-dbi.org/reference/dbAppendTable.html">DBI::dbAppendTable()</a></code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">append_file &lt;- function(path) {
<pre data-type="programlisting" data-code-language="r">append_file &lt;- function(path) {
df &lt;- readxl::read_excel(path)
df$year &lt;- parse_number(basename(path))
@@ -938,15 +938,15 @@ DBI::dbCreateTable(con, "gapminder", template)</pre>
</div>
<p>Now we need to call <code>append_csv()</code> once for each element of <code>paths</code>. Thats certainly possible with <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">paths |&gt; map(append_file)</pre>
<pre data-type="programlisting" data-code-language="r">paths |&gt; map(append_file)</pre>
</div>
<p>But we dont care about the output of <code>append_file()</code>, so instead of <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> its slightly nicer to use <code><a href="https://purrr.tidyverse.org/reference/map.html">walk()</a></code>. <code><a href="https://purrr.tidyverse.org/reference/map.html">walk()</a></code> does exactly the same thing as <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> but throws the output away:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">paths |&gt; walk(append_file)</pre>
<pre data-type="programlisting" data-code-language="r">paths |&gt; walk(append_file)</pre>
</div>
<p>Now we can see if we have all the data in our table:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">con |&gt;
<pre data-type="programlisting" data-code-language="r">con |&gt;
tbl("gapminder") |&gt;
count(year)
#&gt; # Source: SQL [?? x 2]
@@ -968,7 +968,7 @@ DBI::dbCreateTable(con, "gapminder", template)</pre>
Writing csv files</h2>
<p>The same basic principle applies if we want to write multiple csv files, one for each group. Lets imagine that we want to take the <code><a href="https://ggplot2.tidyverse.org/reference/diamonds.html">ggplot2::diamonds</a></code> data and save one csv file for each <code>clarity</code>. First we need to make those individual datasets. There are many ways you could do that, but theres one way we particularly like: <code><a href="https://dplyr.tidyverse.org/reference/group_nest.html">group_nest()</a></code>.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">by_clarity &lt;- diamonds |&gt;
<pre data-type="programlisting" data-code-language="r">by_clarity &lt;- diamonds |&gt;
group_nest(clarity)
by_clarity
@@ -985,7 +985,7 @@ by_clarity
</div>
<p>This gives us a new tibble with eight rows and two columns. <code>clarity</code> is our grouping variable and <code>data</code> is a list-column containing one tibble for each unique value of <code>clarity</code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">by_clarity$data[[1]]
<pre data-type="programlisting" data-code-language="r">by_clarity$data[[1]]
#&gt; # A tibble: 741 × 9
#&gt; carat cut color depth table price x y z
#&gt; &lt;dbl&gt; &lt;ord&gt; &lt;ord&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
@@ -999,7 +999,7 @@ by_clarity
</div>
<p>While were here, lets create a column that gives the name of output file, using <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code> and <code><a href="https://stringr.tidyverse.org/reference/str_glue.html">str_glue()</a></code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">by_clarity &lt;- by_clarity |&gt;
<pre data-type="programlisting" data-code-language="r">by_clarity &lt;- by_clarity |&gt;
mutate(path = str_glue("diamonds-{clarity}.csv"))
by_clarity
@@ -1016,7 +1016,7 @@ by_clarity
</div>
<p>So if we were going to save these data frames by hand, we might write something like:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">write_csv(by_clarity$data[[1]], by_clarity$path[[1]])
<pre data-type="programlisting" data-code-language="r">write_csv(by_clarity$data[[1]], by_clarity$path[[1]])
write_csv(by_clarity$data[[2]], by_clarity$path[[2]])
write_csv(by_clarity$data[[3]], by_clarity$path[[3]])
...
@@ -1024,7 +1024,7 @@ write_csv(by_clarity$by_clarity[[8]], by_clarity$path[[8]])</pre>
</div>
<p>This is a little different to our previous uses of <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> because there are two arguments that are changing, not just one. That means we need a new function: <code><a href="https://purrr.tidyverse.org/reference/map2.html">map2()</a></code>, which varies both the first and second arguments. And because we again dont care about the output, we want <code><a href="https://purrr.tidyverse.org/reference/map2.html">walk2()</a></code> rather than <code><a href="https://purrr.tidyverse.org/reference/map2.html">map2()</a></code>. That gives us:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">walk2(by_clarity$data, by_clarity$path, write_csv)</pre>
<pre data-type="programlisting" data-code-language="r">walk2(by_clarity$data, by_clarity$path, write_csv)</pre>
</div>
</section>
@@ -1033,7 +1033,7 @@ write_csv(by_clarity$by_clarity[[8]], by_clarity$path[[8]])</pre>
Saving plots</h2>
<p>We can take the same basic approach to create many plots. Lets first make a function that draws the plot we want:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">carat_histogram &lt;- function(df) {
<pre data-type="programlisting" data-code-language="r">carat_histogram &lt;- function(df) {
ggplot(df, aes(carat)) + geom_histogram(binwidth = 0.1)
}
@@ -1044,7 +1044,7 @@ carat_histogram(by_clarity$data[[1]])</pre>
</div>
<p>Now we can use <code><a href="https://purrr.tidyverse.org/reference/map.html">map()</a></code> to create a list of many plots<span data-type="footnote">You can print <code>by_clarity$plot</code> to get a crude animation — youll get one plot for each element of <code>plots</code>. NOTE: this didnt happen for me.</span> and their eventual file paths:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">by_clarity &lt;- by_clarity |&gt;
<pre data-type="programlisting" data-code-language="r">by_clarity &lt;- by_clarity |&gt;
mutate(
plot = map(data, carat_histogram),
path = str_glue("clarity-{clarity}.png")
@@ -1052,7 +1052,7 @@ carat_histogram(by_clarity$data[[1]])</pre>
</div>
<p>Then use <code><a href="https://purrr.tidyverse.org/reference/map2.html">walk2()</a></code> with <code><a href="https://ggplot2.tidyverse.org/reference/ggsave.html">ggsave()</a></code> to save each plot:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">walk2(
<pre data-type="programlisting" data-code-language="r">walk2(
by_clarity$path,
by_clarity$plot,
\(path, plot) ggsave(path, plot, width = 6, height = 6)
@@ -1060,7 +1060,7 @@ carat_histogram(by_clarity$data[[1]])</pre>
</div>
<p>This is shorthand for:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">ggsave(by_clarity$path[[1]], by_clarity$plot[[1]], width = 6, height = 6)
<pre data-type="programlisting" data-code-language="r">ggsave(by_clarity$path[[1]], by_clarity$plot[[1]], width = 6, height = 6)
ggsave(by_clarity$path[[2]], by_clarity$plot[[2]], width = 6, height = 6)
ggsave(by_clarity$path[[3]], by_clarity$plot[[3]], width = 6, height = 6)
...