Re-render book for O'Reilly
This commit is contained in:
Hadley Wickham
@@ -4,11 +4,20 @@
|
||||
<h1>
|
||||
Introduction</h1>
|
||||
<p>Numeric vectors are the backbone of data science, and you’ve already used them a bunch of times earlier in the book. Now it’s time to systematically survey what you can do with them in R, ensuring that you’re well situated to tackle any future problem involving numeric vectors.</p>
|
||||
<p>We’ll start by giving you a couple of tools to make numbers if you have strings, and then going into a little more detail of <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code>. Then we’ll dive into various numeric transformations that pair well with <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code>, including more general transformations that can be applied to other types of vector, but are often used with numeric vectors. We’ll finish off by covering the summary functions that pair well with <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarise()</a></code> and show you how they can also be used with <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code>.</p>
|
||||
<p>We’ll start by giving you a couple of tools to make numbers if you have strings, and then going into a little more detail of <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code>. Then we’ll dive into various numeric transformations that pair well with <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code>, including more general transformations that can be applied to other types of vector, but are often used with numeric vectors. We’ll finish off by covering the summary functions that pair well with <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarize()</a></code> and show you how they can also be used with <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code>.</p>
|
||||
|
||||
<section id="prerequisites" data-type="sect2">
|
||||
<h2>
|
||||
Prerequisites</h2>
|
||||
<div data-type="important"><div class="callout-body d-flex">
|
||||
<div class="callout-icon-container">
|
||||
<i class="callout-icon"/>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<p>This chapter relies on features only found in dplyr 1.1.0, which is still in development. If you want to live on the edge, you can get the dev versions with <code>devtools::install_github("tidyverse/dplyr")</code>.</p></div>
|
||||
|
||||
<p>This chapter mostly uses functions from base R, which are available without loading any packages. But we still need the tidyverse because we’ll use these base R functions inside of tidyverse functions like <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/filter.html">filter()</a></code>. Like in the last chapter, we’ll use real examples from nycflights13, as well as toy examples made with <code><a href="https://rdrr.io/r/base/c.html">c()</a></code> and <code><a href="https://tibble.tidyverse.org/reference/tribble.html">tribble()</a></code>.</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">library(tidyverse)
|
||||
@@ -20,7 +29,7 @@ library(nycflights13)</pre>
|
||||
<section id="making-numbers" data-type="sect1">
|
||||
<h1>
|
||||
Making numbers</h1>
|
||||
<p>In most cases, you’ll get numbers already recorded in one of R’s numeric types: integer or double. In some cases, however, you’ll encounter them as strings, possibly because you’ve created them by pivoting from column headers or something has gone wrong in your data import process.</p>
|
||||
<p>In most cases, you’ll get numbers already recorded in one of R’s numeric types: integer or double. In some cases, however, you’ll encounter them as strings, possibly because you’ve created them by pivoting from column headers or because something has gone wrong in your data import process.</p>
|
||||
<p>readr provides two useful functions for parsing strings into numbers: <code><a href="https://readr.tidyverse.org/reference/parse_atomic.html">parse_double()</a></code> and <code><a href="https://readr.tidyverse.org/reference/parse_number.html">parse_number()</a></code>. Use <code><a href="https://readr.tidyverse.org/reference/parse_atomic.html">parse_double()</a></code> when you have numbers that have been written as strings:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">x <- c("1.2", "5.6", "1e3")
|
||||
@@ -53,7 +62,7 @@ Counts</h1>
|
||||
#> # … with 99 more rows</pre>
|
||||
</div>
|
||||
<p>(Despite the advice in <a href="#chp-workflow-style" data-type="xref">#chp-workflow-style</a>, we usually put <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code> on a single line because it’s usually used at the console for a quick check that a calculation is working as expected.)</p>
|
||||
<p>If you want to see the most common values add <code>sort = TRUE</code>:</p>
|
||||
<p>If you want to see the most common values, add <code>sort = TRUE</code>:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |> count(dest, sort = TRUE)
|
||||
#> # A tibble: 105 × 2
|
||||
@@ -68,11 +77,11 @@ Counts</h1>
|
||||
#> # … with 99 more rows</pre>
|
||||
</div>
|
||||
<p>And remember that if you want to see all the values, you can use <code>|> View()</code> or <code>|> print(n = Inf)</code>.</p>
|
||||
<p>You can perform the same computation “by hand” with <code><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarise()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/context.html">n()</a></code>. This is useful because it allows you to compute other summaries at the same time:</p>
|
||||
<p>You can perform the same computation “by hand” with <code><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarize()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/context.html">n()</a></code>. This is useful because it allows you to compute other summaries at the same time:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
group_by(dest) |>
|
||||
summarise(
|
||||
summarize(
|
||||
n = n(),
|
||||
delay = mean(arr_delay, na.rm = TRUE)
|
||||
)
|
||||
@@ -100,7 +109,7 @@ Counts</h1>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
group_by(dest) |>
|
||||
summarise(
|
||||
summarize(
|
||||
carriers = n_distinct(carrier)
|
||||
) |>
|
||||
arrange(desc(carriers))
|
||||
@@ -121,7 +130,7 @@ Counts</h1>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
group_by(tailnum) |>
|
||||
summarise(miles = sum(distance))
|
||||
summarize(miles = sum(distance))
|
||||
#> # A tibble: 4,044 × 2
|
||||
#> tailnum miles
|
||||
#> <chr> <dbl>
|
||||
@@ -153,7 +162,7 @@ Counts</h1>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
group_by(dest) |>
|
||||
summarise(n_cancelled = sum(is.na(dep_time)))
|
||||
summarize(n_cancelled = sum(is.na(dep_time)))
|
||||
#> # A tibble: 105 × 2
|
||||
#> dest n_cancelled
|
||||
#> <chr> <int>
|
||||
@@ -171,7 +180,7 @@ Counts</h1>
|
||||
<h2>
|
||||
Exercises</h2>
|
||||
<ol type="1"><li>How can you use <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code> to count the number rows with a missing value for a given variable?</li>
|
||||
<li>Expand the following calls to <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code> to instead use <code><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarise()</a></code>, and <code><a href="https://dplyr.tidyverse.org/reference/arrange.html">arrange()</a></code>:
|
||||
<li>Expand the following calls to <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code> to instead use <code><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarize()</a></code>, and <code><a href="https://dplyr.tidyverse.org/reference/arrange.html">arrange()</a></code>:
|
||||
<ol type="1"><li><p><code>flights |> count(dest, sort = TRUE)</code></p></li>
|
||||
<li><p><code>flights |> count(tailnum, wt = distance)</code></p></li>
|
||||
</ol></li>
|
||||
@@ -210,20 +219,20 @@ x * c(1, 2, 3)
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
filter(month == c(1, 2))
|
||||
#> # A tibble: 25,977 × 19
|
||||
#> year month day dep_time sched_…¹ dep_d…² arr_t…³ sched…⁴ arr_d…⁵ carrier
|
||||
#> <int> <int> <int> <int> <int> <dbl> <int> <int> <dbl> <chr>
|
||||
#> 1 2013 1 1 517 515 2 830 819 11 UA
|
||||
#> 2 2013 1 1 542 540 2 923 850 33 AA
|
||||
#> 3 2013 1 1 554 600 -6 812 837 -25 DL
|
||||
#> 4 2013 1 1 555 600 -5 913 854 19 B6
|
||||
#> 5 2013 1 1 557 600 -3 838 846 -8 B6
|
||||
#> 6 2013 1 1 558 600 -2 849 851 -2 B6
|
||||
#> # … with 25,971 more rows, 9 more variables: flight <int>, tailnum <chr>,
|
||||
#> # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
|
||||
#> # minute <dbl>, time_hour <dttm>, and abbreviated variable names
|
||||
#> # ¹sched_dep_time, ²dep_delay, ³arr_time, ⁴sched_arr_time, ⁵arr_delay</pre>
|
||||
#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
|
||||
#> <int> <int> <int> <int> <int> <dbl> <int> <int>
|
||||
#> 1 2013 1 1 517 515 2 830 819
|
||||
#> 2 2013 1 1 542 540 2 923 850
|
||||
#> 3 2013 1 1 554 600 -6 812 837
|
||||
#> 4 2013 1 1 555 600 -5 913 854
|
||||
#> 5 2013 1 1 557 600 -3 838 846
|
||||
#> 6 2013 1 1 558 600 -2 849 851
|
||||
#> # … with 25,971 more rows, and 11 more variables: arr_delay <dbl>,
|
||||
#> # carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
|
||||
#> # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
|
||||
#> # time_hour <dttm></pre>
|
||||
</div>
|
||||
<p>The code runs without error, but it doesn’t return what you want. Because of the recycling rules it finds flights in odd numbered rows that departed in January and flights in even numbered rows that departed in February. And unforuntately there’s no warning because <code>flights</code> has an even number of rows.</p>
|
||||
<p>The code runs without error, but it doesn’t return what you want. Because of the recycling rules it finds flights in odd numbered rows that departed in January and flights in even numbered rows that departed in February. And unfortunately there’s no warning because <code>flights</code> has an even number of rows.</p>
|
||||
<p>To protect you from this type of silent failure, most tidyverse functions use a stricter form of recycling that only recycles single values. Unfortunately that doesn’t help here, or in many other cases, because the key computation is performed by the base R function <code>==</code>, not <code><a href="https://dplyr.tidyverse.org/reference/filter.html">filter()</a></code>.</p>
|
||||
</section>
|
||||
|
||||
@@ -277,7 +286,7 @@ Modular arithmetic</h2>
|
||||
1:10 %% 3
|
||||
#> [1] 1 2 0 1 2 0 1 2 0 1</pre>
|
||||
</div>
|
||||
<p>Modular arithmetic is handy for the flights dataset, because we can use it to unpack the <code>sched_dep_time</code> variable into and <code>hour</code> and <code>minute</code>:</p>
|
||||
<p>Modular arithmetic is handy for the flights dataset, because we can use it to unpack the <code>sched_dep_time</code> variable into <code>hour</code> and <code>minute</code>:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
mutate(
|
||||
@@ -300,9 +309,9 @@ Modular arithmetic</h2>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
group_by(hour = sched_dep_time %/% 100) |>
|
||||
summarise(prop_cancelled = mean(is.na(dep_time)), n = n()) |>
|
||||
summarize(prop_cancelled = mean(is.na(dep_time)), n = n()) |>
|
||||
filter(hour > 1) |>
|
||||
ggplot(aes(hour, prop_cancelled)) +
|
||||
ggplot(aes(x = hour, y = prop_cancelled)) +
|
||||
geom_line(color = "grey50") +
|
||||
geom_point(aes(size = n))</pre>
|
||||
<div class="cell-output-display">
|
||||
@@ -323,13 +332,13 @@ Logarithms</h2>
|
||||
interest <- 1.05
|
||||
|
||||
money <- tibble(
|
||||
year = 2000 + 1:50,
|
||||
money = starting * interest^(1:50)
|
||||
year = 1:50,
|
||||
money = starting * interest ^ year
|
||||
)</pre>
|
||||
</div>
|
||||
<p>If you plot this data, you’ll get an exponential curve:</p>
|
||||
<p>If you plot this data, you’ll get an exponential curve showing how your money grows year by year at an interest rate of 1.05:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">ggplot(money, aes(year, money)) +
|
||||
<pre data-type="programlisting" data-code-language="r">ggplot(money, aes(x = year, y = money)) +
|
||||
geom_line()</pre>
|
||||
<div class="cell-output-display">
|
||||
<p><img src="numbers_files/figure-html/unnamed-chunk-22-1.png" width="576"/></p>
|
||||
@@ -337,15 +346,15 @@ money <- tibble(
|
||||
</div>
|
||||
<p>Log transforming the y-axis gives a straight line:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">ggplot(money, aes(year, money)) +
|
||||
<pre data-type="programlisting" data-code-language="r">ggplot(money, aes(x = year, y = money)) +
|
||||
geom_line() +
|
||||
scale_y_log10()</pre>
|
||||
<div class="cell-output-display">
|
||||
<p><img src="numbers_files/figure-html/unnamed-chunk-23-1.png" width="576"/></p>
|
||||
</div>
|
||||
</div>
|
||||
<p>This a straight line because a little algebra reveals that <code>log(money) = log(starting) + n * log(interest)</code>, which matches the pattern for a line, <code>y = m * x + b</code>. This is a useful pattern: if you see a (roughly) straight line after log-transforming the y-axis, you know that there’s underlying exponential growth.</p>
|
||||
<p>If you’re log-transforming your data with dplyr you have a choice of three logarithms provided by base R: <code><a href="https://rdrr.io/r/base/Log.html">log()</a></code> (the natural log, base e), <code><a href="https://rdrr.io/r/base/Log.html">log2()</a></code> (base 2), and <code><a href="https://rdrr.io/r/base/Log.html">log10()</a></code> (base 10). We recommend using <code><a href="https://rdrr.io/r/base/Log.html">log2()</a></code> or <code><a href="https://rdrr.io/r/base/Log.html">log10()</a></code>. <code><a href="https://rdrr.io/r/base/Log.html">log2()</a></code> is easy to interpret because difference of 1 on the log scale corresponds to doubling on the original scale and a difference of -1 corresponds to halving; whereas <code><a href="https://rdrr.io/r/base/Log.html">log10()</a></code> is easy to back-transform because (e.g) 3 is 10^3 = 1000.</p>
|
||||
<p>This a straight line because a little algebra reveals that <code>log10(money) = log10(interest) * year + log10(starting)</code>, which matches the pattern for a line, <code>y = m * x + b</code>. This is a useful pattern: if you see a (roughly) straight line after log-transforming the y-axis, you know that there’s underlying exponential growth.</p>
|
||||
<p>If you’re log-transforming your data with dplyr you have a choice of three logarithms provided by base R: <code><a href="https://rdrr.io/r/base/Log.html">log()</a></code> (the natural log, base e), <code><a href="https://rdrr.io/r/base/Log.html">log2()</a></code> (base 2), and <code><a href="https://rdrr.io/r/base/Log.html">log10()</a></code> (base 10). We recommend using <code><a href="https://rdrr.io/r/base/Log.html">log2()</a></code> or <code><a href="https://rdrr.io/r/base/Log.html">log10()</a></code>. <code><a href="https://rdrr.io/r/base/Log.html">log2()</a></code> is easy to interpret because a difference of 1 on the log scale corresponds to doubling on the original scale and a difference of -1 corresponds to halving; whereas <code><a href="https://rdrr.io/r/base/Log.html">log10()</a></code> is easy to back-transform because (e.g.) 3 is 10^3 = 1000.</p>
|
||||
<p>The inverse of <code><a href="https://rdrr.io/r/base/Log.html">log()</a></code> is <code><a href="https://rdrr.io/r/base/Log.html">exp()</a></code>; to compute the inverse of <code><a href="https://rdrr.io/r/base/Log.html">log2()</a></code> or <code><a href="https://rdrr.io/r/base/Log.html">log10()</a></code> you’ll need to use <code>2^</code> or <code>10^</code>.</p>
|
||||
</section>
|
||||
|
||||
@@ -383,7 +392,7 @@ floor(x)
|
||||
ceiling(x)
|
||||
#> [1] 124</pre>
|
||||
</div>
|
||||
<p>These functions don’t have a digits argument, so you can instead scale down, round, and then scale back up:</p>
|
||||
<p>These functions don’t have a <code>digits</code> argument, so you can instead scale down, round, and then scale back up:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r"># Round down to nearest two digits
|
||||
floor(x / 0.01) * 0.01
|
||||
@@ -439,7 +448,7 @@ cut(y, breaks = c(0, 5, 10, 15, 20))
|
||||
<p>See the documentation for other useful arguments like <code>right</code> and <code>include.lowest</code>, which control if the intervals are <code>[a, b)</code> or <code>(a, b]</code> and if the lowest interval should be <code>[a, b]</code>.</p>
|
||||
</section>
|
||||
|
||||
<section id="cumulative-and-rolling-aggregates" data-type="sect2">
|
||||
<section id="sec-cumulative-and-rolling-aggregates" data-type="sect2">
|
||||
<h2>
|
||||
Cumulative and rolling aggregates</h2>
|
||||
<p>Base R provides <code><a href="https://rdrr.io/r/base/cumsum.html">cumsum()</a></code>, <code><a href="https://rdrr.io/r/base/cumsum.html">cumprod()</a></code>, <code><a href="https://rdrr.io/r/base/cumsum.html">cummin()</a></code>, <code><a href="https://rdrr.io/r/base/cumsum.html">cummax()</a></code> for running, or cumulative, sums, products, mins and maxes. dplyr provides <code><a href="https://dplyr.tidyverse.org/reference/cumall.html">cummean()</a></code> for cumulative means. Cumulative sums tend to come up the most in practice:</p>
|
||||
@@ -477,7 +486,7 @@ Exercises</h2>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
filter(month == 1, day == 1) |>
|
||||
ggplot(aes(sched_dep_time, dep_delay)) +
|
||||
ggplot(aes(x = sched_dep_time, y = dep_delay)) +
|
||||
geom_point()
|
||||
#> Warning: Removed 4 rows containing missing values (`geom_point()`).</pre>
|
||||
<div class="cell-output-display">
|
||||
@@ -580,13 +589,95 @@ lead(x)
|
||||
</ul><p>You can lead or lag by more than one position by using the second argument, <code>n</code>.</p>
|
||||
</section>
|
||||
|
||||
<section id="consecutive-identifiers" data-type="sect2">
|
||||
<h2>
|
||||
Consecutive identifiers</h2>
|
||||
<p>Sometimes you want to start a new group every time some event occurs. For example, when you’re looking at website data, it’s common to want to break up events into sessions, where a session is defined as a gap of more than x minutes since the last activity.</p>
|
||||
<p>For example, imagine you have the times when someone visited a website:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">events <- tibble(
|
||||
time = c(0, 1, 2, 3, 5, 10, 12, 15, 17, 19, 20, 27, 28, 30)
|
||||
)</pre>
|
||||
</div>
|
||||
<p>And you’ve the time lag between the events, and figured out if there’s a gap that’s big enough to qualify:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">events <- events |>
|
||||
mutate(
|
||||
diff = time - lag(time, default = first(time)),
|
||||
gap = diff >= 5
|
||||
)
|
||||
events
|
||||
#> # A tibble: 14 × 3
|
||||
#> time diff gap
|
||||
#> <dbl> <dbl> <lgl>
|
||||
#> 1 0 0 FALSE
|
||||
#> 2 1 1 FALSE
|
||||
#> 3 2 1 FALSE
|
||||
#> 4 3 1 FALSE
|
||||
#> 5 5 2 FALSE
|
||||
#> 6 10 5 TRUE
|
||||
#> # … with 8 more rows</pre>
|
||||
</div>
|
||||
<p>But how do we go from that logical vector to something that we can <code><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by()</a></code>? <code><a href="https://rdrr.io/r/base/cumsum.html">cumsum()</a></code> from <a href="#sec-cumulative-and-rolling-aggregates" data-type="xref">#sec-cumulative-and-rolling-aggregates</a> comes to the rescue as each occurring gap, i.e., <code>gap</code> is <code>TRUE</code>, increments <code>group</code> by one (see <a href="#sec-numeric-summaries-of-logicals" data-type="xref">#sec-numeric-summaries-of-logicals</a> on the numerical interpretation of logicals):</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">events |> mutate(
|
||||
group = cumsum(gap)
|
||||
)
|
||||
#> # A tibble: 14 × 4
|
||||
#> time diff gap group
|
||||
#> <dbl> <dbl> <lgl> <int>
|
||||
#> 1 0 0 FALSE 0
|
||||
#> 2 1 1 FALSE 0
|
||||
#> 3 2 1 FALSE 0
|
||||
#> 4 3 1 FALSE 0
|
||||
#> 5 5 2 FALSE 0
|
||||
#> 6 10 5 TRUE 1
|
||||
#> # … with 8 more rows</pre>
|
||||
</div>
|
||||
<p>Another approach for creating grouping variables is <code><a href="https://dplyr.tidyverse.org/reference/consecutive_id.html">consecutive_id()</a></code>, which starts a new group every time one of its arguments changes. For example, inspired by <a href="https://stackoverflow.com/questions/27482712">this stackoverflow question</a>, imagine you have a data frame with a bunch of repeated values:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">df <- tibble(
|
||||
x = c("a", "a", "a", "b", "c", "c", "d", "e", "a", "a", "b", "b"),
|
||||
y = c(1, 2, 3, 2, 4, 1, 3, 9, 4, 8, 10, 199)
|
||||
)
|
||||
df
|
||||
#> # A tibble: 12 × 2
|
||||
#> x y
|
||||
#> <chr> <dbl>
|
||||
#> 1 a 1
|
||||
#> 2 a 2
|
||||
#> 3 a 3
|
||||
#> 4 b 2
|
||||
#> 5 c 4
|
||||
#> 6 c 1
|
||||
#> # … with 6 more rows</pre>
|
||||
</div>
|
||||
<p>You want to keep the first row from each repeated <code>x</code>. That’s easier to express with a combination of <code><a href="https://dplyr.tidyverse.org/reference/consecutive_id.html">consecutive_id()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/slice.html">slice_head()</a></code>:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">df |>
|
||||
group_by(id = consecutive_id(x)) |>
|
||||
slice_head(n = 1)
|
||||
#> # A tibble: 7 × 3
|
||||
#> # Groups: id [7]
|
||||
#> x y id
|
||||
#> <chr> <dbl> <int>
|
||||
#> 1 a 1 1
|
||||
#> 2 b 2 2
|
||||
#> 3 c 4 3
|
||||
#> 4 d 3 4
|
||||
#> 5 e 9 5
|
||||
#> 6 a 4 6
|
||||
#> # … with 1 more row</pre>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="exercises-2" data-type="sect2">
|
||||
<h2>
|
||||
Exercises</h2>
|
||||
<ol type="1"><li><p>Find the 10 most delayed flights using a ranking function. How do you want to handle ties? Carefully read the documentation for <code><a href="https://dplyr.tidyverse.org/reference/row_number.html">min_rank()</a></code>.</p></li>
|
||||
<li><p>Which plane (<code>tailnum</code>) has the worst on-time record?</p></li>
|
||||
<li><p>What time of day should you fly if you want to avoid delays as much as possible?</p></li>
|
||||
<li><p>What does <code>flights |> group_by(dest() |> filter(row_number() < 4)</code> do? What does <code>flights |> group_by(dest() |> filter(row_number(dep_delay) < 4)</code> do?</p></li>
|
||||
<li><p>What does <code>flights |> group_by(dest) |> filter(row_number() < 4)</code> do? What does <code>flights |> group_by(dest) |> filter(row_number(dep_delay) < 4)</code> do?</p></li>
|
||||
<li><p>For each destination, compute the total minutes of delay. For each flight, compute the proportion of the total delay for its destination.</p></li>
|
||||
<li>
|
||||
<p>Delays are typically temporally correlated: even once the problem that caused the initial delay has been resolved, later flights are delayed to allow earlier flights to leave. Using <code><a href="https://dplyr.tidyverse.org/reference/lead-lag.html">lag()</a></code>, explore how the average flight delay for an hour is related to the average delay for the previous hour.</p>
|
||||
@@ -594,7 +685,7 @@ Exercises</h2>
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
mutate(hour = dep_time %/% 100) |>
|
||||
group_by(year, month, day, hour) |>
|
||||
summarise(
|
||||
summarize(
|
||||
dep_delay = mean(dep_delay, na.rm = TRUE),
|
||||
n = n(),
|
||||
.groups = "drop"
|
||||
@@ -602,7 +693,7 @@ Exercises</h2>
|
||||
filter(n > 5)</pre>
|
||||
</div>
|
||||
</li>
|
||||
<li><p>Look at each destination. Can you find flights that are suspiciously fast? (i.e. flights that represent a potential data entry error). Compute the air time of a flight relative to the shortest flight to that destination. Which flights were most delayed in the air?</p></li>
|
||||
<li><p>Look at each destination. Can you find flights that are suspiciously fast (i.e. flights that represent a potential data entry error)? Compute the air time of a flight relative to the shortest flight to that destination. Which flights were most delayed in the air?</p></li>
|
||||
<li><p>Find all destinations that are flown by at least two carriers. Use those destinations to come up with a relative ranking of the carriers based on their performance for the same destination.</p></li>
|
||||
</ol></section>
|
||||
</section>
|
||||
@@ -610,23 +701,23 @@ Exercises</h2>
|
||||
<section id="numeric-summaries" data-type="sect1">
|
||||
<h1>
|
||||
Numeric summaries</h1>
|
||||
<p>Just using the counts, means, and sums that we’ve introduced already can get you a long way, but R provides many other useful summary functions. Here are a selection that you might find useful.</p>
|
||||
<p>Just using the counts, means, and sums that we’ve introduced already can get you a long way, but R provides many other useful summary functions. Here is a selection that you might find useful.</p>
|
||||
|
||||
<section id="center" data-type="sect2">
|
||||
<h2>
|
||||
Center</h2>
|
||||
<p>So far, we’ve mostly used <code><a href="https://rdrr.io/r/base/mean.html">mean()</a></code> to summarize the center of a vector of values. Because the mean is the sum divided by the count, it is sensitive to even just a few unusually high or low values. An alternative is to use the <code><a href="https://rdrr.io/r/stats/median.html">median()</a></code>, which finds a value that lies in the “middle” of the vector, i.e. 50% of the values is above it and 50% are below it. Depending on the shape of the distribution of the variable you’re interested in, mean or median might be a better measure of center. For example, for symmetric distributions we generally report the mean while for skewed distributions we usually report the median.</p>
|
||||
<p><a href="#fig-mean-vs-median" data-type="xref">#fig-mean-vs-median</a> compares the mean vs the median when looking at the hourly vs median departure delay. The median delay is always smaller than the mean delay because because flights sometimes leave multiple hours late, but never leave multiple hours early.</p>
|
||||
<p><a href="#fig-mean-vs-median" data-type="xref">#fig-mean-vs-median</a> compares the mean vs. the median when looking at the hourly vs. median departure delay. The median delay is always smaller than the mean delay because flights sometimes leave multiple hours late, but never leave multiple hours early.</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
group_by(year, month, day) |>
|
||||
summarise(
|
||||
summarize(
|
||||
mean = mean(dep_delay, na.rm = TRUE),
|
||||
median = median(dep_delay, na.rm = TRUE),
|
||||
n = n(),
|
||||
.groups = "drop"
|
||||
) |>
|
||||
ggplot(aes(mean, median)) +
|
||||
ggplot(aes(x = mean, y = median)) +
|
||||
geom_abline(slope = 1, intercept = 0, color = "white", size = 2) +
|
||||
geom_point()
|
||||
#> Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
|
||||
@@ -644,12 +735,12 @@ Center</h2>
|
||||
<section id="sec-min-max-summary" data-type="sect2">
|
||||
<h2>
|
||||
Minimum, maximum, and quantiles</h2>
|
||||
<p>What if you’re interested in locations other than the center? <code><a href="https://rdrr.io/r/base/Extremes.html">min()</a></code> and <code><a href="https://rdrr.io/r/base/Extremes.html">max()</a></code> will give you the largest and smallest values. Another powerful tool is <code><a href="https://rdrr.io/r/stats/quantile.html">quantile()</a></code> which is a generalization of the median: <code>quantile(x, 0.25)</code> will find the value of <code>x</code> that is greater than 25% of the values, <code>quantile(x, 0.5)</code> is equivalent to the median, and <code>quantile(x, 0.95)</code> will find a value that’s greater than 95% of the values.</p>
|
||||
<p>What if you’re interested in locations other than the center? <code><a href="https://rdrr.io/r/base/Extremes.html">min()</a></code> and <code><a href="https://rdrr.io/r/base/Extremes.html">max()</a></code> will give you the largest and smallest values. Another powerful tool is <code><a href="https://rdrr.io/r/stats/quantile.html">quantile()</a></code> which is a generalization of the median: <code>quantile(x, 0.25)</code> will find the value of <code>x</code> that is greater than 25% of the values, <code>quantile(x, 0.5)</code> is equivalent to the median, and <code>quantile(x, 0.95)</code> will find the value that’s greater than 95% of the values.</p>
|
||||
<p>For the <code>flights</code> data, you might want to look at the 95% quantile of delays rather than the maximum, because it will ignore the 5% of most delayed flights which can be quite extreme.</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
group_by(year, month, day) |>
|
||||
summarise(
|
||||
summarize(
|
||||
max = max(dep_delay, na.rm = TRUE),
|
||||
q95 = quantile(dep_delay, 0.95, na.rm = TRUE),
|
||||
.groups = "drop"
|
||||
@@ -675,7 +766,7 @@ Spread</h2>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
group_by(origin, dest) |>
|
||||
summarise(
|
||||
summarize(
|
||||
distance_sd = IQR(distance),
|
||||
n = n(),
|
||||
.groups = "drop"
|
||||
@@ -696,13 +787,13 @@ Distributions</h2>
|
||||
<p><a href="#fig-flights-dist" data-type="xref">#fig-flights-dist</a> shows the overall distribution of departure delays. The distribution is so skewed that we have to zoom in to see the bulk of the data. This suggests that the mean is unlikely to be a good summary and we might prefer the median instead.</p>
|
||||
<div>
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
ggplot(aes(dep_delay)) +
|
||||
ggplot(aes(x = dep_delay)) +
|
||||
geom_histogram(binwidth = 15)
|
||||
#> Warning: Removed 8255 rows containing non-finite values (`stat_bin()`).
|
||||
|
||||
flights |>
|
||||
filter(dep_delay < 120) |>
|
||||
ggplot(aes(dep_delay)) +
|
||||
ggplot(aes(x = dep_delay)) +
|
||||
geom_histogram(binwidth = 5)</pre>
|
||||
<div id="fig-flights-dist" class="cell quarto-layout-panel">
|
||||
<figure class="figure"><div class="quarto-layout-row quarto-layout-valign-top">
|
||||
@@ -719,14 +810,14 @@ flights |>
|
||||
</figure>
|
||||
</div>
|
||||
</div>
|
||||
<p/><figcaption class="figure-caption">Figure 13.3: The distribution of <code>dep_delay</code> appears highly skewed to the right in both histograms.</figcaption><p/>
|
||||
<p/><figcaption class="figure-caption">Figure 15.3: The distribution of <code>dep_delay</code> appears highly skewed to the right in both histograms.</figcaption><p/>
|
||||
</figure></div>
|
||||
</div>
|
||||
<p>It’s also a good idea to check that distributions for subgroups resemble the whole. <a href="#fig-flights-dist-daily" data-type="xref">#fig-flights-dist-daily</a> overlays a frequency polygon for each day. The distributions seem to follow a common pattern, suggesting it’s fine to use the same summary for each day.</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
filter(dep_delay < 120) |>
|
||||
ggplot(aes(dep_delay, group = interaction(day, month))) +
|
||||
ggplot(aes(x = dep_delay, group = interaction(day, month))) +
|
||||
geom_freqpoly(binwidth = 5, alpha = 1/5)</pre>
|
||||
<div class="cell-output-display">
|
||||
|
||||
@@ -735,18 +826,18 @@ flights |>
|
||||
</figure>
|
||||
</div>
|
||||
</div>
|
||||
<p>Don’t be afraid to explore your own custom summaries specifically tailored for the data that you’re working with. In this case, that might mean separately summarizing the flights that left early vs the flights that left late, or given that the values are so heavily skewed, you might try a log-transformation. Finally, don’t forget what you learned in <a href="#sec-sample-size" data-type="xref">#sec-sample-size</a>: whenever creating numerical summaries, it’s a good idea to include the number of observations in each group.</p>
|
||||
<p>Don’t be afraid to explore your own custom summaries specifically tailored for the data that you’re working with. In this case, that might mean separately summarizing the flights that left early vs. the flights that left late, or given that the values are so heavily skewed, you might try a log-transformation. Finally, don’t forget what you learned in <a href="#sec-sample-size" data-type="xref">#sec-sample-size</a>: whenever creating numerical summaries, it’s a good idea to include the number of observations in each group.</p>
|
||||
</section>
|
||||
|
||||
<section id="positions" data-type="sect2">
|
||||
<h2>
|
||||
Positions</h2>
|
||||
<p>There’s one final type of summary that’s useful for numeric vectors, but also works with every other type of value: extracting a value at specific position. You can do this with the base R <code>[</code> function, but we’re not going to cover it in detail until <a href="#sec-subset-many" data-type="xref">#sec-subset-many</a>, because it’s a very powerful and general function. For now we’ll introduce three specialized functions that you can use to extract values at a specified position: <code>first(x)</code>, <code>last(x)</code>, and <code>nth(x, n)</code>.</p>
|
||||
<p>There’s one final type of summary that’s useful for numeric vectors, but also works with every other type of value: extracting a value at a specific position. You can do this with the base R <code>[</code> function, but we’re not going to cover it in detail until <a href="#sec-subset-many" data-type="xref">#sec-subset-many</a>, because it’s a very powerful and general function. For now we’ll introduce three specialized functions that you can use to extract values at a specified position: <code>first(x)</code>, <code>last(x)</code>, and <code>nth(x, n)</code>.</p>
|
||||
<p>For example, we can find the first and last departure for each day:</p>
|
||||
<div class="cell">
|
||||
<pre data-type="programlisting" data-code-language="r">flights |>
|
||||
group_by(year, month, day) |>
|
||||
summarise(
|
||||
summarize(
|
||||
first_dep = first(dep_time),
|
||||
fifth_dep = nth(dep_time, 5),
|
||||
last_dep = last(dep_time)
|
||||
@@ -775,18 +866,18 @@ Positions</h2>
|
||||
filter(r %in% c(1, max(r)))
|
||||
#> # A tibble: 1,195 × 20
|
||||
#> # Groups: year, month, day [365]
|
||||
#> year month day dep_time sched_…¹ dep_d…² arr_t…³ sched…⁴ arr_d…⁵ carrier
|
||||
#> <int> <int> <int> <int> <int> <dbl> <int> <int> <dbl> <chr>
|
||||
#> 1 2013 1 1 517 515 2 830 819 11 UA
|
||||
#> 2 2013 1 1 2353 2359 -6 425 445 -20 B6
|
||||
#> 3 2013 1 1 2353 2359 -6 418 442 -24 B6
|
||||
#> 4 2013 1 1 2356 2359 -3 425 437 -12 B6
|
||||
#> 5 2013 1 2 42 2359 43 518 442 36 B6
|
||||
#> 6 2013 1 2 458 500 -2 703 650 13 US
|
||||
#> # … with 1,189 more rows, 10 more variables: flight <int>, tailnum <chr>,
|
||||
#> # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
|
||||
#> # minute <dbl>, time_hour <dttm>, r <int>, and abbreviated variable names
|
||||
#> # ¹sched_dep_time, ²dep_delay, ³arr_time, ⁴sched_arr_time, ⁵arr_delay</pre>
|
||||
#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
|
||||
#> <int> <int> <int> <int> <int> <dbl> <int> <int>
|
||||
#> 1 2013 1 1 517 515 2 830 819
|
||||
#> 2 2013 1 1 2353 2359 -6 425 445
|
||||
#> 3 2013 1 1 2353 2359 -6 418 442
|
||||
#> 4 2013 1 1 2356 2359 -3 425 437
|
||||
#> 5 2013 1 2 42 2359 43 518 442
|
||||
#> 6 2013 1 2 458 500 -2 703 650
|
||||
#> # … with 1,189 more rows, and 12 more variables: arr_delay <dbl>,
|
||||
#> # carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
|
||||
#> # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
|
||||
#> # time_hour <dttm>, r <int></pre>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -794,7 +885,7 @@ Positions</h2>
|
||||
<h2>
|
||||
With<code>mutate()</code>
|
||||
</h2>
|
||||
<p>As the names suggest, the summary functions are typically paired with <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarise()</a></code>. However, because of the recycling rules we discussed in <a href="#sec-recycling" data-type="xref">#sec-recycling</a> they can also be usefully paired with <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code>, particularly when you want do some sort of group standardization. For example:</p>
|
||||
<p>As the names suggest, the summary functions are typically paired with <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarize()</a></code>. However, because of the recycling rules we discussed in <a href="#sec-recycling" data-type="xref">#sec-recycling</a> they can also be usefully paired with <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code>, particularly when you want do some sort of group standardization. For example:</p>
|
||||
<ul><li>
|
||||
<code>x / sum(x)</code> calculates the proportion of a total.</li>
|
||||
<li>
|
||||
|
||||
Reference in New Issue
Block a user