course/r4ds
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix code language

Browse Source
This commit is contained in:
Hadley Wickham
2022-11-18 11:26:25 -06:00
parent 69b4597f3b
commit 868a35ca71
29 changed files with 912 additions and 907 deletions
Download Patch File Download Diff File Expand all files Collapse all files

100
oreilly/numbers.html
View File

@@ -11,7 +11,7 @@ Introduction</h1>
Prerequisites</h2>
<p>This chapter mostly uses functions from base R, which are available without loading any packages. But we still need the tidyverse because we’ll use these base R functions inside of tidyverse functions like <code><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/filter.html">filter()</a></code>. Like in the last chapter, we’ll use real examples from nycflights13, as well as toy examples made with <code><a href="https://rdrr.io/r/base/c.html">c()</a></code> and <code><a href="https://tibble.tidyverse.org/reference/tribble.html">tribble()</a></code>.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">library(tidyverse)
<pre data-type="programlisting" data-code-language="r">library(tidyverse)
library(nycflights13)</pre>
</div>
</section>
@@ -23,13 +23,13 @@ Making numbers</h1>
<p>In most cases, you’ll get numbers already recorded in one of R’s numeric types: integer or double. In some cases, however, you’ll encounter them as strings, possibly because you’ve created them by pivoting from column headers or something has gone wrong in your data import process.</p>
<p>readr provides two useful functions for parsing strings into numbers: <code><a href="https://readr.tidyverse.org/reference/parse_atomic.html">parse_double()</a></code> and <code><a href="https://readr.tidyverse.org/reference/parse_number.html">parse_number()</a></code>. Use <code><a href="https://readr.tidyverse.org/reference/parse_atomic.html">parse_double()</a></code> when you have numbers that have been written as strings:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x &lt;- c("1.2", "5.6", "1e3")
<pre data-type="programlisting" data-code-language="r">x &lt;- c("1.2", "5.6", "1e3")
parse_double(x)
#&gt; [1] 1.2 5.6 1000.0</pre>
</div>
<p>Use <code><a href="https://readr.tidyverse.org/reference/parse_number.html">parse_number()</a></code> when the string contains non-numeric text that you want to ignore. This is particularly useful for currency data and percentages:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x &lt;- c("$1,234", "USD 3,513", "59%")
<pre data-type="programlisting" data-code-language="r">x &lt;- c("$1,234", "USD 3,513", "59%")
parse_number(x)
#&gt; [1] 1234 3513 59</pre>
</div>
@@ -40,7 +40,7 @@ parse_number(x)
Counts</h1>
<p>It’s surprising how much data science you can do with just counts and a little basic arithmetic, so dplyr strives to make counting as easy as possible with <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code>. This function is great for quick exploration and checks during analysis:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt; count(dest)
<pre data-type="programlisting" data-code-language="r">flights |&gt; count(dest)
#&gt; # A tibble: 105 × 2
#&gt; dest n
#&gt; &lt;chr&gt; &lt;int&gt;
@@ -55,7 +55,7 @@ Counts</h1>
<p>(Despite the advice in <a href="#chp-workflow-style" data-type="xref">#chp-workflow-style</a>, we usually put <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code> on a single line because it’s usually used at the console for a quick check that a calculation is working as expected.)</p>
<p>If you want to see the most common values add <code>sort = TRUE</code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt; count(dest, sort = TRUE)
<pre data-type="programlisting" data-code-language="r">flights |&gt; count(dest, sort = TRUE)
#&gt; # A tibble: 105 × 2
#&gt; dest n
#&gt; &lt;chr&gt; &lt;int&gt;
@@ -70,7 +70,7 @@ Counts</h1>
<p>And remember that if you want to see all the values, you can use <code>|&gt; View()</code> or <code>|&gt; print(n = Inf)</code>.</p>
<p>You can perform the same computation “by hand” with <code><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarise()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/context.html">n()</a></code>. This is useful because it allows you to compute other summaries at the same time:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(dest) |&gt;
summarise(
n = n(),
@@ -89,7 +89,7 @@ Counts</h1>
</div>
<p><code><a href="https://dplyr.tidyverse.org/reference/context.html">n()</a></code> is a special summary function that doesn’t take any arguments and instead accesses information about the “current” group. This means that it only works inside dplyr verbs:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">n()
<pre data-type="programlisting" data-code-language="r">n()
#&gt; Error in `n()`:
#&gt; ! Must only be used inside data-masking verbs like `mutate()`,
#&gt; `filter()`, and `group_by()`.</pre>
@@ -98,7 +98,7 @@ Counts</h1>
<ul><li>
<p><code>n_distinct(x)</code> counts the number of distinct (unique) values of one or more variables. For example, we could figure out which destinations are served by the most carriers:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(dest) |&gt;
summarise(
carriers = n_distinct(carrier)
@@ -119,7 +119,7 @@ Counts</h1>
<li>
<p>A weighted count is a sum. For example you could “count” the number of miles each plane flew:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(tailnum) |&gt;
summarise(miles = sum(distance))
#&gt; # A tibble: 4,044 × 2
@@ -135,7 +135,7 @@ Counts</h1>
</div>
<p>Weighted counts are a common problem so <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code> has a <code>wt</code> argument that does the same thing:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt; count(tailnum, wt = distance)
<pre data-type="programlisting" data-code-language="r">flights |&gt; count(tailnum, wt = distance)
#&gt; # A tibble: 4,044 × 2
#&gt; tailnum n
#&gt; &lt;chr&gt; &lt;dbl&gt;
@@ -151,7 +151,7 @@ Counts</h1>
<li>
<p>You can count missing values by combining <code><a href="https://rdrr.io/r/base/sum.html">sum()</a></code> and <code><a href="https://rdrr.io/r/base/NA.html">is.na()</a></code>. In the <code>flights</code> dataset this represents flights that are cancelled:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(dest) |&gt;
summarise(n_cancelled = sum(is.na(dep_time)))
#&gt; # A tibble: 105 × 2
@@ -189,7 +189,7 @@ Arithmetic and recycling rules</h2>
<p>We introduced the basics of arithmetic (<code>+</code>, <code>-</code>, <code>*</code>, <code>/</code>, <code>^</code>) in <a href="#chp-workflow-basics" data-type="xref">#chp-workflow-basics</a> and have used them a bunch since. These functions don’t need a huge amount of explanation because they do what you learned in grade school. But we need to briefly talk about the <strong>recycling rules</strong> which determine what happens when the left and right hand sides have different lengths. This is important for operations like <code>flights |&gt; mutate(air_time = air_time / 60)</code> because there are 336,776 numbers on the left of <code>/</code> but only one on the right.</p>
<p>R handles mismatched lengths by <strong>recycling,</strong> or repeating, the short vector. We can see this in operation more easily if we create some vectors outside of a data frame:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x &lt;- c(1, 2, 10, 20)
<pre data-type="programlisting" data-code-language="r">x &lt;- c(1, 2, 10, 20)
x / 5
#&gt; [1] 0.2 0.4 2.0 4.0
# is shorthand for
@@ -198,7 +198,7 @@ x / c(5, 5, 5, 5)
</div>
<p>Generally, you only want to recycle single numbers (i.e. vectors of length 1), but R will recycle any shorter length vector. It usually (but not always) gives you a warning if the longer vector isn’t a multiple of the shorter:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x * c(1, 2)
<pre data-type="programlisting" data-code-language="r">x * c(1, 2)
#&gt; [1] 1 4 10 40
x * c(1, 2, 3)
#&gt; Warning in x * c(1, 2, 3): longer object length is not a multiple of shorter
@@ -207,7 +207,7 @@ x * c(1, 2, 3)
</div>
<p>These recycling rules are also applied to logical comparisons (<code>==</code>, <code>&lt;</code>, <code>&lt;=</code>, <code>&gt;</code>, <code>&gt;=</code>, <code>!=</code>) and can lead to a surprising result if you accidentally use <code>==</code> instead of <code>%in%</code> and the data frame has an unfortunate number of rows. For example, take this code which attempts to find all flights in January and February:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
filter(month == c(1, 2))
#&gt; # A tibble: 25,977 × 19
#&gt; year month day dep_time sched_…¹ dep_d…² arr_t…³ sched…⁴ arr_d…⁵ carrier
@@ -232,7 +232,7 @@ x * c(1, 2, 3)
Minimum and maximum</h2>
<p>The arithmetic functions work with pairs of variables. Two closely related functions are <code><a href="https://rdrr.io/r/base/Extremes.html">pmin()</a></code> and <code><a href="https://rdrr.io/r/base/Extremes.html">pmax()</a></code>, which when given two or more variables will return the smallest or largest value in each row:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df &lt;- tribble(
<pre data-type="programlisting" data-code-language="r">df &lt;- tribble(
~x, ~y,
1, 3,
5, 2,
@@ -253,7 +253,7 @@ df |&gt;
</div>
<p>Note that these are different to the summary functions <code><a href="https://rdrr.io/r/base/Extremes.html">min()</a></code> and <code><a href="https://rdrr.io/r/base/Extremes.html">max()</a></code> which take multiple observations and return a single value. You can tell that you’ve used the wrong form when all the minimums and all the maximums have the same value:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df |&gt;
<pre data-type="programlisting" data-code-language="r">df |&gt;
mutate(
min = min(x, y, na.rm = TRUE),
max = max(x, y, na.rm = TRUE)
@@ -272,14 +272,14 @@ df |&gt;
Modular arithmetic</h2>
<p>Modular arithmetic is the technical name for the type of math you did before you learned about real numbers, i.e. division that yields a whole number and a remainder. In R, <code>%/%</code> does integer division and <code>%%</code> computes the remainder:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">1:10 %/% 3
<pre data-type="programlisting" data-code-language="r">1:10 %/% 3
#&gt; [1] 0 0 1 1 1 2 2 2 3 3
1:10 %% 3
#&gt; [1] 1 2 0 1 2 0 1 2 0 1</pre>
</div>
<p>Modular arithmetic is handy for the flights dataset, because we can use it to unpack the <code>sched_dep_time</code> variable into and <code>hour</code> and <code>minute</code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
mutate(
hour = sched_dep_time %/% 100,
minute = sched_dep_time %% 100,
@@ -298,7 +298,7 @@ Modular arithmetic</h2>
</div>
<p>We can combine that with the <code>mean(is.na(x))</code> trick from <a href="#sec-logical-summaries" data-type="xref">#sec-logical-summaries</a> to see how the proportion of cancelled flights varies over the course of the day. The results are shown in <a href="#fig-prop-cancelled" data-type="xref">#fig-prop-cancelled</a>.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(hour = sched_dep_time %/% 100) |&gt;
summarise(prop_cancelled = mean(is.na(dep_time)), n = n()) |&gt;
filter(hour &gt; 1) |&gt;
@@ -319,7 +319,7 @@ Modular arithmetic</h2>
Logarithms</h2>
<p>Logarithms are an incredibly useful transformation for dealing with data that ranges across multiple orders of magnitude. They also convert exponential growth to linear growth. For example, take compounding interest — the amount of money you have at <code>year + 1</code> is the amount of money you had at <code>year</code> multiplied by the interest rate. That gives a formula like <code>money = starting * interest ^ year</code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">starting &lt;- 100
<pre data-type="programlisting" data-code-language="r">starting &lt;- 100
interest &lt;- 1.05
money &lt;- tibble(
@@ -329,7 +329,7 @@ money &lt;- tibble(
</div>
<p>If you plot this data, you’ll get an exponential curve:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">ggplot(money, aes(year, money)) +
<pre data-type="programlisting" data-code-language="r">ggplot(money, aes(year, money)) +
geom_line()</pre>
<div class="cell-output-display">
<p><img src="numbers_files/figure-html/unnamed-chunk-22-1.png" width="576"/></p>
@@ -337,7 +337,7 @@ money &lt;- tibble(
</div>
<p>Log transforming the y-axis gives a straight line:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">ggplot(money, aes(year, money)) +
<pre data-type="programlisting" data-code-language="r">ggplot(money, aes(year, money)) +
geom_line() +
scale_y_log10()</pre>
<div class="cell-output-display">
@@ -354,12 +354,12 @@ money &lt;- tibble(
Rounding</h2>
<p>Use <code>round(x)</code> to round a number to the nearest integer:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">round(123.456)
<pre data-type="programlisting" data-code-language="r">round(123.456)
#&gt; [1] 123</pre>
</div>
<p>You can control the precision of the rounding with the second argument, <code>digits</code>. <code>round(x, digits)</code> rounds to the nearest <code>10^-n</code> so <code>digits = 2</code> will round to the nearest 0.01. This definition is useful because it implies <code>round(x, -3)</code> will round to the nearest thousand, which indeed it does:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">round(123.456, 2) # two digits
<pre data-type="programlisting" data-code-language="r">round(123.456, 2) # two digits
#&gt; [1] 123.46
round(123.456, 1) # one digit
#&gt; [1] 123.5
@@ -370,13 +370,13 @@ round(123.456, -2) # round to nearest hundred
</div>
<p>There’s one weirdness with <code><a href="https://rdrr.io/r/base/Round.html">round()</a></code> that seems surprising at first glance:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">round(c(1.5, 2.5))
<pre data-type="programlisting" data-code-language="r">round(c(1.5, 2.5))
#&gt; [1] 2 2</pre>
</div>
<p><code><a href="https://rdrr.io/r/base/Round.html">round()</a></code> uses what’s known as “round half to even” or Banker’s rounding: if a number is half way between two integers, it will be rounded to the <strong>even</strong> integer. This is a good strategy because it keeps the rounding unbiased: half of all 0.5s are rounded up, and half are rounded down.</p>
<p><code><a href="https://rdrr.io/r/base/Round.html">round()</a></code> is paired with <code><a href="https://rdrr.io/r/base/Round.html">floor()</a></code> which always rounds down and <code><a href="https://rdrr.io/r/base/Round.html">ceiling()</a></code> which always rounds up:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x &lt;- 123.456
<pre data-type="programlisting" data-code-language="r">x &lt;- 123.456
floor(x)
#&gt; [1] 123
@@ -385,7 +385,7 @@ ceiling(x)
</div>
<p>These functions don’t have a digits argument, so you can instead scale down, round, and then scale back up:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit"># Round down to nearest two digits
<pre data-type="programlisting" data-code-language="r"># Round down to nearest two digits
floor(x / 0.01) * 0.01
#&gt; [1] 123.45
# Round up to nearest two digits
@@ -394,7 +394,7 @@ ceiling(x / 0.01) * 0.01
</div>
<p>You can use the same technique if you want to <code><a href="https://rdrr.io/r/base/Round.html">round()</a></code> to a multiple of some other number:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit"># Round to nearest multiple of 4
<pre data-type="programlisting" data-code-language="r"># Round to nearest multiple of 4
round(x / 4) * 4
#&gt; [1] 124
@@ -409,20 +409,20 @@ round(x / 0.25) * 0.25
Cutting numbers into ranges</h2>
<p>Use <code><a href="https://rdrr.io/r/base/cut.html">cut()</a></code><span data-type="footnote">ggplot2 provides some helpers for common cases in <code><a href="https://ggplot2.tidyverse.org/reference/cut_interval.html">cut_interval()</a></code>, <code><a href="https://ggplot2.tidyverse.org/reference/cut_interval.html">cut_number()</a></code>, and <code><a href="https://ggplot2.tidyverse.org/reference/cut_interval.html">cut_width()</a></code>. ggplot2 is an admittedly weird place for these functions to live, but they are useful as part of histogram computation and were written before any other parts of the tidyverse existed.</span> to break up a numeric vector into discrete buckets:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x &lt;- c(1, 2, 5, 10, 15, 20)
<pre data-type="programlisting" data-code-language="r">x &lt;- c(1, 2, 5, 10, 15, 20)
cut(x, breaks = c(0, 5, 10, 15, 20))
#&gt; [1] (0,5] (0,5] (0,5] (5,10] (10,15] (15,20]
#&gt; Levels: (0,5] (5,10] (10,15] (15,20]</pre>
</div>
<p>The breaks don’t need to be evenly spaced:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">cut(x, breaks = c(0, 5, 10, 100))
<pre data-type="programlisting" data-code-language="r">cut(x, breaks = c(0, 5, 10, 100))
#&gt; [1] (0,5] (0,5] (0,5] (5,10] (10,100] (10,100]
#&gt; Levels: (0,5] (5,10] (10,100]</pre>
</div>
<p>You can optionally supply your own <code>labels</code>. Note that there should be one less <code>labels</code> than <code>breaks</code>.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">cut(x,
<pre data-type="programlisting" data-code-language="r">cut(x,
breaks = c(0, 5, 10, 15, 20),
labels = c("sm", "md", "lg", "xl")
)
@@ -431,7 +431,7 @@ cut(x, breaks = c(0, 5, 10, 15, 20))
</div>
<p>Any values outside of the range of the breaks will become <code>NA</code>:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">y &lt;- c(NA, -10, 5, 10, 30)
<pre data-type="programlisting" data-code-language="r">y &lt;- c(NA, -10, 5, 10, 30)
cut(y, breaks = c(0, 5, 10, 15, 20))
#&gt; [1] &lt;NA&gt; &lt;NA&gt; (0,5] (5,10] &lt;NA&gt;
#&gt; Levels: (0,5] (5,10] (10,15] (15,20]</pre>
@@ -444,13 +444,13 @@ cut(y, breaks = c(0, 5, 10, 15, 20))
Cumulative and rolling aggregates</h2>
<p>Base R provides <code><a href="https://rdrr.io/r/base/cumsum.html">cumsum()</a></code>, <code><a href="https://rdrr.io/r/base/cumsum.html">cumprod()</a></code>, <code><a href="https://rdrr.io/r/base/cumsum.html">cummin()</a></code>, <code><a href="https://rdrr.io/r/base/cumsum.html">cummax()</a></code> for running, or cumulative, sums, products, mins and maxes. dplyr provides <code><a href="https://dplyr.tidyverse.org/reference/cumall.html">cummean()</a></code> for cumulative means. Cumulative sums tend to come up the most in practice:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x &lt;- 1:10
<pre data-type="programlisting" data-code-language="r">x &lt;- 1:10
cumsum(x)
#&gt; [1] 1 3 6 10 15 21 28 36 45 55</pre>
</div>
<p>If you need more complex rolling or sliding aggregates, try the <a href="https://davisvaughan.github.io/slider/">slider</a> package by Davis Vaughan. The following example illustrates some of its features.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">library(slider)
<pre data-type="programlisting" data-code-language="r">library(slider)
# Same as a cumulative sum
slide_vec(x, sum, .before = Inf)
@@ -475,7 +475,7 @@ Exercises</h2>
<li>
<p>Currently <code>dep_time</code> and <code>sched_dep_time</code> are convenient to look at, but hard to compute with because they’re not really continuous numbers. You can see the basic problem in this plot: there’s a gap between each hour.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
filter(month == 1, day == 1) |&gt;
ggplot(aes(sched_dep_time, dep_delay)) +
geom_point()
@@ -499,18 +499,18 @@ General transformations</h1>
Ranks</h2>
<p>dplyr provides a number of ranking functions inspired by SQL, but you should always start with <code><a href="https://dplyr.tidyverse.org/reference/row_number.html">dplyr::min_rank()</a></code>. It uses the typical method for dealing with ties, e.g. 1st, 2nd, 2nd, 4th.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x &lt;- c(1, 2, 2, 3, 4, NA)
<pre data-type="programlisting" data-code-language="r">x &lt;- c(1, 2, 2, 3, 4, NA)
min_rank(x)
#&gt; [1] 1 2 2 4 5 NA</pre>
</div>
<p>Note that the smallest values get the lowest ranks; use <code>desc(x)</code> to give the largest values the smallest ranks:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">min_rank(desc(x))
<pre data-type="programlisting" data-code-language="r">min_rank(desc(x))
#&gt; [1] 5 3 3 2 1 NA</pre>
</div>
<p>If <code><a href="https://dplyr.tidyverse.org/reference/row_number.html">min_rank()</a></code> doesn’t do what you need, look at the variants <code><a href="https://dplyr.tidyverse.org/reference/row_number.html">dplyr::row_number()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/row_number.html">dplyr::dense_rank()</a></code>, <code><a href="https://dplyr.tidyverse.org/reference/percent_rank.html">dplyr::percent_rank()</a></code>, and <code><a href="https://dplyr.tidyverse.org/reference/percent_rank.html">dplyr::cume_dist()</a></code>. See the documentation for details.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df &lt;- tibble(x = x)
<pre data-type="programlisting" data-code-language="r">df &lt;- tibble(x = x)
df |&gt;
mutate(
row_number = row_number(x),
@@ -531,7 +531,7 @@ df |&gt;
<p>You can achieve many of the same results by picking the appropriate <code>ties.method</code> argument to base R’s <code><a href="https://rdrr.io/r/base/rank.html">rank()</a></code>; you’ll probably also want to set <code>na.last = "keep"</code> to keep <code>NA</code>s as <code>NA</code>.</p>
<p><code><a href="https://dplyr.tidyverse.org/reference/row_number.html">row_number()</a></code> can also be used without any arguments when inside a dplyr verb. In this case, it’ll give the number of the “current” row. When combined with <code>%%</code> or <code>%/%</code> this can be a useful tool for dividing data into similarly sized groups:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">df &lt;- tibble(x = runif(10))
<pre data-type="programlisting" data-code-language="r">df &lt;- tibble(x = runif(10))
df |&gt;
mutate(
@@ -557,7 +557,7 @@ df |&gt;
Offsets</h2>
<p><code><a href="https://dplyr.tidyverse.org/reference/lead-lag.html">dplyr::lead()</a></code> and <code><a href="https://dplyr.tidyverse.org/reference/lead-lag.html">dplyr::lag()</a></code> allow you to refer the values just before or just after the “current” value. They return a vector of the same length as the input, padded with <code>NA</code>s at the start or end:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x &lt;- c(2, 5, 11, 11, 19, 35)
<pre data-type="programlisting" data-code-language="r">x &lt;- c(2, 5, 11, 11, 19, 35)
lag(x)
#&gt; [1] NA 2 5 11 11 19
lead(x)
@@ -566,14 +566,14 @@ lead(x)
<ul><li>
<p><code>x - lag(x)</code> gives you the difference between the current and previous value.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x - lag(x)
<pre data-type="programlisting" data-code-language="r">x - lag(x)
#&gt; [1] NA 3 6 0 8 16</pre>
</div>
</li>
<li>
<p><code>x == lag(x)</code> tells you when the current value changes.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">x == lag(x)
<pre data-type="programlisting" data-code-language="r">x == lag(x)
#&gt; [1] NA FALSE FALSE TRUE FALSE FALSE</pre>
</div>
</li>
@@ -591,7 +591,7 @@ Exercises</h2>
<li>
<p>Delays are typically temporally correlated: even once the problem that caused the initial delay has been resolved, later flights are delayed to allow earlier flights to leave. Using <code><a href="https://dplyr.tidyverse.org/reference/lead-lag.html">lag()</a></code>, explore how the average flight delay for an hour is related to the average delay for the previous hour.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
mutate(hour = dep_time %/% 100) |&gt;
group_by(year, month, day, hour) |&gt;
summarise(
@@ -618,7 +618,7 @@ Center</h2>
<p>So far, we’ve mostly used <code><a href="https://rdrr.io/r/base/mean.html">mean()</a></code> to summarize the center of a vector of values. Because the mean is the sum divided by the count, it is sensitive to even just a few unusually high or low values. An alternative is to use the <code><a href="https://rdrr.io/r/stats/median.html">median()</a></code>, which finds a value that lies in the “middle” of the vector, i.e. 50% of the values is above it and 50% are below it. Depending on the shape of the distribution of the variable you’re interested in, mean or median might be a better measure of center. For example, for symmetric distributions we generally report the mean while for skewed distributions we usually report the median.</p>
<p><a href="#fig-mean-vs-median" data-type="xref">#fig-mean-vs-median</a> compares the mean vs the median when looking at the hourly vs median departure delay. The median delay is always smaller than the mean delay because because flights sometimes leave multiple hours late, but never leave multiple hours early.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(year, month, day) |&gt;
summarise(
mean = mean(dep_delay, na.rm = TRUE),
@@ -647,7 +647,7 @@ Minimum, maximum, and quantiles</h2>
<p>What if you’re interested in locations other than the center? <code><a href="https://rdrr.io/r/base/Extremes.html">min()</a></code> and <code><a href="https://rdrr.io/r/base/Extremes.html">max()</a></code> will give you the largest and smallest values. Another powerful tool is <code><a href="https://rdrr.io/r/stats/quantile.html">quantile()</a></code> which is a generalization of the median: <code>quantile(x, 0.25)</code> will find the value of <code>x</code> that is greater than 25% of the values, <code>quantile(x, 0.5)</code> is equivalent to the median, and <code>quantile(x, 0.95)</code> will find a value that’s greater than 95% of the values.</p>
<p>For the <code>flights</code> data, you might want to look at the 95% quantile of delays rather than the maximum, because it will ignore the 5% of most delayed flights which can be quite extreme.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(year, month, day) |&gt;
summarise(
max = max(dep_delay, na.rm = TRUE),
@@ -673,7 +673,7 @@ Spread</h2>
<p>Sometimes you’re not so interested in where the bulk of the data lies, but in how it is spread out. Two commonly used summaries are the standard deviation, <code>sd(x)</code>, and the inter-quartile range, <code><a href="https://rdrr.io/r/stats/IQR.html">IQR()</a></code>. We won’t explain <code><a href="https://rdrr.io/r/stats/sd.html">sd()</a></code> here since you’re probably already familiar with it, but <code><a href="https://rdrr.io/r/stats/IQR.html">IQR()</a></code> might be new — it’s <code>quantile(x, 0.75) - quantile(x, 0.25)</code> and gives you the range that contains the middle 50% of the data.</p>
<p>We can use this to reveal a small oddity in the <code>flights</code> data. You might expect the spread of the distance between origin and destination to be zero, since airports are always in the same place. But the code below makes it looks like one airport, <a href="https://en.wikipedia.org/wiki/Eagle_County_Regional_Airport">EGE</a>, might have moved.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(origin, dest) |&gt;
summarise(
distance_sd = IQR(distance),
@@ -695,7 +695,7 @@ Distributions</h2>
<p>It’s worth remembering that all of the summary statistics described above are a way of reducing the distribution down to a single number. This means that they’re fundamentally reductive, and if you pick the wrong summary, you can easily miss important differences between groups. That’s why it’s always a good idea to visualize the distribution before committing to your summary statistics.</p>
<p><a href="#fig-flights-dist" data-type="xref">#fig-flights-dist</a> shows the overall distribution of departure delays. The distribution is so skewed that we have to zoom in to see the bulk of the data. This suggests that the mean is unlikely to be a good summary and we might prefer the median instead.</p>
<div>
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
ggplot(aes(dep_delay)) +
geom_histogram(binwidth = 15)
#&gt; Warning: Removed 8255 rows containing non-finite values (`stat_bin()`).
@@ -724,7 +724,7 @@ flights |&gt;
</div>
<p>It’s also a good idea to check that distributions for subgroups resemble the whole. <a href="#fig-flights-dist-daily" data-type="xref">#fig-flights-dist-daily</a> overlays a frequency polygon for each day. The distributions seem to follow a common pattern, suggesting it’s fine to use the same summary for each day.</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
filter(dep_delay &lt; 120) |&gt;
ggplot(aes(dep_delay, group = interaction(day, month))) +
geom_freqpoly(binwidth = 5, alpha = 1/5)</pre>
@@ -744,7 +744,7 @@ Positions</h2>
<p>There’s one final type of summary that’s useful for numeric vectors, but also works with every other type of value: extracting a value at specific position. You can do this with the base R <code>[</code> function, but we’re not going to cover it in detail until <a href="#sec-subset-many" data-type="xref">#sec-subset-many</a>, because it’s a very powerful and general function. For now we’ll introduce three specialized functions that you can use to extract values at a specified position: <code>first(x)</code>, <code>last(x)</code>, and <code>nth(x, n)</code>.</p>
<p>For example, we can find the first and last departure for each day:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(year, month, day) |&gt;
summarise(
first_dep = first(dep_time),
@@ -769,7 +769,7 @@ Positions</h2>
<p>If you’re familiar with <code>[</code>, you might wonder if you ever need these functions. There are two main reasons: the <code>default</code> argument and the <code>order_by</code> argument. <code>default</code> allows you to set a default value that’s used if the requested position doesn’t exist, e.g. you’re trying to get the 3rd element from a two element group. <code>order_by</code> lets you locally override the existing ordering of the rows, so you can get the element at the position in the ordering by <code><a href="https://dplyr.tidyverse.org/reference/order_by.html">order_by()</a></code>.</p>
<p>Extracting values at positions is complementary to filtering on ranks. Filtering gives you all variables, with each observation in a separate row:</p>
<div class="cell">
<pre data-type="programlisting" data-code-language="downlit">flights |&gt;
<pre data-type="programlisting" data-code-language="r">flights |&gt;
group_by(year, month, day) |&gt;
mutate(r = min_rank(desc(sched_dep_time))) |&gt;
filter(r %in% c(1, max(r)))