(Despite the advice in Chapter \@ref(code-style), I usually put `count()` on a single line because I'm usually using it at the console for a quick check that my calculation is working as expected.)
Alternatively, you can also count "by hand" by using `n()` with `group_by()` and `summarise()`.
This has a couple of advantages: you can combine it with other summary functions and it's easier to control
```{r}
flights |>
group_by(dest) |>
summarise(n = n())
```
`n()` is a special a summary function because it doesn't take any arguments and instead reads information from the current group.
This means you can't use it outside of dplyr verbs:
```{r, error = TRUE}
n()
```
There are a couple of related counts that you might find useful:
- `n_distinct(x)` counts the number of distinct (unique) values of one or more variables:
There are many functions for creating new variables that you can use with `mutate()`.
The key property is that the function must be vectorised: it must take a vector of values as input, return a vector with the same number of values as output.
R also provides all the trigonometry functions that you might expect.
I'm not going to discuss them here since it's rare that you need them for data science, but you can sleep soundly at night knowing that they're available if you need them.
But we've used them in two subtly different ways: `air_time / 60` and `air_time / distance`.
In the first case we're dividing a vector of numbers by a single number, and in the second case we're working with a pair of vectors that have the same length.
R handles the first case by transforming it to the second case:
```{r}
x <- c(1, 2, 10, 20)
x / 5
# Short hand for
x / c(5, 5, 5, 5)
```
Whenever you're working with a pair of vectors that have different lengths R uses the so called **recycling rules**.
In general, there's only one way you actually want to use these: with a vector and a scalar.
But R supports a somewhat more general rule where it will recycle any shorter length vector:
```{r}
x * c(1, 2)
x * c(1, 2, 3)
```
In most cases you'll get a warning if the longer vector is not a integer multiple of the shower.
The most common way this can bite you is if you accidentally use `==` instead of `%in%` and the data frame has an unfortunate number of row.
For example, this code works but it's unlikely that the result is what you want:
```{r}
flights |>
filter(month == c(1, 2))
```
It returns odd rows in January and even in February.
To protect you from this silently failure, tidyverse functions generally use a stricter set of rules that only recycles single numbers, but it doesn't help in this case because you're using the base R function `==`.
### Minimum and maximum
`pmin()`, `pmax()`
### Modular arithmetic
`%/%` (integer division) and `%%` (remainder), where `x == y * (x %/% y) + (x %% y)`.
`log2()` is easy to interpret because difference of 1 on the log scale corresponds to doubling on the original scale and a difference of -1 corresponds to halving; whereas `log10()` is easy to back-transform because (e.g) 3 is 10\^3 = 1000.
R provides functions for running sums, products, mins and maxes: `cumsum()`, `cumprod()`, `cummin()`, `cummax()`; and dplyr provides `cummean()` for cumulative means.
If you need more complex rolling or sliding aggregates (i.e. a sum computed over a rolling window), try the slider package.
If you're familiar with `[`, these function work similarly to but they let you set a default value if that position does not exist (i.e. you're trying to get the 3rd element from a group that only has two elements).
1. Find the 10 most delayed flights using a ranking function.
How do you want to handle ties?
Carefully read the documentation for `min_rank()`.
2. Which plane (`tailnum`) has the worst on-time record?
3. What time of day should you fly if you want to avoid delays as much as possible?
4. For each destination, compute the total minutes of delay.
For each flight, compute the proportion of the total delay for its destination.
5. Delays are typically temporally correlated: even once the problem that caused the initial delay has been resolved, later flights are delayed to allow earlier flights to leave.
Using `lag()`, explore how the delay of a flight is related to the delay of the immediately preceding flight.
6. Look at each destination.
Can you find flights that are suspiciously fast?
(i.e. flights that represent a potential data entry error).
Compute the air time of a flight relative to the shortest flight to that destination.
Which flights were most delayed in the air?
7. Find all destinations that are flown by at least two carriers.
Just using means, counts, and sum can get you a long way, but R provides many other useful summary functions.
### Center
We've used `mean(x)`, but `median(x)` is also useful.
The mean is the sum divided by the length; the median is a value where 50% of `x` is above it, and 50% is below it.
```{r}
flights |>
group_by(month) |>
summarise(
med_arr_delay = median(arr_delay, na.rm = TRUE),
med_dep_delay = median(dep_delay, na.rm = TRUE)
)
```
Don't forget what you learned in Section \@ref(sample-size): whenever creating numerical summaries, it's a good idea to include the number of observations in each group.
### Minimum, maximum, and quantiles
Quantiles are a generalization of the median.
For example, `quantile(x, 0.25)` will find a value of `x` that is greater than 25% of the values, and less than the remaining 75%.
`min()` and `max()` are like the 0% and 100% quantiles: they're the smallest and biggest numbers.
The interquartile range `IQR(x)` and median absolute deviation `mad(x)` are robust equivalents that may be more useful if you have outliers.
IQR is `quantile(x, 0.75) - quantile(x, 0.25)`.
`mad()` is derivied similarly to `sd()`, but inside being the average of the squared distances from the mean, it's the median of the absolute differences from the median.
### With `mutate()`
As the names suggest, the summary functions are typically paired with `summarise()`, but they can also be usefully paired with `mutate()`, particularly when you want do some sort of group standardisations.
Arithmetic operators are also useful in conjunction with the aggregate functions you'll learn about later.
For example, `x / sum(x)` calculates the proportion of a total, and `y - mean(y)` computes the difference from the mean, `y / y[1]` indexes using the first observation.
### Exercises
1. Currently `dep_time` and `sched_dep_time` are convenient to look at, but hard to compute with because they're not really continuous numbers.
Convert them to a more convenient representation of number of minutes since midnight.
2. What trigonometric functions does R provide?
3. Brainstorm at least 5 different ways to assess the typical delay characteristics of a group of flights.
Consider the following scenarios:
- A flight is 15 minutes early 50% of the time, and 15 minutes late 50% of the time.
- A flight is always 10 minutes late.
- A flight is 30 minutes early 50% of the time, and 30 minutes late 50% of the time.
- 99% of the time a flight is on time.
1% of the time it's 2 hours late.
Which is more important: arrival delay or departure delay?
## Variants
We've seen a few variants of different functions
| Summary | Cumulative | Paired |
|---------|------------|--------|
| `sum` | `cumsum` | `+` |
| `prod` | `cumprod` | `*` |
| `all` | `cumall` | `&` |
| `any` | `cumany` | `\|` |
| `min` | `cummin` | `pmin` |
| `max` | `cummax` | `pmax` |
- Summary functions take a vector and always return a length 1 vector. Typically used with `summarise()`
- Cumulative functions take a vector and return the same length. Used with `mutate()`.
- Paired functions take a pair of functions and return a vector the same length (using the recycling rules if the vectors aren't the same length). Used with `mutate()`
