r4ds/datetimes.Rmd

# Dates and times

## Introduction

This chapter will show you how to work with dates and times in R. At first glance, dates and times seem simple. You use them all the time in your regular life, and they don't seem to cause much confusion. However, the more you learn about dates and times, the more complicated they seem to get. For example:

* Does every year have 365 days?
* Does every day have 24 hours?
* Does every minute have 60 seconds?

I'm sure you know that not every year has 365 days, but but do you know the full rule for determining if a year is a leap year? You might have remembered that many parts of the world use daylight savings time (DST), so that some days have 23 hours, and others have 25. You probably didn't know that some minutes have 61 seconds because every now and then leap seconds are added to keep because the Earth's rotation is gradually slowing down.

Dates and times are hard because they have to reconcile two physical phenonmen (the rotation of the Earth and its orbit around the sun) with a whole raft of geopolitical phenonmeon including months, time zones, and DST. This chapter won't teach you every last detail about dates and times, but it will give you a solid grounding of practical skills that will help you with common data analysis challenges.

### Prerequisites

This chapter will focus on the __lubridate__ package, which makes it easier to work with dates and times in R. We will use nycflights13 for practice data, and some packages for EDA.

```{r setup, message = FALSE}
library(lubridate)

# Data
library(nycflights13)

# EDA
library(dplyr)
library(ggplot2)
```

## Creating date/times

There three types of date/time data that refer to an instant in time:

* A __date__. Tibbles print this as `<date>`.

* A __time__ within a day. Tibbles print this as `<time>`.

* A __date-time__ is a date plus a time: it uniquely identifies an
  instant in time (typically to the nearest second). Tibbles print this
  as `<dttm>`. Elsewhere in R these are called POSIXct, but I don't think
  that's a very useful name.
  
In this chapter we are only going to focus on dates and date-times. R doesn't have a native class for storing times. If you need one, you can use the hms package.

You should always use the simplest possible data type that works for your needs. That means if you can use a date instead of a date-time, you should. Date-times are substantially more complicated because of the need to handle time zones, which we'll come back to at the end of the chapter.

To get the current date or date-time you can use `today()` or `now()`:

```{r}
today()
now()
```

Otherwise, there are three ways you're likely to create a date/time:

* From a character vector.
* From numeric vectors of each component.
* From an existing date/time object.

### From strings

Time data often comes as strings. You've seen one approach to parsing date times with readr package, in [date-times](#readr-datetimes). Another approach is to use the helper functions provided by lubridate. They automatically work out the format once you tell them the order of the day, month, and year components. To use them, identify the order in which the year, month, and day appears in your dates. Now arrange "y", "m", and "d" in the same order. This is the name of the function in lubridate that will parse your dates. For example:

```{r}
ymd("2017-01-31")
mdy("January 31st, 2017")
dmy("31-Jan-2017")
```

If you want to create a single date object for use in comparisons (e.g. in `dplyr::filter()`), I recommend using `ymd()` with numeric input. It's short and unambiguous:

```{r}
ymd(20170131)
```

If you have a date-time that also contains hours, minutes, or seconds, add an underscore and then one or more of "h", "m", and "s" to the name of the parsing function.

```{r}
ymd_hms("2017-01-31 20:11:59")
mdy_hm("01/31/2017 08:01")
```

### From individual components

Sometimes you'll get the individual components of the date time spread acros multiple column. This is what we have in the flights data:

```{r}
flights %>% 
  select(year, month, day, hour, minute)
```

To create a date-time from this sort of input, use `make_datetime()`:

```{r}
flights %>% 
  select(year, month, day, hour, minute) %>% 
  mutate(departure = make_datetime(year, month, day, hour, minute))
```

Let's do the same thing for each of the four times column in `flights`. The times are represented in a slightly odd format, so we use modulus arithmetic to pull out the hour and minute components. Once that's done, I focus in on the variables we'll explore in the rest of the chapter.

```{r}
make_datetime_100 <- function(year, month, day, time) {
  make_datetime(year, month, day, time %/% 100, time %% 100)
}

flights_dt <- flights %>% 
  filter(!is.na(dep_time), !is.na(arr_time)) %>% 
  mutate(
    dep_time = make_datetime_100(year, month, day, dep_time),
    arr_time = make_datetime_100(year, month, day, arr_time),
    sched_dep_time = make_datetime_100(year, month, day, sched_dep_time),
    sched_arr_time = make_datetime_100(year, month, day, sched_arr_time)
  ) %>% 
  select(origin, dest, ends_with("delay"), ends_with("time"))

flights_dt
```

Now I can start to visualise the distribution of departure times across the year:

```{r}
flights_dt %>% 
  ggplot(aes(dep_time)) + 
  geom_freqpoly(binwidth = 86400) # 86400 seconds = 1 day
```

Or within a single day:

```{r}
flights_dt %>% 
  filter(dep_time < ymd(20130102, tz = "UTC")) %>% 
  ggplot(aes(dep_time)) + 
  geom_freqpoly(binwidth = 600) # 600 s = 10 minutes
```

Note the two tricks I needed to create these plots:

1.  When you use date-times in a numeric context (like in a histogram), 1 
    means 1 second, so a binwidth of 86400 means one day. For dates, 1
    means 1 day.
   
1.  R doesn't like to compare date-times with dates, so you can force
    `ymd()` to geneate a date-time by supplying a `tz` argument.

### From other types

You may want to switch between a date-time and a date. That's the job of `as_datetime()` and `as_date()`:

```{r}
# as_datetime(today())
as_date(now())
```

### Exercises

1.  What happens if you parse a string that contains invalid dates?

    ```{r, eval = FALSE}
    ymd(c("2010-10-10", "bananas"))
    ```

1.  What does the `tzone` argument to `today()` do? Why is it important?

1.  Use lubridate to parse each of the following dates:

    ```{r}
    d1 <- "January 1, 2010"
    d2 <- "2015-Mar-07"
    d3 <- "06-Jun-2017"
    d4 <- c("August 19 (2015)", "July 1 (2015)")
    d5 <- "12/30/14" # Dec 30, 2014
    ```

## Date-time components

Now that you know how to get date-time data in R's date-time datastructures let's explore what you can do with them. This section will focus on the accessor functions that let you get and set individual components of the date. The next section will look at how arithmetic works with date-times.

### Getting components

You can pull out individual parts of the date with the acccessor functions `year()`, `month()`, `mday()` (day of the month), `yday()` (day of the year)`, `wday()` (day of the week), `hour()`, `minute()`, `second()`. 

```{r}
datetime <- ymd_hms("2016-07-08 12:34:56")

year(datetime)
month(datetime)
mday(datetime)

yday(datetime)
wday(datetime)
```

For `month()` and `wday()` you can set `label = TRUE` to return the name of the month or day of the week. Set `abbr = TRUE` to return an abbreviated version of the name, which can be helpful in plots.

```{r}
month(datetime, label = TRUE)
wday(datetime, label = TRUE, abbr = TRUE)
```

We can use the `wday()` accessor to see that more flights depart on weekdays than weekend days. 

```{r}
flights_dt %>% 
  mutate(wday = wday(dep_time, label = TRUE)) %>% 
  ggplot(aes(x = wday)) +
    geom_bar()
```

There's an interesting pattern if we look at the average departure delay by minute within the hour. It looks like flights leaving in minutes 20-30 and 50-60 have much lower delays that otherwise!

```{r}
flights_dt %>% 
  mutate(minute = minute(dep_time)) %>% 
  group_by(minute) %>% 
  summarise(
    avg_delay = mean(arr_delay, na.rm = TRUE),
    n = n()) %>% 
  ggplot(aes(minute, avg_delay)) +
    geom_line()
```

Interestingly, if we look at the _scheduled_ departure time we don't see such a strong pattern:

```{r}
sched_dep <- flights_dt %>% 
  mutate(minute = minute(sched_dep_time)) %>% 
  group_by(minute) %>% 
  summarise(
    avg_delay = mean(arr_delay, na.rm = TRUE),
    n = n())

ggplot(sched_dep , aes(minute, avg_delay)) +
  geom_line()
```

So we do we see such a strong pattern in the delays of actual departure times? Well, like much data collected by humans, there's a strong bias towards flights leaving at "nice" departure times. Always be alert for this sort of pattern whenever you data involves human judgement.

```{r}
ggplot(sched_dep, aes(minute, n)) +
  geom_line()
```

What we're probably seeing is the impact of flights scheduled to leave on the hour or half past the hour leaving a few minutes early.

### Rounding

An alternative approach to plotting individual components is to round the date, using `floor_date()`, `round_date()`, and `ceiling_date()` to round a date to a nearby unit of time. Each function takes a vector of dates to adjust and then the name of the unit to floor, ceiling, or round them to.

This allows us to, for example, plot the number of flights per week:

```{r}
flights_dt %>% 
  count(week = floor_date(dep_time, "week")) %>% 
  ggplot(aes(week, n)) +
    geom_line()
```

### Setting components

You can also use each accessor function to set the components of a date or date-time. 

```{r}
datetime
year(datetime) <- 2001
datetime
month(datetime) <- 01
datetime
hour(datetime) <- hour(datetime) + 1
```

Alternatively, rather than modifying in place, you can create a new date-time with `update()`. This also allows you to set multiple values at once.

```{r}
update(datetime, year = 2002, month = 2, mday = 2, hour = 2)
```

If values are too big, they will roll-over:

```{r}
ymd("2015-02-01") %>% update(mday = 30)
ymd("2015-02-01") %>% update(hour = 400)
```

You can use `update()` if you want to see the distribution of flights across the course of the day for every day of year:

```{r}
flights_dt %>% 
  mutate(dep_hour = update(dep_time, month = 1, day = 1)) %>% 
  ggplot(aes(dep_hour)) +
    geom_freqpoly(binwidth = 300)
```

### Exercises

1.  Does the distribution of flight times within a day change over the course
    of the year?
    
1.  How does the average delay time change over the course of a day?
    When exploring that pattern is it better to use `dep_time` or
    `sched_dep_time`? Which is more informative.

1.  On what day of the week should you leave if you want to minimise the
    chance of a delay?

1.  Confirm my hypthosese that the early departures of flights from 20-30 and 
    50-60 are caused by scheduled flights that leave early.  Hint: create a
    a new categorical variable that tells you whether or not the flight
    was delayed, and group by that.

## Time spans

Next you'll learn about how arithmetic with dates works, including substraction, addition, and division. Along the way, you'll learn about three important classes that represent time spans:

* __durations__, which represent an exact number of seconds.
* __periods__, which represent human units like weeks and months.
* __intervals__, which represent a starting and ending point.

### Durations

In R, when you subtract two dates, you get a difftime object:

```{r}
# How old is Hadley?
h_age <- today() - ymd(19791014)
h_age
```

A difftime class object records a time span of seconds, minutes, hours, days, or weeks. This can ambiguity makes difftimes a little painful to work with, so lubridate provides an alternative which always uses seconds: the __duration__.

```{r}
as.duration(h_age)
```

Durations also come with a bunch of convenient constructors:

```{r}
dseconds(15)
dminutes(10)
dhours(c(12, 24))
ddays(0:5)
dweeks(3)
dyears(1)
```

Durations always record the time space in seconds. Larger units are created by converting minutes, hours, days, weeks, and years to seconds at the standard rate (60 seconds in a minute, 60 minutes in an hour, 24 hours in day, 7 days in a week, 365 days in a year).  You can add and multiple durations:

```{r}
2 * dyears(1)
dyears(1) + dweeks(12) + dhours(15)
```

You can add and subtract durations to and from days:

```{r}
tomorrow <- today() + ddays(1)
last_year <- today() - dyears(1)
```

However, because durations represent an exact number of seconds, sometimes you might get an unexpected result:

```{r}
one_pm <- ymd_hms("2016-03-12 13:00:00", tz = "America/New_York")

one_pm
one_pm + ddays(1)
```

Why is one day after 1pm on March 12 2pm on March 13?!  If you look carefully at the date you might also notice that the time zones have changed. Because of DST, March 12 only has 23 hours, so if add a full days worth of seconds we end up with a different hour.

### Periods

You can use __periods__ to handle irregularities in the timeline. Periods are time spans that are work with "human" times, like days, months, and seconds. Periods don't have fixed length in seconds, which lets them work in an intuitive, human friendly way. 

```{r}
one_pm
one_pm + days(1)
```

Like durations, periods can be created with a number of friendly constructor functions. 

```{r}
seconds(15)
minutes(10)
hours(c(12, 24))
days(7)
months(1:6)
weeks(3)
years(1)
```

You can add and multiply periods:

```{r}
10 * (months(6) + days(1))
days(50) + hours(25) + minutes(2)
```

And of course, add them to dates. Compared to durations, periods will usually do what you expect:

```{r}
# A leap year
ymd("2016-01-01") + dyears(1)
ymd("2016-01-01") + years(1)

# Daylight Savings Time
one_pm + ddays(1)
one_pm + days(1)
```

Let's use periods to fix an oddity related to our flight dates. Some planes appear to have arrived at their destination _before_ they departed from New York City.

```{r}
flights_dt %>% 
  filter(arr_time < dep_time) 
```

These are overnight flights. We used the same date information for both the departure and the arrival times, but these flights arrived on the following day. We can fix this by adding `days(1)` to the arrival time of each overnight flight.

```{r}
flights_dt <- flights_dt %>% 
  mutate(
    overnight = arr_time < dep_time,
    arr_time = arr_time + days(overnight * 1),
    sched_arr_time = sched_arr_time + days(overnight * 1)
  )
```

Now all of our flights obey the laws of physics.

```{r}
flights_dt %>% 
  filter(overnight, arr_time < dep_time) 
```

### Intervals

It's obvious what `dyears(1) / ddays(365)` should return. It should return one because durations are always represented by seconds, and a duration of a year is defined as 365 days worth of seconds.

What should `years(1) / days(1)` return? Well, if the year was 2015 it should return 365, but if it was 2016, it should return 366! There's not quite enough information for lubridate to give a single clear answer. What it does instead is give an estimate, with a warning:

```{r}
years(1) / days(1)
```

If you want a more accurate measurement, you'll have to use an __interval__ instead of a a duration. An interval is a duration with a starting point - that makes it precise so you can determine exactly how long it is:

```{r}
next_year <- today() + years(1)
(today() %--% next_year) / ddays(1)
```

To find out how many periods fall into an interval, you need to use integer division:

```{r}
(today() %--% next_year) %/% days(1)
```

### Summary

How do you pick between duration, periods, and intervals? As always, pick the simplest data structure that solves your problem. If you only care about physical time, use a duration; if you need to add human times, use a period; if you need to figure out how long a span is in human units, use an interval.

The following diagram summarises the interelationships between the different data types:

```{r, echo = FALSE}
knitr::include_graphics("diagrams/datetimes-arithmetic.png")
```

### Exercises

1.  Why is there `months()` but no `dmonths()`?

1.  Create a vector of dates giving the first day of every month in 2015.
    Create a vector of dates giving the first day of every month
    in the _current_ year.

1.  Write a function that given your birthday (as a date), returns 
    how old you are in years.

1.  Why can't `(today() %--% next_year) / months(1)` work?

## Time zones 

Time zones are an enormously complicated topic because of their interaction with geopolitical entities. Fortunately we don't need to dig into all the details as they're not all important for data analysis, but there are a few challenges we'll need to tackle head on.

<https://github.com/valodzka/tzcode/blob/master/Theory>

### Time zone names

The first challange is that the names of time zones that you're familiar with are not very general. For example, if you're an American you're probably familiar with EST, or Eastern Standard Time. However, both Australia and Canada also have Eastern standard times which mean different things! To avoid confusion R uses the international standard IANA time zones. These don't have a terribly consistent naming scheme, but tend to fall in one of three camps:

*   "Continent/City", e.g. "America/Chicago", "Europe/Paris", "Australia/NSW".
    Sometimes there are three parts if there have been multiple rules over time 
    for a smaller region (e.g. "America/North_Dakota/New_Salem" 
    vs"America/North_Dakota/Beulah"). 
    

*   "Country/Region" and "Country", e.g. "US/Central", "Canada/Central",
    "Australia/Sydney", "Japan". These are generally easiest to use if the
    time zone you want is present in the database.

  
*   Other, e.g. "CET", "EST".  These are best avoided as they are confusing
    and ambiguous.

You can see a complete list of all time zone names that R knows about with `OlsonNames()`:

```{r}
length(OlsonNames())
head(OlsonNames())
```

And find out what R thinks your current time zone is with `Sys.timezone()`:

```{r}
Sys.timezone()
```

### Daylight Savings Time

An additional complication of time zones is daylight savings time (DST): many time zones shift by an hour during summer time.  For example, the same instants may be the same time or difference times in Denver and Phoenix over the course of the year:

```{r}
x1 <- ymd_hm("2015-01-10 13:00", "2015-05-10 13:00")
with_tz(x1, tzone = "America/Denver")
with_tz(x1, tzone = "America/Phoenix")
```

DST is also challening because it creates discontinuities. What is one day after 1am on  March 13 in New York city? There are two possibilities!

```{r}
nyc <- function(x) {
  ymd_hms(x, tz = "America/New_York")
}
nyc("2016-03-13 01:00:00") + ddays(1)
nyc("2016-03-13 01:00:00") + days(1)
```

This also creates a challenge for determining how much time has elapsed between two date-times. Lubridate also offers solution for this: the __interval__, which you can coerce into either a duration or a period:

```{r}
inst <- nyc("2016-03-13 01:00:00") %--% nyc("2016-03-14 01:00:00")
as.duration(inst)
as.period(inst)
```

### Changing the time zone

In R, time zone is an attribute of the date-time that only controls printing. For example, these three objects represent the same instant in time:

```{r}
x1 <- ymd_hms("2015-06-01 12:00:00", tz = "America/New_York")
x2 <- ymd_hms("2015-06-01 18:00:00", tz = "Europe/Copenhagen")
x3 <- ymd_hms("2015-06-02 04:00:00", tz = "Pacific/Auckland")
```

If you don't specify the time zone, lubridate always assumes UTC.

You can check that's true by subtracting them (we'll talk more about that in the next section)

```{r}
x1 - x2
x1 - x3
```

Operations that drop attributes, such as `c()` will drop the time zone attribute from your date-times. In that case, the date-times will display in your local time zone:

```{r}
x4 <- c(x1, x2, x3)
x4
```

There are two ways to change the time zone:

*   Keep the instant in time the same, and change how it's displayed.
  
    ```{r}
    x4a <- with_tz(x4, tzone = "Australia/Lord_Howe")
    x4a
    x4a - x4
    ```
    
    (This nicely illustrates another possible incorrect believe you might hold:
    that time zones are always whole number changes.)

*   Change the underlying instant in time:

    ```{r}
    x4b <- force_tz(x4, tzone = "Australia/Lord_Howe")
    x4b
    x4b - x4
    ```

### UTC

If you do not set the time zone, lubridate will automatically assign the date-time to Coordinated Universal Time (UTC). Coordinated Universal Time is the standard time zone used by the scientific community and roughly equates to its predecessor, Greenwich Meridian Time. Since Coordinated Universal time does not follow Daylight Savings Time, it is straightforward to work with times saved in this time zone.

```{r}
ymd_hms("2015-06-02 04:00:00")
```