r4ds/intro.Rmd

---
layout: default
title: Welcome
output: bookdown::html_chapter
---
  
# Welcome
  
## Overview

The goal of "R for Data Science" is to give you a solid foundation into using R to do data science. The goal is not to be exhaustive, but to instead focus on what we think are the critical skills for data science:

* Getting your data into R so you can work with it. On disk, in database,
  on the web. 

* Wrangling your data into a tidy form, so it's easier to work with. This let's 
  you spend your time struggling with your questions, not fighting to get data
  into the right form for different functions.
  
* Transforming your data to add variables and compute basic summaries. 

* Visualising your data to gain insight. Visualisations are one of the most
  important tools of data science because they can surprise you: you can
  see something in a visualisation that you did not expect. Visualisations
  are also really helpful for helping you refine your questions of the data.

* Modelling your data to scale visualisations to larger datasets, and to
  remove strong patterns. Modelling is a very deep topic - we can't possibly
  cover all the details, but we'll give you a taste of how you can use it,
  and where you can go to learn more.

* Communicating your results to others. It doesn't matter how great your
  analysis is unless you can communicate the results to others. We'll show
  how you can create static reports with rmarkdown, and interactive apps with
  shiny.

[Hadley's standard data science diagram]

## Learning data science

Above, I've listed the components of the data science process in roughly the order you'll encounter them in an analysis (although of course you'll iterate multiple times). This, however, is not the order you'll encounter them in this book. This is because:

* Starting with data ingest is boring. It's much more interesting to learn
  some new visualisation and manipulation tools on data that's already been
  imported and cleaned. You'll later learn the skills to apply these new ideas
  to your own data.
  
* Some topics, like modelling, are best explained with other tools, like
  visualisation and manipulation. These topics need to come later in the book.

We've honed this order based on our experience teaching live classes, and it's been carefully designed to keep you motivated. We try and stick to a similar pattern within each chapter: give some bigger motivating examples so you can see the bigger picture, and then dive into the details.

Each section of the book also comes with exercises to help you practice what you've learned. It's tempting to skip these, but there's no better way to learn than practicing. If you were taking a class with either of us, we'd force you to do them by making them homework. (Sometimes I feel like teaching is the art of tricking people to do what's in their own best interests.)

## Talking about data science

Throughout the book, we will discuss the principles of data that will help you become a better scientist. That begins here. We will refer to the terms below throughout the book because they are so useful. 

* A _variable_ is a quantity, quality, or property that you can measure. 

* A _value_ is the state of a variable when you measure it. The value of a 
  variable may change from measurement to measurement.

* An _observation_ is a set of measurments you make under similar conditions 
  (usually all at the same time or on the same object). Observations contain 
  values that you measure on different variables. 

These terms will help us speak precisely about the different parts of a data set. They will also provide a system for turning data into insights.

This book focuses exclusively on structured data sets: collections of values that are each associated with a variable and an observation. There are lots of data that doesn't naturally fit in this paradigm: images, sounds, trees, text. But data frames are extremely common in science and in industry and we believe that they're a great place to start your data analysis journey.

## R and big data

This book also focuses almost exclusively on in-memory datasets.

* Small data: data that fits in memory on a laptop, ~10 GB. Note that small
  data is still big! R is great with small data. Pointer to data.table.
  
* Medium data: data that fits in memory on a powerful server, ~5 TB. It's
  possible to use R with this much data, but it's challenging. Dealing
  effectively with medium data requires effective use of all cores on a
  computer. It's not that hard to do that from R, but it requires some thought,
  and many packages do not take advantage of R's tools.
  
* Big data: data that must be stored on disk or spread across the memory of
  multiple machines. Writing code that works efficiently with this sort of data
  is a very challenging. Tools for this sort of data will never be written in
  R: they'll be written in a language specially designed for high performance
  computing like C/C++, Fortran or Scala. But R can still talk to these systems.
  
The other thing to bear in mind, is that while all your data might be big, typically you don't need all of it to answer a specific question:

* Many questions can be answered with the right small dataset. It's often
  possible to find a subset, subsample, or summary that fits in memory and
  still allows you to answer the question you're interested in. The challenge
  here is finding the right small data, which often requires a lot of iteration.
  
* Other challenges are because an individual problem might fit in memory,
  but you have hundreds of thousands or millions of them. For example, you 
  might want to fit a model to each person in your dataset. That would be
  trivial if you had just 10 or 100 people, but instead you have a million.
  Fortunately each problem is independent (sometimes called embarassingly
  parallel), so you just need a system (like hadoop) that allows you to
  send different datasets to different computers for processing.

## Prerequisites

To run the code in this book, you will need to install both R and the RStudio IDE, an application that makes R easier to use. Both are free and easy to install.

### R

To install R, visit <http://cran.r-project.org> and click the link that matches your operating system. What you do next will depend on your operating system.

* Mac users should click the `.pkg` file at the top of the page. This file contains the most current release of R. Once the file is downloaded, double click it to open an R installer. Follow the directions in the installer to install R.

* Windows users should click "base" and then download the most current version of R, which will be linked at the top of the page.

* Linux users should select their distribution and then follow the distribution specific instructions to install R. <https://cran.r-project.org/bin/linux/> includes these instructions alongside the files to download.

### RStudio

After you install R, visit <http://www.rstudio.com/download> to download the RStudio IDE. Choose the installer for your system. Then click the link to download the application. Once you have the application, installation is easy. Once RStudio IDE is installed, open it as you would open any other application.

Brief RStudio orientation (code, console, and output). Pointers to where to learn more.

Important keyboard shortcuts:

* Cmd + Enter: sends current line from editor to console.
* Tab: suggest possible completions for the text you've typed.
* Cmd + ↑: in the console, searches all commands you've typed that start with 
  those characters.
* Cmd + Shift + F10: restart.
* Alt + Shift + K: the keyboard shortcut that shows all the keyboard shortcuts.

Note about turning on save/load session off.

### R Packages

An R _package_ is a collection of functions, data sets, and help files that extends the R language. We will a lot of R packages in this book. To install them all, open RStudio and run:

```{r eval = FALSE}
install.packages(c(
  "DBI", "devtools", "dplyr", "ggplot2", "haven", "knitr", "lubridate", 
  "packrat", "readr", "rmarkdown", "RSQLite", "rvest", "scales", "shiny", 
  "stringr", "tidyr"
))
```

R will download the packages from CRAN and install them in your system library. If you have problems installing, make that you are connected to the internet, and that you haven't blocked <http://cran.r-project.org> in your firewall or proxy.

After you have downloaded the packages, you can load any of the packages into your current R session with the `library()` command, e.g.

```{r eval = FALSE}
library(tidyr)
```

You will not be able to use the functions, objects, and help files in a package until you load it with `library()`. You will need to reload the package if you start a new R session.

### Getting help

*   Google. Always a great place to start! Adding "R" to a query is usually
    enough to filter it down. If you ever hit an error message that you 
    don't know how to handle, great idea to google it. 
    
    If your operating system defaults to another language, you can use 
    `Sys.setenv(LANGUAGE = "en")` to tell R to use english. That's likely to
    get you to common solutions more quickly.
  
*   StackOverflow. How to make a reproducible example. 
    ([reprex](https://github.com/jennybc/reprex))
    
    Unfortunately the R stackoverflow community is not always the friendliest.
  
*   Twitter. #rstats hashtag is very welcoming. Great way to keep up with 
    what's happening in the community.

## Acknowledgements

* Jenny Bryan and Lionel Henry for many helpful discussions around working
  with lists and list-columns.