---
title: Joins
author: David Kane
tutorial:
  id: joins
output:
  learnr::tutorial:
    progressive: yes
    allow_skip: yes
runtime: shiny_prerendered
description: 'Tutorial for Chapter 19: Joins'
---

```{r setup, include = FALSE}
library(learnr)
library(tutorial.helpers)
library(knitr)
library(dplyr)
library(babynames)
library(lubridate)

knitr::opts_chunk$set(echo = FALSE)
knitr::opts_chunk$set(out.width = '90%')
options(tutorial.exercise.timelimit = 60, 
        tutorial.storage = "local") 

library(nycflights13)

flights2 <- flights |> 
  select(year, time_hour, origin, dest, tailnum, carrier)
flights2

x <- tribble(
  ~key, ~val_x,
     1, "x1",
     2, "x2",
     3, "x3"
)
y <- tribble(
  ~key, ~val_y,
     1, "y1",
     2, "y2",
     4, "y3"
)

df1 <- tibble(key = c(1, 2, 2), val_x = c("x1", "x2", "x3"))
df2 <- tibble(key = c(1, 2, 2), val_x = c("x1", "x2", "x3"))

parties <- tibble(
  q = 1:4,
  party = ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03"))
)

employees <- tibble(
  name = sample(babynames::babynames$name, 100),
  birthday = ymd("2022-01-01") + (sample(365, 100, replace = TRUE) - 1)
)

parties2 <- tibble(
  q = 1:4,
  party = ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03")),
  start = ymd(c("2022-01-01", "2022-04-04", "2022-07-11", "2022-10-03")),
  end = ymd(c("2022-04-03", "2022-07-11", "2022-10-02", "2022-12-31"))
)

parties3 <- tibble(
  q = 1:4,
  party = ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03")),
  start = ymd(c("2022-01-01", "2022-04-04", "2022-07-11", "2022-10-03")),
  end = ymd(c("2022-04-03", "2022-07-10", "2022-10-02", "2022-12-31"))
)

```

```{r copy-code-chunk, child = system.file("child_documents/copy_button.Rmd", package = "tutorial.helpers")}
```

```{r info-section, child = system.file("child_documents/info_section.Rmd", package = "tutorial.helpers")}
```

## Introduction
### 

This tutorial covers [Chapter 20: Strings](https://r4ds.hadley.nz/joins.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund. 
This chapter will introduce you to two important types of joins:
Mutating joins, which add new variables to one data frame from matching observations in another.

Filtering joins, which filter observations from one data frame based on whether or not they match an observation in another.

You will also learn about basic joins, keys, foreign keys and more which is basically like creating a relational database in SQL.


## Keys
### 

To understand joins, you need to first understand how two tables can be connected through a pair of keys, within each table. In this section, you’ll learn about the two types of key and see examples of both in the datasets of the ["nycflights13"](https://www.rdocumentation.org/packages/nycflights13/versions/1.0.1) package. You’ll also learn how to check that your keys are valid, and what to do if your table lacks a key.

### Exercise 1

Every join involves a pair of keys: a primary key and a foreign key. A primary key is a variable or set of variables that uniquely identifies each observation.

As a example run `airlines`.

```{r keys-1, exercise = TRUE}

```

```{r keys-1-hint-1, eval = FALSE}
airlines
```

### 

`airlines` records two pieces of data about each airline: its `carrier` code and its full `name`. You can identify an airline with its two letter carrier code, making carrier the primary key.

### Exercise 2

Run `airports`.

```{r keys-2, exercise = TRUE}

```

```{r keys-2-hint-1, eval = FALSE}
airports
```

### 

`airports` records data about each airport. You can identify each airport by its three letter airport code, making `faa` the primary key.

### Exercise 3

Run `planes`.

```{r keys-3, exercise = TRUE}

```

```{r keys-3-hint-1, eval = FALSE}
planes
```

### 

`planes` records data about each plane. You can identify a plane by its tail number, making `tailnum` the primary key.

### Exercise 4

Run `weather`.

```{r keys-4, exercise = TRUE}

```

```{r keys-4-hint-1, eval = FALSE}
weather
```

### 

`weather` records data about the weather at the origin airports. You can identify each observation by the combination of location and time, making `origin` and `time_hour` the compound primary key.

When more than one variable is needed, the key is called a `compound key`.

### Exercise 5

Now that we have seen couple of examples of what `primary keys` are, let's explore what a `foreign key` is. A `foreign key` is a variable (or set of variables) that corresponds to a `primary key` in another table.

Run `flights$tailnum`.

```{r keys-5, exercise = TRUE}

```

```{r keys-5-hint-1, eval = FALSE}
flights$tailnum
```

```{r include = FALSE}
flights$tailnum
```

### 

`flights$tailnum` is a foreign key that corresponds to the primary key `tailnum` in the `planes` tibble.

### Exercise 6

Run `flights$carrier`.

```{r keys-6, exercise = TRUE}

```

```{r keys-6-hint-1, eval = FALSE}
flights$carrier
```

```{r include = FALSE}
flights$carrier
```

### 

`flights$carrier` is a foreign key that corresponds to the primary key `carrier` in the airlines tibble.

### Exercise 7

Run `flights$origin`.

```{r keys-7, exercise = TRUE}

```

```{r keys-7-hint-1, eval = FALSE}
flights$origin
```

```{r include = FALSE}
flights$origin
```

### 

`flights$origin` is a foreign key that corresponds to the primary key `faa` in the `airpors` tibble.

The relationships which we saw can be visualized like:

```{r}
knitr::include_graphics("images/relational.png")
```

### Exercise 8

Now that that we’ve identified the primary keys in each table, it’s good practice to verify that they do indeed uniquely identify each observation. One way to do that is to `count()` the primary keys and look for entries where `n` is greater than one.

Start a pipe with `planes` to the `count()` function and within count set the argument as `tailnum`. Start a pipe  from `count()` to filter and Apply the `filter()` function to `n > 1`. This will filter out any elements where the count (n) is not greater than 1.

```{r keys-8, exercise = TRUE}

```

```{r keys-8-hint-1, eval = FALSE}
... |> 
  count(tailnum) |> 
  filter(...)
```

```{r include = FALSE}
planes |> 
  count(tailnum) |> 
  filter(n > 1)
```

### 

We see no observations or records which reveals that `tailnum` is good primary key and the planes tibble  look good to go.

### Exercise 9

Let's perform a similar check using `weather` data. Please replicate the previous code and modify the dataset to `weather`, while adjusting the `count()` argument to include `time_hour` and `origin` only.

```{r keys-9, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r keys-9-hint-1, eval = FALSE}
... |> 
  count(time_hour, ...) |> 
  filter(n > 1)
```

```{r include = FALSE}
weather |> 
  count(time_hour, origin) |> 
  filter(n > 1)
```

### 

There are also no records when we run the previous code which verifies that time_hour and origin are good primary keys. But what if there were some missing values with the primary keys?

### Exercise 10

Start a pipe with `planes` to `filter()` and set `is.na(tailnum)` as the argument.

```{r keys-10, exercise = TRUE}

```

```{r keys-10-hint-1, eval = FALSE}
planes |> 
  filter(is.na(...))
```

```{r include = FALSE}
planes |> 
  filter(is.na(tailnum))
```

### 

We should be getting no records when we run the code since the table has no NULL values, if there were any then `is.na()` would display them.

### Exercise 11

Let's perform a similar check using `weather` data. Please replicate the previous code and modify the data set to `weather`, while adjusting the `is.na()` argument to include `time_hour`and a `|` or operator then pass `origin` for another `is.na()` argument.

```{r keys-11, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r keys-11-hint-1, eval = FALSE}
... |> 
  filter(is.na(time_hour) | is.na(...))
```

```{r include = FALSE}
planes |> 
  filter(is.na(tailnum) | is.na(origin))
```

### 

So far we haven’t talked about the primary key for `flights`. It’s not super important here, because there are no data frames that use it as a foreign key, but it’s still useful to consider because it’s easier to work with observations if we have some way to describe them to others.

### Exercise 12

After a little thinking and experimentation, we determined that there are three variables that together uniquely identify each flight: `time_hour`, `carrier` and `flight`.

Let's verify if the following variables can be used a primary keys, start a pipe with `flights` to `count()` with the three variable we talked about earlier and then pipe that to `filter()` and set n to be greater than 1 .

```{r keys-12, exercise = TRUE}

```

```{r keys-12-hint-1, eval = FALSE}
... |> 
  count(time_hour, ..., flight) |> 
  ...
```

```{r include = FALSE}
flights |> 
  count(time_hour, carrier, flight) |> 
  filter(n>1)
```

### 

Does the absence of duplicates automatically make `time_hour`-`carrier`-`flight` a primary key? It’s certainly a good start, but it doesn’t guarantee it. For example, are altitude and latitude a good primary key for airports?

### Exercise 13

Let's check if altitude and latitude are good primary keys for airports, so copy the previous code and change the dataset to `airports` and change the argument within `count()` to `alt , lat`.

```{r keys-13, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r keys-13-hint-1, eval = FALSE}
... |> 
  count(..., ...) |> 
  filter(...)
```

```{r include = FALSE}
airports |> 
  count(alt, lat) |> 
  filter(n>1)
```

### 

Using altitude and latitude to identify an airport is unreliable, and it's generally not possible to determine from data alone whether a combination of variables can serve as a good primary key. However, for flights, the combination of time_hour, carrier, and flight appears reasonable. Having multiple flights with the same flight number in the air simultaneously would be confusing for both the airline and its customers.

### Exercise 14

That said, we might be better off introducing a simple numeric surrogate key using the row number instead of confusing the airlines and customers by using three columns are primary keys. 

Let's make a new primary key `id` and to add a new column `id` with row numbers to the flights dataset, start a pipe with `flights` and use `mutate()` function. Within `mutate()` assign `row_number()` function to the `id` column and set `.before` to 1.

```{r keys-14, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r keys-14-hint-1, eval = FALSE}
flights |> 
  mutate(id = ..., .before = ...)
```

```{r include = FALSE}
flights |> 
  mutate(id = row_number(), .before = 1)
```

### 

This code adds a new column id with row numbers to the `flights` dataset, positioning it as the first column. In conclusion, Surrogate keys can be particular useful when communicating to other humans: it’s much easier to tell someone to take a look at flight 2001 than to say look at UA430 which departed 9am 2013-01-03.

## Basic Joins
### 

Now that you understand how data frames are connected via keys, we can start using joins to better understand the flights dataset. In this section, you’ll learn how to use one mutating join, `left_join()`, and two filtering joins, `semi_join()` and `anti_join()`.

### Exercise 1

Start a pipe with `flights` to `select()` where you will select the following variables: `year, time_hour, origin, dest, tailnum, carrier` and then assign the whole code to `flights2` and run `flights2` on a new line.

```{r basic-joins-1, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-1-hint-1, eval = FALSE}
flights2 <- ... |> 
 ...(year, ..., origin, dest, ....,...)
flights2
```

```{r include = FALSE}
flights2 <- flights |> 
select(year, time_hour, origin, dest, tailnum,carrier)
```

### 

A mutating join allows you to combine variables from two data frames: it first matches observations by their keys, then copies across variables from one data frame to the other. 

### Exercise 2

There are four types of mutating join, but there’s one that you’ll use almost all of the time: `left_join()`. It’s special because the output will always have the same rows as x. The primary use of `left_join()` is to add in additional metadata. 

To add full airline name to the flights2 data, let's start a pipe with `flights2` to `left_join()` and pass in `airlines` as argument.

```{r basic-joins-2, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-2-hint-1, eval = FALSE}
... |>
  left_join(...)
```

```{r include = FALSE}
flights2 |>
  left_join(airlines)
```

### 

Like `mutate()`, the join functions add variables to the right, so if your dataset has many variables, you won’t see the new ones as we saw inb the previous code.

### Exercise 3

Instead of filling in the names of the airlines, let's add metadata about the wind speed, temperature and the time the plane departed.

Copy the previous code, modify the arguemet for `left_join()` to `weather |> select(origin,time_hour, temp, wind_speed)` which just select the origin, time, temperature and wind speed from `weather` data set.

```{r basic-joins-3, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-3-hint-1, eval = FALSE}
... |> 
  ...(weather |> select(origin, time_hour, temp, wind_speed))
```

```{r include = FALSE}
flights2 |> 
  left_join(weather |> select(origin, time_hour, temp, wind_speed))
```

### 

If you want to explore `left_join()` more in depth, check out this [website](https://dplyr.tidyverse.org/reference/mutate-joins.html).

### Exercise 4

What if we want ot know the size of the plane? 
Copy the previous code and modify the argument in `left_join()` to `planes |> select(tailnum, type, engines, seats)` which selects the variable which are associated with a size of a plane.

```{r basic-joins-4, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-4-hint-1, eval = FALSE}
... |> 
  ...(...)
```

```{r include = FALSE}
flights2 |> 
  left_join(planes |> select(tailnum, type, engines, seats))
```

### 

What happens when `left_join()` fails to find a match for a row in x?

### Exercise 5

To demonstrate this, start a pipe with `flights2` to `filter()` where you only select the records where the `tailnum` is equal to `N3ALAA` which doesn't exist and then create another pipe from `filter()` to `left_join()` and pass in `planes |> select(tailnum, type, engines, seats)` as the argument 

```{r basic-joins-5, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-5-hint-1, eval = FALSE}
flights2 |> 
  filter(...) |>
  left_join(...)
```

```{r include = FALSE}
flights2 |> 
  filter(tailnum == "N3ALAA") |> 
  left_join(planes |> select(tailnum, type, engines, seats))
```

### 

When `left_join()` fails to find a match for a row in x, it fills in the new variables with missing values. For example, there’s no information about the plane with tail number `N3ALAA` so the type, engines, and seats will be missings


### Exercise 6

By default, `left_join()` will use all variables that appear in both data frames as the join key, the so called natural join.  This is a useful heuristic, but it doesn’t always work. 

To demostarte the flaw, join the `flights2` with the `planes` dataset using `left_join()`.
To do so, start of by creating pipe with `flights2` to `left_join()` and pass in `planes` as the arguement.

```{r basic-joins-6, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-6-hint-1, eval = FALSE}
... |>
  left_join(...)
```

```{r include = FALSE}
flights2 |>
  left_join(planes)
```

### 

We get a lot of missing matches because our join is trying to use tailnum and year as a compound key. Both flights and planes have a year column but they mean different things: `flights$year` is the year the flight occurred and `planes$year` is the year the plane was built.

### Exercise 7

We only want to join on `tailnum` so we need to provide an explicit specification with `join_by()`.

Copy the previous code and within `left_join()`, add `join_by()` and pass in `tailnum` as the argument.

```{r basic-joins-7, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-7-hint-1, eval = FALSE}
... |> 
  ...(planes, join_by(...))
```

```{r include = FALSE}
flights2 |>
  left_join(planes, join_by(tailnum))
```

### 

`join_by(tailnum)` is short for `join_by(tailnum == tailnum)`. It’s important to know about this fuller form for two reasons. Firstly, it describes the relationship between the two tables: the keys must be equal. That’s why this type of join is often called an equi join.

### Exercise 8

As we saw `join_by()` gives us the flexibility to join  based on the variables we want. For example we know that we can join `flights2` and `airports` by `dest` or `origin`.

Copy the previous code change `planes` to `airports` and in `join_by()` set `dest == faa`.

```{r basic-joins-8, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-8-hint-1, eval = FALSE}
... |> 
  ...(airports, join_by(...))
```

```{r include = FALSE}
flights2 |>
  left_join(airports, join_by(dest == faa))
```

### 

Now let's change `join_by()` to change the tables to join by `origin`.


### Exercise 9

Copy the previous code and change `join_by()` arguement to `origin == faa`.

```{r basic-joins-9, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-9-hint-1, eval = FALSE}
... |> 
  ...(airports, join_by(...== ...))
```

```{r include = FALSE}
flights2 |>
  left_join(airports, join_by(origin == faa))
```

### 

If we compare both the tibbbles we can see the difference in the `name` variable.

### Exercise 10

As you might guess the primary action of a filtering join is to filter the rows. There are two types: semi-joins and anti-joins. Semi-joins keep all rows in x that have a match in y. For example, we could use a semi-join to filter the airports dataset to show just the origin airports.

Start a pipe with `airports` to `semi_join()` and for the table which we will join, pass in `flights2` and also pass in `join_by(faa == dest)` which will collect all the rows that match in both tibbles.

```{r basic-joins-10, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-10-hint-1, eval = FALSE}
... |> 
  ...(flights2, join_by(faa == ...))
```

```{r include = FALSE}
airports |> 
  semi_join(flights2, join_by(faa == dest))
```

### 

Compared to `semi_join()` , Anti joins are the opposite: they return all rows in x that don’t have a match in y. In next few excises we will talk about anti joins which are opposite of semi joins.

### Exercise 11

Let's find rows that are missing from airports by looking for flights that don’t have a matching destination airport using `anti_join()`. First start a pipe with `flights2` to `anti_join()` which takes in `airports` as the joining tibble and we will tell it to join by `dest == faa`.
 
```{r basic-joins-11, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-11-hint-1, eval = FALSE}
... |> 
  ...(airports, join_by(... == faa)
```

```{r, include = FALSE}
flights2 |> 
  anti_join(airports, join_by(dest == faa)) |> 
  distinct(dest)
```

### 

We found four rows that are missing from `airports` but not in `flights`.

### Exercise 12

Let's now find which `tailnums` are missing from planes. Copy the previous code, change the the table we will join from `airports` to `planes` and then modify `join_by()` argument to `tailnum`. Continue the pipe to `distinct()` and add `tailnum` as an argument/

```{r basic-joins-12, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r basic-joins-12-hint-1, eval = FALSE}
... |>
  anti_join(..., join_by(tailnum)) |> 
  ...(tailnum)
```

```{r, include = FALSE}
flights2 |> 
  anti_join(planes, join_by(tailnum)) |> 
  distinct(tailnum)
```

### 

Good Work! You now have the knowledge of basic joins that you will maybe need in the future for your final project.

## How do joins work?
Now that you’ve used joins a few times it’s time to learn more about how they work, focusing on how each row in x matches rows in y. We’ll begin by introducing a visual representation of joins, using the simple tibbles defined below  In these examples we’ll use a single key called key and a single value column (val_x and val_y), but the ideas all generalize to multiple keys and multiple values.

### Exercise 1

These are the tibbles we will use to demonstrate or visualize the images that will be shown the exercises going on.

```{r how-do-joins-work-1}
x <- tribble(
  ~key, ~val_x,
     1, "x1",
     2, "x2",
     3, "x3"
)
y <- tribble(
  ~key, ~val_y,
     1, "y1",
     2, "y2",
     4, "y3"
)
```

### 

The graphical representation of the `x` and `y` tibble:

```{r}
knitr::include_graphics("images/photo1.png")
```

### Exercise 2

To understand how joins work, it’s useful to think of every possible match. Here we show that with a grid of connecting lines.

```{r}
knitr::include_graphics("images/setup2.png")
```

### 

The image above depicts the foundation of our visual representation, showing potential matches between x and y as intersecting lines. The output's rows and columns align with the horizontal x table.

### Exercise 3

To describe a join type, we use dots to indicate matches. The matches determine the output rows, which include the key, x values, and y values. Image below displays an inner join, retaining rows only when the keys are equal.

```{r}
knitr::include_graphics("images/inner.png")
```

### 

An inner join matches each row in x to the row in y that has the same value of key. Each match becomes a row in the output.We can apply the same principles to explain the outer joins, which keep observations that appear in at least one of the data frames. 

### Exercise 4

Outer joins retain observations that appear in at least one of the data frames, following the same principles. They add a "virtual" observation to each frame with a key that matches when no other key matches, and NA values. There are three types of outer joins.

The image below is a visual representation of the left join where every row in x appears in the output.

```{r}
knitr::include_graphics("images/left.png")
```

### 

A left join keeps all observations in x, like in the image below. Every row of x is preserved in the output because it can fall back to matching a row of `NAs` in y.

### Exercise 5

The image below is a visual representation of the right join where every row of y appears in the output.

```{r}
knitr::include_graphics("images/right.png")
```

### 

A right join keeps all observations in y, like the image above. Every row of y is preserved in the output because it can fall back to matching a row of `NAs` in x. The output still matches x as much as possible; any extra rows from y are added to the end.

### Exercise 6

The image below is  a visual representation of the full join where every row in x and y appears in the output.

```{r}
knitr::include_graphics("images/full.png")
```

### 

A full join keeps all observations that appear in x or y, like the image above. Every row of x and y is included in the output because both x and y have a fall back row of `NAs`. Again, the output starts with all rows from x, followed by the remaining unmatched y rows.

### Exercise 7

The image below is  a visual representation Venn diagrams showing the difference between inner, left, right, and full joins.

```{r}
knitr::include_graphics("images/venn.png")
```

### 

The joins shown here are the so-called equi joins, where rows match if the keys are equal. Equi joins are the most common type of join, so we’ll typically omit the equi prefix, and just say “inner join” rather than “equi inner join”.

### Exercise 8

So far we’ve explored what happens if a row in x matches zero or one rows in y. What happens if it matches more than one row?

The image below is shows how two records in Y tibble connect to the same record in X tibble.

```{r}
knitr::include_graphics("images/match-types.png")
```

### 

There are three possible outcomes for a row in x:

If it doesn’t match anything, it’s dropped.
If it matches 1 row in y, it’s preserved.
If it matches more than 1 row in y, it’s duplicated once for each match.

### Exercise 9

In practice when there are two records connecting to same record, there are no issue but there is, however, one particularly dangerous case which can cause a combination explosion of rows. Imagine joining the following two tables:

Create two variables `df1` and `df2` on different lines. Assign `tibble(key = c(1, 2, 2), val_x = c("x1", "x2", "x3"))` to `df1` and assign `tibble(key = c(1, 2, 2), val_x = c("x1", "x2", "x3"))` to `df2`.

```{r how-do-joins-work-9, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r how-do-joins-work-9-hint-1, eval = FALSE}
df1 <- 
df2 <- 
```

```{r, include = FALSE}
df1 <- tibble(key = c(1, 2, 2), val_x = c("x1", "x2", "x3"))
df2 <- tibble(key = c(1, 2, 2), val_x = c("x1", "x2", "x3"))
```

### 

As you can see we have two duplicate keys in both tibbles.

### Exercise 10

Copy the previous code and on a new line create a pipe with `df1` to `inner_join()` and pass in `df2` and then join the tibble by `key` using `join_by()`.

```{r how-do-joins-work-10, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r how-do-joins-work-10-hint-1, eval = FALSE}
... |> 
  inner_join(df2, ...(key))
```

```{r, include = FALSE}
df1 |> 
  inner_join(df2, join_by(key))
```

### 

While the first row in df1 only matches one row in df2, the second and third rows both match two rows. This is sometimes called a many-to-many join, and will cause dplyr to emit a warning. If you are doing this deliberately, you can set relationship = "many-to-many", as the warning suggests.

## Non-equi joins
### 

### Exercise 1

Let's relax the restriction of equi joins and explore different criteria for matching rows. Before that, let's revisit a simplification we made earlier. In equi joins, the x keys and y keys are always equal, so we only display one key in the output. By requesting dplyr to keep both keys with `keep = TRUE`, we can modify the inner_join() code.

To join `x` with `y` using left_join(), start a pipe and specify y as the tibble to join. Use `key` as the joining criterion using `by` and include `keep = TRUE` to retain both keys.

```{r nonequi-joins-1, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-1-hint-1, eval = FALSE}
x |> 
  left_join(..., ... = "key", keep = ...)
```

```{r, include = FALSE}
x |> 
  left_join(y, by = "key", keep = TRUE)
```

### 

Below is the representation of when you do equi joins:

```{r}
knitr::include_graphics("images/inner-both.png")
```

### Exercise 2

When using non-equi joins, we always display both keys because their values can differ. For instance, instead of requiring equality between x$key and y$key, we can match when x$key is greater than or equal to y$key, as shown in image below. dplyr's join functions recognize this distinction and automatically show both keys in non-equi joins.

```{r}
knitr::include_graphics("images/gte.png")
```

### 

Non-equi join isn’t a particularly useful term because it only tells you what the join is not, not what it is. dplyr helps by identifying four particularly useful types of non-equi join: Cross joins match every pair of rows which we will explore in next few exercises.


### Exercise 3

A cross join matches everything, as the image below, generating the Cartesian product of rows. This means the output will have nrow(x) * nrow(y) rows.

```{r}
knitr::include_graphics("images/cross.png")
```

### 

The image above demonstrates a cross join matching each row in x with every row in y. Looking at the image we cans ee that cross joins are useful when generating permutations.

### Exercise 4

To make use of the permutations skills that this function has, let's first create a tibble `df` and assign `tibble(name = c("John", "Simon", "Tracy", "Max"))` to it.

```{r nonequi-joins-4, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-4-hint-1, eval = FALSE}
df <- tibble(name = ...(...))
```

```{r, include=FALSE}
df <- tibble(name = c("John", "Simon", "Tracy", "Max"))
```


### 

Looking at the tibble you can understand that we will be generating all the possible name combinations.

### Exercise 5

Copy the previous code and on a new line start a pipe with `df` to `cross_join()` and pass in `df` as the argument.

```{r nonequi-joins-5, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-5-hint-1, eval = FALSE}
df <- tibble(name = c("John", "Simon", "Tracy", "Max")) 
df |> 
  cross_join(...)
```

```{r, include = FALSE}
df <- tibble(name = c("John", "Simon", "Tracy", "Max"))
df |> 
  cross_join(df)
```

### 

Since we’re joining df to itself, this is sometimes called a self-join. Cross joins use a different join function because there’s no distinction between inner/left/right/full when you’re matching every row. 

Next up we will talk about the second type of non-equi join that dplyr has which is inequality joins.

### Exercise 6

Inequality joins use `<, <=, >=, or >` to restrict the set of possible matches. Below is what it looks like visually.

```{r}
knitr::include_graphics("images/lt.png")
```

### 

The image above is an inequality join where x is joined to y on rows where the key of x is less than the key of y. This makes a triangular shape in the top-left corner.

### Exercise 7

Inequality joins are extremely general, so general that it’s hard to come up with meaningful specific use cases. One small useful technique is to use them to restrict the cross join so that instead of generating all permutations, we generate all combinations:

Create a variable `df` and assign  `tibble(id = 1:4, name = c("John", "Simon", "Tracy", "Max"))` to it.

```{r nonequi-joins-7, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-7-hint-1, eval = FALSE}
df <- tibble(..., name = ...(...))
```

```{r, include = FALSE}
tibble(id = 1:4, name = c("John", "Simon", "Tracy", "Max"))
```


### 

Compared to cross joins, we can see that with the inequality join, we were able to stop the names from being duplicated and instead got combinations.

The next type of non equi joins are rolling joins which we will explore in the next few exercises.

### Exercise 8

Rolling joins return the closest match instead of all satisfying the inequality. to have  rolling join add `closest() to the inequality condition. For example, join_by(closest(x <= y)) finds the smallest y ≥ x, and join_by(closest(x > y)) finds the largest y < x. Example:

```{r}
knitr::include_graphics("images/closest.png")
```

### 

Rolling joins are particularly useful when you have two tables of dates that don’t perfectly line up and you want to find (e.g.) the closest date in table 1 that comes before (or after) some date in table 2.

### Exercise 9

To demonstrate a use case of rolling joins, let imagine that as the party planning commission for your office, you're responsible for organizing quarterly parties due to budget constraints. The rules for determining the party dates are as follows: parties are held on Mondays, the first week of January is skipped, and the first Monday of Q3 2022 falls on July 4, which needs to be postponed by a week. Here are the resulting party days: `2022-01-10`, `2022-04-04`, `2022-07-11`, `2022-10-03`.

Create a tibble `parties` and add columns `q` and pass in `1:4` for it then add another column `party` and pass in `ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03"))`.

```{r nonequi-joins-9, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-9-hint-1, eval = FALSE}
parties <- tibble(q = ..., party = ymd(c(...)))
```

```{r, include = FALSE}
parties <- tibble(q = 1:4, party = ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03")))
```

### 

Now that we have the party dates, we now have to have employee birthdays.

### Exercise 10

Let's now create a new tibble `employees` using `tibble()`. The tibble contains two columns `name` and `birthday`. Pass in `name = sample(babynames::babynames$name, 100)`, which takes sample of 100 baby names, to `name`. Then pass `ymd("2022-01-01") + (sample(365, 100, replace = TRUE) - 1)`, which generates birthdays, to `birthday`. 

On a new line call employees to have a look at the tibble.

```{r nonequi-joins-10, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-10-hint-1, eval = FALSE}
... <- tibble(
  name = sample(babynames::...$name, 100),
 ... = ymd(....) + (sample(365, 100, replace = ...) - 1)
)
employees
```

```{r, include = FALSE}
employees <- tibble(
  name = sample(babynames::babynames$name, 100),
 birthdays = ymd("2022-01-01") + (sample(365, 100, replace = TRUE) - 1)
)
employees
```

### 

We generated the `employees` tibble using babynames dataset and also generated their birthdays. Let's now use roller joins to find first party date that comes after their birthday.

### Exercise 11

To join the employees and parties tables, start a pipe with employees and use `left_join()`. Use `join_by()` to find the closest party dates after the birthday by applying `closest(birthday >= party)`.

```{r nonequi-joins-11, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-11-hint-1, eval = FALSE}
... |> 
  left_join(..., join_by(...(birthday >= party)))
```

```{r, include = FALSE}
employees <- tibble(
     name = sample(babynames::babynames$name, 100),
     birthday = ymd("2022-01-01") + (sample(365, 100, replace = TRUE) - 1)
)

parties <- tibble(
     q = 1:4,
     party = ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03")))

employees |> 
  left_join(parties, join_by(closest(birthday >= party)))
```

### 

There is, however, one problem with this approach: the folks with birthdays before January 10 don’t get a party.

### Exercise 12

Like mentioned previously this approach only looks for birthdays after January 10th and leave out birthdays before January 10th. To demonstrate this, Copy the previous code and change the join from `left_join()` to `right_join()`.

```{r nonequi-joins-12, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-12-hint-1, eval = FALSE}
employees |> 
  ...(parties, join_by(closest(birthday >= party)))
```

```{r, include = FALSE}
employees |> 
  right_join(parties, join_by(closest(birthday >= party)))
```

### 

To resolve that issue we’ll need to tackle the problem a different way, with overlap joins: the fourth type of non equi joins.

### Exercise 13

Let’s continue the birthday example to see how you might use them. There’s one problem with the strategy we used above: there’s no party preceding the birthdays Jan 1-9. So it might be better to be explicit about the date ranges that each party spans, and make a special case for those early birthdays.

Make a new tibble `parties2` using `tibble()` and define three columns: party, start, and end. Set the key as `q` and assign `1:4` to it. For the party column, pass in `ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03"))`. For the start column, pass in `ymd(c("2022-01-01", "2022-04-04", "2022-07-11", "2022-10-03"))`. For the end column, pass in `ymd(c("2022-04-03", "2022-07-11", "2022-10-02", "2022-12-31"))`. On a new line call `parties2` to check the new tibble.

```{r nonequi-joins-13, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-13-hint-1, eval = FALSE}
parties2 <- tibble(
  q = ...,
  ... = ymd(c(..., "2022-04-04", "2022-07-11", "2022-10-03")),
  ... = ymd(c("2022-01-01", "2022-04-04",..., "2022-10-03")),
  end = ymd(c("2022-04-03", "2022-07-11", "2022-10-02", ...))
)
parties2
```

```{r, include = FALSE}
parties2 <- tibble(
  q = 1:4,
  party = ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03")),
  start = ymd(c("2022-01-01", "2022-04-04", "2022-07-11", "2022-10-03")),
  end = ymd(c("2022-04-03", "2022-07-11", "2022-10-02", "2022-12-31"))
)
parties2
```


### 

Overlap joins provide three helpers that use inequality joins to make it easier to work with intervals:

`between(x, y_lower, y_upper)` is short for x >= y_lower, x <= y_upper.
`within(x_lower, x_upper, y_lower, y_upper)` is short for x_lower >= y_lower, x_upper <= y_upper.
`overlaps(x_lower, x_upper, y_lower, y_upper)` is short for x_lower <= y_upper, x_upper >= y_lower.


### Exercise 14

Hadley is hopelessly bad at data entry so he also wanted to check that the party periods don’t overlap. One way to do this is by using a self-join to check if any start-end interval overlap with another.

Start a pipe with `parties2` to `inner_join()`, we will join `parties2` to itself and we will join them by `overlaps(start, end, start, end), q < q`.

```{r nonequi-joins-14, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-14-hint-1, eval = FALSE}
... |> 
  inner_join(..., join_by(...(start, end, start, end), ...))
```

```{r, include = FALSE}
parties2 |> 
  inner_join(parties2, join_by(overlaps(start, end, start, end), q < q))
```

### 

If you want to explore move about overlap joins, check out [dplyr](https://dplyr.tidyverse.org/reference/join_by.html#overlap-joins).

### Exercise 15

When we ran the previous code, we see other info which we dont need, so let's remove those using `select()`. So copy the previous code and continue the pipe from `inner_join()` to `select()` and only select the `start.x, end.x, start.y, end.y`.

```{r nonequi-joins-15, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-15-hint-1, eval = FALSE}
parties2 |> 
  inner_join(parties2, join_by(overlaps(start, end, start, end), q < q)) |> 
  select(..., ..., ..., ...)
```

```{r, include = FALSE}
parties2 |> 
  inner_join(parties2, join_by(overlaps(start, end, start, end), q < q)) |>
  select(start.x, end.x, start.y, end.y)
```

### 

Ooops, there is an overlap, so let’s fix that problem and continue:

### Exercise 16

Go back to exercise 13 and copy your answer from there and paste it here, change the name of the tibble to`parties3` and change the 2nd data in `end` to `"2022-07-10"`.

```{r nonequi-joins-16, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-16-hint-1, eval = FALSE}
... <- tibble(
  ...,
  party = ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03")),
  ... = ymd(c("2022-01-01", "2022-04-04", ..., "2022-10-03")),
  end = ymd(c(..., "2022-07-10", ..., "2022-12-31"))
)
```

```{r, include = FALSE}
parties4 <- tibble(
  q = 1:4,
  party = ymd(c("2022-01-10", "2022-04-04", "2022-07-11", "2022-10-03")),
  start = ymd(c("2022-01-01", "2022-04-04", "2022-07-11", "2022-10-03")),
  end = ymd(c("2022-04-03", "2022-07-10", "2022-10-02", "2022-12-31"))
)
parties2
```

### 

Now that we have the dates ready, let's match each employee to their party.

### Exercise 17

Let's join `employees` and `parties3` by first creating a pipe with `employees` to `inner_join()`. We will be joining them with `between(birthday, start, end)` using `join_by()` and in addition also add `unmatched = "error"`.

```{r nonequi-joins-17, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r nonequi-joins-17-hint-1, eval = FALSE}
... |> 
      inner_join(parties3, ...(between(birthday, start, end)), unmatched = "...")
```

```{r, include = FALSE}
employees |> 
      inner_join(parties3, 
                 join_by(between(birthday, start, end)), unmatched = "error")
```

### 

The reason we added `unmatched = "error"` is because we want to quickly find out if any employees didn’t get assigned a party.

## Summary
### 

This tutorial covered [Chapter 19: Joins](https://r4ds.hadley.nz/joins.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund. 
This chapter introduced you to two important types of joins:
Mutating joins, which add new variables to one data frame from matching observations in another.

Filtering joins, which filter observations from one data frame based on whether or not they match an observation in another.

You also learned about basic joins, keys, foreign keys and more which is basically like creating a relational database in SQL.


```{r download-answers, child = system.file("child_documents/download_answers.Rmd", package = "tutorial.helpers")}
```
