dplyr's philosophy is to have small functions that do one thing well. The five simple functions (filter, arrange, select, mutate, and summarise) can be used to reveal new ways to describe data. When combined with group_by, these functions can be used to calculate group wise summary statistics.
Syntax commonalities
All these functions have a similar syntax:
The first argument to all these functions is always a data frame
Columns can be referred directly using bare variable names (i.e., without using $)
These functions do not modify the original data itself, i.e., they don't have side effects. Hence, the results should always be saved to an object.
We will use the built-in mtcars(opens new window) dataset to explore dplyr's single table verbs. Before converting the type of mtcars to tbl_df (since it makes printing cleaner), we add the rownames of the dataset as a column using rownames_to_column function from the tibble(opens new window) package.
library(dplyr)# This documentation was written using version 0.5.0
mtcars_tbl <- as_data_frame(tibble::rownames_to_column(mtcars,"cars"))# examine the structure of data
head(mtcars_tbl)# A tibble: 6 x 12# cars mpg cyl disp hp drat wt qsec vs am gear carb# <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>#1 Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4#2 Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4#3 Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1#4 Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1#5 Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2#6 Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
filter
filter helps subset rows that match certain criteria. The first argument is the name of the data.frame and the second (and subsequent) arguments are the criteria that filter the data (these criteria should evaluate to either TRUE or FALSE)
filter selects rows based on criteria, to select rows by position, use slice.
slice takes only 2 arguments: the first one is a data.frame and the second is integer row values.
This results in the same output as slice(mtcars_tbl, 6:9)
n() represents the number of observations in the current group
arrange
arrange is used to sort the data by a specified variable(s). Just like the previous verb (and all other functions in dplyr), the first argument is a data.frame, and consequent arguments are used to sort the data. If more than one variable is passed, the data is first sorted by the first variable, and then by the second variable, and so on..
For datasets that contain several columns, it can be tedious to select several columns by name. To make life easier, there are a number of helper functions (such as starts_with(), ends_with(), contains(), matches(), num_range(), one_of(), and everything()) that can be used in select. To learn more about how to use them, see ?select_helpers and ?select.
Note: While referring to columns directly in select(), we use bare column names, but quotes should be used while referring to columns in helper functions.
To rename columns while selecting:
select(mtcars_tbl, cylinders = cyl, displacement = disp)# A tibble: 32 x 2# cylinders displacement# <dbl> <dbl>#1 6 160.0#2 6 160.0#3 4 108.0#4 6 258.0#5 8 360.0#6 6 225.0# ... with 26 more rows
As expected, this drops all other variables.
To rename columns without dropping other variables, use rename:
mutate can be used to add new columns to the data. Like all other functions in dplyr, mutate doesn't add the newly created columns to the original data. Columns are added at the end of the data.frame.
Note the use of weight_ton while creating weight_pounds. Unlike base R, mutate allows us to refer to columns that we just created to be used for a subsequent operation.
To retain only the newly created columns, use transmute instead of mutate:
transmute(mtcars_tbl, weight_ton = wt/2, weight_pounds = weight_ton *2000)# A tibble: 32 x 2# weight_ton weight_pounds# <dbl> <dbl>#1 1.3100 2620#2 1.4375 2875#3 1.1600 2320#4 1.6075 3215#5 1.7200 3440#6 1.7300 3460# ... with 26 more rows
summarise
summarise calculates summary statistics of variables by collapsing multiple values to a single value. It can calculate multiple statistics and we can name these summary columns in the same statement.
To calculate the mean and standard deviation of mpg and disp of all cars in the dataset:
group_by can be used to perform group wise operations on data. When the verbs defined above are applied on this grouped data, they are automatically applied to each group separately.
We select columns from cars through hp and gear, order the rows by cyl and from highest to lowest mpg, group the data by gear, and finally subset only those cars have mpg > 20 and hp > 75
selected <- select(mtcars_tbl, cars:hp, gear)
ordered <- arrange(selected, cyl, desc(mpg))
by_cyl <- group_by(ordered, gear)
filter(by_cyl, mpg >20, hp >75)
Source: local data frame [9 x 6]
Groups: gear [3]# cars mpg cyl disp hp gear# <chr> <dbl> <dbl> <dbl> <dbl> <dbl>#1 Lotus Europa 30.4 4 95.1 113 5#2 Porsche 914-2 26.0 4 120.3 91 5#3 Datsun 710 22.8 4 108.0 93 4#4 Merc 230 22.8 4 140.8 95 4#5 Toyota Corona 21.5 4 120.1 97 3# ... with 4 more rows
Maybe we are not interested the intermediate results, we can achieve the same result as above by wrapping the function calls:
This can be a little difficult to read. So, dplyr operations can be chained using the pipe(opens new window)%>% operator. The above code transalates to:
mtcars_tbl %>%
select(cars:hp)%>%
arrange(cyl, desc(mpg))%>%
group_by(cyl)%>%
filter(mpg >20, hp >75)
summarise multiple columns
dplyr provides summarise_all() to apply functions to all (non-grouping) columns.
To find the number of distinct values for each column:
mtcars_tbl %>%
summarise_all(n_distinct)# A tibble: 1 x 12# cars mpg cyl disp hp drat wt qsec vs am gear carb# <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>#1 32 25 3 27 22 22 29 30 2 2 3 6
To find the number of distinct values for each column by cyl:
Note that we just had to add the group_by statement and the rest of the code is the same. The output now consists of three rows - one for each unique value of cyl.
To summarise specific multiple columns, use summarise_at
The pipe (%>%) operator(opens new window) could be used in combination with dplyr functions. In this example we use the mtcars dataset (see help("mtcars") for more information) to show how to sumarize a data frame, and to add variables to the data with the result of the application of a function.
library(dplyr)
library(magrittr)
df <- mtcars
df$cars <- rownames(df)#just add the cars names to the df
df <- df[,c(ncol(df),1:(ncol(df)-1))]# and place the names in the first column
1. Sumarize the data
To compute statistics we use summarize and the appropriate functions. In this case n() is used for counting the number of cases.
dplyr uses Non-Standard Evaluation(NSE), which is why we normally can use the variable names without quotes. However, sometimes during the data pipeline, we need to get our variable names from other sources such as a Shiny selection box. In case of functions like select, we can just use select_ to use a string variable to select
dplyr is an iteration of plyr that provides a flexible "verb" based functions to manipulate data in R. The latest version of dplyr can be downloaded from CRAN using
install.package("dplyr")
The key object in dplyr is a tbl, a representation of a tabular data structure. Currently dplyr (version 0.5.0) supports: