Data Cleaning is the process of transforming raw data into consistent data that can be analyzed. It is aimed at improving the content of statistical statements based on the data as well as their reliability. Data cleaning may profoundly influence the statistical statements based on the data.
Tidy Evaluation with rlang Cheatsheet. Tidy Evaluation (Tidy Eval) is a framework for doing non-standard evaluation in R that makes it easier to program with tidyverse functions. Non-standard evaluation, better thought of as “delayed evaluation,” lets you capture a user’s R code to run later in a new environment or against a new data frame. Often 80% of data analysis time is spent on data preparation and data cleaning 1. Data entry, importing data set to R, assigning factor labels, 2. Data screening: checking for errors, outliers, 3. Fitting models & diagnostics: whoops! Something wrong, go back to step 1. Whatever you can do to reduce this, gives more time for.
R has a set of comprehensive tools that are specifically designed to clean data in an effective and comprehensive manner.
STEP 1: Initial Exploratory Analysis
The first step to the overall data cleaning process involves an initial exploration of the data frame that you have just imported into R. It is very important to understand how you can import data into R and save it as a data frame.
setwd(“C:/Users/NAGRAJ/Desktop/House Pricing“)
dir()
data<-read.csv(“Regression-Analysis-House Pricing.csv“,na.strings = “”)
View(data)
The first thing that you should do is check the class of your data frame:
class(data)
This renders an output as shown below in which we can clearly see that our dataset is saved as a data frame.
[1] “data.frame”
Next, we want to check the number of rows and columns the data frame has.
The code give us & its result:
dim(data)
[1] 932 10
Here we can see that the data frame has 932 rows and 10 columns.
We can view the summary statistics for all the columns of the data frame using the code shown below:
summary(data)
This renders an output as shown below:
STEP 2: Visual Exploratory Analysis
There are 2 types of plots that you should use during your cleaning process –The Histogram and the BoxPlot
- Histogram
The histogram is very useful in visualizing the overall distribution of a numeric column. We can determine if the distribution of data is normal or bi-modal or unimodal or any other kind of distribution of interest. We can also use Histograms to figure out if there are outliers in the particular numerical column under study. In order to plot a histogram for any particular column we need to use the code shown below:
install.packages(“plyr”)
library(plyr)
hist(data$Dist_Taxi)
- BoxPlot
Boxplots are super useful because it shows you the median, along with the first, second and third quartiles. BoxPlots are the best way of spotting outliers in your data frame. In order to visualize a box plot we need to use the code shown below:
boxplot(data$Dist_Taxi)
STEP 3: Correcting the errors!
This step focuses on the methods that you can use to correct all the errors that you have seen.
If we want to change the name of our data frame we can do so using the code shown below:
data$carpet_area<-data$Carpet
In the code above we renamed the Carpet column as “Carpet_area”.
Sometimes columns have an incorrect type associated with them. For example, a column containing text elements stored as a numeric column. In such a case we can change the type of column by using the code shown below:
data$Dist_Taxi<-as.character(data$Dist_Taxi)
class(data$Dist_Taxi)
“character”
There are a wide array of type conversions you can carry out in R. They are listed below.
as.character()
as.numeric()
as.integer()
as.logical()
as.factor()
String manipulation in R comes in handy when you are working with datasets that have a lot of text based elements.
In order to change all the text to uppercase or lowercase in a particular column we need to execute the code shown below:
#Making all uppercase
data$Parking<-toupper(data$Parking)
#Making all Lowercase
data$Parking<-toupper(data$Parking)
If we want to trim the whitespaces in the next under a column we need to use the code shown below:
#Installing and loading the required packages
install.packages(“stringr”)
library(stringr)
#Trimming all whitespaces
data$Dist_Taxi<-str_trim(data$Dist_Taxi)
If we want to replace a particular word or letter under a column we can do so using the code below:
#Replacing “Not Provided” with “Not Available”
data$Parking<-str_replace(data$Parking,”Not Provided”,”Not Available”)
In order to replace the outliers with the summary statistics like median the following code is used.
#Replacing the outliers of a particular column with median
vec1<-boxplot.stats(data$Dist_Taxi)$out;
data$Dist_Taxi[data$Dist_Taxi %in% vec1]<-median(data$Dist_Taxi)
The next section will show you how to deal with your missing values:
#Checking for missing values in the entire dataframe
any(is.na(data))
#Checking for the total number of missing values in the entire dataframe
sum(is.na(data))
#Checking for the total number of missing values in a particular column
sum(is.na(data$Dist_Taxi))
#Eliminating missing values completely from the entire dataframe
na.omit(data)
#Eliminating missing values completely from a particular column
na.omit(data$Dist_Taxi)
#Replacing the NA’s in the entire dataframe with ‘0’s
data[is.na(data)]<- 0
#Replacing the NA’s in a particular column with ‘0’s
data$Dist_Taxi[is.na(data$Dist_Taxi)]<-0
#Replacing the NA’s in a particular column with a summary statistics like median
data$Dist_Taxi[is.na(data$Dist_Taxi)]<-median(data$Dist_Taxi)
Suppose we want to unite two columns in our data frame we can do so using the code shown below:
#Installing and loading the required package
install.packages(“tidyr”)
library(tidyr)
data1<-unite(data = data,col = city_category_with_parking,City_Category,Parking)
View(data1)
The unite() function takes 4 arguments – The data frame, the new column name, the first column and the second column name that you want to unite.
Conversely we can also separate a column as shown below:
data2<-separate(data = data1,city_category_with_parking,c(“City_Category”,”Parking”), sep = “-“)
View(data2)
R Data Transformation Cheat Sheet
The separate() function takes 4 arguments – The data frame, the column that we want to separate, the names of the new columns and the indicator at which we want the column to be separated at.
steps 1 to 3 above gives you a relatively clean dataset. Always keep exploring new ways that you can clean your data and never stop exploring.
R Data Frame Cheat Sheet
Happy Cleaning!
About Chandana:
R Data Cleaning Cheat Sheet Pdf
Chandana is B.E. She was working as Analyst Intern with Nikhil Guru Consulting Analytics Service LLP (Nikhil Analytics), Bangalore.