LAB: Sampling in R

• 1. Import “Census Income Data/Income_data.csv”
• 2. Create a new dataset by taking a random sample of 5000 records

Income_data<- read.csv("~\\Census Income Data\\Income_data.csv")
sample <- Income_data[sample(1:nrow(Income_data),5000),]
dim(sample)

## [1] 5000   15

Central tendencies

Income  <- read.csv("~\\Census Income DataIncome_data.csv")

mean(Income$capital.gain)

## [1] 1077.649

median(Income$capital.gain)

## [1] 0

LAB: Mean and Median on R

• 1. Import Dataset: “./Online Retail Sales Data/Online Retail.csv”
• 2. What is the mean of “UnitPrice”
• 3. What is the median of “UnitPrice”
• 4. Is mean equal to median? Do you suspect the presence of outliers in the data?
• 5. What is the mean of “Quantity”
• 6. What is the median of “Quantity”
• 7. Is mean equal to median? Do you suspect the presence of outliers in the data?

Solutions

• 1. Import Dataset: “./Online Retail Sales Data/Online Retail.csv”

Online_Retail<-read.csv("~\\Online Retail Sales Data\\Online Retail.csv")

• 2. What is the mean of “UnitPrice”

mean(Online_Retail$UnitPrice)

## [1] 4.611114

• 3. What is the median of “UnitPrice”

median(Online_Retail$UnitPrice)

## [1] 2.08

• 4.Is mean equal to median? Do you suspect the presence of outliers in the data?Yes, in this case mean and median are close. However we still cannot conclude on the absence of outlier because if there are balancing outliers on the either side of median, then also the mean and median can be close.
• 5. What is the mean of “Quantity”

mean(Online_Retail$Quantity)

## [1] 9.55225

• 6. What is the median of “Quantity”

median(Online_Retail$Quantity)

## [1] 3

• 7.Is mean equal to median? Do you suspect the presence of outliers in the data?No.Looks like there are outliers.

Dispersion

• Just knowing the central tendency is not enough.
• Two variables might have same mean, but they might be very different.
• Dispersion gives us an idea about the spread of the data.
• Look at these two variables. Profit details of two companies A & B for last 14 Quarters in MMs

• Though the average profit is 15 in both the cases
• Company B has performed consistently than company A.
• There was even loses for company A
• Measures of dispersion become very vital in such cases

Variance Calculation

Standard Deviation

• Standard deviation is just the square root of variance
• Variance gives a good idea on dispersion, but it is of the order of squares.
• Its very clear from the formula, variance unites are squared than that of original data.
• Standard deviation is the variance measure that is in the same units as the original data

Variance and Standard Deviation in R

• Divide the Income data into two sets. USA vs Others

usa_income<-Income[(Income$native.country==" United-States"), ] 
other_income<-Income[!(Income$native.country==" United-States"),]
nrow(usa_income)

## [1] 29170

nrow(other_income)

## [1] 3391

• Find the variance of “education.num” in those two sets. Which one has higher variance?
• Variance and SD for USA

var(usa_income$education.num)

## [1] 5.735863

sd(usa_income$education.num)

## [1] 2.394966

• Variance and SD for Other

var(other_income$education.num)

## [1] 13.56761

sd(other_income$education.num)

## [1] 3.683424

other_income dataset has a higher variance

LAB: Variance and Standard deviation

• 1. Import Dataset: “./Online Retail Sales Data/Online Retail.csv”
• 2. What is the variance and s.d of “UnitPrice”
• 3. What is the variance and s.d of “Quantity”
• 4. Which one these two variables is consistent?

Online_Retail<-read.csv("~\\Online Retail Sales Data\\Online Retail.csv")

Solutions

• 2. What is the variance and standard dediation of “UnitPrice”

var(Online_Retail$UnitPrice)

## [1] 9362.469

sd(Online_Retail$UnitPrice)

## [1] 96.75985

• 3. What is the variance and s.d of “Quantity”

var(Online_Retail$Quantity)

## [1] 47559.39

sd(Online_Retail$Quantity)

## [1] 218.0812

• 4.Which one these two variables is consistent?UnitPrice

Percentiles

• A student attended an exam along with 1000 others.
• He got 68% marks? How good or bad he performed in the exam?
• What will be his rank overall?
• What will be his rank if there were 100 students overall?
• For example, with 68 marks, he stood at 90th position. There are 910 students who got less than 68, only 89 students got more marks than him
• He is standing at 91 percentile.
• Instead of stating 68 marks, 91% gives a good idea on his performance
• Percentiles make the data easy to read
• (p^{th}) percentile: p percent of observations below it, (100 – p)% above it.
• Marks are 40 but percentile is 80%, what does this mean?
• 80% of CAT exam percentile means
• 20% are above & 80% are below
• Percentiles help us in getting an idea on outliers.
• For example the highest income value is 400,000 but 95th percentile is 20,000 only. That means 95% of the values are less than 20,000. So the values near 400,000 are clearly outliers

Quartiles

• Percentiles divide the whole population into 100 groups where as quartiles divide the population into 4 groups
• p = 25: First Quartile or Lower quartile (LQ)
• p = 50: second quartile or Median
• p = 75: Third Quartile or Upper quartile (UQ)

summary(Income$capital.gain)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##       0       0       0    1078       0  100000

quantile(Income$capital.gain, c(0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)) 

##    0%   10%   20%   30%   40%   50%   60%   70%   80%   90%  100% 
##     0     0     0     0     0     0     0     0     0     0 99999

quantile(Income$capital.loss, c(0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)) 

##   0%  10%  20%  30%  40%  50%  60%  70%  80%  90% 100% 
##    0    0    0    0    0    0    0    0    0    0 4356

quantile(Income$hours.per.week, c(0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)) 

##   0%  10%  20%  30%  40%  50%  60%  70%  80%  90% 100% 
##    1   24   35   40   40   40   40   40   48   55   99

LAB: Percentiles & Quartiles in R

• 1. ImportDataset: “./Bank Marketing/bank_market.csv”
• 2. Get the summary of the balance variable
• 3. Do you suspect any outliers in balance ?
• 4. Get relevant percentiles and see their distribution.
• 5. Are there really some outliers present?
• 6. Get the summary of the age variable
• 7. Do you suspect any outliers in age?
• 8. Get relevant percentiles and see their distribution.
• 9. Are there really some outliers present?

Solutions

• 1. ImportDataset: “./Bank Marketing/bank_market.csv”

bank_market <- read.csv("~\\Bank Marketing\\bank_market.csv")

• 2. Get the summary of the balance variable

summary(bank_market$balance)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   -8019      72     448    1362    1428  102100

• 3.Do you suspect any outliers in balance ?

Yes

• 4. Get relevant percentiles and see their distribution.

quantile(bank_market$balance, c(0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)) 

##     0%    10%    20%    30%    40%    50%    60%    70%    80%    90% 
##  -8019      0     22    131    272    448    701   1126   1859   3574 
##   100% 
## 102127

• 5.Are there really some outliers present?

Yes

• 6. Get the summary of the age variable

summary(bank_market$age)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   18.00   33.00   39.00   40.94   48.00   95.00

• 7.Do you suspect any outliers in age?

No

• 8. Get relevant percentiles and see their distribution.

quantile(bank_market$age, c(0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)) 

##   0%  10%  20%  30%  40%  50%  60%  70%  80%  90% 100% 
##   18   29   32   34   36   39   42   46   51   56   95

• 9.Are there really some outliers present?

Yes

Box Plots and Outlier Detection

• Box plots have box from LQ to UQ, with median marked.
• They portray a five-number graphical summary of the data Minimum, LQ, Median, UQ, Maximum
• Helps us to get an idea on the data distribution
• Helps us to identify the outliers easily
• 25% of the population is below first quartile,
• 75% of the population is below third quartile
• If the box is pushed to one side and some values are far away from the box then it’s a clear indication of outliers

• Some set of values far away from box, is gives us a clear indication of outliers.
• In this example the minimum is 5, maximum is 120, and 75% of the values are less than 15
• Still there are some records reaching 120. Hence a clear indication of outliers

• Sometimes the outliers are so evident that, the box appear to be a horizontal line in box plot.

boxplot(usa_income$capital.gain)

boxplot(usa_income$education.num)

LAB: Box plots and outlier detection

• 1. Import Dataset: “./Bank Marketing/bank_market.csv”
• 2. Draw a box plot for balance variable
• 3. Do you suspect any outliers in balance ?
• 4. Get relevant percentiles and see their distribution.
• 5. Draw a box plot for age variable
• 6. Do you suspect any outliers in age?
• 7. Get relevant percentiles and see their distribution.

Solutions

• 1. Import Dataset: “./Bank Marketing/bank_market.csv”

bank_market <- read.csv("~\\Bank Marketing\\bank_market.csv")

• 2. Draw a box plot for balance variable

boxplot(bank_market$balance)

– 3.Do you suspect any outliers in balance ? Yes – 4.Get relevant percentiles and see their distribution.

quantile(bank_market$balance, c(0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)) 

##     0%    10%    20%    30%    40%    50%    60%    70%    80%    90% 
##  -8019      0     22    131    272    448    701   1126   1859   3574 
##   100% 
## 102127

• 5. Draw a box plot for age variable

boxplot(bank_market$age)

6.Do you suspect any outliers in balance ? Yes – 7.Get relevant percentiles and see their distribution.

quantile(bank_market$age, c(0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)) 

##   0%  10%  20%  30%  40%  50%  60%  70%  80%  90% 100% 
##   18   29   32   34   36   39   42   46   51   56   95

LAB: Graphs

• 1. Import Dataset: “./Sporting_goods_sales/Sporting_goods_sales.csv”
• 2. Draw a scatter plot between Average_Income and Sales. Is there any relation between two variables?
• 3. Draw a scatter plot between Under35_Population_pect and Sales. Is there any relation between two?

Solutions

• 1. Import Dataset: “./Sporting_goods_sales/Sporting_goods_sales.csv”

Sporting_goods_sales <- read.csv("~\\Sporting_goods_sales\\Sporting_goods_sales.csv")

• 2. Draw a scatter plot between Average_Income and Sales. Is there any relation between two variables?

plot(Sporting_goods_sales$Average_Income, Sporting_goods_sales$Sales)

Looks like there is as such no relation between Average_Income and Sales.

• 3. Draw a scatter plot between Under35_Population_pect and Sales. Is there any relation between two?

plot(Sporting_goods_sales$Under35_Population_pect, Sporting_goods_sales$Sales)

There is a strong positive relationship between Under35_Population_pect and Sales.

Bar Chart

bar_table<-table(mtcars$cyl)
barplot(bar_table, main="Cars Data Details",   xlab="Cylinders", ylab="Cars Count")

bar_table_gear<-table(mtcars$gear)
barplot(bar_table_gear, main="Cars Data Details",   xlab="gears", ylab="Cars Count" , col = 4)

LAB: Bar Chart

• 1. Import Dataset: “./Sporting_goods_sales/Sporting_goods_sales.csv”
• 2. Create a bar chart summarizing the information on family size.

Solutions

Sporting_goods_sales <- read.csv("~\\Sporting_goods_sales\\Sporting_goods_sales.csv")

• 2. Create a bar chart summarizing the information on family size.

table_family.size<-table(Sporting_goods_sales$Avg_family_size)
barplot(table_family.size, main="Family Size Details",   xlab="Avg_family_size", ylab="Count" , col = 4)

Trend chart – Trend chart is used for time series datasets.

plot(AirPassengers)

Lab : Trend chart

• 1. Draw trend chart for Ukgas data from datasets package
• 2. Plot JohnsonJohnson data in a trend chart

Solutions

• 1. Draw trend chart for Ukgas data from datasets package

plot(UKgas)

– 2. Plot JohnsonJohnson data in a trend chart

plot(JohnsonJohnson)