Exploratory Data Analysis & Unsuperivsed Learning Course: Dr. HAS Sothea TP: Dr. Ms. UANN Sreyvi
Objective: To equip you all with the skills to analyze and interpret the relationship between two variables. You will explore these relationships as a function of time using the Gapminder dataset.
#%pip install gapminder # This is for installing the packagefrom gapminder import gapminderimport pandas as pdimport numpy as npimport plotly.io as piopio.renderers.default ='notebook'gapminder.head()
country
continent
year
lifeExp
pop
gdpPercap
0
Afghanistan
Asia
1952
28.801
8425333
779.445314
1
Afghanistan
Asia
1957
30.332
9240934
820.853030
2
Afghanistan
Asia
1962
31.997
10267083
853.100710
3
Afghanistan
Asia
1967
34.020
11537966
836.197138
4
Afghanistan
Asia
1972
36.088
13079460
739.981106
1. Pearson and Spearman’s correlations
a. Compute Pearson correlation matrix of the three quantitative variables on year \(1952\), \(1987\) and then \(2007\) using pd.corr(). Give a brief intuition of the relationship between these variables.
# To do
Description:
b. Compute Spearman’s Rank Correlation of the previous columns in 1952, 1987 and 2007. What do you observe?
# To do
c. From the previous result, pick the most interesting pair of variables and plot a graph illustrating their relationship on each year, using proper axis scaling and title.
# To do
d. Revisit your intuition of the correlation matrix in year 1952 from question (a), can you see why we observed such a (poor) correlation in 1952?!
Remark: Pearson correlation matrix can summarize linear relationship between pairs of quantitative variables but it might be inacurate and influenced by
outliers,
non-linearity,
small sample size,
confounding (causal) variables…
Conclusion:
# To do
2. Visualization with more information
Scatterplot is the primary graphic type for visualizing the relationship between two quantitative variables. Additionally, you can include other factors of interest using color, shape, size, facets, etc., depending on the type of those variables.
The following questions apply to the years 1952, 1987, and 2007:
a. Do you think health conditions and economies differ across continents? Compute some indicators to illustrate this.
# To do
Description:
c. We will try to confirm this using Analysis of Variance (ANOVA).
In 2007,
Perform one way ANOVA on lifeExp and continent (you can use f_oneway from scipy.stats module).
# To do
Do the same with gdpPercap and continent.
Are these results reliable? Why?
Your response:
Propose ideas that might solve this problem or an alternative method.
Your response:
d. We have previously described the relationship between population and life expectancy or GDP per capita. Now, illustrate the effect of population defining size of points according to variable pop.
# To do
e. Now, directly visualize the relation between pop and lifeExp, then the relation between pop and gdpPercap. Explain the resulting graphs.
# To do
3. Quantitative vs Qualitative
a. We have seen how life expectancy and economy vary across continents in 2007 (Slide 25 of the course). Compute \(\eta\)-squared correlation between continent and lifeExp, then continent with gdpPercap in 1952, 1987 and 2007.
Do you find the results reasonable?
# To do
4. Time evolution
a. Draw the evolution of the following correaltions from 1952 to 2007:
Person and Spearman corerlation between life expectancy and GDP per capita
\(\eta\)-squared coefficients of continent vs life expectancy, and continents vs GDP per capita.
Hint: the following figure is expected.
import plotly.io as piopio.renderers.default ='notebook'import plotly.express as px# To do
b. Compare side by side the three aspects: population, life expectancy and GDP per Capita between Cambodia and Thailand from 1952 to 2007.
# To do
c. Describe what you observed: the world from 1952 to 2007.
