Exploratory Data Analysis & Unsupervised Learning
Gapminder dataset (1704, 5)| country | continent | lifeExp | pop | gdpPercap |
|---|---|---|---|---|
| Afghanistan | Asia | 43.828000 | 31889923 | 974.580338 |
| Albania | Europe | 76.423000 | 3600523 | 5937.029526 |
| Algeria | Africa | 72.301000 | 33333216 | 6223.367465 |
| Angola | Africa | 42.731000 | 12420476 | 4797.231267 |
| Argentina | Americas | 75.320000 | 40301927 | 12779.379640 |
| pop | lifeExp | gdpPercap | |
|---|---|---|---|
| mean | 4.402122e+07 | 67.007423 | 11680.071820 |
| std | 1.476214e+08 | 12.073021 | 12859.937337 |
| min | 1.995790e+05 | 39.613000 | 277.551859 |
| 50% | 1.051753e+07 | 71.935500 | 6124.371108 |
| max | 1.318683e+09 | 82.603000 | 49357.190170 |
Gapminder dataset (1704, 5)from plotly.subplots import make_subplots
import plotly.graph_objects as go
fig = make_subplots(rows=1, cols=3,
subplot_titles=("Boxplot of pop", "Violinplot of lifeExp", "Histogram of GDP Per Capita"))
fig.add_trace(go.Box(y=data2007['pop'], name="pop"), col=1, row=1)
fig.add_trace(go.Violin(y=data2007['lifeExp'], name="lifeExp"), row=1, col=2)
fig.add_trace(go.Histogram(x=data2007['gdpPercap'],
name="gdpPercap"), row=1, col=3)
fig.update_layout(height=280, width=900)
fig.update_yaxes(type="log", row=1, col=1)
fig.update_xaxes(title="Population", row=1, col=1)
fig.update_xaxes(title="Life Expectancy", row=1, col=2)
fig.update_xaxes(title="GDP Per Capita", row=1, col=3)
fig.show()\[\text{Cov}(X,Y)=\frac{1}{n-1}\sum_{i=1}(\text{x}_i-\overline{x})(\text{y}_i-\overline{y}).\]
\[\text{Cov}(X,Y)=\frac{1}{n-1}\sum_{i=1}(\text{x}_i-\overline{x})(\text{y}_i-\overline{y}).\]
fig.update_layout(title=f"Tips vs total bill (Cov = {float(np.cov(df['tip'].values, df['total_bill'].values).round(2)[0,1])}) & gender ")
fig.update_layout(width=380, height=260, title="Tips vs total bill & gender")
fig.add_annotation(
x=20, y=8, showarrow=False, font=dict(size=12),
text=f"Cov = {float(np.cov(df['tip'].values, df['total_bill'].values).round(2)[0,1])}"
)
fig.show()| tip | total_bill | sex | |
|---|---|---|---|
| 0 | 1.01 | 16.99 | Female |
| 1 | 1.66 | 10.34 | Male |
| 2 | 3.50 | 21.01 | Male |
| 3 | 3.31 | 23.68 | Male |
| 4 | 3.61 | 24.59 | Female |
Itās hard to interpret the value of covariance as it can be large or small according to the scale of \(X\) and \(Y\).
tip example: \(\text{Corr}(\text{tip}, \text{bill})=\) 0.676.
Gapminder in 2007:LifeExp and GDP appear to be positively related.Population may be decorrelated with the other two columns.\[\rho_{\color{blue}{X},\color{red}{Y}}=r_{R[{\color{blue}{X}}],R[{\color{red}{Y}}]}=\frac{\text{Cov}(R[{\color{blue}{X}}],R[{\color{red}{Y}}])}{s_{R[{\color{blue}{X}}]},s_{R[{\color{red}{Y}}]}}=1-\frac{6\sum_{i=1}^nd_i^2}{n(n^2-1)},\]
where \(R[{\color{blue}{\text{x}_i}}]\) and \(R[{\color{red}{\text{y}_i}}]\) be the rank of observations \(\color{blue}{\text{x}_i}\) and \(\color{red}{\text{y}_i}\) in their own list, and \(d_i=R[{\color{blue}{\text{x}_i}}]-R[{\color{red}{\text{y}_i}}]\) be the distance in rank of observation \(i\)-th.
Spearmanās Rank Correlation is large when most of the ranks of the two variables match.
Gapminder, 2007):| Aspect | Pearson |
Spearman |
|---|---|---|
| Type | Parametric | Non-parametric |
| Measure | Linear relationship | Monotonic relationship |
| Data Type | Numerical | Ordinal or Numerical |
| Outliers | Sensitive to outliers | Less sensitive to outliers |
| Range | \([-1,1]\) | \([-1,1]\) |
| Interpretation | \(\approx\) 1: Perfect positive linear relationship \(\approx\) -1: Perfect negative linear relationship \(\approx\) 0: No linear trend or interrupted by noise |
\(\approx\) 1: Perfect increasing monotonic relationship \(\approx\) -1: Perfect decreasing monotonic relationship \(\approx\) 0: No monotonic relationship at all |
ScatterplotgdpPercap wih lifeExp & pop.import plotly.graph_objects as go
import plotly.express as px
data2007 = gapminder.query("year == 2007")
fig1 = px.scatter(
data2007,
x="gdpPercap", y="lifeExp",
hover_name="country", opacity=0.7)
fig1.update_traces(marker=dict(size=10))
fig1.update_layout(
height=370, width=450,
title="The world GDP vs LifeExp in 2007")
fig1.show()ScatterplotgdpPercap wih lifeExp & pop. Log transformation is useful for visualizing data with dense data. It requires a clear description to avoid misinterpretation.
ScatterplotGPD vs Life Expectancy:
GPD vs Population:
lifeExp and gdpPercap on different continent:| LifeExp | GDP | |
|---|---|---|
| Continent | 0.635 | 0.424 |
Continent seems more related to lifeExp than to gdpPercap.GDP on each continent?import plotly.figure_factory as ff
group_labels = list(data2007.continent.unique())
hist_data = [data2007.lifeExp[data2007.continent == x] for x in group_labels]
colors = ["#f1ab17", "#f13c26", "#9be155", "#4ab8dc", "#d567f3"]
sorted_data = data2007.sort_values(by='gdpPercap')
hist_data_gdp = [data2007.gdpPercap[data2007.continent == x] for x in group_labels]
fig4 = px.box(data2007,
x="continent", y="gdpPercap", hover_name="country", points='all',
color="continent", category_orders={'continent': sorted_data['continent']})
fig4.update_layout(title="GDP per Capita on each continent in 2007", height=350, width=500)
fig4.show()sorted_data = data2007.sort_values(by='gdpPercap')
hist_data_gdp = [data2007.gdpPercap[data2007.continent == x] for x in group_labels]
fig_ = px.violin(data2007, x="continent", y="gdpPercap",
hover_name="country", color="continent", points='all',
category_orders={'continent': sorted_data['continent']})
fig_.update_layout(title="GDP per Capita on each continent in 2007", height=350, width=500)
fig_.show()gdpPercap between coninents is not as clear as lifeExp, yet one can still see the differences.Continent is useful for predicting / explaining gdpPercap and visa versa, but not as much as for lifeExp.import plotly.express as px
fig = px.box(data2007,
x='continent',
y='lifeExp',
points='all',
category_orders={'continent': sorted_data['continent']},
color='continent')
fig.update_layout(
height=450,
width=450,
title=f"LifeExp per continent with eta-squared {np.round(df_eta['LifeExp'].values[0], 3)}")
fig.show()GDP is categorized into three groups: [Developing, Emerging, Developed] at 33.33% and 66.66% quantiles.continent is related to categorial GDP?GDP category of a country.import pandas as pd
import plotly.express as px
# 1. Prepare your data (Binning GDP as in your original code)
data2007['gdp_category'] = pd.qcut(data2007['gdpPercap'], q=3, labels=['Developing', 'Emerging', 'Developed'])
# 2. Group the data to get counts for each combination
# (Plotly needs explicit aggregate sizes for a treemap)
df_grouped = data2007.groupby(['continent', 'gdp_category']).size().reset_index(name='count')
df_grouped['Percentage'] = df_grouped.groupby('continent')['count'].transform(lambda x: (x / x.sum() * 100).round(2))
# 3. Define your custom color map
color_map = {
'Asia': '#51cb4b',
'Africa': '#e35441',
'Americas': '#41b4e3',
'Europe': '#dda63e',
'Oceania': '#b374df'
}
# 4. Build the Treemap
fig = px.treemap(
df_grouped,
path=['continent', 'gdp_category'], # Hierarchy: Outer block -> Inner block
values='Percentage', # Sets block size based on data density
color='continent', # Groups color by continent
color_discrete_map=color_map, # Applies your exact hex codes
title="Treemap of cat. GDP vs Continent"
)
# 5. Fine-tune layout styling
fig.update_layout(
width=450, height=420,
font=dict(size=15),
margin=dict(t=50, l=10, r=10, b=10)
)
# 6. Show the interactive plot
fig.show()df_freq = data2007.groupby(['continent','gdp_category']).size().reset_index(name='Freq')
df_freq['Percent'] = df_freq.groupby('continent')['Freq'].apply(lambda x: x/x.sum() * 100).reset_index(level=0, drop=True)
fig = px.bar(
df_freq,
x="continent",
y="Percent",
color="gdp_category",
barmode='stack',
text= df_freq['Percent'].round(2).astype(str) + '%')
fig.update_layout(width=450, height=400,
title='Stacked Barplot of Categorical GDP vs Continent')
fig.show()distinct the bars, the stronger the association between the variables.| \(\color{blue}{X}\) - \(\color{red}{Y}\) | \(\color{red}{Y_1}\) | \(\dots\) | \(\color{red}{Y_J}\) | Total |
|---|---|---|---|---|
| \(\color{blue}{X_1}\) | \(n_{1,1}\) | \(\dots\) | \(n_{1,J}\) | \(\color{blue}{n_{1,.}}\) |
| \(\vdots\) | \(\vdots\) | \(\ddots\) | \(\vdots\) | \(\color{blue}{\vdots}\) |
| \(\color{blue}{X_I}\) | \(n_{I,1}\) | \(\dots\) | \(n_{I,J}\) | \(\color{blue}{n_{I,.}}\) |
| Total | \(\color{red}{n_{.,1}}\) | \(\color{red}{\dots}\) | \(\color{red}{n_{.,J}}\) | \(N\) |
where \(n_{i,j}\): frequency of \(\color{blue}{X_i}\) and \(\color{red}{Y_j}\).
Under the assumption that \(H_0\) is true, then \(\chi^2(\color{blue}{X},\color{red}{Y})\sim \chi^2(\text{df})\) with \(\text{df}=(\color{blue}{I}-1)(\color{red}{J}-1)\).
| \(\color{blue}{X}\) - \(\color{red}{Y}\) | \(\color{red}{Y_1}\) | \(\dots\) | \(\color{red}{Y_J}\) | Total |
|---|---|---|---|---|
| \(\color{blue}{X_1}\) | \(O_{1,1}\) | \(\dots\) | \(O_{1,J}\) | \(\color{blue}{O_{1,.}}\) |
| \(\vdots\) | \(\vdots\) | \(\ddots\) | \(\vdots\) | \(\color{blue}{\vdots}\) |
| \(\color{blue}{X_I}\) | \(O_{I,1}\) | \(\dots\) | \(O_{I,J}\) | \(\color{blue}{O_{I,.}}\) |
| Total | \(\color{red}{O_{.,1}}\) | \(\color{red}{\dots}\) | \(\color{red}{O_{.,J}}\) | \(1\) |
| \(\color{blue}{X}\) - \(\color{red}{Y}\) | \(\color{red}{Y_1}\) | \(\dots\) | \(\color{red}{Y_J}\) |
|---|---|---|---|
| \(\color{blue}{X_1}\) | \(\color{red}{O_{.,1}}\color{blue}{O_{1,.}}\) | \(\dots\) | \(\color{red}{O_{.,J}}\color{blue}{O_{1,.}}\) |
| \(\vdots\) | \(\vdots\) | \(\ddots\) | \(\vdots\) |
| \(\color{blue}{X_I}\) | \(\color{red}{O_{.,1}}\color{blue}{O_{I,.}}\) | \(\dots\) | \(\color{red}{O_{.,J}}\color{blue}{O_{I,.}}\) |
import plotly.graph_objects as go
from scipy.stats import chi2
# Create x-axis values (domain for chi-squared)
x = np.linspace(0, 50, 100)
# Degrees of freedom to display
dfs = [1, 5, 10, 15, 20, 30]
# Create figure
fig = go.Figure()
# Add trace for each degree of freedom
for df in dfs:
y = chi2.pdf(x, df)
# Add line to plot
fig.add_trace(
go.Scatter(
x=x,
y=y,
mode='lines',
name=f'df = {df}',
line=dict(width=2)
)
)
# Update layout
fig.update_layout(
title=r'$\text{Different distributions of }\chi^2(\text{df})$',
xaxis_title='x',
yaxis_title='Density',
legend_title='DFs',
template='plotly_white',
hovermode='closest',
width=450,
height=249
)
fig.show()Under the assumption that \(H_0\) is true, then \(\chi^2(\color{blue}{X},\color{red}{Y})\sim \chi^2(\text{df})\) with \(\text{df}=(\color{blue}{I}-1)(\color{red}{J}-1)\).
| F | SE | SEP | MEP | E | MW | NW | |
|---|---|---|---|---|---|---|---|
| 3 | 263 | 1085 | 7704 | 4346 | 4052 | 1551 | 2319 |
| 2 | 616 | 2265 | 10088 | 8889 | 11264 | 4713 | 3892 |
| 1 | 222 | 950 | 2864 | 3238 | 4517 | 1910 | 1145 |
| 0 | 126 | 562 | 1105 | 1420 | 2596 | 1531 | 622 |
df_vaccine = df.melt(value_name='Count', var_name='Job')
df_vaccine['Vaccine Favor'] = list(range(4)) * 7
df_vaccine['Vaccine Favor'] = df_vaccine[['Vaccine Favor']].astype(object)
temp = df_vaccine.groupby('Job')['Count'].apply(lambda x: x/x.sum() * 100).reset_index(level=1)
temp.columns = ['level_1', 'Percent']
df_vaccine = pd.merge(temp, df_vaccine, left_on='level_1', right_index=True, how='left')
fig = px.bar(
df_vaccine,
x="Job",
y="Percent",
color="Vaccine Favor",
barmode='stack',
text= df_vaccine['Percent'].round(2).astype(str) + '%')
fig.update_layout(width=450, height=300,
title='Stacked Barplot of Job vs Vaccine Favor')
fig.show()Property
continent (quantitative).log_pop (quantitative).import numpy as np
data2007[' '] = np.repeat('Data 2007', data2007.shape[0])
fig = px.scatter(
data2007, x="gdpPercap", y="lifeExp",
hover_name="country", size_max=80, color=" ")
fig.update_layout(width=500, height=370, title='Life Expectancy vs GDP per Capita & Continent')
fig.update_xaxes(type="log")
fig.show()Shape for representing qualitative data.
Example:
Symbol = gdp_category.Color = continent.Combining numerous colors and symbols can complicate a graph.
Use them carefully and only when appropriate.
Size for representing quantitative data.
Example:
Size = pop.Color = continent.Colors and size are common, and the resulting graphs are often called Bubble chart.
One shoule choose suitable max size to have a nice graph.
Oceania from 1952 to 2007. def cat_gdp(yearly_data):
return pd.qcut(yearly_data, q=3, labels=['Developing', 'Emerging', 'Developed'])
df = gapminder
# Apply the function to each year
df['GDP_Category'] = df.groupby('year').apply(lambda x: cat_gdp(x.gdpPercap)).reset_index(level=0, drop=True)
df_Af = df.query("continent == 'Asia'")
# Aggregate the data
df_agg = df_Af.groupby(['year', 'GDP_Category']).size().reset_index(name='Count')
# Create the stacked bar chart
fig = px.bar(
df_agg, x='year', y='Count',
color='GDP_Category', barmode='stack',
title="Evolution of Asian Countries' GDP Categories from 1952 to 2007",
labels={'Count': 'Number of Countries', 'year': 'Year'})
fig.update_layout(height=240, width=900)
fig.show()
import plotly.express as px
df = px.data.gapminder()
fig_anime = px.scatter(df, x="gdpPercap", y="lifeExp", animation_frame="year", animation_group="country",
size="pop", color="continent", hover_name="country",
log_x=True, size_max=50, range_x=[100,100000],
range_y=[25,90])
fig_anime.update_layout(height=370, width=950,
title="The Evolution of the World in a Single Graph")Begin the analysis with indicators,
then visualize the ones with potential impact.
š„³ Yeahhhhā¦.
Party timeā¦ š„