TP3 - Data Preprocessing

Exploratory Data Analysis & Unsuperivsed Learning
M1-DAS
Lecturer: HAS Sothea, PhD

Objective: Preprocessing is important in data related tasks. In this TP, you will explore different challanges you may encounted during when performing data preprocessing. We will discuss reasonable solution to these challanges.

The Jupyter Notebook for this TP can be downloaded here: TP3_Preprocessing.ipynb.

1. Titanic dataset

The Titanic dataset contains information on the passengers aboard the RMS Titanic, which sank in \(1912\). It includes details like age, gender, class, and survival status.

I bet you have heard about or watched Tiannic movie at least once. How about we take a look at the real dataset of Titanic available in Kaggle. For more information about the dataset and the columns, read Titanic dataset. Let’s import it into our Jupyter Notebook by running the following code.

import kagglehub

# Download latest version
path = kagglehub.dataset_download("surendhan/titanic-dataset")

# Import data
import pandas as pd
data = pd.read_csv(path + "/titanic.csv")
data.head()
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked
0 892 0 3 Kelly, Mr. James male 34.5 0 0 330911 7.8292 NaN Q
1 893 1 3 Wilkes, Mrs. James (Ellen Needs) female 47.0 1 0 363272 7.0000 NaN S
2 894 0 2 Myles, Mr. Thomas Francis male 62.0 0 0 240276 9.6875 NaN Q
3 895 0 3 Wirz, Mr. Albert male 27.0 0 0 315154 8.6625 NaN S
4 896 1 3 Hirvonen, Mrs. Alexander (Helga E Lindqvist) female 22.0 1 1 3101298 12.2875 NaN S

A. What’s the dimension of this dataset? How many quantitative and qualitative variables are there in this dataset (read about the data here)?

# To do

B. Are there any missing values? If so,

  • Study the impact of missing value removal on the quantitative variables.
  • Study the impact of missing value removal on the qualitative variables.
  • Conclude the dynamic of the missing values and handle them.
# To do

C. What are the most common passenger names? Were Rose and Jack on the ship?

Hint: WordCloud is a useful graph for such text summary. For more, read here.

2. Bivariate/Multivariate Analysis

We are primarily interested in exploring the relationship between each column and the likelihood of passenger survival. The following questions will guide you through this exploration. In each question, try to give some comments on what you observe in the graphs.

A. Survival Analysis: How did the survival rates vary by gender? How about by class?

Hint: Create bar charts or stacked bar charts showing the survival rates for different genders and different passenger classes.

# To do

B. Fare and Survival: Is there a relationship between the fare paid and the likelihood of survival?

Hint: Create boxplots to analyze the fare distribution among survivors and non-survivors.

# To do

C. Family Size: How does family size (number of siblings/spouses and parents/children) impact the chances of survival?

# To do

D. Embarkation Points: How do survival rates differ based on the port of embarkation (C, Q, S)?

# To do

E. Pclass and Age: How does passenger class correlate with age?

# To do

F. Gender and Age: How does age distribution differ between male and female passengers?

# To do

G. Age, Fare and Gender: View the connection of Age, Fare and Gender in one graph.

# To do

H. Age, Fare and Class: View the connection of Age, Fare and Class in one graph.

# To do

I. Based on your analysis, which variables appear to have the greatest impact on the likelihood of survival?

# To do

Further readings