Lab3: Data Quality & Preprocessing

Course: INF-604: Data Analysis I
Lecturer: Sothea HAS, PhD

Objective: In this lab, you will delve deeper into assessing the quality of datasets and employing preprocessing techniques to properly clean them.

1. Kaggle Heart Failure Dataset

Let’s consider Heart Disease Dataset dataset discussed in the previous Lab2.

Code 
import kagglehub
import pandas as pd

# Download latest version
path = kagglehub.dataset_download("fedesoriano/heart-failure-prediction")
data = pd.read_csv(path + "/heart.csv")
data.head()
Age Sex ChestPainType RestingBP Cholesterol FastingBS RestingECG MaxHR ExerciseAngina Oldpeak ST_Slope HeartDisease
0 40 M ATA 140 289 0 Normal 172 N 0.0 Up 0
1 49 F NAP 160 180 0 Normal 156 N 1.0 Flat 1
2 37 M ATA 130 283 0 ST 98 N 0.0 Up 0
3 48 F ASY 138 214 0 Normal 108 Y 1.5 Flat 1
4 54 M NAP 150 195 0 Normal 122 N 0.0 Up 0

A. Using quick statistical summary of each columns, identify the following problems:

  • Encode each column into its suitable data type.
  • Detect missing values using data.isna().sum(). What do you observe?
  • Identify columns with invalid values. Count how many invalid data are there in each column?

B. From now, convert invalid data using NA encoding.

  • Study its nature: MCAR, MAR or MNAR? [Hint: You should compuare the clean columns before and after dropping the currupted values.]
  • Handle them according to your analysis.
  • Outliers: How many columns contain outliers? List them in a list called outlier_list = [...].

C. General Information:

  • Visualize if cholesterol is correlated with age or not.
  • Visualize if there is any chest pain type that strongly links to heart disease status or not.
  • Visualize if diabetes is correlated with heart disease status or not.

2. Kaggle AutoMPG dataset

  • Your job is to address if there is any columns containing missing values, invalid values or outliers or not?
  • How would you handle the missing values if there is any?
import kagglehub
import pandas as pd
# Download latest version
path = kagglehub.dataset_download("uciml/autompg-dataset")
auto = pd.read_csv(path + '/auto-mpg.csv')
auto.head()
mpg cylinders displacement horsepower weight acceleration model year origin car name
0 18.0 8 307.0 130 3504 12.0 70 1 chevrolet chevelle malibu
1 15.0 8 350.0 165 3693 11.5 70 1 buick skylark 320
2 18.0 8 318.0 150 3436 11.0 70 1 plymouth satellite
3 16.0 8 304.0 150 3433 12.0 70 1 amc rebel sst
4 17.0 8 302.0 140 3449 10.5 70 1 ford torino

Further Reading