{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# **TP3 - Data Preprocessing**\n", "\n", "Exploratory Data Analysis & Unsuperivsed Learning
\n", "**Course: PHAUK Sokkey, PhD**
\n", "**TP: HAS Sothea, PhD**\n", "\n", "-------" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Objective**: Preprocessing is important in data related tasks. In this TP, you will explore different challanges you may encounted during when performing data preprocessing. We will discuss reasonable solution to these challanges.\n", "\n", "---------\n", "\n", "> **The `Jupyter Notebook` for this TP can be downloaded here: [TP3-Data-Preprocessing](https://hassothea.github.io/EDA_ITC/TPs/TP3_Data_Preprocessing.ipynb)**.\n", "\n", "-------" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Missing Data\n", "\n", "We will begin with missing data which are very common within real-world datasets. One common question about missing values is \"Should we remove them\"? We will delve into possible proper solution to this problem.\n", "\n", "We will work with `Enfants` dataset available here: [Enfants dataset](https://github.com/hassothea/EDA_ITC/blob/main/data/Enfants.txt)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**a.** Import this data and name it `data`." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "<>:3: SyntaxWarning: invalid escape sequence '\\d'\n", "<>:3: SyntaxWarning: invalid escape sequence '\\d'\n", "C:\\Users\\hasso\\AppData\\Local\\Temp\\ipykernel_18808\\610560642.py:3: SyntaxWarning: invalid escape sequence '\\d'\n", " data = pd.read_table(\"D:/Sothea_PC/Teaching_ITC/EDA\\data/Enfants.txt\", sep=\"\\t\")\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
GENREAGETAILLEMASSE
0F68020
1M7411618
2M6912023
3M7212125
4M7311417
\n", "
" ], "text/plain": [ " GENRE AGE TAILLE MASSE\n", "0 F 68 0 20\n", "1 M 74 116 18\n", "2 M 69 120 23\n", "3 M 72 121 25\n", "4 M 73 114 17" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "import numpy as np\n", "data = pd.read_table(\"D:/Sothea_PC/Teaching_ITC/EDA\\data/Enfants.txt\", sep=\"\\t\")\n", "data.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**b.** Compute statistical values and visualize the distribution of each variable.\n", "\n", "- Stastistics" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# To do" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Visualization" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# To do" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**c.** Did you find anything strange in the previous graphs?\n", "\n", "> Your response:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**d.** Create new data called `data_NoNA` by replacing all missing values with `np.nan`. Remove all the rows containing missing values. \n", "- Repeat point **(b)** on `data_NoNA` data.\n", "- Compare the distribution of these variables before and after removing missing values. " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# TO do" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# To do" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- What do you observe?\n", "\n", "> Your response:\n", "\n", "- What's the mechanism/type of these missing values?\n", "\n", "> Your response:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**e.** Do you spot any outliers in this dataset? \n", "\n", "> Your response:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Outliers & high leverage data\n", "\n", "In an unsupervised framework, outliers are data points that significantly deviate from the majority of observations. In a supervised framework, inputs with extreme values (but not their target) are known as high leverage points. Both outliers and high leverage points may obscure (but not always) the true underlying patterns in the data, often complicating analysis and leading to potential inaccuracies. We will start hunting outliers and high leverage points using [`Abalone` dataset](https://www.kaggle.com/datasets/rodolfomendes/abalone-dataset)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**a.** Download and import the dataset." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# To do" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**b.** Compute and visualize the correaltion matrix. Provide your first impression on the correlation matrix." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "# To do" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**c.** Study the relation between `Type` and the most interesting variable (target) `Rings`." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "# To do" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**d.** Insepct outliers and high leverage data points (considering `Rings` as the target) in the dataset. You might be interested in the following interesting lessons from STAT 501:\n", "- [Using Leverages to Help Identify Extreme x Values, STAT 501](https://online.stat.psu.edu/stat501/lesson/11/11.2)\n", "- [Identifying Outliers (Unusual y Values), STAT 501](https://online.stat.psu.edu/stat501/lesson/11/11.3)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "# To do" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**e.** Once again, study the correlation matrix of the data after removing outliers and high leverage data points. Conclude." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "# To do\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Further Readings\n", "\n", "- [Outliers & high leverage observations](https://online.stat.psu.edu/stat501/lesson/11/11.1#:~:text=A%20data%20point%20has%20high,is%20particularly%20high%20or%20low.)\n", "- [Using Leverages to Help Identify Extreme x Values, STAT 501](https://online.stat.psu.edu/stat501/lesson/11/11.2)\n", "- [Identifying Outliers (Unusual y Values), STAT 501](https://online.stat.psu.edu/stat501/lesson/11/11.3)\n", "- [Missing data: types, explanation, Imputation](https://www.scribbr.com/statistics/missing-data/)\n", "- [Kaggle: Abalone dataset](https://www.kaggle.com/datasets/rodolfomendes/abalone-dataset)\n", "- [Chapet 3, section 3.3.3 of The Introduction to Statistical Learning](https://www.stat.berkeley.edu/~rabbee/s154/ISLR_First_Printing.pdf)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.5" } }, "nbformat": 4, "nbformat_minor": 2 }