Lab 3: Probability, Distributions & Simulation

Course: INS-605: Data Analysis II
Lecturer: Sothea HAS, PhD

Student name: …
ID: …

Objective

In this lab, we use real NYC 311 complaint data to connect a simple data-analysis workflow with probability:

Select the relevant observations and organize them into time intervals.
Visualize a random variable.
Pick a reasonable distribution family.
Estimate its parameter(s) and compare the observed data with the probability model.
Compute probabilities from the fitted model.
Simulate website traffic using a transition matrix and decide where an advertisement could be placed.

The data can be downloaded here: Lab3: Probability and Simulation.ipynb

0. Load and understand the NYC 311 data

The data come from NYC Open Data – 311 Service Requests. The downloaded period covers 20 August through 1 September 2026 (the end date is exclusive).

For this lab, we are not interested in all 311 complaints. We will keep only complaints handled by the New York City Police Department and related to noise.

Columns to look at

Pay particular attention to:

created_date — when the complaint was created. Use this column to organize complaints into time intervals.
agency_name — the full name of the responsible agency. We want New York City Police Department.
complaint_type — the main complaint category. We want complaint types beginning with Noise.
descriptor — a more detailed description of the complaint, such as Loud Music/Party.
location_type — where the complaint was reported.

We will mainly use created_date, agency_name, and complaint_type in this lab.

import pandas as pd
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
from scipy import stats
from urllib.parse import quote

target_date = "2026-08-20"
end_date = "2026-09-01"

dataset_id = "erm2-nwe9"

where_clause = (
    f"created_date >= '{target_date}T00:00:00' "
    f"AND created_date < '{end_date}T00:00:00'"
)

url = (
    f"https://data.cityofnewyork.us/resource/{dataset_id}.csv"
    f"?$where={quote(where_clause)}&$limit=50000"
)

data = pd.read_csv(url)

print(f"Downloaded {len(data):,} records.")
print(data.shape)
data.head()

Downloaded 50,000 records.
(50000, 44)

/var/folders/7g/054lbs0d6cj31wgldycfb31w0000gn/T/ipykernel_32503/1261155316.py:23: DtypeWarning: Columns (35) have mixed types. Specify dtype option on import or set low_memory=False.
  data = pd.read_csv(url)

	unique_key	created_date	closed_date	agency	agency_name	complaint_type	descriptor	descriptor_2	location_type	incident_zip	...	vehicle_type	taxi_company_borough	taxi_pick_up_location	bridge_highway_name	bridge_highway_direction	road_ramp	bridge_highway_segment	latitude	longitude	location
0	70259761	2026-08-31T23:59:50.000	2026-09-02T14:11:15.000	TLC	Taxi and Limousine Commission	Lost Property	Bag/Wallet	Wallet	Taxi	10013.0	...	NaN	NaN	231 HUDSON STREET, MANHATTAN (NEW YORK), NY, 1...	NaN	NaN	NaN	NaN	40.724282	-74.007840	POINT (-74.007839675607 40.724281751473)
1	70255217	2026-08-31T23:59:30.000	NaN	TLC	Taxi and Limousine Commission	For Hire Vehicle Complaint	Driver Complaint - Non Passenger	Unsafe Driving	Street	11211.0	...	NaN	NaN	636 GRAND STREET, BROOKLYN, NY, 11211	NaN	NaN	NaN	NaN	40.711369	-73.946691	POINT (-73.946691151302 40.711369303559)
2	70257205	2026-08-31T23:59:27.000	2026-09-01T01:34:39.000	NYPD	New York City Police Department	Noise - Street/Sidewalk	Loud Music/Party	NaN	Street/Sidewalk	11691.0	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	40.603358	-73.753866	POINT (-73.75386629509 40.603358017906)
3	70258319	2026-08-31T23:59:25.000	NaN	HPD	Department of Housing Preservation and Develop...	DOOR/WINDOW	WINDOW FRAME	LOOSE OR DEFECTIVE	RESIDENTIAL BUILDING	11368.0	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	40.748623	-73.860494	POINT (-73.860494398392 40.748623091636)
4	70259363	2026-08-31T23:59:21.000	2026-09-01T00:40:00.000	DEP	Department of Environmental Protection	Sewer Maintenance	Backup	NaN	Sewer	10467.0	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	40.881926	-73.862873	POINT (-73.862872685544 40.881926161908)

5 rows × 44 columns

Question 0a — Inspect the important columns

Display/ inspect the following columns:

created_date, agency_name, complaint_type, descriptor, location_type

Then check how many different values appear in agency_name and complaint_type.

Which columns will you need to answer the questions in this lab?

data.agency_name.value_counts()

agency_name
New York City Police Department                       25174
Department of Housing Preservation and Development     7492
Department of Sanitation                               4631
Department of Transportation                           3184
Department of Environmental Protection                 2956
Department of Parks and Recreation                     2147
Department of Buildings                                1513
Department of Health and Mental Hygiene                1324
Department of Homeless Services                         712
Taxi and Limousine Commission                           503
Department of Consumer and Worker Protection            265
Office of the Sheriff                                    66
Economic Development Corporation                         31
Office of Technology and Innovation                       2
Name: count, dtype: int64

data.complaint_type.value_counts()[:15]

complaint_type
Illegal Parking            7809
Noise - Residential        5525
Noise - Street/Sidewalk    4384
Blocked Driveway           2389
UNSANITARY CONDITION       1969
Street Condition           1303
Dirty Condition            1179
Abandoned Vehicle          1107
Noise - Commercial         1022
PAINT/PLASTER               951
Water Maintenance           947
PLUMBING                    886
Noise - Vehicle             846
Noise                       774
Encampment                  734
Name: count, dtype: int64

Question 0b — Keep only NYPD noise complaints

Filter the data so that:

agency_name is exactly New York City Police Department
complaint_type starts with Noise

Save the result as noise.

Hint: str.startswith("Noise", na=False) can be useful.

Check the shape of the resulting DataFrame and display its first few rows.

df_noise = data.query("agency_name == 'New York City Police Department' and complaint_type.str.contains('Noise')")
df_noise.head()

	unique_key	created_date	closed_date	agency	agency_name	complaint_type	descriptor	descriptor_2	location_type	incident_zip	...	vehicle_type	taxi_company_borough	taxi_pick_up_location	bridge_highway_name	bridge_highway_direction	road_ramp	bridge_highway_segment	latitude	longitude	location
2	70257205	2026-08-31T23:59:27.000	2026-09-01T01:34:39.000	NYPD	New York City Police Department	Noise - Street/Sidewalk	Loud Music/Party	NaN	Street/Sidewalk	11691.0	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	40.603358	-73.753866	POINT (-73.75386629509 40.603358017906)
5	70254867	2026-08-31T23:59:15.000	2026-09-01T01:12:10.000	NYPD	New York City Police Department	Noise - Residential	Banging/Pounding	NaN	Residential Building/House	10452.0	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	40.831527	-73.928863	POINT (-73.928863032215 40.831527104823)
8	70251888	2026-08-31T23:59:06.000	2026-09-01T00:28:25.000	NYPD	New York City Police Department	Noise - Commercial	Loud Music/Party	NaN	Store/Commercial	10457.0	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	40.846753	-73.891781	POINT (-73.891780971498 40.846753375601)
9	70261830	2026-08-31T23:59:06.000	2026-09-01T00:16:46.000	NYPD	New York City Police Department	Noise - Residential	Loud Music/Party	NaN	Residential Building/House	11367.0	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	40.724342	-73.811411	POINT (-73.811410930776 40.724341663078)
12	70250608	2026-08-31T23:58:21.000	2026-09-01T00:06:56.000	NYPD	New York City Police Department	Noise - Street/Sidewalk	Loud Music/Party	NaN	Street/Sidewalk	11213.0	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	40.670798	-73.934752	POINT (-73.934751792532 40.670798197587)

5 rows × 44 columns

1. Organize complaints into time intervals

The raw data contains one row per complaint. For probability analysis, we will turn the event data into wating time interval.

Question 1

Use the created_date column.

Create column wating_time in minutes indicating the waiting time between two consecutive complaints.

For example:

00:02 → first complaint arrived
00:15 → second complaint arrived

Then the waiting time between these two complaints would be: 13 minutes.

Create statistical summary of waiting_time using describe().

waiting_time = pd.to_datetime(
    df_noise\
        .created_date\
        .dropna()\
        .sort_values(ascending=True))\
        .diff()\
        .dt\
        .total_seconds() / 60
waiting_time.describe()

count    12097.000000
mean         0.552924
std          0.956760
min          0.000000
25%          0.083333
50%          0.216667
75%          0.583333
max         20.000000
Name: created_date, dtype: float64

2. Visualize the variable

Question 2

Create a histogram of waiting_time with clear axis name.

Then answer briefly:

Is the variable concentrated around a particular value?
Is it symmetric or right-skewed?
Are there intervals with unusually large counts?

import plotly.graph_objects as go

waiting_time = waiting_time

fig = go.Figure()
fig.add_trace(
    go.Histogram(
        x = waiting_time,
        autobinx=False, 
        histnorm='probability',
        marker_color='red', 
        xbins=dict(
            start= 0,
            size=0.1,
            end=20
        )
    )
)
fig.update_layout(
    width=600,
    height=400,
    title='Distribution of the waiting time'
).show()

Unable to display output for mime type(s): application/vnd.plotly.v1+json

3. Pick a distribution family

Question 3:

Let variable \(T>0\) be the waiting times above (you should remove all the zeros from the data).

What is the type of variable \(T\)?
What is its sample space?
What is the distribution family of \(T\)? Propose a probablistic distribution to model this data.

4. Estimate the parameter and generate the model PDF

Question 4

Let \(t_1,t_2,...\) be the observed waiting times.
Write log-likelihood function of this observation for any value of parameter of the model.
Optimize it and find the parameter as a function of these waiting times.
Compute this MLE as a number.
Plot the density of the estimated density above on top of the histogram of the observed waiting times.

# TODO

5. Compute probabilities

Now use the fitted distribution to answer the following questions.

What is the chance that a complaint arrives within the next 5 minutes?
What is the chance that no complaint arrives within the next 30 minutes?
What is the chance that a complaint arrives within the next 10 minutes?

# TODO

6. Network traffic and ad placement

Now we move from a count distribution to simulation.

Imagine a website with four states:

Home
Product
Cart
Exit

A visitor moves from one state to another according to the transition matrix below.

Each row gives the probability of the next state, given the current state.

From / To	Home	Product	Cart	Exit
Home	0.10	0.75	0.05	0.10
Product	0.20	0.45	0.25	0.10
Cart	0.05	0.20	0.25	0.50
Exit	0.00	0.00	0.00	1.00

This is a simple transition-matrix / Markov-chain simulation.

Question 6a

Create the transition matrix in Python and check that every row sums to 1.

states = ["Home", "Product", "Cart", "Exit"]

P = np.array([
    [0.10, 0.75, 0.05, 0.10],
    [0.20, 0.45, 0.25, 0.10],
    [0.05, 0.20, 0.25, 0.50],
    [0.00, 0.00, 0.00, 1.00]
])

transition = pd.DataFrame(P, index=states, columns=states)
transition

Question 6b — Simulate one visitor

Start at Home and repeatedly choose the next state using the transition probabilities.

Stop when the visitor reaches Exit or after 20 steps.

Hint: np.random.choice(states, p=...).

Write a function simulate_visit() that returns the sequence of states.

# TODO

def simulate_visit(max_steps=20):
    # Start at Home
    # At each step, use the appropriate row of P
    # Stop at Exit or max_steps
    pass

simulate_visit()

Question 6c — Simulate many visitors

Simulate 5,000 visitors.

Count how often each state is visited and create a bar chart.

Question 6d — Where should we place an ad?

Suppose we can show an advertisement when a visitor is on a page.

Use your simulation results to answer:

Which non-Exit state receives the most traffic, and why might it be a good place to show an ad?

Do not use the Exit state as an ad location.

n_visitors = 5000
all_visits = []

for _ in range(n_visitors):
    all_visits.extend(simulate_visit())

visit_counts = pd.Series(all_visits).value_counts().reindex(states, fill_value=0)

print(visit_counts)

# TODO: create a bar chart

7. Quick reflection

Answer briefly:

Why is Exponential Distribution a reasonable first model for waiting times T?
What does lambda_hat represent in this lab? What is a large \(\lambda\) mean for the data?
What is the difference between a PMF and a PDF?
What does P(T >= 10) mean in the context of the complaint data?
Why does the transition matrix allow us to simulate website traffic?
Is the most visited page automatically the best advertising location? Give one reason why traffic alone may not be enough.