Lab 3: Probability, Distributions & Simulation

Course: INS-605: Data Analysis II
Lecturer: Sothea HAS, PhD

Student name: …
ID: …

Objective

In this lab, we use real NYC 311 complaint data to connect a simple data-analysis workflow with probability:

  1. Select the relevant observations and organize them into time intervals.
  2. Visualize a random variable.
  3. Pick a reasonable distribution family.
  4. Estimate its parameter(s) and compare the observed data with the probability model.
  5. Compute probabilities from the fitted model.
  6. Simulate website traffic using a transition matrix and decide where an advertisement could be placed.

The data can be downloaded here: Lab3: Probability and Simulation.ipynb

0. Load and understand the NYC 311 data

The data come from NYC Open Data – 311 Service Requests. The downloaded period covers 20 August through 1 September 2026 (the end date is exclusive).

For this lab, we are not interested in all 311 complaints. We will keep only complaints handled by the New York City Police Department and related to noise.

Columns to look at

Pay particular attention to:

  • created_date — when the complaint was created. Use this column to organize complaints into time intervals.
  • agency_name — the full name of the responsible agency. We want New York City Police Department.
  • complaint_type — the main complaint category. We want complaint types beginning with Noise.
  • descriptor — a more detailed description of the complaint, such as Loud Music/Party.
  • location_type — where the complaint was reported.

We will mainly use created_date, agency_name, and complaint_type in this lab.

import pandas as pd
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
from scipy import stats
from urllib.parse import quote

target_date = "2026-08-20"
end_date = "2026-09-01"

dataset_id = "erm2-nwe9"

where_clause = (
    f"created_date >= '{target_date}T00:00:00' "
    f"AND created_date < '{end_date}T00:00:00'"
)

url = (
    f"https://data.cityofnewyork.us/resource/{dataset_id}.csv"
    f"?$where={quote(where_clause)}&$limit=50000"
)

data = pd.read_csv(url)

print(f"Downloaded {len(data):,} records.")
print(data.shape)
data.head()
Downloaded 50,000 records.
(50000, 44)
unique_key created_date closed_date agency agency_name complaint_type descriptor descriptor_2 location_type incident_zip ... vehicle_type taxi_company_borough taxi_pick_up_location bridge_highway_name bridge_highway_direction road_ramp bridge_highway_segment latitude longitude location
0 70248760 2026-08-31T01:59:39.000 NaN DOT Department of Transportation Street Condition Pothole NaN NaN 11412.0 ... NaN NaN NaN NaN NaN NaN NaN 40.694215 -73.753087 POINT (-73.753087331085 40.694214514285)
1 70243572 2026-08-31T01:50:52.000 NaN NYPD New York City Police Department Noise - Street/Sidewalk Loud Music/Party NaN Street/Sidewalk 10472.0 ... NaN NaN NaN NaN NaN NaN NaN 40.827242 -73.874057 POINT (-73.874056693475 40.827242433541)
2 70239573 2026-08-31T01:50:48.000 NaN DOHMH Department of Health and Mental Hygiene Smoking or Vaping Allowed in Smoke Free Area Cannabis Smoking or Vaping Residential Building 11221.0 ... NaN NaN NaN NaN NaN NaN NaN 40.693773 -73.915534 POINT (-73.91553395065 40.693773172836)
3 70249101 2026-08-31T01:50:26.000 NaN NYPD New York City Police Department Noise - Commercial Loud Music/Party NaN Club/Bar/Restaurant 11226.0 ... NaN NaN NaN NaN NaN NaN NaN 40.654245 -73.952867 POINT (-73.952867166126 40.654245046963)
4 70248458 2026-08-31T01:50:21.000 NaN NYPD New York City Police Department Noise - Street/Sidewalk Loud Music/Party NaN Street/Sidewalk 11354.0 ... NaN NaN NaN NaN NaN NaN NaN 40.766446 -73.829089 POINT (-73.829089290124 40.766446350276)

5 rows × 44 columns

data.query("agency_name == 'New York City Police Department' and complaint_type.str.startswith('Noise')").shape
(12534, 44)

Question 0a — Inspect the important columns

Display/ inspect the following columns:

created_date, agency_name, complaint_type, descriptor, location_type

Then check how many different values appear in agency_name and complaint_type.

  • Which columns will you need to answer the questions in this lab?
# TODO

Question 0b — Keep only NYPD noise complaints

Filter the data so that:

  • agency_name is exactly New York City Police Department
  • complaint_type starts with Noise

Save the result as noise.

Hint: str.startswith("Noise", na=False) can be useful.

Check the shape of the resulting DataFrame and display its first few rows.

# TODO

1. Organize complaints into time intervals

The raw data contains one row per complaint. For probability analysis, we will turn the event data into wating time interval.

Question 1

Use the created_date column.

  1. Create column wating_time in minutes indicating the waiting time between two consecutive complaints.

For example:

  • 00:02 → first complaint arrived
  • 00:15 → second complaint arrived

Then the waiting time between these two complaints would be: 13 minutes.

  1. Create statistical summary of waiting_time using describe().
# TODO

2. Visualize the variable

Question 2

Create a histogram of waiting_time with clear axis name.

Then answer briefly:

  1. Is the variable concentrated around a particular value?
  2. Is it symmetric or right-skewed?
  3. Are there intervals with unusually large counts?
# TODO
# Hint: px.histogram(...)

3. Pick a distribution family

Question 3:

Let variable \(T>0\) be the waiting times above.

  1. What is the type of variable \(T\)?
  2. What is its sample space?
  3. What is the distribution family of \(T\)? Propose a probablistic distribution to model this data.

4. Estimate the parameter and generate the model PMF

Question 4

  1. Let \(t_1,t_2,...\) be the observed waiting times.
  2. Write log-likelihood function of this observation for any value of parameter of the model.
  3. Optimize it and find the parameter as a function of these waiting times.
  4. Compute this MLE as a number.
  5. Plot the density of the estimated density above on top of the histogram of the observed waiting times.
# TODO

5. Compute probabilities

Now use the fitted distribution to answer the following questions.

  1. What is the chance that a complaint arrives within the next 5 minutes?
  2. What is the chance that no complaint arrives within the next 30 minutes?
  3. What is the chance that a complaint arrives within the next 10 minutes?
# TODO

6. Network traffic and ad placement

Now we move from a count distribution to simulation.

Imagine a website with four states:

  • Home
  • Product
  • Cart
  • Exit

A visitor moves from one state to another according to the transition matrix below.

Each row gives the probability of the next state, given the current state.

From / To Home Product Cart Exit
Home 0.10 0.75 0.05 0.10
Product 0.20 0.45 0.25 0.10
Cart 0.05 0.20 0.25 0.50
Exit 0.00 0.00 0.00 1.00

This is a simple transition-matrix / Markov-chain simulation.

Question 6a

Create the transition matrix in Python and check that every row sums to 1.

states = ["Home", "Product", "Cart", "Exit"]

P = np.array([
    [0.10, 0.75, 0.05, 0.10],
    [0.20, 0.45, 0.25, 0.10],
    [0.05, 0.20, 0.25, 0.50],
    [0.00, 0.00, 0.00, 1.00]
])

transition = pd.DataFrame(P, index=states, columns=states)
transition

Question 6b — Simulate one visitor

Start at Home and repeatedly choose the next state using the transition probabilities.

Stop when the visitor reaches Exit or after 20 steps.

Hint: np.random.choice(states, p=...).

Write a function simulate_visit() that returns the sequence of states.

# TODO

def simulate_visit(max_steps=20):
    # Start at Home
    # At each step, use the appropriate row of P
    # Stop at Exit or max_steps
    pass

simulate_visit()

Question 6c — Simulate many visitors

Simulate 5,000 visitors.

Count how often each state is visited and create a bar chart.

Question 6d — Where should we place an ad?

Suppose we can show an advertisement when a visitor is on a page.

Use your simulation results to answer:

Which non-Exit state receives the most traffic, and why might it be a good place to show an ad?

Do not use the Exit state as an ad location.

n_visitors = 5000
all_visits = []

for _ in range(n_visitors):
    all_visits.extend(simulate_visit())

visit_counts = pd.Series(all_visits).value_counts().reindex(states, fill_value=0)

print(visit_counts)

# TODO: create a bar chart

7. Quick reflection

Answer briefly:

  1. Why did we convert individual complaints into counts per 30-minute interval?
  2. Why is a Poisson distribution a reasonable first model for X?
  3. What does lambda_hat represent in this lab?
  4. What is the difference between a PMF and a PDF?
  5. What does P(X >= 10) mean in the context of the complaint data?
  6. Why does the transition matrix allow us to simulate website traffic?
  7. Is the most visited page automatically the best advertising location? Give one reason why traffic alone may not be enough.