Lecture 2: EDA & visualization

Applied Statistics: From Data to Decisions

Professor Madeleine Udell

Wednesday, April 1, 2026

the perils of summary statistics

four datasets. identical mean, SD, correlation, regression line.

so they must look the same, right?

Let this land. “Same statistics, completely different data.” This is the single most important lesson in EDA: numbers alone are not enough. You must visualize.

same stats, different graphs

Matejka & Fitzmaurice, “Same Stats, Different Graphs” (Autodesk Research, 2017)

The modern update to Anscombe. All 12 datasets have the same mean, SD, and correlation to two decimal places. One is a dinosaur. Spend 15 seconds — the image does the work.

always look at your data

today: the EDA workflow

1. first look: shape, types, first impressions
2. distributions: what does each variable look like?
3. relationships: how do variables relate?
4. categorical comparisons: groups and compositions
5. missing data: how much? any patterns?
6. confounding: when associations mislead

Six parts, following the book chapter exactly. Each one builds on the last. We’ll use one primary dataset — Airbnb listings — throughout.

four modes of statistics

• summary: describe the data (today)
• prediction: forecast new values
• inference: quantify uncertainty
• causation: establish cause and effect

today we focus on summary — what’s in the data, and what traps are hiding in it

Brief framing. We’ll return to prediction, inference, and causation throughout the course. Today is about looking before you model.

part 1: first look at the data

the dataset: NYC Airbnb

29,142 listings from Inside Airbnb — every active rental in New York City

you’re a traveler trying to find a good deal. where do you start?

Set the scene. This is real data — not a cleaned textbook dataset. It has all the problems real data has.

Demo: first look at Airbnb data

open in Colab

DEMO (~5 min). Walk through chapter 2, Part 1.

Key cells to run: - df.shape → (29,142, 96) — that’s 96 columns! - Select 11 columns: name, neighbourhood_group_cleansed, neighbourhood_cleansed, room_type, price, bedrooms, beds, security_deposit, number_of_reviews, review_scores_rating, accommodates - df.dtypes → note that most are int64/float64, text columns are object - df.describe() → scan the count row: different counts for different columns (missing data!) - df.head() → first impressions

End with: “we have 29,000+ listings, 11 columns, and already some gaps in the count row.”

what the demo showed

• 29,142 rows, 96 columns → selected 11
• dtypes: mix of int64, float64, object
• .describe() count row: some columns have fewer entries — missing data

six semantic types: continuous, discrete, nominal, ordinal, text, identifier

The six semantic types from the book. Price is continuous, bedrooms is discrete, borough is nominal, review scores could be treated as ordinal. The key lesson: pandas dtype tells you how it’s stored, not what it means.

part 2: distributions

Demo: price distribution

open in Colab

DEMO (~5 min). Walk through Part 2 of the chapter.

Key cells to run: - Histogram of all prices → bunched at left, invisible tail - Zoom to $1–$500 → reveals the shape: peak near $100, long right tail - Before computing: ask students to estimate the mean vs. median - Show the histogram with mean (red dashed) and median (orange) marked

Key moment: “Most listings are $50–$200, but a few are $999. Does the mean represent a typical listing?”

what does “typical” mean?

the histogram tells a different story

dramatic right skew — long tail to $999/night

• most listings: $50–$200
• a few listings: $500+

which single number represents a “typical” listing?

This motivates the definitions that follow. Students have seen the histogram — now they need vocabulary to describe what they saw.

mean vs. median

mean

\(\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i\)

• the arithmetic average
• sensitive to extreme values

median

the middle value when sorted

• half below, half above
• barely affected by outliers

Show these AFTER the histogram. The definitions land because students already have a visual intuition for why the mean is misleading here.

which summary do you trust?

the mean is $133. the median is $100.

if you price your listing at the mean, roughly what fraction of listings are cheaper than yours?

well over half — the right tail pulls the mean above most of the data

Give 15 seconds. The answer is well over half. This is the core lesson: when data is right-skewed, the mean overrepresents expensive listings. The median is a better measure of “typical.”

quartiles and IQR

quartiles

divide sorted data into four equal parts

• Q1 (25th percentile), Q2 (median), Q3 (75th percentile)

interquartile range (IQR)

Q3 − Q1 — the width of the middle 50%. a robust measure of spread.

for Airbnb prices: Q1 = $67, Q3 = $165, IQR = $98

Robust = barely affected by extreme values. Mean and SD are not robust. Median and IQR are. When you have heavy tails, robust summaries are more reliable.

same data, different lens

raw scale

histogram bunched at left, invisible tail

mean pulled far from center

log scale

roughly symmetric

structure in the tail becomes visible

use a log axis to look at skewed data; log transform to model it

The log scale doesn’t change the data — it changes the ruler. Each tick mark represents a multiplication (×10) rather than an addition (+100). The book shows three panels: linear histogram, log-axis with log-spaced bins, and log-transformed histogram.

part 3: relationships between variables

I’m about to plot price (log scale, y-axis) vs. number of guests (x-axis)

• sketch what you expect
• does each additional guest add a fixed dollar amount, or multiply the price?
• does the spread stay constant or grow?

1 min sketch or discuss. then I’ll reveal the plot.

DISCUSSION: Predict-then-reveal (3 min) After 1 min, show the scatter plot from the demo. The key question is additive vs. multiplicative: if each guest adds $50, the relationship is linear on the raw scale but curved on the log scale. If each guest multiplies the price by 1.3×, the relationship is linear on the log scale. Most students will predict “goes up” but won’t distinguish additive from multiplicative. The reveal: it’s roughly linear on the log scale — multiplicative. The spread also increases with group size.

Demo: scatter plots and 2D histograms

open in Colab

DEMO (~3 min). Walk through Part 3 of the chapter.

Key cells to run: - Price vs. accommodates scatter: linear scale (overplotted mess) → log scale on y-axis (clear upward trend) - 2D histogram: overplotted scatter vs. 2D histogram with log color scale - Key insight: most listings accommodate 1–4 guests at $50–$200/night

The 2D histogram is the tool for dense data — scatter plots fail when points pile up.

when scatter plots fail

29,000 points stacked on top of each other — you can’t see the pattern

2D histogram: bin both axes, color by count

• reveals that the overwhelming majority of listings are 1–4 guests at $50–$200
• the expensive outliers are rare, not representative

Brief slide to anchor what the demo showed. The 2D histogram is one of the most useful and least-taught EDA tools.

part 4: categorical variables

room types and boroughs

room types

• Entire home/apt: 15,022
• Private room: 13,469
• Shared room: 651

boroughs

• 5 boroughs, 212 neighborhoods
• Manhattan has the most listings

bar charts make these comparisons easy — pie charts don’t

Set up the pie chart critique. The numbers come directly from the book chapter.

how to lie like Steve Jobs

Steve Jobs presenting US smartphone market share. Apple has 19.5% of the market, but the 3D perspective and positioning make it look comparable to RIM’s 39%. A bar chart would instantly show the truth. Pie charts mislead because humans are bad at comparing angles and areas — especially when 3D perspective distorts them further.

Demo: categorical comparisons

open in Colab

DEMO (~5 min). Walk through Part 4 of the chapter.

Key cells to run: - Bar chart of room type counts - Bar chart of top 10 neighborhoods - Pie chart vs. bar chart for neighborhoods (pie makes Williamsburg, Bed-Stuy, Harlem indistinguishable) - Dodged bar chart: room type by borough - Box plots: price by room type (entire homes ~2× private rooms) - Violin plots: price by borough (Manhattan has wider spread)

Key moment: the pie vs. bar comparison. “Can you tell which neighborhood has more listings — Williamsburg or Bed-Stuy? In the pie chart, they look identical. The bar chart makes it obvious.”

you’ve seen price distributions for the whole city

• predict: do entire homes or private rooms have higher median price?
• by how much? (2×? 5×? 10×?)
• does the spread differ?

30 sec think. 1 min pair. then I’ll show the box plot.

DISCUSSION: Predict-then-reveal (3 min) Reveal: entire homes are roughly 2× private rooms in median price. Private room prices cluster tightly around $60; entire home prices spread from $100 to $300. The violin plot makes this especially clear — the private room violin is narrow and peaked, the entire home violin is wide and flat.

comparing compositions

three ways to show room type by borough:

1. dodged bar chart → compare counts across room types
2. stacked bar chart → see total listings per borough
3. standardized bar chart → compare proportions

Manhattan has a higher proportion of entire homes than Brooklyn

The three bar chart types from the book. Each one answers a different question. The standardized chart is the one that reveals composition differences — and sets up the confounding story in Part 6.

what’s missing?

40% of security deposits are missing

11,827 out of 29,142 listings have no deposit listed

is that random, or does it mean something?

The book shows overlapping density histograms comparing prices for listings with vs. without a deposit. If the distributions look the same, missingness might be random. If they differ, the missingness is informative.

three patterns of missing data

MCAR (missing completely at random)

missingness has no pattern — unrelated to any variable

MAR (missing at random)

missingness depends on observed variables

MNAR (missing not at random)

missingness depends on the unobserved value itself — the most dangerous pattern

Brief definitions here. Chapter 3 covers these in depth with the hospital data. For now: “MCAR is rare, MAR is manageable, MNAR introduces bias.”

a host lists a $500/night apartment but doesn’t fill in the security deposit field

• is the deposit more likely to be high or low?
• does this look like MCAR, MAR, or MNAR?
• if we drop listings with missing deposits, what happens to our price statistics?

1 min think. 2 min pair. 1 min share.

DISCUSSION: Think-pair-share (4 min) Key insight: expensive listings might skip the deposit field because they handle deposits differently (through a property manager, or via Airbnb’s built-in damage protection). If high-price listings are more likely to have missing deposits, dropping them would shift our price distribution downward. This is a possible MAR pattern — missingness depends on observed price.

If a student says “maybe hosts with no deposit are more casual” — that’s also a valid hypothesis. The point is to think about why data is missing, not just that it’s missing.

MNAR in the wild: hospital readmissions

CMS suppresses hospitals with “too few” readmissions to report — the value itself determines whether you see it

Brief callback to the hospital data. The left panel shows all hospitals. The right panel shows what happens when the smallest hospitals are suppressed — the reported mean shifts. This is MNAR: the count drives the suppression, and we want to analyze counts. Full treatment in Chapter 3.

association is not causation

Manhattan vs. Brooklyn

median price: Manhattan $135, Brooklyn $90

Manhattan is $45/night more expensive

but is that comparing like with like?

Set up the confounding reveal. The overall gap is $45. But what happens when we control for room type?

control for room type

	Manhattan	Brooklyn	gap
Entire home	$180	$140	$40
Private room	$85	$62	$23
Shared room	$60	$35	$25
Overall	$135	$90	$45

the overall gap ($45) exceeds every room-type gap ($23–$40)

This is a composition effect. Manhattan has a higher proportion of entire homes (the most expensive room type), which pulls up its overall median. When comparing “like with like” — entire homes to entire homes, private rooms to private rooms — the borough difference shrinks. The overall gap overstates the true effect of location on price.

the confounder

confounder

a variable that affects both the treatment and the outcome

• inflates (or deflates) the observed association beyond the true causal effect

room type drives both location distribution and price

Manhattan isn’t as much more expensive as it looks — room type is doing some of the work

Room type is the confounder. Manhattan has more entire homes, and entire homes cost more. If you don’t control for room type, you attribute the composition difference to the borough. This is a causal concept; we’ll formalize it in Act 3. For now: “when you see an association, ask what else could drive both variables.”

students who attend office hours get higher grades

how much of the grade boost is from office hours, and how much from the kind of student who shows up?

motivation drives both OH attendance and studying — the raw correlation overstates the causal effect

A second example to reinforce the pattern. The association is real. Office hours probably do help. But the observed effect overstates the true causal benefit, because the same motivation that drives students to attend also drives them to study harder independently.

how to lie with data

what’s wrong with each of these?

Quick warm-up. Show the two bad viz slides back-to-back, giving 20-30 seconds each. Students call out what they see. No formal activity structure — just hands up.

• y-axis is inverted — 0 at top, 1,000 at bottom

Reuters / C. Chan, 2014. Data: Florida Dept. of Law Enforcement

The red fill evokes blood dripping down — visually powerful but deeply misleading. At first glance, the downward slope after 2005 looks like a decrease. In fact, the y-axis runs from 0 (top) to 1,000 (bottom), so downward means more deaths. Gun deaths in Florida roughly doubled after Stand Your Ground.

• dates are not in chronological order, but sorted by bar height to fake a downward trend

Georgia Dept. of Public Health, May 2020

The x-axis reads: 28Apr, 27Apr, 29Apr, 01May, 30Apr, etc. This is sorted by total case count (decreasing), not by date. The visual impression is “cases are declining” — but the dates are scrambled. When plotted chronologically, cases were roughly flat or rising. Georgia DPH later corrected the chart after public outcry.

takeaways

• EDA workflow: shape → types → distributions → relationships → missing data
• distributions reveal what summaries hide: always plot; use log scale for heavy tails
• missing data tells a story: ask why before dropping rows
• an association can be real without being causal: confounders inflate observed effects

Map each bullet back to a specific moment from the lecture. EDA checklist → the first look at Airbnb. Distributions → mean vs. median. Missing data → the 40% missing deposits. Association ≠ causation → Manhattan vs. Brooklyn, office hours.

next time

we can describe and diagnose the data

how do we clean it? what do we do with missing values, wrong types, messy text?

Forward pointer to lecture 3 (data munging). The questions raised today — missing deposits, messy data — become actionable in the next lecture.

before next class

• read Chapter 2 in the course notes
• try the notebook exercises on your own
• HW 1 due next week

one-minute feedback

1. what was the most useful thing you learned today?
2. what was the most confusing?

give feedback

Last 1-2 minutes. Anonymous. QR code goes to the same form. Entry field pre-filled with “2” for lecture number.