Classification Meets Inference

Open in ColabOpen in Colab  Download notebook (.ipynb)

Every inference tool from Act 2 — bootstrap, permutation tests, hypothesis tests, multiple testing corrections — also works for classification. This optional chapter applies those tools to classification problems, reinforcing the ideas from Chapters 8–12 in a new context.

NoteOptional chapter

This chapter is not covered in lecture. It serves as review and self-assessment: work through it if you want to solidify your understanding of Act 2 tools, or skip it if you feel confident.

Bootstrap confidence interval for AUC

How uncertain is your classifier’s performance?

Permutation test: does the classifier beat random guessing?

Is this model actually better than flipping a coin?

Hypothesis test on logistic regression coefficients

Which features actually matter?

Multiple testing in feature selection

With 50 features, how many “significant” ones are real?

Confounding in classification

Simpson’s paradox for logistic regression