Explains how a random forest fixes a single decision tree's weaknesses on MNIST, through bootstrap sampling, random feature subsets, and averaging. Covers scikit-learn defaults, out-of-bag scoring, feature and permutation importance, and a sweep exercise. This episode is narrated by AI-generated synthetic voices. Episode page & show notes Visit website Why one tree falls short Last time a single decision tree on MNIST reached about 87–88 percent on validation images, and its shape changed when a random five percent of rows was dropped. A random forest grows many trees on slightly different versions of the data and averages them. RandomForestClassifier on raw pixels gets close to 97 percent, level with nearest neighbors and above logistic regression's 92. Bagging and out-of-bag rows Each tree trains on a bootstrap sample, drawn with replacement. A worked ten-row example shows some rows repeated and others missing. About 63 percent of distinct rows land in a sample, and about 37 percent are left out. Those out-of-bag rows serve as a check on that tree. Why averaging helps, and why it stops Averaging M trees leaves variance of rho times sigma squared plus (1 minus rho) times sigma squared over M. With sigma squared of 1, 100 trees and rho of 0.5, variance only falls to about 0.505. Shared mistakes cannot be averaged away. So each split also looks at a random subset of features, the square root of 784, which is 28 pixels. Trees get slightly weaker but disagree more. Breiman's 2001 paper bounds forest error by tree strength and correlation, and shows adding trees does not cause overfitting. Building one and reading the defaults Fit on 50,000 images, validate on 10,000, keep the last 10,000 sealed, no scaling needed. Defaults covered: n_estimators 100 (10 before version 0.22), max_features "sqrt" (earlier "auto"), bootstrap True, min_samples_leaf 1, oob_score False, n_jobs and random_state. Fully grown trees use lots of memory; min_samples_leaf or max_depth shrinks them. One published benchmark reports 97.20 percent test accuracy on raw MNIST, with an unstated tree count. A forest predicts much faster than nearest neighbors, so time both. Out-of-bag score With oob_score on, each training image is scored only by trees that missed it. Expect a slightly pessimistic number, optimism if you tune on it, and NaN rows in oob_decision_function_ with very few trees. Importances and probabilities feature_importances_ is computed on training data and can favor features with many distinct values. Permutation importance uses held-out data, but correlated neighboring pixels can hide each other. predict_proba averages leaf fractions, giving smoother scores that are not necessarily calibrated. ExtraTreesClassifier draws random thresholds and defaults to no bootstrap. Exercise Sweep n_estimators (10, 50, 100, 300) against max_features ("sqrt", 0.5) on validation only, with oob_score on and a fixed seed. Guess first, then record out-of-bag score, validation accuracy, fit time and prediction time. Gradient boosting, which adds trees one after another to fix earlier errors, comes next.