🌳 From Decision Trees to Random Forests: Key ML Concepts Explained

Machine learning can feel overwhelming with all its jargon — bias, variance, entropy, random forests, transfer learning — the list goes on. In this post, we’ll walk through some of the most important concepts, explained simply using the same style as common exam questions.

🔍 Diagnostics in Machine Learning

A diagnostic is a test you run to gain insight into what is (or isn’t) working with a learning algorithm.

• If training error is low but cross-validation error is high → the model has high variance (overfitting).
• If training error is high compared to the baseline → the model has high bias (underfitting).

⚖️ Bias vs. Variance

Balancing bias and variance is at the heart of machine learning:

• High Bias (Underfitting) → Model is too simple.
  Fix by: adding features, using a more complex model, or decreasing regularization.
• High Variance (Overfitting) → Model is too complex.
  Fix by: collecting more data, using regularization, or simplifying the model.

Problem	Symptoms	Solutions
High Bias	High training error	More features, more complex model, reduce λ
High Variance	Low training error but high CV error	More data, increase λ, simplify model

🐱 Decision Trees Basics

Decision trees split data based on features that give the highest information gain using entropy.

H(p) = -p log2(p) - (1 - p) log2(1 - p)

Information Gain:

IG = H(root) - (w_left H(left) + w_right H(right))

Example: If 10 animals = 6 cats + 4 not cats →

H = -0.6 log2(0.6) - 0.4 log2(0.4)

This measures “impurity.” Splits aim to reduce impurity.

🌲 Random Forests

A random forest is just a collection of decision trees, but each tree is built slightly differently to avoid them being identical:

• Bootstrap sampling: Train each tree on data sampled with replacement.
• Feature randomness: Each tree only sees a random subset of features at each split.

This diversity makes random forests more powerful and less prone to overfitting.