Random forests

Trees, forests and variable importance

Dr Massimiliano Canzi

Random forests

Did you know: orangutans sometimes hold a bundle of leaves in front of their mouth when they give a “kiss squeak” at a predator. The leaves lower the frequency of the call, and lower calls come from bigger orangutans.

Source: Hardus et al. (2009), Proceedings of the Royal Society B

Where we are

Every model so far started from a formula that we wrote: which predictors, which interactions, which curves. A random forest works the other way round. It searches the predictors for splits that separate the outcomes, and finds interactions and non-linear effects without being told.

In linguistics, random forests are popular for data with many predictors that are correlated with each other, such as corpus and sociolinguistic data (Tagliamonte & Baayen, 2012). This lecture explains how they work; next time you build one yourselves.

Today

  1. Decision trees
  2. From trees to forests
  3. Variable importance
  4. Forests and regression models

The data

A simulated sociolinguistic study of /t, d/ deletion, as in mis’ for mist: 40 speakers, 50 tokens each.

set.seed(2032)
speakers <- tibble(speaker = sprintf("S%02d", 1:40), age = round(runif(40, 18, 80)),
                   gender = sample(rep(c("female", "male"), 20)), s0 = rnorm(40, 0, 0.5)) |>
  mutate(education = if_else(age + rnorm(40, 0, 10) > 50, "school", "university"))
td <- crossing(speakers, token = 1:50) |>
  mutate(following = sample(c("consonant", "vowel", "pause"), n(), TRUE, prob = c(0.5, 0.3, 0.2)),
         morph = sample(c("monomorpheme", "past"), n(), TRUE, prob = c(0.6, 0.4)),
         rate = round(rnorm(n(), 5, 0.8), 2), log_freq = round(runif(n(), 1, 6), 2),
         noise_cont = rnorm(n()), noise_bin = sample(c("a", "b"), n(), TRUE),
         eta = -0.8 + s0 + 1.8 * (following == "consonant") + 0.4 * (following == "pause") -
           1 * (morph == "past") + 0.9 * (rate - 5) * (following == "consonant") +
           1.2 * plogis((45 - age) / 5) + 0.25 * (log_freq - 3.5),
         deleted = factor(if_else(rbinom(n(), 1, plogis(eta)) == 1, "deleted", "retained"),
                          levels = c("retained", "deleted"))) |>
  mutate(across(c(speaker, gender, education, following, morph, noise_bin), factor)) |>
  select(speaker, deleted, following, morph, rate, log_freq, age, gender, education, noise_cont, noise_bin)

What we built in

Deletion is more likely before a consonant, especially in fast speech (left), less likely in past tense forms, and more likely among speakers under about 45 (right). Education has no effect, but older speakers more often left school early (filled points). noise_cont and noise_bin are pure noise.

Decision trees

Did you know: orangutan kiss squeaks are highest when made with the lips alone, lower with a hand in front of the mouth, and lowest with leaves.

Source: Hardus et al. (2009), Proceedings of the Royal Society B

Splitting the data

A decision tree asks one yes/no question about one predictor, splits the data in two, and repeats the process in each half. The split chosen at each step is the one that makes the two halves as pure as possible, with mostly deleted or mostly retained tokens.

f <- deleted ~ following + morph + rate + log_freq + age + gender + education + noise_cont + noise_bin
tree2 <- rpart(f, data = td, method = "class", control = rpart.control(maxdepth = 2))

A tree with two levels

tree2
n= 2000 

node), split, n, loss, yval, (yprob)
      * denotes terminal node

1) root 2000 882 deleted (0.4410000 0.5590000)  
  2) following=pause,vowel 1002 414 retained (0.5868263 0.4131737)  
    4) age>=44.5 646 212 retained (0.6718266 0.3281734) *
    5) age< 44.5 356 154 deleted (0.4325843 0.5674157) *
  3) following=consonant 998 294 deleted (0.2945892 0.7054108) *

Read each line as a rule: the condition, the number of tokens, the misclassified tokens, the majority class, and the proportions of retained and deleted.

What a tree can do

  • It finds interactions by itself: the split below a split applies only to the tokens in that branch
  • It handles non-linear effects, categorical predictors and outliers without any transformation
  • Its predictions are steps: inside each final group, the leaf, every token gets the same prediction

Trees overfit

Trained on 30 speakers, tested on 10 new ones. The deep tree learns the noise of its training data and does worst on new speakers. The forest fits its training data perfectly too, so its training accuracy says little; on new speakers it matches the depth-3 tree. Its advantage is stability: small changes in the data can change a single tree completely.

From trees to forests

Did you know: the leaf technique is found only in some orangutan populations, while kiss squeaks themselves are universal. Hardus and colleagues see it as a cultural variant.

Source: Hardus et al. (2009), Proceedings of the Royal Society B

Bagging

Bagging, short for bootstrap aggregating, grows many deep trees, each on a bootstrap sample: as many observations as in the data, drawn with replacement (Breiman, 1996). Each tree overfits in its own way, and averaging their votes cancels much of that noise.

But if one predictor is very strong, every tree starts with the same split, and the trees stay similar to each other.

Random forests

A random forest adds one more source of randomness (Breiman, 2001). At each split, a tree may only choose among a random subset of the predictors.

  • The size of the subset is mtry. In {ranger} the default is the square root of the number of predictors, rounded down: 3 of our 9
  • Weaker predictors get their chance to split, so the trees differ more, and their average predicts better

Fitting a forest

rf <- ranger(f, data = td, num.trees = 1000, importance = "permutation", seed = 1)
rf
Ranger result

Call:
 ranger(f, data = td, num.trees = 1000, importance = "permutation",      seed = 1) 

Type:                             Classification 
Number of trees:                  1000 
Sample size:                      2000 
Number of independent variables:  9 
Mtry:                             3 
Target node size:                 1 
Variable importance mode:         permutation 
Splitrule:                        gini 
OOB prediction error:             30.80 %

Out-of-bag error

A bootstrap sample leaves out about a third of the observations (on average 1 − 1/e ≈ 37%). Each tree can therefore be tested on the tokens it never saw, its out-of-bag (OOB) tokens. The OOB prediction for a token is the vote of the trees that did not train on it.

  • The OOB prediction error, here 0.308, estimates the error on new data without a separate test set
  • Compare it with a baseline: always guessing the more common outcome would be wrong 0.441 of the time

Tuning

  • Extra trees (num.trees) cost computing time but do not make the forest worse. Use enough that results do not change between seeds
  • The settings mtry and min.node.size matter more. The defaults are often close to the best values, as here (the default in orange)

Variable importance

Did you know: in 2022, a wild Sumatran orangutan called Rakus chewed the leaves of a climbing plant used in traditional medicine and applied them to a wound on his face. The wound closed within five days.

Source: Laumer et al. (2024), Scientific Reports

Permutation importance

A forest has no coefficients. To see which predictors it relies on, shuffle one predictor at a time and measure how much the OOB accuracy drops.

  • A predictor the forest uses gives a large drop: its permutation importance
  • A useless predictor gives a drop around zero, sometimes slightly negative
  • The scale has no fixed meaning; only the ranking and the distance from zero do

Two kinds of importance

Impurity importance adds up how much each predictor improved the splits. It is fast, but it favours continuous predictors and factors with many levels (Strobl et al., 2007). Compare noise_cont and noise_bin: both are noise.

Correlated predictors

Education has no effect in our data. Whatever importance it gets comes from its correlation with age: shuffling education breaks that link, and the forest loses some of the information it took from the pair.

  • Permutation importance is marginal: it credits a predictor for what it shares with others
  • Conditional permutation importance shuffles a predictor only within groups of similar values of the correlated predictors, so it asks what a predictor adds on top of the others (Strobl et al., 2008; Debeer & Strobl, 2020). In R: permimp::permimp() on a forest from party::cforest()

Partial dependence

A partial dependence plot sets every token to the same value of one predictor, predicts, averages, and repeats for each value (Friedman, 2001).

rf_prob <- ranger(f, data = td, num.trees = 500, probability = TRUE, seed = 1)
partial <- function(model, data, var, values) {
  map(values, \(v) {
    data[[var]] <- v
    tibble(value = v, p = mean(predict(model, data)$predictions[, "deleted"]))
  }) |> list_rbind()
}

The effect of age

The simulation lowers deletion gradually between about 35 and 55, and the curve falls in that range without being given a shape. The later drop at 75 is not built in: age belongs to speakers, and with 40 speakers the curve also follows individuals.

Interactions

Before a consonant, deletion rises with speech rate; elsewhere it hardly changes. Partial dependence curves are flatter than the true effects, because forests average many step functions.

Forests and regression models

Did you know: wild orangutans have the longest interbirth intervals of any mammal. Across seven field sites, mothers gave birth on average every 7.6 years.

Source: van Noordwijk et al. (2018), Journal of Human Evolution

What forests are good at

  • Many predictors, even correlated ones or more predictors than observations, with interactions and non-linear effects that nobody thought to put in a formula
  • Prediction, with an honest estimate of the error from the OOB data

What forests do not give you

  • No coefficients, standard errors or p-values. Importance is a ranking, and says nothing about the direction or shape of an effect
  • No random effects. Tokens from the same speaker are not independent, and the OOB error ignores that: a forest can recognise speakers instead of learning patterns that generalise (next lecture)
  • The results depend on the seed, the settings and the importance measure, so they need to be reported in full

Both together

Tagliamonte and Baayen (2012) analysed was/were variation in York with a mixed-effects logistic regression, a conditional inference tree and a random forest. The forest ranked the predictors, including a correlated set that the regression could not handle together, and the regression estimated the effects with a random effect for speaker.

A common workflow: use a forest to explore which predictors matter and how, then test specific hypotheses with a (mixed) regression model.

Reporting

We fitted a random forest with 1,000 trees ({ranger}; Wright & Ziegler, 2017) to predict deletion from nine predictors, with mtry = 3 and a minimum node size of 1. The OOB error was 0.31, against 0.44 for always predicting the more frequent outcome. Variable importance was assessed by permutation.

Add the seed, how importance changed between seeds, and partial dependence plots for the predictors you discuss.

Summary

  • A decision tree splits the data with one predictor at a time; it finds interactions but overfits and is unstable
  • A random forest averages many trees grown on bootstrap samples, each split choosing among mtry random predictors
  • The OOB error estimates prediction error; always compare it with the majority-class baseline
  • Use permutation importance over impurity importance, and conditional importance when predictors are correlated
  • Partial dependence plots show the direction and shape of effects; regression models still give you effect sizes and random effects

References

  • Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.
  • Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.
  • Debeer, D., & Strobl, C. (2020). Conditional permutation importance revisited. BMC Bioinformatics, 21, 307.
  • Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5), 1189–1232.
  • Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics, 8, 25.
  • Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T., & Zeileis, A. (2008). Conditional variable importance for random forests. BMC Bioinformatics, 9, 307.
  • Strobl, C., Malley, J., & Tutz, G. (2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods, 14(4), 323–348.
  • Tagliamonte, S. A., & Baayen, R. H. (2012). Models, forests, and trees of York English: Was/were variation as a case study for statistical practice. Language Variation and Change, 24(2), 135–178.
  • Wright, M. N., & Ziegler, A. (2017). ranger: A fast implementation of random forests for high dimensional data in C++ and R. Journal of Statistical Software, 77(1), 1–17.

Sources of the animal facts

  • Hardus, M. E., Lameira, A. R., van Schaik, C. P., & Wich, S. A. (2009). Tool use in wild orang-utans modifies sound production: A functionally deceptive innovation? Proceedings of the Royal Society B, 276(1673), 3689–3694. https://doi.org/10.1098/rspb.2009.1027
  • Laumer, I. B., Rahman, A., Rahmaeti, T., Azhari, U., Hermansyah, Atmoko, S. S. U., & Schuppli, C. (2024). Active self-treatment of a facial wound with a biologically active plant by a male Sumatran orangutan. Scientific Reports, 14, 8932. https://doi.org/10.1038/s41598-024-58988-7
  • van Noordwijk, M. A., Atmoko, S. S. U., Knott, C. D., Kuze, N., Morrogh-Bernard, H. C., Oram, F., Schuppli, C., van Schaik, C. P., & Willems, E. P. (2018). The slow ape: High infant survival and long interbirth intervals in wild orangutans. Journal of Human Evolution, 125, 38–49. https://doi.org/10.1016/j.jhevol.2018.09.004

Slides and code

QR code linking to these slides

These slides

mcanzi.de/ss27_advanced_statistics/06_random_forests.html

Code and all lectures

github.com/mcanzi/ss27_advanced_statistics