How today works

As in lecture 5: you work in pairs, the task slides say what to do, checkpoints show what you should get, and the solutions are in the .Rmd file of these slides on GitHub.

Task 1: look at the data

1. How many tokens are there per speaker and per group?
2. Compute speaker means for each measure, and plot them by L1. Which measures separate the groups?
3. What accuracy would you get by always guessing the same L1? This is your baseline

Checkpoint 1

VOT, the vowel ratio and, less clearly, articulation rate differ between the groups. f0 mostly separates women from men. The baseline is 50%.

Task 2: fit a forest

f_all <- l1 ~ vot + vowel_ratio + rate + f0 + h1h2 + intensity
rf_all <- ranger(f_all, data = accent, num.trees = 1000, importance = ___, seed = 1)
rf_all
1 - rf_all$prediction.error

• Ask for permutation importance, as in lecture 6
• The last line is the OOB accuracy. How does it compare with the baseline?
• Plot the variable importance. Which predictors does the forest use most?

Checkpoint 2

OOB accuracy: 0.892. Before you go on: f0 and H1–H2 have no link with L1 in the simulation. Why could the forest find them useful?

Task 3: the trap

Every speaker’s 40 tokens are in the data. A tree’s bootstrap sample therefore almost always contains other tokens of the speaker whose OOB token it predicts.

• f0 and H1–H2 vary between speakers but little within them, like a speaker fingerprint. The forest can recognise the speaker and look up their L1
• That does not help with a new speaker, which is what the research question is about

Speaker fingerprints

Each column holds one speaker’s 40 tokens. They cluster tightly, while English and Italian speakers are mixed along the whole range. A token’s f0 tells the forest who is speaking, and the speaker’s other tokens tell it the L1.

Task 4: cross-validation by speaker

set.seed(2)
folds <- distinct(accent, speaker, l1) |>
  mutate(fold = sample(rep(1:5, length.out = n())), .by = l1)
cv_speakers <- function(formula) {
  map(1:5, \(k) {
    train <- filter(accent, speaker %in% folds$speaker[folds$fold != k])
    test <- filter(accent, speaker %in% ___)
    m <- ranger(formula, data = train, num.trees = 500, seed = k)
    tibble(speaker = test$speaker, l1 = test$l1, pred = predict(m, test)$predictions)
  }) |> list_rbind()
}

Complete the test set, run cv_speakers(f_all), and compute the accuracy on held-out speakers. Each fold leaves out three speakers from each group.

Task 5: fewer predictors

Refit with only the three measures that differed between the groups in Task 1: vot, vowel_ratio and rate.

1. Compare the OOB accuracy and the speaker-wise cross-validated accuracy of both forests
2. Which forest would you trust for new speakers?

Checkpoint 3

Only cross-validation by speaker estimates how well the forest classifies speakers it has not seen. The dashed line is the baseline.

Task 6: classify speakers

The research question asks about speakers, so the last step goes from tokens to people. Use the cross-validated predictions of the three-measure forest:

• For each held-out speaker, take the L1 that most of their 40 tokens were assigned to
• How many of the 30 speakers are classified correctly? How does this compare with the token accuracy?

Task 7: how do the measures matter?

Fit a probability forest with the three measures on all data, and draw a partial dependence plot for VOT, as in lecture 6.

rf_prob <- ranger(f_l1, data = accent, num.trees = 500, probability = ___, seed = 1)

• In which VOT range does the predicted probability of an Italian L1 change most?
• Does that match what you know about VOT in English and Italian?

Checkpoint 4

30 of 30 held-out speakers classified correctly by majority vote.

Task 8: report

Write a short methods and results paragraph. It should say:

• the number of trees, mtry and the seed, and why you chose the predictors you kept
• how accuracy was estimated, and why OOB accuracy was not enough
• token and speaker accuracy against the baseline, and what the partial dependence plot shows

If you finish early

• Tune mtry for the three-measure forest, using speaker-wise cross-validation and not the OOB error
• Fit lme4::glmer(l1 ~ scale(vot) + scale(vowel_ratio) + scale(rate) + (1 | speaker), family = binomial). What does the mixed model tell you that the forest does not, and what does it fail to see?

Summary

Slides and code

These slides

mcanzi.de/ss27_advanced_statistics/07_random_forests_practical.html

Code and all lectures

github.com/mcanzi/ss27_advanced_statistics