Frequentist Inference

• Uses only the data and compares it to an idealized model to make inferences about the data
• Example Problem: You lose your cellphone in your house. You have a friend call it and you listen for the sound to find it. Where do you search in the house?
• Frequentist solution: You can hear the phone ringing. You also have a mental model of the house that helps you identify where the sound is coming from. When you hear the phone ringing, you infer the location with the help of your model.

Bayesian Inference

• Draws conclusions from data and all prior knowledge about the data including idealized models
• Uses prior knowledge to make bets on the outcome
• Bayesian solution: You can hear the phone ringing but you also know where you were today in the house and where you tend to lose things. You also have a mental model of the house. You combine the inferences from the ringing with your prior knowledge to find the phone

Bayes’ Theorem

• \(P(A\mid B) = \frac{P(B \mid A) \, P(A)}{P(B)}\)
• The probability of observing event A given that B is true equals the probability of observing event B given the A is true times the probability of observing event A divided by the probability of event B
• Can be reformulated as Bayes’ Rule:
• Prior Odds * Relative Likelihoods = Posterior Odds

Bayesian Terms

• Prior odds = our beliefs about the events prior to looking at the new data
• Relative likelihoods = how likely an observation is when comparing one hypothesis to another
• Posterior odds = our state of belief after comparing the prior information to the likelihoods (and updating with new data)

What they look like

Applying Bayes’ Theorem: Example Problem

• You are treating students with Diseasitis
• 20% of students will have Diseasitis right now based on previous research
• You have a test that turns black if a student has Diseasitis
• If a student has Diseasitis, only 90% cause the test to turn black
• The test also turns black 30% of the time for healthy students
• A student comes in to take the test and it turns black. What is the probability they have Diseasitis?

Solution

• We know
• The probability of black given sick = 0.9
• The probability of black given not sick = 0.3
• The probability of sick = 0.2
• Prior odds X Relative likelihoods = Posterior odds
• Relative likelihoods compare the probabilities of the two observed outcomes (prob of black = sick and prob of black = healthy)
• \(\frac{P(sick)}{P(healthy)} \times \frac{P(black\mid sick)}{P(black\mid healthy)} = \frac{P(sick\mid black)}{P(healthy\mid black)}\)
• \((20\% : 80\%) \times (90\%:30\%) = \frac{3}{4}\) odds or \(\frac{3}{7} =43\%\) probability of sick given blackened test
• Remember: odds \((A:B)\) = probability \(\frac{A}{A+B}\)

Solution (con’t)

• Another way to look at it:
• Imagine a waterfall with two different colored waters
• At the top of the waterfall, 20 gallons/second of red water are flowing down, and
• 80 gallons/second of blue water are coming down.
• 90% of the red water makes it to the bottom.
• 30% of the blue water makes it to the bottom.
• Of the purplish water that makes it to the bottom of the pool, how much was originally from the red stream and how much was originally from the blue stream?

Solution visualized

Applying Bayesian Inference

• Bayesian inference can be applied to most models (ANOVA, linear regression, GLMERs, LMERs, GAMMs, CLMMs, etc.)
• Bayesian inference does not give p-values (more on this later)
• It can run models that otherwise don’t converge with frequentist models
• Easier to intrepret
• Provide distribution of values for each parameter rather than a single output

Applying Bayesian Inference

• Real life language data is much more complex than the earlier examples
• We can’t directly calculate the posterior distribution
• Instead we have to approximate accurate posterior distributions
• The rstanarm package uses the Stan language to generate a posterior distribution via Markov chain Monte Carlo sampling
• This process takes time (sometimes a long time!) and requires thousands of samples
• Stan is much more efficient than BUGS or JAGS (other Bayesian samplers) and is in active development

rstanarm

• Use library(rstanarm)
• stan_lmer() or stan_glmer()
• Similar syntax to lme4 but have to specify
• priors
• number of chains
• iterations
• adapt_delta (optional)

Selecting priors

• No matter what we research, we already know something about the data we will collect before we do
• For phonetics, we know range of sounds human can produce and hear
• We know that it is very unlikely to get meaningful speech sounds that are 100,000 Hz
• VOTs are typical not seconds long or conversely not extremely short like 1 ns
• We pick priors that are weakly informative but still helpful
• We want to make some bets but not bet too strongly to overly influence the results
• We pick priors that are distributions (typically normal) that have a reasonable mean and standard deviation but not too narrow nor too wide

Selecting priors

• The priors tell the model where to start guessing at the data and how strong of a bet to make
• If we have lots of data that strongly diverge from the priors, then the results won’t reflect the priors as much, but if the priors are too strong then they can heavily influence our results

VOT example

• We need to pick priors for the intercept and slopes for VOT data
• What do we know about VOT that are measured in ms?
• We can pick a prior with a normal distribution centered at 0ms with a standard deviation of 50ms: normal(0,50)
• that says we are 68% sure the mean is between -50ms and 50ms, 95% sure between -100ms and 100ms and 99.7% sure between -150ms and 150ms
• We can tweak it as needed
• What if we said normal(0,5)? Is that too strong? Or normal(0,500)? Is that too weak?

Sampling the posterior distribution

• We need to specify how many samples we need in addition to the number of chains
• Stan is quite efficient so we can use a relatively small amount depending on how nice the data are
• Typically use chains = 4 (number of unique samplers) and iter = 2000
• The sampling process requires warmup samples to get a taste of the distribution, which we throw out since not very good (warmup = 1000)
• Sometimes there are divergent samples which are bad so we add adapt_delta=0.999 (if still divergent keep adding 9s until it works)

Let’s make some models!

• These can take a bit of time to run

data_vot <- read_csv("t_d_vot.csv") # import data
options(mc.cores=parallel::detectCores ()) # Run on multiple cores
# strong priors
model1 <-stan_lmer(vot*1000 ~ consonant2 + (1+consonant2|speaker_id) + (1+consonant2|word),
                   data=data_vot,
                   prior_intercept = normal(0, 5,autoscale = F),
                   prior = normal(0, 5,autoscale = F), 
                   prior_covariance = decov(regularization = 2),
                   chains = 4,
                   iter = 2000, adapt_delta=0.999,warmup=1000)

More models

# weaker priors
model2 <-stan_lmer(vot*1000 ~ consonant2 + (1+consonant2|speaker_id) + (1+consonant2|word),
                   data=data_vot,
                   prior_intercept = normal(0, 50,autoscale = F),
                   prior = normal(0, 50,autoscale = F), 
                   prior_covariance = decov(regularization = 2),
                   chains = 4,
                   iter = 2000, adapt_delta=0.999,warmup=1000)

One last model

# very weak priors
model3 <-stan_lmer(vot*1000 ~ consonant2 + (1+consonant2|speaker_id) + (1+consonant2|word),
                   data=data_vot,
                   prior_intercept = normal(0, 500,autoscale = F),
                   prior = normal(0, 500,autoscale = F), 
                   prior_covariance = decov(regularization = 2),
                   chains = 4,
                   iter = 2000, adapt_delta=0.999,warmup=1000)

What are p-values?

• From http://www.labstats.net/articles/pvalue.html
• The probability of getting the results you did (or more extreme results) given that the null hypothesis is true.
• P = 0.05 does not mean there is only a 5% chance that the null hypothesis is true.
• P = 0.05 does not mean there is a 5% chance of a Type I error (i.e. false positive).
• P = 0.05 does not mean there is a 95% chance that the results would replicate if the study were repeated.
• P > 0.05 does not mean there is no difference between groups.
• P < 0.05 does not mean you have proved your experimental hypothesis.

Intrepreting p-values

• If you are studying the difference in VOT between two stops and the p-value is 0.03 that means:
• if there were no difference between the VOT of the two stops, you’d get the observed difference or more in 3% of the studies due to random sampling error.
• It does not give us any information about the alternate hypothesis that there is a difference between the stops’ VOTs.

Bayesian Approach

• No p-values!
• Use Bayesian Highest Density Interval (HDI)
• Area of posterior distribution with 95% highest probability [2.5% to 97.5% of posterior distribution]
• Bayesian approach answers “The probability of the Null Hypothesis being true given the data” (Frequentist is “The probability of observing the data or anything more extreme given the null hypothesis”)
• Bayesian approach is more inuitive
• Multiple comparisons also aren’t an issue!

## Intrepreting HDI

• If there is no overlap of HDIs, then probable difference
• If overlap can say there is probable similarity but also probable difference (can now make these claims rather than just binary answers!)
• Much more nuanced and like real life

Model 1 Results

Model 2 Results

Model 3 Results

Compared to frequentist model

model_freq <- lmer(vot*1000 ~ consonant2 + (1+consonant2|speaker_id) + (1+consonant2|word),data=data_vot)
sjPlot::sjt.lmer(model_freq)

Plotting Posterior Distribution: Model1

Plotting Posterior Distribution: Model2

Plotting Posterior Distribution: Model3

Model Diagnostics

• In summary output want Rhat to be 1.0 or very close to 1 (like 0.98) [measure of how well the chains mix]
• Use launch_shinystan(model1) to explore Stan’s built in diagnostic tools (launches page in your web browser)

Example of Shinystan

How well did the model sample

Model Comparisons

• Like frequentist models, we can compare two different models to see how different they are (like comparing random effects)
• Instead of anova() we use loo() (leave one out) and compare_models()
• Can take a little bit to run
• loo() is actually more like comparing AIC values
• If elpd_diff is small, then little difference between models
• loo is an approximation of k_fold(model,K=10) but is much faster

Loo

• The LOO Information Criterion (LOOIC) has the same purpose as the Akaike Information Criterion (AIC) that is used by frequentists.
• Both are intended to estimate the expected log predictive density (ELPD) for a new dataset.
• However, the AIC ignores priors and assumes that the posterior distribution is multivariate normal, whereas the functions from the loo package do not make this distributional assumption and integrate over uncertainty in the parameters.
• This only assumes that any one observation can be omitted without having a major effect on the posterior distribution

ELPD

• The difference in ELPD will be negative if the expected out-of-sample predictive accuracy of the first model is higher.
• If the difference is be positive then the second model is preferred.

Example loo

model1a <-stan_lmer(vot*1000 ~ consonant2 + (1+consonant2|speaker_id),
data=data_vot,
prior_intercept = normal(0, 5),
prior = normal(0, 5), 
prior_covariance = decov(regularization = 2),
chains = 4,
iter = 2000, adapt_delta=0.999,warmup=1000)

loo1 <- loo(model1) 
loo1a <- loo(model1a)

Compare models

compare_models(loo1,loo1a)

## elpd_diff        se 
##       0.4       1.1

Pros and cons of Bayesian analyses

• Pros
• Easier to intrpret
• No p-values
• Uses past knowledge (more scientific)
• Can use more complex models
• Can do multiple comparisons
• Have distributions of results
• Can make more nuanced claims
• Cons
• Slower
• New for linguistics so have to explain more

Resources

• https://cran.rstudio.com/web/packages/rstanarm/vignettes/lm.html
• https://cran.rstudio.com/web/packages/rstanarm/vignettes/continuous.html
• https://arbital.com/p/bayes_rule/?l=693
• http://www.kevinboone.net/bayes.html