Showing posts with label Empirical Bayes. Show all posts
Showing posts with label Empirical Bayes. Show all posts

04 December, 2015

Correcting Parametric Empirical Bayesian Intervals using a Bootstrap

In a previous post I discussed empirical Bayes for the beta-binomial model. Empirical Bayesian estimates are just expected values of a posterior distribution - suppose instead that you want interval estimates. The empirical Bayesian method can be used, but the intervals potentially have to be adjusted. In this post I want to show how to construct empirical Bayesian intervals for the beta-binomial model, correct them using a parametric bootstrap, and give a baseball example.

Annotated code for the procedure I will describe in this post can be found on my github.

As a side note, I just want to say how much I love this procedure. It uses parametric bootstrapping to correct Bayesian intervals to achieve frequentist coverage. God bless America.


Empirical Bayesian Intervals



In my previous post on beta-binomial empirical Bayes analysis, I used the model

$y_i \sim Bin(n_i, \theta_i)$
$\theta_i \sim Beta(\alpha, \beta)$

The empirical Bayes method says to get estimates $\hat{\alpha}$ and $\hat{\beta}$ of the prior parameters - I give a method of moments estimator in the post, but marginal maximum likelihood may also be used - and then calculate posterior distributions for each of the $\theta_i$ using $Beta(\hat{\alpha}, \hat{\beta})$ as the prior, essentially using the data itself to estimate the prior.

The empirical Bayesian estimate is the mean of this posterior distribution. If an interval estimate is desired, a credible interval can be calculated by taking quantiles directly from this posterior - see my post on credible intervals. This is what I will call a "naive" empirical Bayesian interval.


The Empirical Bayes Problem



The fundamental problem with naive empirical Bayesian intervals is that they often end up too short, inappropriately centered, or both. This is because the uncertainty of the prior parameters themselves has not been accounted for. From the law of total variance, the posterior variance is given by

$Var(\theta_i | y_i) = E_{\alpha, \beta|y_i}[Var(\theta_i|y_i, \alpha, \beta)] + Var_{\alpha, \beta|y_i}[E(\theta_i|y_i, \alpha, \beta)]$

Taking quantiles from the empirical Bayesian posterior estimates the first term, but not the second. For small samples this second term can be significant, and empirical Bayesian generally won't achieve nominal coverage (for more information, see Carlin and Louis's book Bayesian Methods for Data Analysis)

One way to correct for the uncertainty is to perform a hierarchical Bayesian analysis, but it's not clear what the "correct" hyperpriors should be - and just using noninformative priors doesn't guarantee that you'll get nominal frequentist coverage.

An alternative is to use the bootstrap. Since I'm working in the parametric empirical Bayes case, a parametric bootstrap will be used, though this doesn't necessarily have to be the case. For more information on the technique I will use (and a discussion on how it applies to the normal-normal case), see Laird, N., and Louis, T. (1987), "Empirical Bayes Con fidence Intervals Based on Bootstrap Samples," Journal of the American Statistical Association, 82(399), 739-750.

I want to emphasize that this is a technique for small samples. For even moderate samples, the uncertainty in the parameters will be small enough that naive empirical Bayesian intervals will have good frequentist properties, and this can be checked by simulation if you desire.


Parametric Bootstrap



The idea of the parametric bootstrap is that we can account for $Var_{\alpha, \beta|y_i}[E(\theta_i|y_i, \alpha, \beta)]$ by resampling. In a traditional bootstrap, data is sampled with replacement from the original data set. Since this is a parametric bootstrap instead, we will resample by generating observations from the model assuming that our estimates $\hat{\alpha}$ and $\hat{\beta}$ are correct.

  1. Generate $\theta^*_1, \theta^*_2, ..., \theta^*_k$ from $Beta(\hat{\alpha}, \hat{\beta})$
  2. Generate $y^*_1, y^*_2,..., y^*_k$ from $Bin(n_i, \theta^*_i)$
  3. Estimate $\alpha$ and $\beta$ from the bootstrapped $(y^*_i, n_i)$ observations using the same method as you initially used. Call these estimates $\alpha^*_j, \beta^*_j$


In that way, we get a set of $N$ bootstrap estimates $\alpha^*, \beta^*$ of the parameters of the underlying beta distribution. The posterior density that accounts for uncertainty of $\hat{\alpha}$ and $\hat{\beta}$ can then be estimated as

$p^*(\theta_i | y_i, \hat{\alpha}, \hat{\beta}) \approx \dfrac{ \sum_{j = 1}^N p(\theta_i | y_i, \alpha^*_j, \beta^*_j)}{N}$

Essentially, just the raw average density at each point, averaging over all the bootstrapped parameters values. The corrected 95% empirical Bayesian interval is given by solving

$\displaystyle \int_{-\infty}^{l} p^*(\theta_i | y_i, \hat{\alpha}, \hat{\beta}) = 0.025$

$\displaystyle \int_{u}^{\infty} p^*(\theta_i | y_i, \hat{\alpha}, \hat{\beta}) = 0.975$


For lower and upper bounds $l$ and $u$ using numerical techniques.


Baseball Example



In my previous post, I analyzed the famous Morris baseball data set that with respect to loss functions to show why empirical Bayes works. Analyzing it with respect to interval estimation also provides an interesting example.

Using a beta-binomial model with the method of moments estimator I described in the previous post, this data set has parameters $\hat{\alpha} = 97.676$ and $\hat{\beta} = 270.312$.  Each player had $n_i = 45$ at-bats, so the posterior distribution for the batting average $\theta_i$ of player $i$ is

$\theta_i | y_i, \hat{\alpha}, \hat{\beta} \sim Beta(y_i + 97.676, 45 - y_i + 270.312)$

Naive intervals can be taken directly as central 95% quantiles from the posterior distribution - again, see my article on Bayesian credible intervals for more explanation on this.

\begin{array}{l c c c c c c c c} \hline
\textrm{Player} & y_i & y_i/n_i &  \textrm{EB Estimate} & \textrm{Naive Lower} &  \theta_i & \textrm{Naive Upper}\\ \hline
Clemente & 18 & .400  & .280 & 0.238 & .346 &  0.324 \\
F. Robinson & 17 & .378   & .278 & 0.236   & .298 &  0.322 \\
F. Howard & 16 & .356 &  .275 & 0.233   & .276 &  0.319 \\
Johnstone & 15 & .333  &  .273 & 0.231   & .222 & 0.317  \\
Barry & 14 & .311 & .270 &  0.229    & .273 & 0.314  \\
Spencer & 14 & .311  & .270 & 0.229  & .270 & 0.314 \\
Kessinger & 13 & .289  & .268 & 0.226    & .263 &  0.312 \\
L. Alvarado & 12 & .267 & .266 & 0.224    & .210 &  0.309 \\
Santo & 11 & .244 &  .263 &  0.222  & .269 &  0.307\\
Swoboda & 11 & .244 &  .263 & 0.222   & .230 & 0.307  \\
Unser & 10 &.222 & .261 &  0.220  & .264 & 0.304  \\
Williams & 10 & .222 & .261 & 0.220  & .256 & 0.304 \\
Scott & 10 & .222 & .261 & 0.220    & .303 & 0.304 \\
Petrocelli & 10 & .222  & .261 & 0.220   & .264 & 0.304  \\
E. Rodriguez & 10 & .222  & .261 &  0.220   & .226 &  0.304\\
Campaneris & 9 & .200 & .258 &  0.217   & .285 & 0.302 \\
Munson & 8 & .178 & .256 & 0.215   & .316 & 0.299  \\
Alvis & 7 & .156 & .253 &  0.213    & .200 & 0.296 \\ \hline
\end{array}

Thirteen out of the eighteen intervals captured the hitter's true average for the rest of the year, for an observed coverage of 72.22%. Roberto Clemente and Thurman Munson managed to overperform with respect to their intervals, while Jay Johnstone, Luis Alvarado, and Max Alvis underperformed.

The parametric bootstrap procedure fixes this as follows:

  1. Simulate a set of 18 new $\theta^*_i$ from a $Beta(97.676, 270.312)$ distribution.
  2. Simulate a set of 18 new $y^*_i$ from a $Bin(45, \theta^*_i)$ distribution
  3. Estimate $\alpha^*$ and $\beta^*$ using the same method used on the original data set.
I repeated this for 5000 bootstrap samples. The bootstrapped posterior is then


$p^*(\theta_i | y_i, 97.676, 270.312) \approx \dfrac{ \sum_{j = 1}^{5000} p(\theta_i | y_i, \alpha^*_j, \beta^*_j)}{5000}$

The effect of this is to create a posterior distribution that's centered around the same empirical Bayesian estimate $\hat{\theta_i}$, but more spread out. This is shown in the naive (solid line) and bootstrapped (dashed line) posterior distributions for $y_i = 10$.


Quantiles taken from this bootstrapped distribution will give wider intervals than the naive empirical Bayesian intervals (though it is possible to come up with "odd" data sets where the bootstrap interval is shorter). The bootstrap interval is given by solving


$\displaystyle \int_{-\infty}^{l} p^*(\theta_i | y_i, 97.676, 270.312) = 0.025$

$\displaystyle \int_{u}^{\infty} p^*(\theta_i | y_i, 97.676, 270.312) = 0.975$


For lower and upper bounds $l$ and $u$ - and in full disclosure, I didn't actually perform the full numerical integration. Instead, I averaged over the pbeta(x, alpha, beta) function in R and solved for the value where the averaged CDF is equal to 0.025 or 0.975.

Doing this,  95% bootstrapped intervals are given by

\begin{array}{l c c c c c c c c} \hline
\textrm{Player} & y_i & y_i/n_i &  \textrm{EB Estimate} & \textrm{Bootstrap Lower} &  \theta_i & \textrm{Bootstrap Upper}\\ \hline
Clemente & 18 & .400  & .280 &  0.231  & .346 &  0.391 \\
F. Robinson & 17 & .378   & .278 & 0.227   & .298 &  0.382 \\
F. Howard & 16 & .356 &  .275 & 0.222   & .276 &  0.372 \\
Johnstone & 15 & .333  &  .273 & 0.217   & .222 & 0.363  \\
Barry & 14 & .311 & .270 &  0.211    & .273 & 0.355  \\
Spencer & 14 & .311  & .270 & 0.211  & .270 & 0.355 \\
Kessinger & 13 & .289  & .268 & 0.205    & .263 &  0.346 \\
L. Alvarado & 12 & .267 & .266 & 0.199    & .210 &  0.338 \\
Santo & 11 & .244 &  .263 &  0.192  & .269 &  0.330\\
Swoboda & 11 & .244 &  .263 & 0.192   & .230 & 0.330  \\
Unser & 10 &.222 & .261 &  0.184  & .264 & 0.323 \\
Williams & 10 & .222 & .261 & 0.184  & .256 & 0.323 \\
Scott & 10 & .222 & .261 & 0.184  & .303 & 0.323 \\
Petrocelli & 10 & .222  & .261 & 0.184   & .264 & 0.323  \\
E. Rodriguez & 10 & .222  & .261 &  0.184   & .226 &  0.323\\
Campaneris & 9 & .200 & .258 &  0.177   & .285 & 0.317 \\
Munson & 8 & .178 & .256 & 0.169  & .316 & 0.310  \\
Alvis & 7 & .156 & .253 &  0.161  & .200 & 0.305 \\ \hline
\end{array}

Only one player out of eighteen is not captured by the interval - Thurman Munson, who managed to hit 0.178 in is his first 45 at-bats and 0.316 the rest of the season -  for an observed coverage of 94.44%.

(As a reminder, annotated code to perform estimation in the beta-binomial model and calculate bootstrapped empirical Bayesian intervals for the beta-binomial model is available on my github)

24 July, 2015

Normal-Normal Shrinkage Estimation by Empirical Bayes

Shrinkage estimation is a very common technique in baseball statistics. So is the normal model. It turns out that one of the one way to shrink is to assume that both the data you see and the distribution of means of the data you see are normal - and then estimate the distribution of means prior by use of the data itself.

 The (non-annotated) code I used to generate these results may be found on my github.

The Normal-Normal Model


The basic normal-normal model assumes two "stages" of data - the first is the observed data, which we will call $Y_i$ - this is assumed to be normally distributed with mean $\theta_i$ and variance $\sigma^2$.

$Y_i \sim N(\theta_i, \sigma^2)$
$\theta_i \sim N(\mu, \tau^2)$

Bayes' rule says that if you assume the "second" level - the distribution of the $\theta_i$ - is also normal, and treat it as a prior, the posterior distribution of $\theta_i$ (that is, the distribution in belief of values of $\theta_i$ after taking into account the data $y_i$ - see my previous post on Bayesian inference) is also normal with distribution

$\theta_i | y_i, \mu, \sigma^2 \sim N(B\mu + (1-B)y_i, (1-B)\sigma^2)$

where
$B = \left(\dfrac{\sigma^2}{\sigma^2 + \tau^2}\right)$

The quantity $B$ gives the amount that the mean of the data shrinks towards the mean of the prior. If $\sigma^2$ is large compared to $\tau^2$ (and so the data has a large amount of variance relative to the prior) then the mean of the data gets shrunk towards the prior mean by quite a bit. If $\sigma^2$ is small compared to $\tau^2$ (and so the data has a small amount of variance relative to the prior) then the mean of the data doesn't get shrunk much at all - it tends to stay near the observed $y_i$.

No matter what the prior is, this Bayesian estimator can be thought of as a shrinkage estimator. It's just a question of what you're shrinking to, and by how much. A Bayesian, if he or she wishes to be "noninformative" (and I'm going to completely ignore all the controversy of naming and choosing priors) might pick something like $\theta_i \sim N(0,1000)$ as a prior, so $B$ is very close to zero and the shrinkage is very small.

Empirical Bayes


What we're going to do, however, is focus on using the data to choose the prior distribution, by assuming the normal-normal model as described above and estimating the parameters of the prior from the data itself - this is known as empirical Bayes. The effect  of using the data to choose the prior is that the data is shrunk towards the mean of the data, in an amount determined by the variance(s) of the data.

How do we estimate $\mu$ and $\sigma^2$? One nice property of the normal-normal model is that the marginal distribution of $y_i$ is also normal:

$y_i | \sigma^2, \tau^2, \mu \sim N( \mu, \sigma^2 + \tau^2 )$

There are three quantities that need to be estimated to perform this - $\mu$, $\sigma^2$, and $\tau^2$. The formula above gives us two - the first, the population mean, can be estimated by the sample mean $\hat{\mu} = \bar{y}$. The variance is a bit trickier. If I just take the standard variance estimator of the $y_i$

$Var(Y) = \dfrac{\sum (y_i - \bar{y})^2}{N-1}$

then that gives an estimate $\hat{\sigma^2 + \tau^2}$ (the hat is over the entire thing - it's not estimating two individual variances and summing them, it's estimating the sum of two individual variances), assuming $\sigma^2$ is the same for every observation. So we're going to need to get some information from somewhere about what $\sigma^2$  is. If the $y_i$ are sums or averages of observations, we can use that. If not, we have to get creative.


Baseball Example


Let's say $Y_i$ is the distribution of a player's batting average in $n_i$ at-bats (the $Y_i$ are already divided by $n_i$, so they are averaged), and the player has true batting average $\theta_i$. We know that a true 0.300 hitter isn't going to always hit 0.300 - he will hit sometimes above, sometimes below. The model says that the player's observed batting average follows a normal distribution with true batting average $\theta_i$ and variance $\sigma^2/n_i$ (since it is an average) - this is the "first" normal distribution as described above.

The second stage is the underlying distribution of the $\theta_i$ - that is, the distribution of all players' true batting averages. It is normal with mean $\mu$ (the true mean league average) and variance $\tau^2$. So using this two-stage model is equivalent to saying that if I selected a random, unknown batter from the pool of all major league baseball players and observed his batting average $y_i$, it will follow a normal distribution with mean $\mu$ (the true league mean batting average) and variance $\sigma^2/n_i + \tau^2$ (the sum of the variation due naturally to a player's luck and the variation in batting averages between all players). I understand that this is a bit weird to think about, but imagine trying to describe the distribution of a player's observed batting average when we don't even know his name - you have to figure his true average is somewhere between 0.200 and 0.330, with an mean of around 0.265, and then on top of that add in the average amount of natural variation around his true average (sometimes above, sometimes below) at $n_i$ at-bats. That's what's going on here.

In baseball terms, $\sigma^2/n_i$ can be thought of as the "within-player" variance or "luck" and $\tau^2$ can be thought of as the "between-player" variance or "talent." If a batter is very consistent in hitting near his true ability and the distribution of all batting averages is very spread out, then not much shrinkage will occur. Conversely, if a batter has a lot of variation in his observed batting average but there's not much apparent variation in the distribution of all batting averages, then the player will be shrunk heavily towards the league mean.

We need three estimates in order to do the procedure - an estimate of each of $\mu$, $\sigma^2/n_i$, and $\tau^2$. The first, $\mu$, is the mean batting average of the league - and since $y_i$ is the batting average for player i, the estimate of the league mean batting average is the obvious one - $\hat{\mu} = \bar{y}$.

In the case that $n_i$ is the same for all of your observations, taking $Var(y_i)$ gives us $\hat{\sigma^2/n_i + \tau^2}$. If there are differing $n_i$, the estimation method gets more complex - it's worth its own post at some point to work through it. I'm going to assume that all the players have the same number of at-bats to keep things simple for this example, though it admittedly does make it feel rather artificial.

Now we need an estimate of $\sigma^2/n_i$ - the "within-player" variance. But the normal distribution doesn't tell us what $\sigma^2/n_i$ should be. It's pretty typical to model a player's hits in $n_i$ at-bats as following a binomial distribution with true batting average $\theta_i$. Then the sample batting average (hits over at-bats) has variance

$Var(\textrm{Sample Batting Average}) = \dfrac{\theta_i(1-\theta_i)}{n_i}$

The binomial distribution is just a sum of independent, identical Bernoulli trials (at-bats, in this case) each with probability of getting a hit $\theta$. So the central limit theorem says that for large $n_i$, we can approximate the distribution of the sample batting average with a normal!


$\textrm{Sample Batting Average} \sim N\left(\theta_i, \dfrac{\theta_i(1-\theta_i)}{n_i}\right)$


A value for $\theta_i$ is needed. It feels natural to use $y_i$ - the player's batting average - in the estimation. This is wrong, however - remember, we don't know how the player's true talent level! We need to shrink by the average variance amount. The average variance amount is estimated by the variance at the league batting average - which is a quantity we also have an estimate for. The estimate of the within-player variance is then

$\dfrac{\hat{\sigma^2}}{n_i} \approx \dfrac{\bar{y}(1-\bar{y})}{n_i}$

 Then the empirical Bayes estimator is given by

$\hat{\theta_i} = \hat{B} \bar{y} + (1-\hat{B})y_i$

where

 $\hat{B} = \dfrac{\bar{y}(1-\bar{y})/n_i}{\sum (y_i - \bar{y})^2/(N-1)}$

Comparison


Using the famous Morris data set to compare these estimators (I'll call them $\hat{\theta}^{NN}$ for normal-normal) to the shrinkage estimators from the Beta-Binomial (see post here - I'll call these $\hat{\theta}^{BB}$) and James-Stein (see post here - and note that the James-Stein estimator is just a specific version of the normal-normal estimator - I'll call them $\hat{\theta}^{JS}$), we see that it performs well.

\begin{array}{l c c c c c} \hline
\textrm{Player} & y_i & \hat{\theta}^{NN} & \hat{\theta}^{BB} & \hat{\theta}^{JS} & \theta \\ \hline
Clemente        &0.400             &0.280             &0.280            &0.290 &0.346 \\
F. Robinson        &0.378             &0.277             &0.278            &0.286 &0.298 \\
F. Howard        &0.356             &0.275             &0.275            &0.282 &0.276 \\
Johnstone        &0.333             &0.273            &0.273            &0.277 &0.222\\
Barry        &0.311             &0.270             &0.270            &0.273 &0.273\\
Spencer        &0.311             &0.270             &0.270            &0.273 &0.270\\
Kessinger        &0.289             &0.268             &0.268            &0.268 &0.263\\
L.Alvarado        &0.267             &0.266             &0.266            &0.264 &0.210\\
Santo        &0.244             &0.263             &0.263            &0.259 &0.269\\
Swoboda       &0.244             &0.263             &0.263            &0.259 &0.230\\
Unser       &0.222             &0.261             &0.261            &0.254 &0.264\\
Williams       &0.222             &0.261             &0.261            &0.254 &0.256\\
Scott      &0.222             &0.261             &0.261            &0.254 &0.303\\
Petrocelli &0.222             &0.261             &0.261            &0.254 &0.264\\
E. Rodriguez &0.222             &0.261             &0.261            &0.254 &0.226\\
Campaneris       &0.200             &0.258             &0.258            &0.249 &0.285\\
Munson &0.178             &0.256             &0.256            &0.244 &0.316\\
Alvis       &0.156             &0.254             &0.253            &0.239 &0.200\\ \hline
\end{array}

For this data set, the normal-normal and beta-binomial estimates are almost identical. This shouldn't be a surprise - both the distribution of batting average talent and variation around batting averages is roughly bell-shaped and symmetric, so the normal-normal model and the beta-binomial models are both flexible enough to take that shape.

The normal-normal and beta-binomial estimators shrinks the most while the James-Stein shrinks a moderate amount. For this specific data set, the James-Stein estimator seems to hold a slight-advantage - not by much, though.

  • $\sum (\hat{\theta}^{NN}_i - \theta_i)^2 = 0.0218$
  • $\sum (\hat{\theta}^{BB}_i - \theta_i)^2 = 0.0218$
  • $\sum (\hat{\theta}^{JS}_i - \theta_i)^2 = 0.0215$
  • $\sum (y_i - \theta_i)^2 = 0.0753$.

Whatever the method you use to shrink, estimates are produced that, when judged using the squared error loss function, are far superior to using the raw batting averages.

Advantages/Disadvantages

This estimator relies on the assumption of normality for both the data and underlying distribution of means - this means it will work well for batting statistics (which tend to be constructed as sums, weighted or otherwise, of assumed independent, identical events) but not as well for other statistics which don't naturally look "normal." Furthermore, if I have to estimate $\sigma^2$ with a binomial variance -why don't I just use a beta-binomial model? That doesn't depend on a large number of at-bats for normality of the distribution of the sample batting average. Overall, I think it will give nice results when used appropriately, but in many situations a different model will fit more naturally to the data.

21 May, 2015

Beta-Binomial Empirical Bayes

When you have a set of random variables - say, batting averages of a group of players - building a model that allows for "borrowing" of information from can greatly improve estimation. In the case of batting averages, say a player has a batting average of 0.400 over 45 at-bats. Any baseball fan knows that's unsustainably high - but the raw prediction doesn't "know." All the raw prediction knows is that a player went to bat 45 times and got 18 hits - so the best prediction of that player's future is 0.400. But if it also can somehow know that 0.400 is the highest batting average in a group of players, then that information can be used - possibly, to predict that some amount of regression towards the mean is due.

A basic statistical model that involves a group of similar probability distributions can often be improved by adding something that allows the model from one player to access the information from another player. One technique common in statistical modeling is to add a distribution "underneath" (or on top of, depending on how you visualize it) the individual probability distributions for each player. This is known as a "mixing" distribution, and it represents the probability distribution of the "true" parameters for each player.



In the batting averages example, it represents the distribution of batting averages. For example, you could use the mixing distribution to answer to questions such as "If I randomly select a batter, what is the probability that he has a true batting average greater than 0.250?" or "If I randomly select a batter, what is the probability that I see 5 hits in 30 at bats?" - essentially, sampling from the pool of all batters rather than looking at an individual one.

The Beta-Binomial Model


A very common statistical model of this form is called the beta-binomial model (I'm going to assume you're familiar with both the beta and binomial distributions individually). The basic formulation is

$y_i \sim Bin(\theta_i, n_i)$
$\theta_i \sim Beta(\alpha, \beta)$

This model assumes player $i$ has a "true" batting average represented by constant proportion $\theta_i$, where $0 \le \theta_i \le 1$. Of course, we don't actually know that each $\theta_i$ exists, any more than we know that it follows a binomial distribution or know that there's a beta distribution underneath it. But what this model does allow is the sharing of information - in this case, the beta distribution is the mixing distribution - which is very, very useful to us.

If we knew the values of the parameters of the beta distribution $\alpha$ and $\beta$, we could use Bayes' rule to get an estimate of the $\theta_i$

$\hat{\theta_i} =\dfrac{y_i + \alpha}{n_i + \alpha + \beta}$

If we don't know and don't want to be full Bayesians, however, another option is to use empirical Bayes - that is, to replace $\alpha$ and $\beta$ with estimates $\hat{\alpha}$ and $\hat{\beta}$ (these can be moment estimates, maximum likelihood estimates - any kind of estimates). Then our estimates of the $\theta_i$ become

$\hat{\theta_i} = \dfrac{y_i + \hat{\alpha}}{n_i + \hat{\alpha } + \hat{\beta}}$

The effect of this is that the raw estimates $y_i/n_i$ are pulled towards the group mean in an amount that is determined by two things - the variance between the raw batting averages (if the raw batting averages are very close together, that will increase the shrinkage) and the number of at-bats. Raw predictions that are further away from the mean are pulled harder - a player batting 0.400 or 0.100 is going to have his predicted batting average reduced or improved drastically, but a player batting 0.250 is not going to see much of a change. Furthermore, no matter what $\hat{\alpha}$ and $\hat{\beta}$ are, their effect will go to zero (and hence, shrinkage will lessen) as the number of at-bats $n_i$ gets larger.


Estimating the Parameters



The two most common procedures used for estimation of the parameters in the beta-binomial model are the method of moments and the method of maximum likelihood. I'm going to briefly describe an iterative method of moments estimators, which is useful if all you need are a point estimate and don't necessarily care about the uncertainty of estimation for the parameters. The disadvantage is that moment estimators tend to be less efficient than likelihood estimators, and for small sample sizes can produce negative estimates of $\alpha$ and $\beta$.

Code in R for this procedure can be found on my github.

As previously stated, the two-stage model for beta-binomial empirical Bayes is given by

$y_i \sim Binomial(n_i, \theta_i)$
$\theta_i \sim Beta(\alpha, \beta)$


For the purpose of estimation, I'm going to rewrite the parameters of the beta distribution as

$\mu = \dfrac{\alpha}{\alpha + \beta}$
$\phi = \dfrac{1}{\alpha + \beta + 1}$

This makes $\mu$ equal to the expected value of the beta distribution with $\phi$ as the dispersion parameter.

Begin by defining


$\tilde{\theta_i} = \dfrac{y_i}{n_i}$


These are the raw, un-shrunk estimates of the $\theta_i$. Estimate the mean as


$\hat{\mu} = \dfrac{\displaystyle \sum w_i \tilde{\theta_i}}{w}$

where $w_i$ are weights and $w = \sum w_i$. Initial values of weights must be chosen - using $w_i = n_i$ is a good choice.

Then define


$S = \displaystyle \sum w_i (\hat{\theta_i} - \hat{\mu})^2$


The dispersion parameter $\phi$ of the beta distribution is estimated as


$\hat{\phi} = \dfrac{\hat{\mu}(1-\hat{\mu})\left[\displaystyle \sum \dfrac{w_i}{n_i} \left(1 - \dfrac{w_i}{w}\right)\right]}{\hat{\mu}(1-\hat{\mu})\left[\displaystyle \sum w_i \left(1 - \dfrac{w_i}{w}\right) -  \displaystyle \sum \dfrac{w_i}{n_i} \left(1 - \dfrac{w_i}{w}\right)\right]}$

The weights $w_i$ are then re-estimated as

$w_i = \dfrac{n_i}{1 + \hat{\phi}(n_i - 1)}$

And the process is repeated (starting at calculating $\hat{\mu}$) until the sum of squared differences between the old and new weights is less than a certain tolerance. The iterated moments estimator produces estimates tend to be closer in value to the estimates produced by maximum likelihood than the naive estimator found on Wikipedia, but both moments estimators will give the same estimates when all sample sizes are equal.

To convert these estimates back to the traditional $\alpha, \beta$ format, apply the fomulas:

$\hat{\alpha} = \hat{\mu} \left(\dfrac{1-\hat{\phi}}{\hat{\phi}}\right)$

$\hat{\beta} = (1- \hat{\mu}) \left(\dfrac{1-\hat{\phi}}{\hat{\phi}}\right)$

 And empirical Bayesian estimates of the $\hat{\theta_i}$ are given by

$\hat{\theta_i} =\dfrac{y_i + \hat{\alpha}}{n_i + \hat{\alpha} + \hat{\beta}}$

Caution: It's important here not to try to use the raw $y_i / n_i$ as observations from the beta distribution in order to estimate $\alpha$ and $\beta$ directly - they're estimates of the $\theta_i$, and you'd be ignoring the uncertainty in those estimates, which could lead to poor estimation of the parameters of the underlying beta distribution when the $n_i$ are small or the noise around the $\theta_i$ is large, such as for batting average on balls in play.


Batting Averages


Recall the batting-averages data used in the James-Stein estimator post. That data required a messy transformation to make it fit the equal variance, normality assumption . The beta-binomial method does not require either. If we say that $y_i$ is the number of hits player $i$ gets in $n_i$ at-bats, then moment estimates of the beta parameters are given by $\hat{\mu} = 0.265$ and $\hat{M} = 367.988$. Also define $\theta_i$ as the batting average over the rest of the season

\begin{array}{l c c c c c c c} \hline
\textrm{Player} & y_i & y_i/n_i & \textrm{JS Estimate} & \textrm{BB Estimate} & \theta_i\\ \hline
Clemente & 18 & .400 & .290  & .280 & .346 \\
F. Robinson & 17 & .378  & .286  & .278 & .298 \\
F. Howard & 16 & .356 &  .282 &  .275 & .276 \\
Johnstone & 15 & .333  &  .277 &  .273 & .222\\
Barry & 14 & .311 &  .273 & .270 & .273\\
Spencer & 14 & .311 &  .273 & .270 & .270\\
Kessinger & 13 & .289 & .268  & .268 & .263\\
L. Alvarado & 12 & .267 &  .264 & .266 & .210\\
Santo & 11 & .244 &  .259 &  .263 & .269\\
Swoboda & 11 & .244 & .259  &  .263 & .230\\
Unser & 10 &.222  &  .254 & .261 & .264\\
Williams & 10 & .222 & .254  & .261 & .256 \\
Scott & 10 & .222 &  .254 & .261 &  .303 \\
Petrocelli & 10 & .222 & .254  & .261 & .264\\
E. Rodriguez & 10 & .222 &  .254 & .261 & .226\\
Campaneris & 9 & .200 &  .249 & .258 & .285\\
Munson & 8 & .178 &  .244 & .256 & .316\\
Alvis & 7 & .156 &  .239  & .253 & .200\\ \hline
\end{array}


The beta-binomial empirical Bayesian estimates tend to shrink a bit harder towards the mean - sometimes for better, sometimes for worse. Both the James-Stein and empirical Bayesian estimates improve dramatically on the naive estimates for prediction - compare

  •  $\sum (\hat{\theta}^{JS}_i - \theta_i)^2 = 0.0215$
  •  $\sum (\hat{\theta}^{BB}_i - \theta_i)^2 = 0.0218$ 
  •  $\sum (y_i/n_i - \theta_i)^2 = 0.0753$. 

This type of estimation that pulls observations towards a group mean is known as shrinkage estimation. Whether by means of the James-Stein estimator or a beta-binomial empirical Bayesian method, it offers a clear improvement on prediction, when the measurement criterion is squared error.

16 May, 2015

The James-Stein Estimator


If you want to make a prediction, an important question is "How do I judge accuracy?" There can be different ways.

For example, suppose you want to take the batting averages of a group of players in April and use them to predict the final batting average at the end of the regular season. How do you want to judge how close you got? You could take the average of the differences between your predictions and the actual batting averages. You could take the maximum difference between your prediction and the actual batting average. You could even focus on one side and acknowledge that hey, you don't care if a player over performs based on the prediction - you just want to make sure to minimize how much a player can under perform. All of these ways of judging your predictions could potentially lead to different prediction methods.

The first method - summing or average some function of the distances between the predicted and actual values - is both common, and has an estimator that goes along with it that isn't obvious.

The James-Stein Esimator


Let's say you have a set of observations $x_i$ that represent occurrences of some phenomenon, each of which has common parameter $\theta_i$ that controls some underlying aspect of that phenomenon - a baseball example would be to let $\theta_i$ be the "true" batting average of player $i$, if we can model each player as having a true, constant batting average representing his ability.

A common problem is then, how do we estimate $\theta_i$ with some estimator $\hat{\theta}_i$? The answer to that, as discussed previously, depends on how you are going to measure the accuracy of your estimate. A common approach is to use a squared loss function

$L_i(\theta_i, \hat{\theta}_i) = (\hat{\theta}_i - \theta_i)^2$

This presents a pretty good balance between bias and variance, and makes a lot of intuitive sense - squaring the distance makes everything positive so when you sum them up you can figure out a minimum, and it makes being very wrong is a lot worse than being a little wrong.

How then, do you measure the accuracy of the set of estimators? You could just sum the losses.

$L({\theta}, \hat{\theta}) = \displaystyle \sum_i (\hat{\theta}_i - \theta_i)^2$

In 1962, statistician Charles Stein showed a surprising result: first, let $x_i$ follow a normal distribution with known variance and mean $\theta_i$

$x_i \sim N(\theta_i, 1)$

where $i = 1, 2, ..., k$, for $k \ge 4$ (the number of $\theta$s has to be four or bigger - this is a hard rule that can't be broken)*. In the absence of any other assumptions about the form of the data, the simplest estimator $\hat{\theta}_i$ is the $\hat{\theta}_i = x_i$ (in the case of multiple observations per group, since the sample mean will also be normal, so we can without loss of generality assume one observation). This estimator tends to work well, an is, again, pretty intuitive - the baseball example of this would be to just use April's batting average to predict the final batting average.

 However, Stein showed that a better estimator is given by

$\tilde{\theta}_i = \overline{x} + \left(1 - \dfrac{k-3}{\sum (x_i - \overline{x})^2}\right)(x_i - \overline{x})$

The effect of this estimator is to shrink the estimate $\hat{\theta}_i$ towards the overall mean of the data $\overline{x}$, in an amount determined by both the variance of the $\hat{\theta}_i$ and the distance of $\hat{\theta}_i$ from the overall mean. On average, using $\tilde{\theta}_i$ actually works better in terms of the average squared loss defined above  than using the naive estimator $\hat{\theta}_i$. In some cases, it works much better.

Baseball


How do we apply this to strictly to baseball? Professors Bradley Efron and Carl Morris gave a classic example. The batting averages of 18 players through their first 45 at-bats of the 1970 baseball season is given below. The problem is to use the first 45 at-bats to predict the batting average for the remainder of the season.

\begin{array}{l c c} \hline
\textrm{Player} & x_i & \theta_i \\ \hline
Clemente & .400 & .346 \\
F. Robinson & .378 & .298\\
F. Howard & .356 & .276\\
Johnstone & .333 & .222\\
Barry & .311 & .273\\
Spencer & .311 & .270\\
Kessinger & .289 & .263\\
L. Alvarado & .267 & .210\\
Santo & .244 & .269\\
Swoboda & .244 & .230\\
Unser & .222 & .264\\
Williams & .222 & .256\\
Scott & .222 & .303\\
Petrocelli & .222 & .264\\
E. Rodriguez & .222 & .226\\
Campaneris & .200 & .285\\
Munson & .178 & .316\\
Alvis & .156 & .200\\ \hline
\end{array}

Where $x_i$ is the batting avg. for first 45 at-bats and $\theta_i$ is the batting average for the remainder of the season.

The first 45 at-bats were taken because the James-Stein estimator requires equal variance, and the variance is going to be dependent on the number of at-bats you look at. Here, the distribution of this data is clearly not normal - it is binomial. Hence, the transformation $f(x_i) = \sqrt{n}\arcsin(2x_i - 1)$ (in this case, n = 45) is used. Then the transformed data does fit the James-Stein model - it will follow a $N(\theta_i, 1)$ distribution.

The maximum likelihood estimate in each scenario of the remainder batting average would be $\hat{\theta}_i = x_i$. Applying the transformation to both $x_i$ and $\theta_i$, the squared loss using the naive estimates is then $17.56$. However, if the James-Stein estimator is applied to the transformed $x_i$, the squared loss is $5.01$ - an efficiency of $3.5 = \dfrac{17.56}{5.01}$.


\begin{array}{c c c c c} \hline
x_i & \textrm{Transformed }x_i & \textrm{JS transformed }x_i & \textrm{Back-transformed JS Estimates} & \theta_i\\ \hline
        0.400     & -1.351   &    -2.906   &         0.290 & 0.346\\
        0.378     & -1.653   &    -2.969   &         0.286 & 0.298\\
        0.356     & -1.960   &    -3.033   &         0.282 & 0.276\\
        0.333     & -2.284   &    -3.101   &         0.277 & 0.222\\
        0.311     & -2.600   &    -3.167   &         0.273 & 0.273\\
        0.311     & -2.600   &    -3.167   &         0.273 & 0.270\\
        0.289     & -2.922   &    -3.235   &         0.268 & 0.263\\
        0.267     & -3.252   &    -3.304   &         0.264 & 0.210\\
        0.244     & -3.606   &    -3.378   &         0.259 & 0.269\\
        0.244     & -3.606   &    -3.378   &         0.259 & 0.230\\
        0.222     & -3.955   &    -3.451   &         0.254 & 0.264\\
       0.222      &-3.955    &   -3.451    &        0.254 & 0.256\\
       0.222      &-3.955    &   -3.451    &        0.254 & 0.303\\
       0.222      &-3.955    &   -3.451    &        0.254 & 0.264\\
       0.222      &-3.955    &   -3.451    &        0.254 & 0.226\\
      0.200      &-4.317     &  -3.526     &       0.249 & 0.285\\
       0.178     & -4.694    &   -3.605    &        0.244 & 0.316\\
       0.156     & -5.090   &    -3.688    &        0.239 & 0.200\\ \hline
\end{array}

(R code for these calculations may be found in my github)

In fact, the James-Stein estimator dominates the least-squares estimator when the way to measure accuracy is the squared error function - outside of statistics speak, that means that no matter what your actual $\theta$s happen to be, on average, the James-Stein estimator will work better.

So what, then, is a situation where you would not want to use the James-Stein estimator? The answer is if you're concerned about the worst-case scenario for individual players. The James-Stein estimator - and shrinkage estimation in general - sacrifice accuracy on the individual level in order to increase accuracy on the group level. Players like Roberto Clemente, who we know are very good, are going to get shrunk the hardest. If the only player you care about is Robert Clemente, then using the raw April batting results is going to have the best average performance - even though on the group level, the James-Stein estimator will still "win" most of the time.


Shrinkage Estimation


In general, techniques that "shrink" estimates towards the mean will work better for group prediction than looking at each prediction individually. The James-Stein estimator, however, requires very specific assumptions - normality and equality of variances - that may be hard to meet. In future posts, I will discuss some ways to perform shrinkage estimation in other scenarios.

*Stein's estimator originally simply shrunk towards a common value and used $k-2$ in the estimator - however, in situations where the data is shrunk towards the estimated mean, $k-3$ must be used as a correction.