Where Does Our Arsenal Grade Miss the Most?

If you’ve been following our arsenal scores series, you’ve seen that we projected the SwStr% on each pitcher’s best fastball, breaking ball and changeup, of guys who threw 40 innings pitched in a season. We then took those projections to project a player’s xFIP in the current and following year. Variations of the model also included BB% and pitch type usage. When assessing model accuracy, I used a train/test split. I then trained the model and projected on all pitchers. Our next articles will show what predictors the models found valuable, but this one will be looking specifically at the biggest misses and why those misses may have occurred.

              First off I wanted to see where our model missed the most. This graph shows that the biggest predictions clearly come on the ends of the xFIP spectrum.

When creating arbitrary FIP groups from 2.50 to 5.50 by every 0.5, we see a similar trend.

We see that on the extremes, the average prediction of the models comes in a lot lower or higher than it should, and it does a better job predicting the middle group. This makes sense, as the model is regressing back extreme performances and therefore not going to predict them. So, if we want to get an interesting idea of the biggest misses, we can zoom in on that group of xFIPs from 3.5 to 5.0, specifically 4.0 to 4.5, to see where the model missed the most on guys it usually got right.

Biggest 3.50 – 4.00 xFIP Misses

             Biggest 4.00 - 4.50 xFIP Misses

 Biggest 4.50 – 5.00 xFIP Misses



From  3.50 to 4.00 some of the names seem to be relievers who broke out, but others appear to be players who may have added a pitch, changed their pitch mitch or added velocity. Lucas Giolito, who we see at the top of this list added almost two miles per hour to his four seamer from 2018 to 2019 (92.4 to 94.2 mph) and completely did away with his sinker while throwing his changeup more and Delgado added 1 mph to his fastball. Some of the misses in the other group appear to be big changes in usage or just general regression from players. The group of 4.50 to 5.00 players consists of guys who declined after a breakout year. Kluber threw only 35 innings in his following year, but his walk rate ballooned in those innings. After that we see guys who either declined in velocity, changed usage or maybe had an extra pitch the model didn’t account for.

              More in depth work can be done going forward to see exactly why the model missed on these players, but it is helpful to try and look at how people may surpass or fall below their projections. Looking at variable importance for each model in the next couple articles will help us to get a better idea of this as well.