Not bad for a summer job, but are baseball’s ‘advanced defensive metrics’ accurate?

Twins’ pinch runner Byron Buxton stole second base as Royals’ Hunter Dozier (17) couldn’t make the play in the ninth inning. The Kansas City Royals lost to the Minnesota Twins 7-6 at Kauffman Stadium Wednesday, Apr. 3, 2019.
Twins’ pinch runner Byron Buxton stole second base as Royals’ Hunter Dozier (17) couldn’t make the play in the ninth inning. The Kansas City Royals lost to the Minnesota Twins 7-6 at Kauffman Stadium Wednesday, Apr. 3, 2019.

Apparently a whole bunch of people have math anxiety: the feeling that they are incapable of understanding or doing math.

In my case it’s more like math laziness; I figure I could do it if the fate of the free world depended on it, but I really don’t want to.

Start explaining advanced baseball metrics with formulas that look like a recipe for rocket fuel and the minds of people who don’t enjoy math tend to wander.

And many of those same people — once again, I’m among them — tend to skip the fine print when agreeing to something that might include forfeiting our first-born male child to a group of devil worshipers.

Our math anxiety, laziness and lack of attention to detail makes us susceptible to loan officers, used car salesmen and the people who come up with advanced baseball metrics.

Read the fine print

If you can fight off your natural inclination to avoid anything resembling homework and dig into the details, you’ll find that the raw data for defensive metrics like Ultimate Zone Rating, Defensive Runs Saved and the Plus/Minus System is collected by a company called Baseball Info Solutions.

BIS also publishes the Fielding Bible, and here’s what the Fielding Bible has to say about that raw data:

“Baseball Info Solutions reviews videotape of every game in Major League Baseball. Every play is entered into the computer where we record the exact direction, distance, speed and type of every batted ball. Direction and distance is done on a computer screen by simply clicking the exact location of the ball on a replica of the field shown on the screen. Speed is simply soft, medium and hard while types of batted balls are groundball, liner, fly and bunt.”

Let’s run through that one more time.

BIS claims to record “exact” direction, distance, speed and type of every batted ball, and then, before the paragraph is over, admits someone is actually estimating where the ball landed, how hard the ball was hit and what kind of trajectory the ball exhibited.

You might also notice that the BIS guys are watching video of games and aren’t there in person, which as you’re about to see matters a lot.

Unless there’s a shot looking straight down from the Goodyear blimp, recording the exact location of the ball on a replica of the field would be difficult given the angles TV cameras typically use.

BIS likes to call the people doing the estimating “video scouts,” which sounds impressive until you find out it’s a seasonal job and starts at $8 an hour. Just for comparison’s sake, McDonald’s pays its employees an average of $9.35 an hour and none of them have the capability of ruining a big-league ballplayer’s reputation.

Now head over to FanGraphs and see what they have to say about Ultimate Zone Rating, the advanced defensive metric they use, which is once again based on Baseball Info Solutions data:

“We don’t know precisely where a ball is hit, we don’t know exactly how long the ball was airborne or on the ground before it lands, is touched, or passes a fielder, and we don’t know exactly where the fielders were positioned when the ball was hit.”

Because the BIS guys are watching video and games on TV rarely show defensive alignment before a ball is hit, they don’t know where a fielder was positioned at the beginning of the play.

So metrics that claim to measure a player’s defensive ability actually have little or no information about how far a player had to run to get to a batted ball.

So how are they calculating range?

According to the Fielding Bible there are “about” 260 “vectors” dividing the field. The velocity of the ball (estimated) and the ball’s trajectory (once again estimated) are then mixed into the soup.

Here’s an example of how it works:

Figure out the league average for converting a “soft ground ball” hit into a particular vector into an out. Then, if a player makes that play more often than average, he gets plus points; if he makes that play less often than average, he gets minus points, expressed in whatever format a particular metric uses.

So what if the player was positioned somewhere in Outer Mongolia and was nowhere near the vector the ball was hit to and had zero chance of making a play that would have been routine if he hadn’t been in a shift?

That’s a problem.

According to FanGraphs, if the “stringer” — aka the $8-an-hour video scout — draws on all his experience and thinks a shift affected a play, the play is ignored.

But unless the TV angle reveals where the player was positioned before the ball was hit and the stringer caught it and scored it correctly, the player is going to get screwed. And more and more shifts are affecting more and more plays.

And that’s not the only problem with these defensive metrics.

Park factors

Continue burrowing down into the FanGraphs fine print and there are some vague allusions to figuring in the speed of the infield and a promise that park factors are updated every time there is a “material” change or a team moves into a new park.

Which is pretty unlikely.

The speed of the infield changes every time the grass is mowed and every time it rains. If it gets wet, an infield that was fast when the game started could be slow by the end of a game; on a hot day, a slow infield that dries out could be fast by the end of a game.

But admit that and all the numbers are questionable, so try not to think about it too much and let’s move on to the disclaimers.

Don’t hold us to it

Advanced metrics usually express their conclusions in very specific, scientific-sounding numbers.

Look up Alex Gordon’s FanGraphs Ultimate Zone Rating in runs above average as a left fielder and you’ll see 91.0, which gives the impression that these numbers are so accurate they need to be expressed with a decimal point.

Then read these excerpts from what FanGraphs has to say about Ultimate Zone Rating:

“Just because UZR or any other defensive metric “says” that someone is X, even if that X is based on many years of data, does not make it so. When you are dealing with sample data, as we almost always are with every metric in baseball that we encounter, there is a certain chance that the metric is going to be ‘wrong.’

“The reason for that is that the data is imperfect.

“We don’t know exactly where each fielder was stationed, we certainly don’t know the exact location of the batted ball to the nearest square inch on the field, and we definitely don’t know how long the ball was in the air or on the ground. In reality, it might have been an easy ball to catch or it might have been a difficult one to catch, or somewhere in between.

“There is no guarantee that our UZR number reflects what the player actually did or his true defensive talent.”

If that strikes you as an awful lot of stuff not to know for a metric that supposedly measures a player’s defense, join the club.

Slog through the fine print of advanced baseball metrics and you often find a passage that says, Yeah, these numbers might not be accurate, but it’s the best information we’ve got, so let’s go with it.

Which is pretty much the same thought process that led to the belief that the Earth was flat, the sun orbited around it and a good way to identify a witch was to tie her up and throw her in the nearest lake to see if she sank.

“Sure, we might be ‘wrong’ and drown a few people, but it’s the best system we’ve got.”

Here’s the problem with that: When we accept and use these numbers without question we give them validity, and players are now being judged by metrics that even the metrics’ advocates admit may not be accurate.

I’d trust the opinion of a coach or scout who has watched thousands and thousands of baseball games and tens of thousands of players far more than the opinion of a “video scout” watching a game on TV, making $8 an hour, doing a summer job and feeding what may or may not be accurate data into a flawed system.

So why do we continue using these numbers?

Because we want to have opinions without going to the trouble of developing expertise.

Developing expertise not only requires watching thousands of baseball games, it also requires you to actually pay attention and know what to look for ... and most of us can’t be bothered.

It’s much easier to look up a number on a website and sound knowledgeable, even if the number we regurgitate is inaccurate.

So after looking into how defensive metrics are put together I wouldn’t trust them as far as I could throw the Royals’ team bus, but I’m guessing that won’t change a thing.

Advanced defensive metrics might not be accurate, but they sure are convenient and that’s usually enough to keep people using them.

And if you didn’t like what I had to say about advanced defensive metrics, just wait until we get to Wins Above Replacement.

Related stories from Kansas City Star