Incentives: Does Replay Review Make Officiating Worse?

Another weekend, and another National Football League team's fans are upset that the league's review system failed to overturn a call that game officials appeared to have gotten wrong.

This week's incensed fans belong to the Washington Redskins, whose receiver Andre Roberts was ruled to have fumbled on this play.

Washington fans cannot complain that the fumble was the deciding play in the the team's 30–20 loss to the Arizona Cardinals—quarterback Kirk Cousins' subsequent three fourth-quarter interceptions had much greater effect on the game's outcome. Still, Washington fans are wondering why replay didn't overrule the initial call. They're understandably dissatisfied by referee Ed Hochuli's explanation that the replay was inconclusive because it didn't show the ball coming out—a justification that sounds a bit like Kramer explaining why he didn't have to pay off his bet with Jerry.

Supporters of the NFL's replay-review system argue that, though replay fails to overturn all erroneous calls, it does overturn some of them, resulting in fewer erroneous calls overall. If we want to geek out and express this idea mathematically, then

1)   (1 – R)E < E

where E is the number of erroneous calls and R is the percentage of erroneous calls that replay corrects (0 < R < 1).

Looking at equation (1), it seems obvious that replay review has improved the game. But what if E is larger in the replay review era (call it ER) than it was before replay review (EBR)? Then it's possible that replay review is resulting in more incorrect calls, to the detriment of the sport.

Why would ER be larger? A fundamental rule of economics is that people tend to act in their self-interest, as best they understand it. That's true regardless of whether the person is a craftsman who wants to trade his wares for money or a government official who wants to protect his job and improve his work conditions.

Field officials are no different than anyone else: they want to keep their jobs and improve their work conditions. Three things that help them achieve those goals are:

1. limit the number of erroneous calls,
2. make calls that are easily defensible [which is not the same as (1), as I'll explain in a moment], and
3. make calls that please the home team's fans.

At first glance, the desire to avoid erroneous calls should be the overarching incentive, and would benefit the sport. But the existence of replay review makes it seem like it's less important that officials get the call right (because replay can do that); instead, officials need only make calls that are easily defensible.

For instance, a defensive back might make a potentially game-changing, "unbelievable" shoestring interception that the official sees correctly, but because the catch is unbelievable and possibly momentous, the "safe" call for the official would be to rule the pass incomplete and let replay review make the "hard" call that it was a catch. The official has even more reason to make the "safe" call if the defensive back plays for the visiting team; why would an official want to draw the ire of a stadium full of spectators (thereby hurting the official's work conditions) when he can make a "safe" call? This, I suspect, is part of "home cooking" officiating.

To be clear, I don't think game officials stand on the playing field, consciously reasoning through each play to determine what is the most defensible call to make. But I do think the above-described perverse incentives, buttressed by the "safety net" idea that replay review will fix mistakes, help to subconsciously bias officials toward "safe" and "home team" calls. Because of those incentives, I believe ER > EBR.

But, someone might respond, replay review is a safety net, and so it shouldn't matter if ER > EBR; review will clean up the messes. The problem for the NFL is that Section 9 Article 3 of the NFL Rulebook states that replay can only overturn the original call if there is "indisputable visual evidence" that the original call was wrong. (Major League Baseball, the National Hockey League, and college football have similar rules for their replay review systems.) That is a very high standard, as evidenced by all the calls that TV game analysts say video raises doubts about, but there isn't enough evidence to overturn the original call. So, if game officials are relying on replay review to make the "hard" calls for them, then many of the incorrect but "safe" calls will stand.

That then raises the question, are there ultimately more incorrect final calls because of instant replay; that is, (1 – R)ER > EBR? My intution is "Probably not"—but I suspect the two numbers are much closer than people realize, and that many of those calls are on critical, potentially game-changing plays.

I would love to test this idea empirically, but I don't yet have an idea for how to do it. If you have one, I'd love to read about it in the comment section.

P.S.: I should probably add, despite the above concern, I've been pleased with the first season of expanded MLB replay review. Besides correcting some calls, review has greatly reduced the pointless but time-consuming manager–umpire argument. It will be interesting to see if, in future seasons, ER appears to increase significantly over EBR, offsetting the benefit of fewer arguments.