The reluctance of NFL teams to adopt analytic approaches is sometimes attributed to the limitations of statistics in the sport. It's true that there is much more Dark Matter in football than in baseball or basketball: Outside of quarterbacks and other skill-position players, we've barely begun to collect statistics on what the 22 players on the field are doing, let alone measure their worth. Fourth-down strategy, however, is one of the exceptions -- a relatively narrow and well-defined problem. To use a familiar analogy, it's quite hard to understand the morass that is American politics but quite easy to predict the results of upcoming elections by looking at the polls.
My view is that NFL coaches aren't irrational or necessarily ignorant of the statistics as much as they are poorly incentivized to get these decisions right. The average NFL team has been owned by the same family or organization since 1980 -- for the past 34 years. (By contrast, the average MLB and NBA team last changed owners in 1999.) Furthermore, because of the NFL's prodigious popularity and its generous revenue-sharing policies, even losing or incompetent owners possess extraordinarily valuable products. (The Jacksonville Jaguars are worth $840 million, according to Forbes.) This is a culture that fosters extreme risk aversion. Going for it on fourth down is risky twice over: in the micro sense of staking more on the result of one play, and in the macro sense of defying custom and tradition.
It was in baseball, instead -- the closest thing to a free market among the major American sports and one in which poorly run franchises often lose money -- where teams like the A's were forced to innovate. Moneyball was less an ideological commitment than a survival strategy.
Dark Matter discoveries have the potential to level the playing field among the sports, at least in theory. If 90 percent of what happens on a baseball diamond is adequately described by box score statistics but only 40 percent of what happens on a football field is captured, the latter sport has more opportunity to make gains through developments like motion-tracking cameras or more sophisticated study of the interactions among teammates. Moreover, the new areas of study are often much richer than Analytics 1.0. Being able to measure the break on Clayton Kershaw's curveball or the mileage Stephen Curry covers as he sprints up and down the court comes much closer to domains traditionally associated with scouting than statistics.
Human beings, however, are both the impetus for technological innovation and its greatest constraint. European soccer, which is as cutthroat economically as it is on the pitch, might benefit substantially from Analytics 2.0, as might the NHL, in which many franchises struggle to make a consistent profit. Baseball and the NBA should continue to make gains; they are already so competitive that teams might soon view analytics as less of a luxury and more of a necessity. But for the NFL, which has failed for more than 20 years to adopt some of the most basic tenets of analytics, Dark Matter is the equivalent of an iPad to a man who still uses a typewriter.