Oct 20, 2013; Indianapolis, IN, USA; Indianapolis Colts quarterback Andrew Luck (12) throws a pass as he is hit by Denver Broncos defensive tackle Terrance Knighton (94) during the first half in the game at Lucas Oil Stadium. Mandatory Credit: Thomas J. Russo-USA TODAY Sports
Let’s play a game.
Who’s the better all-time quarterback? Dan Marino, Brett Favre or Steve Young? How about Johnny Unitas, Joe Montana or John Elway? Good luck reaching a consensus. A quick google search reveals that respected analysts from around the country and across the globe have engaged in this exercise, each of them reaching different conclusions in some form or fashion, often dramatically so. What do you weight more? Super Bowls or MVP awards? Wins or quarterback rating? Signature wins or signature losses? Everybody will have a slightly different answer to these questions and with good reason, it’s a subjective exercise with subjective methodologies and as such, invariably subjective results.
Of course that’s just one position of 22 on a field, and that figure doesn’t include kickoff and punt coverage teams, punt returners, kick returner, field goal snappers, holders and kickers. All of whom are important cogs in the winning or losing of an NFL game.
The NFL is often referred to by players as “the ultimate team game” and yet those same players often, upon retiring into life as an analyst, fall into the same rhetorical traps that so many fans do, pitting this player against that player, comparing stats side-by-side as if somehow equivalent statistical totals represent equivalent ability.
This, we know, is an illusion. It’s a mirage created by sports culture which necessitates instant debate and firm, unequivocal conclusions based on limited evidence. But the NFL isn’t like other sports, and I’m going to tell you why.
Small Sample Sizes
82, 162, 82, 16
No, this isn’t the start of a LOST joke (at least not a good one, assuming such a thing exists), those are the number of games in the regular season of the four major American professional sports leagues, NHL, MLB, NBA, and NFL respectively. I don’t think I need to point out that one is not like the others.
Sample size is a popular buzz word in NFL analysis. I think for many though it just kind of goes over their head, a term that makes sense as two words put together but the significance of which is largely misunderstood. In statistics, a small sample size is another way of saying that the data does not include enough information to properly draw meaningful inferences from it. Yet in the NFL this is often used as a sort of disclaimer, a footnote, “It’s a small sample size BUUUTTTT, here’s why RGIII should be benched.”
Think about the absurdity of Bulls fans and media suggesting Derrick Rose should be benched two games into the NBA season because he’s looked rusty coming off an ACL recovery, yet this is exactly what happens in the NFL.
Yes, when you only play 16 games in a season every one of them takes on added importance, but that’s not the same as saying that the outcome of every game is also somehow more significant as a representation of future performance. Nobody is losing their heads after the Miami Heat lost to a far inferior Sixers team, they simply accept it as one game in many, but in the NFL? Good luck getting a levelheaded analysis if Denver lost a game to Jacksonville (hell, Denver was roundly questioned after failing to cover a 28 point spread in that game).
Consider the 2006 NFL season in which the Colts lost an embarrassing game to Jacksonville in week 14, 44-17, giving up nearly 400 rushing yards in the process despite the opposing QB posting a QBR of 8.7. Following that game Bill Polian famously told Jim Irsay, “It’s over.” A month and a half later they hoisted the Lombardi Trophy as Super Bowl champions.
Was this some crazy fluke? Absolutely not. The NFL is filled with countless examples of heavily favored teams losing in the playoffs, it’s the nature of a single elimination format and a sample size too small to make reliable predictions; fantastic drama, but the “best” team rarely wins the championship.
If you want to know what an MLB season might look like if it was only 16 games long, here you go.
The Complexity of the Sport
The NFL Rulebook is a mind-numbing 121 pages long, and we’re not talking brochure pages with 16 point font, we’re talking 121 8x11 PDF pages with font so small you need a magnifying glass.
But aside from the rules, the game itself is staggeringly complex. I was once asked to write a “brief” explanation of the NFL game for someone who knew nothing about football, the final manuscript was 3,000 words and I didn’t even touch on scheme, formations, personnel packages and any number of other things.
Let’s say the outcome of a play is an incomplete pass. Your friend, who has never seen football before, asks you, “Hey (insert your name), what just happened?” The simple answer of course is, “The quarterback tried to throw the ball to his receiver but it was incomplete.” Great. But we know that this response barely scratches the surface of what may or may not have happened to cause that relatively pedestrian incompletion.
Let’s ignore all the preparation and game planning that go on during the week leading up to a game, which could, and does, fill volumes, here’s just a small sample of things that happen on just about every play in the NFL.
Pre-snap: The offensive coordinator has to establish based on down and distance, position on the field, current situation in the game, defensive tendencies, and his own player personnel (i.e. injuries during a game, unexpected protection issues, etc.) what play to call, and he has about 5 seconds to make that decision. The defensive coordinator is doing the same. This is the chess match of football.
Assuming the play is relayed correctly, and all the players in the huddle understand said call (something that is far from a given in a thunderously loud stadium), the quarterback then reads the defense’s response to the called formation. It then becomes his responsibility to ensure that he makes the correct read based on defensive personnel and formation. If the defense is in a nickel coverage the QB may need to audible into a run, for example.
The center is also making reads, calling out possible protection shifts, identifying the mike, and making adjustments based on information that may or may not be a feint by the defense. Wide receivers are responding to coverage as well, often times having several route options on any given play based on reads of the defense. If the QB, center, and WRs aren’t on the same page disaster can often ensue, a wrong route or incorrect read a potential game changing pick-6.
For their part, the defense is also reading the offense. It’s a never ending contest of strategy, perception and skill.
During the play: All that is just the pre-snap read. Now comes the fun part. Assuming all goes according to plan, now you have to execute it. A perfect play call is meaningless if the execution is sloppy, in fact the Colts spent years running the same concepts over and over and just did so with such precision execution that it was still effective.
Certainly, different positions on the field have more of an impact than others. The quarterback, for example, can overcome a lot of mistakes made by other players on a play, but when analyzing outcome it’s impossible to ignore the importance of a team’s collective success or failure.
We know this play resulted in an incomplete pass (remember our hypothetical football ignorant friend?), but why that pass was incomplete could be attributed to many factors, often ones known only to the coach and players. Did the quarterback and center correctly identify the protections? Did the offensive line and running backs execute those protections? Did the QB throw the ball in the proper location based on the correct pre-snap read of the defense? Did the WR run the correct route? Perhaps the defender just made an excellent play, perhaps the pass was just slightly off target, or maybe the receiver just dropped the ball.
The possibilities here are nearly endless, but the point in all this is that any given play in any given game does not exist in a vacuum. Certainly this is true of other sports as well, but in football this fact is amplified tenfold. A single mistake by any player on the field can domino into a failed play, which in turn can domino into a failed series, which can domino into a lost game, and with 16 games to a season every one is precious. Then again, sometimes a failed play is just a failed play.
All plays are not created equal and not all outcomes should be weighted as such, but that is the conundrum of NFL analysis. Saying, “We just can’t know for sure” simply isn’t good enough for fans demanding answers and expecting solutions.
In the wake of this complexity, many advanced statistical techniques have cropped up in an attempt to quantify performance in a sport with nearly infinite variability. Websites like Football Outsiders, Advanced NFL Stats, ESPN Stats and Information, Cold Hard Football Facts, Pro Football Focus, and of course Pro Football Reference, to name but a few, have taken to measuring the most minute details of every NFL snap in hopes of determining exactly what leads to winning and how it can be predicted in the future.
These tools are invaluable in getting to the heart of what really happens on an NFL football field, without them the task seems almost futile, like taking shots in the dark.
But as useful as these tools undeniably are, none of them are truly capable of cutting through the veil of complexity that shrouds the NFL game, and even when analyzed collectively can sometimes create more confusion than clarity. Trying to parse out the roles of coach vs. player in a given play is hard enough, parsing out player vs. other player is even more difficult.
Take Andrew Luck for example. Depending on which metric you use to measure overall QB performance he falls anywhere from 4th to 11th, and that doesn’t even include traditional metrics like passer rating. It’s also unclear, given the proprietary nature of many of these metrics, exactly how these determinations are reached or what factors are ignored. Does a QB with a porous offensive line get some kind of bonus for his performance? Does an INT that bounces of a WR’s hands weight equally to one that was thrown directly to the defense? Sometimes the answer appears to be yes, sometimes no. Then of course there are things that stats cannot possibly account for, such as an incorrect route by a WR resulting in an anticipated throw that’s intercepted.
Aside from the unknown nature of many of these measurements there’s also the sheer volume. Here’s a quick, and incomplete, list of advanced WR metrics:
Catch rate, drop rate, route depth, yards per route run, estimated points added, yards after catch, yards after catch per reception, target percentage, wide receiver rating, deep pass target percentage, deep pass catch rate, yards above replacement, defense-adjusted yards above replacement, defense-adjusted value over average, effective yards, win probability added, estimated points added, success rate.
That’s just WR and there are many I left off, good luck making a list for every QB metric out there.
Arguments for or against a player can almost certainly be made by picking and choosing whichever metrics fit your particular point of view, a practice that is so prevalent among fans and analysts that it’s just expected at this point.
And these are just individual metrics, team metrics that try to quantify entire team performance have also proliferated prodigiously in recent years. While certainly better than win-loss record alone, the NFL game is just too complicated and unpredictable to accept any one number as representing anything other than a best guess based on a small sample.
The purpose of this article was not to argue that all NFL analysis and discussion is a futile exercise in intellectual self-aggrandizing (though admittedly that’s what a lot of it is), but rather to illustrate, in some small way, the caution one must take in drawing conclusions from data with so many inherent flaws and stemming from such incredible complexity.
Further evidence could certainly be proffered, such as the surprisingly high failure rate of draft picks, or the fluctuation in player performance from one season to the next, but I’m willing to bet at this point you’ve heard enough.
The NFL game is stacked with drama, fascinating in its strategic diversity, and as much fun to watch as anything on TV, but as a member of a fan community and a writer myself (however humble) my fervent hope is that the future of NFL analysis will be marked by measured criticism and not reactionary pulp. We can all have fun debating who the best QB of all-time is or which team is going to win the Super Bowl, but it’s worth remembering that room for merited dissension is not only an essential element of human discourse but a necessary and warranted aspect of any NFL discussion.
You can follow me on Twitter here -
Very timely that this brilliant post came around the same time that "60 Minutes" ran their report on Nick Saban.
Saban's aproach to coaching seems to stem from exactly what this post is highlighting.
You know, reading this and the article on the NFLization of other leagues, I thought about that proposal to add 2 more games to the schedule. And while I think it's a bad idea for the game as it is, I'd really like to see a lot more football, but in a way that doesn't chew up bodies the way the NFL does. Maybe the point is to make the game less violent so the season can be longer. You know those guys who complain it's turning into flag football? Well, I'm not sanctimonious here—my favorite player to watch used to be Steve Atwater (mainly because Ronnie Lott was a Niner and my brother liked them so I had to hate them)—but maybe there's something there. Restructure the game somehow to make injuries less common and extended recovery time a bit less necessary. I don't know what that would mean. But Rugby's still a pretty hard-assed sport without the spectacular violence, and it has fewer stops in the action as well. Maybe the answer is reducing the padding, instead of increasingly legalistic and fine divisions of what hits are acceptable and when, which nobody really seems to understand (I mean, extremes like James Harrison and Brandon Merriweather are easy to see, but that's in the nature of extremes). Of course, if this resulted in moar Dan Dierdorf, I'd have to give up the game altogether, but it would be fun to watch Mark Schlereth's head explode and see if that actually managed to make him less articulate and coherent. We know it wouldn't affect Merril Hoge.
What makes the game so great is how fiercely it resists "Moneyball" approaches. It's 40 percent luck, 40 percent plan, 20 percent skill, or some other combination thereof.
@DougEngland Thanks man. Glad you enjoyed it. I watched that Saban interview as well, seems like a pretty tough taskmaster but I can see why his teams are so successful.
@MarcusDugan Thank you sir, likely to change just about nothing but at least I got my thoughts on the subject out there, some of them anyway.
@andreaallennyc thanks, yes I did.
@ColtsHead_Ben I might flip the luck and skill in that equation but it's something close to that to be sure.
@smonroe Maybe not "anything" exactly, probably better summed up, "Nobody knows as much as they think they do or would have us believe that they do." That's not quite as pithy though, so probably wouldn't work as a Family Guy skit.
@smonroe Sadly the result of "nobody knows anything" is a proliferation of clueless talking heads and faith that empty phrases or concepts like "body blows" and "controlling the clock" have meaning.
@codrutc Well thank you, how kind of you to say so.