Going for 2

In 2008, I worked at the Philadelphia Eagles as a statistician in the front office. I talk a lot about that job despite being there briefly, mostly because more people can understand what I worked on (it's much less of a party topic to say that I work on tooling for data pipelines). When people ask a bit more deeply about it, I say I created their two-point chart (which was a project I worked on). I want to share a bit about this work to better explain how analytics can impact an organization.

What is a 2 point chart?

In 1958, college football implemented a 2 point try. After scoring a touchdown, a team is entitled to a conversion try for additional points. There is a choice between an extra point kick (also know as a PAT, point after touchdown) or a two-point conversion (a play that attempts to advance the ball past the goal line). Typically, the try is more likely converted as a kick but has a lower reward. In 1994, the NFL adopted the rule, enabling teams to go for 1 or 2.

Famously, in the early 1970's (now NFL HoF coach) Dick Vermeil, then-offensive coordinator of the UCLA Bruins, created a 2 point chart. It is likely that other college coaches had a guide (either mental or written), but this chart (see below) had historically been most used as a tool for decision making.

The organization I worked at, the Eagles, ironically had been a subsequent stop in Vermeil's legendary coaching career. My story is that I (as well as many other sports researchers) was able to improve upon Vermeil's chart leveraging my experience with math and statistics.

The insight

In the subsequent 30 years, more data became available and more attention was given to optimizing the strategy for the 2 point conversion. In a crude way, Vermeil's chart is great, though possibly flawed with new rules and the way the game had evolved. So, like many data problems, it requires revisiting as the context changes.

Vermeil's chart gives a binary decision, which could be richened and changed depending on the quality of the offense, the quality of the defense, the time remaining in the game. These sounds obvious, but they create a problem -- they make the decision more complex. You could imagine Vermeil's chart with thousands of dimensions about the teams and game context as providing an improvement, but is that worth the complexity introduced?

In blackjack, basic strategy indicates to hit a 16 against a dealer showing a 7+. However, if you are most of the way through the shoe and there are a lot of high cards remaining (TC >1), the optimal decision changes mathematically. Similar to blackjack, understanding the composition of the remaining decks, or in this case, understanding the remaining time is key and can meaningfully alter decisions.

As an easy thought experiment, a team down 8 (that then scores) with seconds remaining in the game should almost always go for 2, whereas that same scenario in the first quarter might depend on the quality of the teams. Similarly, a team that has a terrible kicker and an amazing offense should at least on the margin, be more inclined to try for 2 when the decision is a close one.

So the insight, is not a remarkable one, but its' that time remaining and (beliefs about the) probability of success matter.

The math

I'll mostly glaze over this part because it's the least interesting (and it's not the point of the story) and because Benjamin Morris (FiveThirtyEight) and other authors have done it so well... https://fivethirtyeight.com/features/when-to-go-for-2-for-real/

Ultimately, you want to take the action that has the greatest expected win value. The way to understand it, is what's the probability for success in the kick * the likelihood of winning given kick success + the probability for failure in the kick * the likelihood of winning given kick failure and compare that vs. the same in the two-point try.

One of the most interesting cases (that I've always been a proponent of) is going for 2 after scoring a touchdown having been down 14 (always an "according to the analytics" play). The difficulty in that thought experiment is that there is an informational advantage that impacts the chance of winning and losing in the event of failure or success (because the coach declares the subsequent decisions after the result of the first event).

Unsurprisingly, there are edge cases to consider -- injury risk, the return of a kick or try by the defense, player morale, momentum, and probably many others I've never thought about that might have meaningful or negligible impact. And more importantly, it is likely that the coach considers the impact to the staff's job security in terms of taking an unconventional approach (even if it is in pursuit of winning).

No need for dynamic programming (though that can help be clear in the approach), it's a set of assumptions that produce a conclusion given the beliefs about conversion.

The analysis implementation and presentation

As great as it is to build the chart, there's something quite unfulfilling about an analysis that never makes it to the field (or the business operator). Almost everyone in their analyst life has thought, if only I could decide what to do here, we would be better off. And in a narrow sense it is true -- there are many in a vacuum decisions that would produce meaningfully different expected wins for a football team. It still frustrates me that teams would spend tens of millions on player improvements to find these expected wins, but "football stubbornly clings to the notion that experience always trumps analysis... [in] basic tactical decisions" that could provide similar value.

If a tree falls in the woods...

Analysis for analysis' sake...

So how could I be a more effective analyst?

According to Benjamin Morris, "I see the value in being cautious about adopting new strategies and generally think the burden should be on the purveyors of new techniques and tactics not only to find rigorous, workable proposals, but also to explain them compellingly." So first, it requires a compelling explanation. I'd argue, it needs to be accessible both ways, in compelling form and in rigorous math (and the ability to toggle between both).

Perhaps translate the math to wins forgone or to salary cap dollars needed to deliver those wins or to player upgrades for those dollars. It's very hard to understand expected value but it's easier to imagine the difference between Devin Singletary and Saquon Barkley (~$7M). The decision to forgo a two-point try when trailing by 4 in the 4th is according to the models the equivalent to losing a player like Christian McCaffrey. The result needs to be presented in visceral terms not just statistics.

The result needs to have the work accessible and checkable -- a black box solution often yields an organization to default to caution. It should also be consumable. There is a beautiful output of many dimensions (assumed success probability, score difference, and time remaining) that the author creates graphically.

In 2008, I simply presented the indifference probability point in the 4th quarter. A classic example is for a 100% kick probability (in 2008, PAT attempts were from the 2 yard line) and 50% OT probability, the decision when down 8 is do you believe that there's a >38% chance of 2-point try success.

A set of percentages was not that easily consumed. I understood the meaning, but Vermeil's chart was more accessible. It's hard to think in multiple dimensions, and it's even harder to communicate it that way.

The outcome

Much like Vermeil's chart, mine was incomplete -- partially for the sake of simplicity and partially because I could not solve all times (a combination of lack of data and lack of strategy). But also because that was all I could communicate effectively. While my chart made it to pockets of the organization, it probably was not used on game days. And for certain, it was not relied on for game days.

The most important skill in analytics is getting it implemented as a non-owner of the metric. Andy Reid owned wins, not the data department. While insightful, clever, and meaningful (perhaps), the analysis was ineffective. While statistically sound, I did not have the organizational clout and trust, and knowing that I did not make it sufficiently consumable or convincing. A failure that I'm proud of.

Epilogue

On January 18th, 2009, the Eagles faced the Cardinals in the NFC Championship. With less than a minute to go in the 3rd quarter and the Eagles trailing by 11, Donovan McNabb threw a touchdown to Brent Celek. The Eagles elected to go for 1 (contrary to the chart) and the outcome of the game was meaningfully impacted (see the 538 graph below). While it's more hindsight given the outcome of the plays, I still think about that decision (as it could have meant a Superbowl).

At the very least it inspires me to think about the ways to make analysis impactful, not just rigorous.