While watching some insanely frigid football last weekend, one Deadspinner wondered if there were fewer flags in cold NFL games, the result of referees deciding, consciously or not, to speed games up and get the hell off the field. Without scrutinizing hundreds of hours of game tape we can't really tell if the refs or players are to blame, but we can ask a general question: Are there fewer penalties in colder games?
Cutting to the chase, the answer is yes! If reading through science-esque methodological explanations isn't exactly your cup of tea, feel free to skip below to the "Results" section, seven paragraphs and two charts below. (I might ask why you're reading Regressing in the first place, though). For everyone else, here's how we got to this answer.
Mike Kania of ProFootballReference was kind enough to provide us with data on the temperature, number of penalties, date, and venue type (outdoors, dome, etc.) of 9,774 games from 1970 to 2013. We only looked at the regular season, as Greg Bedard of MMQB and Chase Stewart of FootballPerpective.com have both found that penalty calls generally drop in the postseason. We also controlled for tempo (fewer snaps in cold weather would mean fewer opportunities for penalties), using an estimate of plays per game to calculate "penalties per 150 plays".* The 7,613 outdoor games in our sample saw an average of 147 plays per game, so this is effectively a tempo-adjusted approximation of penalties per game.
The chart below shows the average penalty rate for outdoor games since 1970, by temperature:
This seems straightforward: As temperature drops, penalties drop. This continues down to 29°, at which point the penalty rate stays roughly stable for a bit. Games with temperatures of 29° or below saw just 11.0 penalties per 150 plays, while outdoor games with a temperature above 29° saw 12.7 penalties per 150 plays—a 1.7 penalty gap!
But cold games tend to take place towards the end of the year. What if the effect we're seeing is just that fewer penalties are called as the regular season progresses? Sure enough, if we look at the 1,672 dome games in the sample (as a control group for temperature), the penalty rate drops at a small but significant rate as the season moves forward (r = -0.12, p < 0.05):
So how can we tell if there's a cold weather effect after all? What we did—and thanks to Michael Lopez of statsbylopez.com for his help here—is to stratify the games in our sample by when each was played, and then to look within each strata for a cold-weather effect.
The methodology: Returning to that 29° cutoff, we broke the time range in which these games generally occurred (Oct. 31st-Jan. 3rd) into four "strata." Each was just 16-17 days long, so we could effectively ignore the "time in season" variable within each strata, and see if there was a difference between the penalty rates of the <=29° and >29° groups anyways. There was, so we then calculated the total difference between the two groups, weighted by each strata's sample size (and pooled the weighted standard error to determine if this difference was significant; you can take a look at our math here).
Outdoor games that were 29° or below saw an average of 1.1 fewer penalties per 150 plays than games above 29° (p = 0.04). This is smaller than the drop we saw before we controlled for "time in season" (1.7 penalties per 150 plays for 29°), but it's still a meaningful figure—about one penalty less in a typical game. Chase Stewart's research shows that the postseason typically sees a couple fewer penalties per game than the regular season; it seems like a good chunk of this could just be a temperature effect.
We wanted to see if penalties continued to drop at lower temperatures, so we performed the same analysis using a cutoff of 23.5°, which represents the coldest 2.5 percent of games. While the penalty gap was larger (-1.8 fewer penalties), there is only margin evidence that it's significant (p = 0.07). The temperature chart at the top certainly implies that this cold effect strengthens at very cold temperatures, but the small sample size of these games means that we can't say for sure.
None of the match-ups this weekend are expected to drop to this 29° threshold, although games in Denver and Foxborough could always surprise you. The AFC Championship could be a cold one though, not to mention this year's New York Super Bowl. Could the temperature, and the flag or so it ends up preventing, end up making all the difference? Quite possibly so.
*PFR doesn't have the total play number for every game going back to 1970, so we used pass attempts + rush attempts + pass sacked + punts + kickoffs + field goals as an approximation.