When Do We Know if a Goalie is Good?
About halfway through the 2013-2014 fantasy hockey season, a LWL regular posed the following question to our staff:
At what point into a goalie's career do we know how good he is?
The question highlights a fundamental issue in the game and has wide-ranging implications for fantasy hockey. I knew immediately that this was a question I wanted to spend quality time answering.
One of the key problems with goaltender evaluation is the small sample size. And by small sample size (at this point in the discussion) I'm not referring to how many shots the goalie has faced at this point in his career - because, in fact, that's what the original question is about after all. By small sample size, I mean how many goalies, for example, have followed the same career trajectory as Sergei Bobrovsky? He posted the following save percentages in his first three seasons: .915, .899, and .932. If I were interested in digging up all NHL goalies who have followed a similar trajectory, how many do you think I would find? No matter the number, it wouldn't be sufficient to build an analysis upon.
To overcome this lack of data, I decided to approach this problem using simulated data. I would programmatically create a league-average goalie (SV% = .9138) and fire a lot of pucks at him. If you consider that a typical starting goalie plays about 60 games and faces about 30 shots per game, then a 10-year career would yield about 18,000 shots against. So, I fired 18,000 shots at my league-average goalie. But if I analyzed only one goalie, I'd be opening myself up to small sample size problems all over again. Instead, I simulated 1000 careers of league-average goalies with each goalie facing 18,000 shots. In a 30-team league, you'd need about 250 seasons of data to match that number of shots.
Below, I've plotted the results of the 1000 simulated goalie careers. It's worth mentioning again that all of these goalies are average. That is, they are capable of stopping the puck 91.38% of the time.
What you see above is 1000 career trajectories for average NHL goalies. The SV% along the vertical axis is cumulative, i.e., it tells you the career SV% of a goalie based on how many shots he has faced up to that point in his career. For convenience, I've placed tick marks along the horizontal axis to represent a single season (using the assumption that a goalie plays 60 games and faces 30 shots in each game - neither of which are in any way critical to the outcome of the simuations).
So, what is the answer to the original question that prompted this article? I'd focus on the part of the graph where the spread of the data becomes relatively constant. Without overcomplicating things, nobody would call you crazy for suggesting that you probably want at least 3000 shots of data before you start making bold claims about a goalie's talent. Anything shy of that and you might find yourself saying future Vezina winners don't belong in the NHL.
A Few More Conclusions
Consider the 300 shots against part of the graph. Imagine a vertical line running through the data at that mark. The simulation suggests that an average goalie is capable of posting almost any imaginable SV% over the course of 300 shots. An average goalie (over a 10-game stretch) can look like a rock star or a complete dud. Consider Frederik Andersen of the Anaheim Ducks; we don't know his talent level very well because he's only faced 783 shots in the NHL. But a lot of fantasy hockey managers already have a great impression of him. In his first 300 shots faced, he posted a .932 SV%. Does this mean Andersen is a good goalie? An average goalie? We don't know. We do know that an average goalie is certainly capable of posting a .932 over 10 games - and even a .923 after nearly 800 shots could still fall within the realm of possibility for an average goalie. If you think these results scream "UNCERTAINTY" then you are beginning to grasp the point of this article.
Another way to look at the data is the following: find the data point with these coordinates (4410, .903) on the chart. If you look closely, you'll see about 1-2 goalies in that spot? 1-2 out of a thousand is about 0.1% - 0.2%. So, if you have a goalie who posted a .903 over that many shots (4410), you can say with confidence that this goalie is definitely not average (instead, he's below average for sure). Those are Martin Brodeur's numbers over the past four seasons.
There are a lot of interesting takeaways buried in the simulation results. We've also run simulations for above-average goalies and below-average goalies and there is a wealth of fantasy hockey information to be explored. We'll be posting these results and our analysis of them in the 2014-2015 fantasy hockey draft kit. If you have any interesting interpretations or questions regarding the simulation, drop us a note in the comments!