Player Evaluation Model Improvements
On Monday I outlined an all-encompassing player evaluation model that allows us to evaluate every forward, defenseman and goalie under the same methodology. In short, the system compares how many goals are scored for and against while a player is on the ice and compares it to how many goals scored for/against one should expect based on the quality of his line mates and opposition. That model, I believe, makes a reasonable attempt at evaluating a players performance, but it can be improved.
The first method of improvement is to utilize the additional information we have about the quality of a players line mates and opposition once we have run the model. Initially I use the goals for and against performance of his line mates and opposition when the player being evaluated is not on the ice at the same time as his line mates and opposition. But now that we have run the model we, at least theoretically, have a better understanding of the quality of his team mates and opposition. I can then take the output of the first model run and use it as the input of the second model run to get new and better results. I can then continue doing this iteratively and the good news is that after every iteration the difference between the player rating from that iteration and the previous iteration trends towards zero which is a very nice result.
For the most part changes in player ratings are not significant and generally speaking the iterations tighten the results (i.e. the magnitude of the extremes are lessened) which is probably desirable and certainly something we find when we increase the size of the dataset (see below). With the new revised results, the top 10 offensive forwards from last season are Daniel Sedin, Henrik Sedin, Crosby, Semin, Backstrom, Ovechkin, Elias, Antropov, Chris Stewart and Downie. Those last 3 or 4 names might surprise some but the first six probably won’t. That said, the model still probably isn’t adequately separating out the skills of the three Washington players but that is difficult since they mostly all play together. The five worst offensive players in the league last year using this model were Stephane Veilleux, Nate Thompson, Steve Begin, Jeff Halpern, and Maxim Lapierre.
The top defensive forwards were Drew Miller, Kyle Wellwood, Dustin Boyd, Manny Malhotra, Curtis Glencross, Taylor Pyatt, Travis Zajac, Darcy Tucker, Michael Ryder and Nick Foligno. The worst defensive forwards were Malkin, Afinogenov, O’Sullivan, Potulny and Umberger. Overall (average of offensive and defensive ratings) the top forwards were D. Sedin, Ovechkin, Malhotra, H. Sedin, Backstrom, Semin, Gaborik, Feht, Pyatt and Zajac.
For defensemen, the top offensive defensemen were Green, Jeff Shultz, Ehrhoff, Markov, Boyle, Morrisonn, Whitney, Keith, Alberts and Goligoski. The top defensive defensemen were Mark Fistric, Mark Giordano, Keith Yandle, Mike Weaver, Sami Lepisto, Jeff Shultz, Francis Bouillon, Andy Greene, Marc-Edouard Vlasic and Ryan Wilson. The top overall defensemen were Mark Fistric, Jeff Schultz, Christian Ehrhoff, Keith Yandle, Mark Giordano, Mike Green, Andrei Markov, Tom Poti, Marc-Edouard Vlasic and Sami Lepisto.
The top goalies last year were Rask, Varlamov, Emery, Niemi, Halak, Kiprusoff, Bryzgalov, Rinne, Lundqvist, and Brodeur. The bottom 5 goalies were Pascal Leclaire, Ondrej Pavelec, Michael Leighton, Jeff Deslauriers and Alex Auld.
One thing to remember when looking at the above lists is ice time is not a factor in the ratings. I am looking at the performance of the player given the ice time he received. So, while some players may have ranked higher they may not be as valuable as a lesser ranked player who had more ice time.
I have produced ratings for every player for each of the last three seasons (each season computed independently). The bad news is that the correlation of a players ranking from one season to the next is not that high which would seem to indicate the model is failing in some manner. Sure players have good seasons and bad seasons and the level of their play fluctuates from one season to another, but one would expect at least a moderate correlation level. For forwards with at least 900 minutes of ice time in a season the correlation in their 2008-09 offensive rating vs 2009-10 offensive rating is r2=0.22 which is a little disappointing. I believe the main reason behind this is because of the small sample size of using goals against which is why many people have switched to utilizing corsi analysis.
So, the second method I utilized in improving the rating system is to use a 3 season dataset as opposed to a single season. The results are as follows:
|HENRIK SEDIN||1.58||MANNY MALHOTRA||1.31||PAVEL DATSYUK||1.41|
|PAVEL DATSYUK||1.57||TRAVIS ZAJAC||1.28||DANIEL SEDIN||1.31|
|ALEX OVECHKIN||1.57||ZACH PARISE||1.25||HENRIK SEDIN||1.3|
|JASON ARNOTT||1.55||PAVEL DATSYUK||1.24||ALEX OVECHKIN||1.29|
|DANIEL SEDIN||1.45||TRAVIS MOEN||1.23||ZACH PARISE||1.28|
|EVGENI MALKIN||1.45||TAYLOR PYATT||1.21||JOE THORNTON||1.27|
|JOE THORNTON||1.44||JORDAN STAAL||1.2||JASON ARNOTT||1.23|
|DANY HEATLEY||1.44||MIKKO KOIVU||1.18||NICKLAS BACKSTROM||1.21|
|J.P. DUMONT||1.43||OWEN NOLAN||1.18||RYAN GETZLAF||1.2|
|NICKLAS BACKSTROM||1.41||VALTTERI FILPPULA||1.18||DANY HEATLEY||1.19|
|MIKE GREEN||1.35||SEAN O’DONNELL||1.36||NICKLAS LIDSTROM||1.25|
|JEFF SCHULTZ||1.33||NICKLAS LIDSTROM||1.27||JEFF SCHULTZ||1.22|
|NICKLAS LIDSTROM||1.24||BRETT LEBDA||1.24||MIKE GREEN||1.17|
|JAROSLAV SPACEK||1.23||MARK STUART||1.22||BRIAN RAFALSKI||1.14|
|MARK STREIT||1.22||ROB SCUDERI||1.21||ZDENO CHARA||1.14|
|DION PHANEUF||1.22||CHRIS PRONGER||1.2||NIKLAS KRONWALL||1.14|
|DUNCAN KEITH||1.21||MARC-EDOUARD VLASIC||1.2||CHRISTIAN EHRHOFF||1.13|
|STEPHANE ROBIDAS||1.21||DUSTIN BYFUGLIEN||1.2||DUNCAN KEITH||1.13|
|NIKLAS KRONWALL||1.2||BRIAN RAFALSKI||1.19||SEAN O’DONNELL||1.12|
|IAN WHITE||1.2||CHRISTIAN EHRHOFF||1.19||ROB SCUDERI||1.11|
Only players with at minimum 750 minutes of five on five even strength ice time in each of the last 3 seasons are included in the lists above and also remember that this is just even strength performance evaluation. There are a few surprises on those lists but for the most part they seem like reasonable evaluations. Ty Conklin might be the biggest surprise but consider that over the past 3 seasons he has had save percentages of .923, .909 and .921 (for three different teams) and you begin to understand why he made it into the top 10. Jason Arnott among the top 10 forwards overall is a bit of a surprise but Datsuk, both Sedins, Ovechkin, Parise, Thornton, Backstrom, Getzlaf and Heatly being there are probably not. Sidney Crosby did not play 750 even 5v5 strength minutes in 2007-08 so didn’t make the cut but his 3 year rankings were 1.59, 0.87 and 1.23 which would put him right at the top of the offensive rankings and among the top 10 overall though his defensive ranking pulls him back a bit.
And now for the worst players:
|TRAVIS MOEN||0.54||ILYA KOVALCHUK||0.77||ROD BRIND’AMOUR||0.84|
|ROB NIEDERMAYER||0.68||PATRICK O’SULLIVAN||0.78||JAY MCCLEMENT||0.86|
|JAY MCCLEMENT||0.79||VINCENT LECAVALIER||0.8||MICHAL HANDZUS||0.87|
|ALES KOTALIK||0.8||TOMAS FLEISCHMANN||0.82||MARK RECCHI||0.89|
|RADEK DVORAK||0.81||THOMAS VANEK||0.82||CHRIS KELLY||0.89|
|ROD BRIND’AMOUR||0.81||MARTIN ERAT||0.83||ROB NIEDERMAYER||0.89|
|MICHAL HANDZUS||0.82||DAVID BACKES||0.83||TRAVIS MOEN||0.89|
|IAN LAPERRIERE||0.83||EVGENI MALKIN||0.83||PATRICK O’SULLIVAN||0.91|
|MARTIN HANZAL||0.83||RICHARD PARK||0.84||MIKE FISHER||0.92|
|CHRIS DRURY||0.85||MARTIN ST. LOUIS||0.84||ALES KOTALIK||0.92|
|ANDREW FERENCE||0.77||BRAD STUART||0.81||STEVE STAIOS||0.84|
|NICLAS WALLIN||0.79||STEVE STAIOS||0.84||NICLAS WALLIN||0.85|
|ANTON VOLCHENKOV||0.81||MIKE COMMODORE||0.85||MARTIN SKOULA||0.86|
|MARTIN SKOULA||0.82||SHAONE MORRISONN||0.85||BRAD STUART||0.89|
|GREG ZANON||0.84||BARRET JACKMAN||0.87||ANDREW FERENCE||0.89|
|SHANE HNIDY||0.85||DAN HAMHUIS||0.87||SCOTT HANNAN||0.9|
|STEVE STAIOS||0.85||DION PHANEUF||0.87||BARRET JACKMAN||0.91|
|NICK SCHULTZ||0.87||BRUNO GERVAIS||0.88||BRUNO GERVAIS||0.91|
|RON HAINSEY||0.88||RYAN WHITNEY||0.88||MIKE COMMODORE||0.92|
|KIMMO TIMONEN||0.88||PAVEL KUBINA||0.88||SHAONE MORRISONN||0.92|
Interesting to see a lot of solid offensive players in the worst defensive list and vice versa. In fact, all 10 of the worst defensive forwards had offensive ratings above 1.00 (i.e. above average), many significantly above, and seven of the 10 worst offensive forwards had defensive ratings above 1.00 (only McClement, Brind’amour and Handzus did not). It is probably an indication that a players performance offensively or defensively is directly tied to the role he is asked to play on the ice. Guys like Moen and Niedermayer are primarily asked to play a defense first role and thus their offense suffers significantly and when players are tasked primarily with producing offense their defense suffers (or if players are allowed to ignore the defensive aspect of their game they can produce offensively). This result may be one of the big reasons why year to year correlations are not that great. Many players see their roles change from year to year either through promotions up or down lines or though a coaching change that might change they style of play a team employs. Many players probably have the raw skills (speed, puck control, size, strength, etc.) to be successful in any role they are assigned but some might be asked to perform a more offensive role and others are asked to perform a more defensive role. The rarity is the player that can do both at the same time. Pavel Datsyuk is clearly one of those guys and Lidstrom another.