So, this whole heated debate started when I criticized using GAR to draw conclusions about hockey players.
Let’s use this metric that rates Mathieu Perreault (11g, 38pts) ahead of Scheifele (29g, 77pts) to show why Foligno and Saad are so good. https://t.co/lFJdXD90Sy
— David Johnson (@hockeyanalysis) March 30, 2017
What has gotten lost in this whole debate and group attacks from the Hockey Graphs writers is that I think GAR is a terrible way to evaluate players. This was my original critique and my primary critique. We simply shouldn’t be drawing any conclusions about players using GAR. It is just too flawed.
In my tweet above I pointed out the Scheifele/Perreault problem which some people correctly point out we can’t throw away a stat because it got one player wrong. So last night I threw out a number of incorrect comparisons which I’ll summarize here. Presented will be the players rank among forwards, the forwards name, and their GAR value.
Patrick Kane trailing Jakob Silfverberg. Does that make sense?
Bozak and Lewis just ahead of Ovechkin and Carter? Really?
William Karlsson among these talented guys? These talented guys ranked this low?
Nylander, Grabner, Drouin and Galchenyuk all rate as 3rd line players? Nylander has more goals than Jesper Fast has points (and almost 3 times). Grabner has 27 goals, Fast has 21 points.
Stepan, Stastny, Duchene, Monahan and Rackell stuffed in between Pittsburgh’s 40 year old 3rd line center and Scott Wilson who apparently is a frequent linemate of Matt Cullen. Who knew? Wilson and Cullen have combined for 19 goals and 52 points while Monahan has 27 goals and 58 points. Monahan faces some of the toughest competition among Flames forwards, Cullen/Wilson among the weakest QoC among Penguins forwards. GAR considers these players more or less equal. Do you agree?
Yes, Phil Kessel and his 22 goals, 66 points are less valuable than teammates Matt Cullen and Scott Wilson according to GAR.
Your Leaf rookies:
Yeah, Zach Hyman better than Nylander and Brown who rank as 3rd line players. William Nylander, 22 goals, 59 points, 202nd most valuable forward in the NHL.
I don’t think you need to know all the details of how GAR is calculated to know that GAR is a significantly flawed metric. I mean, I guess it is possible that the hockey world is completely out to lunch and Scott Wilson is in fact more valuable to the Penguins than Phil Kessel and Ovechkin and Carter are the 79th and 80th best forwards in the league just behind Trevor Lewis and Tyler Bozak. Sure, maybe this is possible but it is going to take far more evidence than GAR to convince me. Extraordinary claims require extraordinary evidence.
This isn’t to criticize @DTMAboutHeart work or the theory behind GAR which may very well be valid. The problem arises because hockey is an incredibly complex sport with limited (and sometimes of poor quality) data to work with. I simply do not believe that it is possible to come up with a single metric to evaluate players with the publicly available data we have. It doesn’t matter how good the math is, I don’t think it is possible. Garbage in, garbage out.
Citing GAR and then claiming the rest of the NHL may not have figured out how good Foligno and Saad are is not helpful at all. It’s bad analytics and I believe it makes hockey analytics look bad.
How do I Evaluate Players?
@hockeyanalysis Serious question: which metrics and/or combo would you suggest?
— Daniel Rodriguez (@hidaniel) April 2, 2017
This is a good question. My answer is all of them. For forwards I would look at 5v5 goals/60, assists/60, and points/60. I would look at IGP, IAP and IPP to get a sense of how involved the player is in his teams offense when he is on the ice. I would look at how much ice time the player gets and whether he is given ice time on the powerplay or penalty kill or against quality opponents (OppGF60 is my QoC metric of choice but I also use OppGF% and OppCF%) because I believe generally coaches have a good feel for who their best players are. I will look at GF60 Rel, GA60 Rel and GF% Rel typically over multiple years. I will look at CF60Rel, CA60Rel, and CF%Rel as well as the Sh%Rel and Sv%Rel to get a deeper sense of what the underlying numbers are suggesting. I also spend a lot of time looking at WOWYs and WOWY charts. Again, for all these stats I’ll look at multiple years to look at longer-term trends. The stronger the trend I see the more confidence I’ll have in the conclusions I draw. I’ll look at how the player ranks on his team and in the league. I’ll look at whether a players results diverge from how they are being used. I have even looked at shot distance and shot locations charts to get a sense of whether the player is a guy who drives the net or whether they are more of a perimeter player. In team building and line makeup this is important as you want a nice mix of players, not all net drivers or all play makers. Regardless of how good WAR/GAR is this information gets lost when we compress everything into a single stat.
Ultimately I want to get a complete picture of the player. I want to get an idea of all the strengths and weaknesses of the player and get to k now what type of player they are. Are they a goal scorer or a play maker or used as a defensive specialist? Are they heavily involved in the offense or might they be contributing to offensive production in indirect ways? Are they defensively responsible a defensive liability?
I have said this a gazillion times but most people who use hockey analytics have way too much confidence in their work and not enough emphasis is put on the limitations that we have to deal. If half of the NHL teams used no analytics and only traditional scouting and the other half of NHL teams used only analytics using publicly available data and no traditional scouting I’d be willing to bet the best teams in the league would be the traditional scouting teams. Hockey Analytics just isn’t good enough or reliable enough yet.
Now, this doesn’t mean hockey analytics is useless. The best run organizations should use both traditional scouting and hockey analytics. A deep statistical analysis of a player can shine a light on some strengths or weaknesses that may have been overlooked. This can force a more in depth evaluation using traditional methods which is never a bad thing. Only when we make use of every tool available to us will we make the best decisions.