GAR and dealing with Critique in Hockey Analytics

The other day I caused a bit of a stir on twitter when I critiqued this article by Dom Luszczyszyn at The Athletic (paywall required unfortunately). My critique went like this:


And I went on to question whether using such a metric to draw conclusions about how good Foligno and Saad are was wise. Let me quote a couple of things from Luszczyszyn’s article.

They might not have the flashy numbers as the Jameses and Crosbys of the league, but when they’re playing, their teams are much better than when they’re sitting.

Yes, Saad and Foligno are being compared to Lebron James and Sidney Crosby in terms of value to their team.

Anyone who shares the ice with Foligno and Saad plays and looks better, and it’s for that reason that GAR loves them. But even though the numbers are big fans of their work, it might still be a while before the rest of the league catches on.

Yes, GAR loves Foligno and Saad but the rest of the NHL hasn’t yet caught on to their star-level value. The entire hockey industry hasn’t figured out that Foligno is as valuable as Crosby but luckily GAR has.

Now these are pretty bold statements to draw from a single statistics, even if it is supposedly an all-encompassing statistic. Nick Foligno in direct comparison to Sidney Crosby? Wow.

Just to be clear if you can’t get by The Athletic paywall, there is no other statistical evidence presented to justify these claims. Just GAR and some wordy justification as to why GAR might like these guys so much.

Now, I am all for analytics shedding light on stuff we may not otherwise realize. I am the guy who has argued the merits of Kris Russell when everyone else was trashing him so have at it. (Side Note: Russell and his on-ice save percentage are doing just fine in Edmonton thank-you very much). However, if you are going to make such bold claims you better have substantial evidence to back them up. More than a single stat that also suggests Mathieu Perreault and his 11 goals, 39 points is as valuable to the Winnipeg Jets as Mark Scheifele and his 30 goals, 78 points.

Of course, when I critiqued the article the attack dogs came out. You see, when you critique a writer from Hockey Graphs (which Luszczyszyn’s is, as is DTMAboutHeart, the creator of GAR) the response is a series of counter-attacks from other Hockey Graph writers. They are very defensive of any critique of their work.



For the record, @domluszczyszyn, @EvolvingWild, @DTMAboutHeart, @iyer_prashanth, @imfleming16 and @RK_Stimp are all writers or have been writers for Hockey Graphs. This isn’t the first time that I have critiqued one member of Hockey Graphs and have been immediately counter-attacked by a number of them. There seems to be a concerted effort on the part of the Hockey Graphs crew to immediately extinguish any critique of their work. Not one person actually stood up and argued that Perreault actually is as good as Scheifele or that Foligno actually is of star value. Instead of defending the core of my critique they attacked me personally.

Some are just pointless attacks to discredit me such as I only use goals or on-ice save percentage to evaluate players which is absolutely not true. These are just petty attacks attempting to divert attention from the real issue at hand. This ‘attack the messenger’ is essentially Donald Trump’s modus operandi towards anyone who criticises him. Sad but true.

The marginally better attacks are of the form of ‘you cannot dismiss a metric just because it does not mesh with popular belief’. This is true, however you cannot automatically accept them either which is what I am critiquing. If you want to put forward a new metric or a new theory you have to be open to critique. This is how science and research works as Stefan Wolejszo pointed out.

Nobody likes to be critiqued but it is part of the process. Among hockey analytics I get critiqued as much as anyone. The thing is, when people critique me (fairly) I use it as a challenge to do better work and find better ways to express my point of view. The Hockey Graphs crew would rather gang up and counter punch to shut down critique than use them as challenges to improve their work.

It should be stated that I have had good conversations with some Hockey Graphs writers. For example, Matt Cane whom I have had twitter discussions with and have met several times at conferences. I like Matt and while we frequently disagree I have found him willing to engage and I haven’t felt like he was in attack mode or trying to shut down dissent. I don’t expect everyone to agree with me and in fact some of the most interesting conversations I have are with people who disagree with something I have written. I use these discussions as learning experiences, even if they just force me to re-think my positions. This is all good.

I’ve even had good conversations with Ryan Stimson and I like the work he does on the passing project so I find it really unfortunate he participates in these group attacks. For example, Stimson’s attack towards me above is completely out of bounds. I simply mentioned that I don’t know all the details of how GAR is calculated and Stimson counter attacked with “Stop wasting your time. He hasn’t done the required reading for class today” as if you need to know all the details of GAR to be able to evaluate the results. This is the analytics version of “You haven’t played hockey, what do you know about evaluating players” argument that the hockey analytics community hates when directed towards them. I have been around long enough and know enough about using hockey analytics in player evaluation to have an opinion on the validity of a ranked list of players without having to know all the details of the underlying ranking system. You simply don’t need to know how to bake a cake to know whether its tastes good or not.

Many of these guys dream of a job with an NHL team. To those who do I ask you to think about how a GM or coach would react if when he questioned your player evaluation you responded with “You can’t critique my results, you don’t know the math behind them”. My guess is you wouldn’t have a job for very long, or maybe you won’t get hired in the first place.

Critique is part of life and we all have to deal with it gracefully. If you think you are right, find a different way to present your argument. If you really think Mathieu Perreault is as good as Mark Scheifele or Nick Foligno is really as valuable as other star players you should be able to find ample evidence to support your claim. If you can’t, maybe that is telling you something.

Throughout this whole GAR episode I was frequently asked “What evidence do you have that Perreault isn’t as good as Scheifele”. Well, to finish off, here is some (though far from a complete list).

Year Player GF% CF% G/60 A/60 Pts/60
2016-17 Scheifele 55.2 50.3 0.78 1.73 2.51
Perreault 40.7 53.0 0.54 1.08 1.62
2015-16 Scheifele 59.4 54.0 1.05 1.46 2.51
Perreault 49.4 57.1 0.20 1.37 1.57

I have bolded where where the stat is better than for the other player. Scheifele is significantly better than Perreault in every metric except CF%. Of course, over emphasising CF% is what I suspected was the problem with GAR. I absolutely believe that Scheifele’s offensive skill is more than enough to offset the difference in CF%. I have always believed hockey analytics puts too much emphasis on shot metrics and not enough on shot quality and this is just another example of that.

Just to be clear, I think Perreault and Foligno are perfectly fine players and useful guys to have on your team however they are not star players nor do they have star-level value. I’d rank Saad above these two but he too likely falls a little short of star level value. Certainly falls short of Sidney Crosby level. These guys aren’t Sidney Crosby and shouldn’t be mentioned in the same sentence with him and if you do, you better have some outstanding proof to support such a claim. Extraordinary claims require extraordinary evidence. Just using GAR to support your claim doesn’t cut it. That’s GARbage analytics in my opinion and I will continue to call it out. Hockey Analytics can, and must, do better than this.