Oct 272011
 

There has been a fair bit of discussion going on regarding shot quality the past few weeks among the hockey stats nuts.  It started with this article about defense independent goalie rating (DIGR) in the wall street journal and several others have chimed in on the discussion so it is my turn.

Gabe Desjardins has a post today talking about his hatred of shot quality and how it really isn’t a significant factor and is dominated by luck and randomness.  Now, generally speaking when others use the shot quality they are mostly talking about thinks like shot distance/location, shot type, whether it was on a rebound, etc.  because that is all data that is relatively easily available or easily calculated.  When I talk shot quality I mean the overall difficulty of the shot including factors that aren’t measurable such as the circumstances (i.e. 2 on 1, one timer on a cross ice pass, goalie getting screened, etc.).  Unfortunately my definition means that shot quality isn’t easily calculated but more on that later.

In Gabe’s hatred post he dismisses pretty much everything related to shot quality in one get to the point paragraph.

 

Alan’s initial observation – the likelihood of a shot going in vs a shooter’s distance from the net – is a good one.  As are adjustments for shot type and rebounds.  But it turned out there wasn’t much else there.  Why?  The indispensable JLikens explained why – he put an upper bound on what we could hope to learn from “shot quality” and showed that save percentage was dominated by luck.  The similarly indispensable Vic Ferrari coined the stat “PDO” – simply the sum of shooting percentage and save percentage – and showed that it was almost entirely luck.  Vic also showed that individual shooting percentage also regressed very heavily toward a player’s career averages.  An exhaustive search of players whose shooting percentage vastly exceeded their expected shooting percentage given where they shot from turned up one winner: Ilya Kovalchuk…Who proceeded to shoot horribly for the worst-shooting team in recent memory last season.

So, what Gabe is suggesting is that players have little or no ability to generate goals aside from their ability to generate shots.  Those who follow me know that I disagree.  The problem with a lot of shot quality and shooting percentage studies is that sample sizes aren’t sufficient to draw conclusions at a high confidence level.  Ilya Kovalchuk may be the only one that we can say is a better shooter than the average NHLer with a high degree of confidence, but it doesn’t mean he is the only one who is an above average shooter.  It’s just that we can’t say that about the others at a statistically significant degree of confidence.

Part of the problem is that goals are very rare events.  A 30 goal scorer is a pretty good player but 30 events is an extremely small sample size to draw any conclusions over.  Making matters worse, of the hundreds of players in the NHL only a small portion of them reach the 30 goal plateau.  The majority would be in the 10-30 goal range and I don’t care how you do your study, you won’t be able to say much of anything at a high confidence level about a 15 goal scorer.

The thing is though, just because you cannot say something at a high confidence level doesn’t mean it doesn’t exist.  What we need to do is find ways of increasing the sample size to increase our confidence levels.  One way I have done that is to use 4 years of day and instead of using individual shooting percentage I use on-ice shooting percentage (this is useful in identifying players who might be good passers and have the ability to improve their linemates shooting percentage).  Just take the list of forwards sorted by on-ice 5v5 shooting percentage over the past 4 seasons.  The top of that list is dominated by players we know to be good offensive players and the bottom of the list is dominated by third line defensive role players.  If shooting percentage were indeed random we would expect some Moen and Pahlsson types to be intermingled with the Sedin’s and Crosby’s, but generally speaking they are not.

A year ago Tom Awad did a series of posts at Hockey Prospectus on “What Makes Good Players Good.”  In the first post of that series he grouped forwards according to their even strength ice time.  Coaches are going to play the good players more than the not so good players so this seems like a pretty legitimate way of stratifying the players.  Tom came up with four tiers with the first tier of players being identified as the good players.  The first tier of players contained 83 players.  It will be much easier to draw conclusions at a high confidence level about a group of 83 players than we can about single players.  Tom’s conclusions are the following:

The unmistakable conclusions from this table? Outshooting, out-qualitying and out-finishing all contribute to why Good Players dominate their opponents. Shot Quality only represents a small fraction of this advantage; outshooting and outfinishing are the largest contributors to good players’ +/-. This means that judging players uniquely by Corsi or Delta will be flawed: some good players are good puck controllers but poor finishers (Ryan Clowe, Scott Gomez), while others are good finishers but poor puck controllers (Ilya Kovalchuk, Nathan Horton). Needless to say, some will excel at both (Alexander Ovechkin, Daniel Sedin, Corey Perry). This is not to bash Corsi and Delta: puck possession remains a fundamental skill for winning hockey games. It’s just not the only skill.

In that paragraph “shot quality” and “out-qualitying” is used to reference a shot quality model that incorporates things like shot location, out-finishing is essentially shooting percentage, and outshooting is self-explanatory.  Tom’s conclusion is that the ability to generate shots from more difficult locations is a minor factor in being a better player but both being able to take more shots and being able to capitalize on those shots is of far greater importance.

In the final table in his post he identifies the variation in +/- due to the three factors.  This is a very telling table because it tells it gives us an indication of how much each factors into scoring goals.  The following is the difference in +/- between the top tier of players and the bottom tier of players:

  • +/- due to Finishing:  0.42
  • +/- due to shot quality:  0.08
  • +/- due to out shooting:  0.30

In percentages, finishing ability accounted for 52.5% of the difference, out shooting 37.5% of the difference and shot quality 10% of the difference.  Just because we can’t identify individual player shooting ability at a high confidence level doesn’t mean it doesn’t exist.

If we use the above as a guide, it is fair to suggest that scoring goals is ~40% shot generation and ~60% the ability to capitalize on those shots (either through shot location or better shooting percentages from those locations).  Shooting percentage matters and matters a lot.  It’s just a talent that is difficult to identify.

A while back I showed that goal rates are better than corsi rates in evaluating players.  In that study I showed that with just 1 season of data goal for rates will predict future goal for rates just as good as fenwick for rates can predict future goal for rates and with 2 years of data goal for rates significantly surpass fenwick for rates in terms of predictability.  I also showed that defensively, fenwick against rates are very poor predictors of future goal against rates (to the point of uselessness) while goals against rates were far better predictors of future goal against rates, even at the single season level.

The Conclusion:  There simply is no reliable way of evaluating a player statistically at even a marginally high confidence level using just a single year of data.  Our choices are either performing a Corsi analysis and doing a good job at predicting 40% of the game or performing a goal based analysis and doing a poor job at predicting 100% of the game.  Either way we end up with a fairly unreliable player evaluation.  Using more data won’t improve a corsi based analysis because sample sizes aren’t the problem, but using more data can significantly improve a goal based analysis.  This is why I cringe when I see people performing a corsi based evaluation of players.  It’s just not, and never will be, a good way of evaluating players.

 

Oct 132011
 

Earlier this week I participated in several conversations elsewhere and thought I’d post a followup to a few of them.

“The Bet”

The other day I posted an article about on ice shooting percentage being a talent and referenced a discussion I had with Gabe Desjardin’s of Arctic Ice Hockey at MC79Hockey.com.  As part of that discussion we were negotiating a “bet” of sorts.  In it I suggested that as a group Crosby, Gaborik, Ryan, St. Louis, H. Sedin, Toews, Heatley, Tanguay, Datsyuk, Horton would have an average on-ice shooting percentage over 10% this season while Gabe suggested that the group of them would regress significantly to the mean.  The problem was, we couldn’t agree on what the mean was that we were regressing to.

I suggested we use 7.94% which is league wide-shooting percentage (all goals divided by all shots at 5v5 over the last 4 seasons) but Gabe suggested we use only players who were on the ice for 1000 shots at 5v5 over the last 4 seasons which resulted in a much higher 9.12%.  He suggested we need to do this because we don’t want to include players who are “sub-replacement” level.  This seems odd to me because he believes that variation in shooting percentage is largely random so using that theory the sub-replacement level player is just as likely to have an elevated shooting percentage than a below average one.

So, in the end, we never seemed to agree on what mean on-ice shooting percentage to use but I wanted to make it known that I still do believe players can drive shooting percentage and thus corsi derived player analysis will never tell us the whole story.  So, with that said I’ll predict the 10 players above will end the season with an on-ice 5v5 even strength shooting percentage above 10% and no more than 2 will have an on-ice shooting percentage below 9.5%.  So, come seasons end lets look back and see what happened and hopefully my prediction will come true and we can all get past the “shooting percentage is random” nonsense.

“Mike Weaver”

Over at Pension Plan Puppets we had another Mike Weaver discussion.  In it I compared how goalies performed when Weaver was on the ice vs when Weaver was not on the ice.  Let me expand on that here.

Year with Weaver without Weaver
Vokoun 2010-11 2.14 2.79
Clemmensen 2010-11 1.70 2.63
Mason 2009-10 1.44 2.58
Conklin 2009-10 1.85 2.72
Mason 2008-09 1.68 2.01
Legace 2008-09 2.38 3.15
Luongo 2007-08 1.45 2.14

The table shows the goals against average of each goalie that Weaver has played at least 200 minutes in front of in each of the past 4 seasons along with the goalies goals against average when Weaver is on the ice in front of them and when Weaver is not on the ice in front of them.  As you can see, in every instance the goalie has had a significantly better goals against average when Weaver is on the ice than when he is not.  This should put to rest some of the “but Weaver has played in front of excellent goalies” argument that came out of my previous Mike Weaver is an excellent defensive defenseman post.

Separating Goalie Performance from Defensemen

As a followup to the Mike Weaver discussion @so_Truculent asked me how I separate out goaltender play in my goal based analysis.  i.e. how do you account for a player who plays against, on average worse goalies, or a player who plays in front of sub-par goalies.  How I do it is a somewhat complex process but it is exactly the same as how I account for quality of competition and quality of teammates for forwards and defensemen.  Let me describe the process as simply as I can.

I assign every player an offensive and defensive rating based on their GF20 (goals for while on ice per 20 minutes) and GA20 (goals against while on ice per 20 minutes) stats.  Now, lets use an example.  Let’s assume we are evaluating Mike Weaver defensively.  I will then calculate an estimated GA20 based on an average GA20 of all the players who played with Mike Weaver (weighted by ice time played with Mike Weaver) and based on the average GF20 of all the players who Mike Weaver played against (weighted by ice time played against Mike Weaver).  This gives me an estimated GA20 based on the players Mike Weaver plays with and against.  I then assign Mike Weaver a new defensive rating equivalent to Mike Weavers estimated GA20 divided by his actual GA20.  So, if Mike Weaver gives up fewer goals against than the average of his teammates and opposition, then Weaver gets a better than average rating and if he gives up more then he gets a below average rating.

I do this for every player in the league for both offensively and defensively.  This included forwards, defensemen and goaltenders (defensively only, I don’t factor goalies into offensive ratings).  That’s the beauty of this system.  It works for all players the same way.  If a goalie make his teammates GA20 ratings better and the oppositions GF20 ratings worse then the goalie gets a better than average rating.

Now, after we do this once we have improved information about the players offensive and defensive abilities and new offensive and defensive ratings.  So, I take these new numbers and run them through the process again.  And again.  And again.  Eventually it comes to a stable state where each iteration has very little change in the players ratings.  At this point we have the HARO+ and HARD+ ratings you see at stats.hockeyanalysis.com.  The HART+ ratings are just an average of each players HARO+ and HARD+ ratings.  Players with a HARO+ rating above 1.00 are above average offensively (though the median is something below 1.00) and players with a HARD+ rating above 1.00 are above average defensively (again, the median is something below 1.00 though).

I don’t know for sure if these are perfect ratings, but generally speaking I am very happy with the results, and I am very happy that the iterative process converges on a ratings solution as opposed to going completely haywire as some iterative processes do.  I also really like the fact that I can include goalies in the process because separating goalie performance from the players in front of them is probably one of the hardest questions to answer in hockey, especially if you are like me and believe that players can suppress shooting percentage (meaning save % is a team stat as much as it is a goalie stat).

 

Oct 102011
 

There has been an interesting discussion of on-ice shooting percentage at Tyler Dellow’s mc79hockey.com.  I have argued that we need to look at on-ice shooting percentage as a talent, and not something that just happens randomly while others have largely dismissed it.  One person in particular is Gabe Desjardin’s who has a followup post on his blog largely dismissing its importance.

In his blog post Gabe first discusses Gaborik’s value just considering his on-ice shooting percentage.

So are these totals 75% skill then?  Let’s do a quick check on how many goals that skill would be worth: 1000 on-ice shots/season * 2.5% above mean * 75% = 18.75 goals above average.  Double that to get to an approximate replacement level of 37.5 goals or just over six wins.  The current price for one win on the free agent market is roughly $3M, so we’d estimate Gaborik’s offensive value at more than $18M.

Gabe doesn’t believe any one player could be worth $18M based just on shooting percentage so he tries to shoot a hole in that by looking at 2yr vs 2yr regression.

Needless to say, anytime you come up with a metric that says a player should get paid $18M, you have to go back and check your math.  I did that by splitting the last four years into two two year periods (2007-08/2008-09 vs 2009-10/2010-11) and comparing on-ice shooting percentage among players who had 1000+ on-ice shots in each period.  I found that player on-ice shooting regressed 80% to the mean from the first set to the second, which puts Gaborik’s apparent talent closer to $5M.

Ok, so we have Gaborik’s value down to $5M.  That still seems pretty large to me but Gabe dismisses that further by suggesting some of that $5M has to be attributed to his linemates, arena bias, and strangely, a players opponents (particularly at home).

This is a key point, of course, and one that may not come through when we talk about team-level effects or try to figure out the value of individual top six forwards: when a #1 line plays against a #4 line, their shooting percentage goes up relative to when they’re playing power-vs-power.

Here is the thing.  The whole reason I participate in these debates is to suggest that players do in fact have the ability to drive or suppress shooting percentage and thus we must consider shooting percentage, in addition to corsi, when evaluating players.  So, it amazes me when I am debating someone that is trying to minimize the ability to drive/suppress shooting percentage that they bring up such observations that a players shooting percentage will go up when they are playing weaker players (i.e. the fourth line) who I presume can’t suppress shooting percentage as well as the stronger players.   There is clearly something wrong with the logic there.  Players don’t have the ability to suppress shooting percentage, but fourth liners are worse at suppressing shooting percentage than first liners??

The other thing I want to discuss is Gabe’s calculation that shooting percentage regresses 80% to the mean based on his 2yr vs 2yr calculation.  I won’t dispute his math because it is probably true, but I will suggest that I don’t believe that a league-wide observation can be applied to individual players.  There are a number of factors that influence a players on-ice shooting percentage.

1.  The quality of his linemates.

2.  The quality of opposition.

3.  Style of play (i.e.  aggressive offensive game vs defensive style).

4.  Score effects

Team building generally revolves around a small number of players.  Pittsburgh has Crosby, Malkin and Staal as their core forwards, everyone else is pretty much interchangeable.  Pretty much every team is like this.  A lot of those interchangeable parts move from team to team or even line to line on the same team or get asked to play different roles on the same team.  An injury to Kunitz and Pascal Dupuis gets bumped from the third line to playing with Crosby.  For these mostly interchangeable parts there can be a lot of variation in who they play with, the team they play on, and who they play against, and the roles they are asked to play.  All these factors are at play when Gabe calculates his 80% regression to the mean.  The good players who have well-defined roles don’t see near the same variation in their on-ice shooting percentages.  Look at the Crosby’s and Gaborik’s.  They are consistently at the top of the list.  Look at the Moen’s and Marchant’s and Pahlsson’s, they are consistently near the bottom of the list.  The players we perceive as good offensive players are at the top of the list.  The players we perceive as weak offensive players, or defensive minded players, are at the bottom of the list.  That’s a talent that we must consider.

 

 

Oct 052011
 

The Leafs traded for David Steckel yesterday and while this is by no means a significant trade my first reaction to it was a very positive one.  A fourth round pick is almost worthless and Steckel is a more than useful quality defensive third/fourth line guy who can kill penalties, something the Leafs desperately need.  Upon further review of the stats, I still like the trade because of it’s low risk but my thoughts on Steckel are a little more mixed than I first believed.

The Good

On the surface, Steckel looks like a premiere defensive forward.  Over the past 4 years, Steckel has the 9thth lowest on ice goals against per 20 minutes of the 250 forwards with 1500 5v5close minutes of ice time and he has been consistently very good at keeping the puck out of his own net at even strength.  His four year HARD+ rating is 1.152 and his HARD+ ratings for the past 4 seasons are 1.112, 1.262, 1,102 and 1.094.  All of these things point to Steckel being a good, or maybe very good, defensive forward.

The Bad

Throwing a damper on everything I just said, his quality of competition is quite weak.  His OppGF20 (opposition goals for per 20 minutes) ranks 227th of 250 and surprisingly he has over the past 4 seasons had slightly more offensive zone starts than defensive zone starts.  Now this isn’t all bad.  His opponents on average scored at a rate of 0.766 goals per 20 minutes of 5v5close ice time while Steckel and his teammates held them to 0.499 goals per 20 minutes but I would have more confidence in his defensive numbers if he was playing against top level opponents.

The Ugly

One of the key roles the Leafs likely acquired Steckel for is to provide some desperately needed help to their woeful penalty kill.  The problem is, Steckel’s PK numbers are quite woeful as well.  Of the 63 forwards with 500 4v5 PK minutes over the past 4 seasons, Steckel ranks in 48th in goals against per 20 minutes though he is a much better, but still average, 28th in fenwick against per 20 minutes.  Furthermore, the quality of his opponents on the PK hasn’t been all that great either as he ranked 57th of 63 in OppGF20 and 60th of 63 in OppFenF20.  Add it all up and it is quite likely that Steckel has been a below (maybe well below) average PK guy over the past four seasons.  That isn’t good news for the Leafs PK in 2011-12.

The Skinny

Although the numbers cast some doubts as to whether Steckel will live up to my initial reaction when I heard the trade, I still like the trade because it is a low risk trade and adds some defensive minded depth and size to the Leafs lineup.  I’ll take a wait and see attitude with regards to Steckel being a quality addition to the Leafs penalty kill unit but at the very least he’ll be a quality addition to the fourth line.  A fourth line that includes Steckel along side Mike Brown and Colton Orr could at the very least be a physically intimidating energy line that hopefully is more than responsible defensively and that isn’t all bad.