Evaluating defence and goaltending revisited

A couple weeks ago I posted an article about the Leafs defence and how it isn’t as bad as many people think. Well, since then I have been working on trying to improve on the methodology by including shot type (slap shot, wrist shot, snap shot, tip in, backhand, wraparound) and in that process I found a few mistakes/issues in what I did previously.

First, I found a bug in my program that caused a number of powerplay goals to be considered even strength goals. When I fixed this the general conclusions of that article remained in tact though the amount of the goals caused by the goalie was reduced for most teams. In general the closer that a teams penalty kill ability was to their even strength ability the more valid the results but where a teams penalty kill ability was seemingly far superior or inferior to their even strength ability the results were skewed. The most notable team was the Philadelphia Flyers who were down right horrible at even strength but for some reason managed to have a pretty good penalty kill. When I fixed the bug the Flyers looked far worse than they did in the previous article.

The second issue I discovered is that not all shot distances and types are created equal and that there is significant (unintentional) bias in how game monitors decide what is a wrist shot vs snap shot as well as the distance a shot is from the goal. It is a bit surprising but it is clear to me now that some game monitors have a real hard time distinguishing between a 10’ shot and a 15’ shot. Since teams play half their games at the same arena (their home arena) if that arena’s game monitor couldn’t judge shot type or shot distance very well their shot difficulty ratings would get significantly biased one way or the other.

In an ideal world I would there would be an easy method for factoring out that bias and using all the data but I cannot think of any such easy method. The quick and dirty solution is to just look at shots against while playing on the road. This will eliminate any significant home arena bias and hopefully and biases found in other arenas will get averaged out on their own. For the most part this is likely true but I am still not completely happy with this solution because I think on some level teams play a bit different at home than on the road. More on this later but for now just looking at road shots is the best solution so lets go with that.

Ok, so what I did was group shots into 19 categories based on shot type and distance.

  • 0-14′ wrist shot
  • 15-29′ wrist shot
  • 30-44′ wrist shot
  • 45+’ wrist shot
  • 0-14′ snap shot
  • 15-29′ snap shot
  • 30-44′ snap shot
  • 45+’ snap shot
  • 0-14′ slap shot
  • 15-29′ slap shot
  • 30-44′ slap shot
  • 45+’ slap shot
  • 0-9′ tip-in
  • 10-25′ tip-in
  • 26+’ tip-in
  • 0-12′ backhand
  • 13-24′ backhand
  • 25+’ backhand
  • wraparound

I then performed an analysis more or less equivalent to the analysis done in the previous article. By doing that I come up with the following results:

Team ExpGA/60m GA/60m Goalie Impact
Philadelphia 2.58 3.15 0.57
Los Angeles 2.68 3.08 0.39
Edmonton 2.49 2.85 0.36
Washington 2.63 2.95 0.32
Phoenix 2.39 2.71 0.32
Tampa Bay 2.52 2.80 0.28
Montreal 2.71 2.98 0.27
Carolina 2.39 2.56 0.17
Chicago 2.66 2.82 0.15
Colorado 2.66 2.76 0.10
Boston 2.87 2.95 0.07
Calgary 2.46 2.50 0.03
San Jose 2.08 2.10 0.02
Pittsburgh 2.50 2.51 0.01
Toronto 2.44 2.42 -0.02
Anaheim 2.23 2.18 -0.05
Florida 2.58 2.52 -0.06
NY Islanders 2.71 2.62 -0.09
Columbus 2.02 1.93 -0.09
Minnesota 2.45 2.33 -0.12
NY Rangers 2.38 2.16 -0.22
Detroit 2.11 1.88 -0.24
Nashville 2.61 2.36 -0.25
Dallas 2.16 1.91 -0.26
Buffalo 2.67 2.38 -0.30
Ottawa 2.60 2.30 -0.30
New Jersey 2.48 2.11 -0.37
Atlanta 2.62 2.21 -0.41
Vancouver 2.34 1.81 -0.53
St. Louis 2.73 2.18 -0.55

Goalie Impact is the amount of goals per 60 minutes of even strength ice time the goalie is responsible for above or below what an average goalie would allow.
There are a lot of similarities between the table above and the table in my previous article but there are some teams that have moved up or down the list.

Toronto: Toronto’s defence (ExpGA/60m) drops a bit in the ratings influenced partially by factoring in the amount of time the Leafs spend at even strength but more significantly because of removing a small home ice bias that made shots at Air Canada be reported as being at a slightly greater distance than they likely actually were (it should be noted the Air Canada Center bias was much lower than some other arenas). The result is that the Leafs dropped to 11th best road defence from 8th. Not a huge drop but I figured since the article was primarily about the Leafs defence I should mention it. Also, because of these changes it makes Andrew Raycroft look much better than under the previous analysis. Using this current approach the net effect of Leaf goaltending is pretty neutral (i.e. On the Leaf goaltending was about average). And this gets to my problem with just looking at road statistics. The Leafs were generally a worse team at home than on the road despite the fact that it is typical for a team to play better (by about 10%) on home ice as road ice. Raycroft may have been the culprit as he had an .898 save percentage on the road and a .890 save percentage at home but it could also be the Leafs as a team played differently and gave up tougher shots at home. Without reliable statistics we will never know which is true or whether it is some combination of the two.

St. Louis: Wow! How did they become the team with the goalies that saved the most goals? That is hard to believe considering the names of the goalies they have on their roster but it seems to be a function of the number of shots and their difficulty. St. Louis goalies also posted a much better road save % than a home save %.

The next thing I looked at is how individual goalies performed. Based on league wide save percentages for the 19 groupings I calculated how many goals a perfectly average goalie should give up given the shot types that each goalie faced. Here are the results sorted by the number of goals the goalie saved per game.

Name Team ExpGoals Goals Diff Diff/game
SANFORD, CURTIS St. Louis 30.80 23.00 7.80 0.42
OSGOOD, CHRIS Detroit 22.82 18.00 4.82 0.34
LUONGO, ROBERTO Vancouver 59.43 42.00 17.43 0.33
BACASHIHUA, JASON St. Louis 22.43 19.00 3.43 0.33
HEDBERG, JOHAN Atlanta 20.96 17.00 3.96 0.32
LEGACE, MANNY St. Louis 35.99 27.00 8.99 0.29
BRODEUR, MARTIN New Jersey 81.80 65.00 16.80 0.28
LEHTONEN, KARI Atlanta 65.11 52.00 13.11 0.27
DIPIETRO, RICK NY Islanders 63.73 52.00 11.73 0.27
TURCO, MARTY Dallas 56.34 44.00 12.34 0.27
MILLER, RYAN Buffalo 54.75 43.00 11.75 0.26
VOKOUN, TOMAS Nashville 40.43 33.00 7.43 0.24
LUNDQVIST, HENRIK NY Rangers 59.50 49.00 10.50 0.21
GERBER, MARTIN Ottawa 29.63 26.00 3.63 0.19
BURKE, SEAN Los Angeles 33.80 31.00 2.80 0.17
EMERY, RAY Ottawa 57.16 50.00 7.16 0.17
NORRENA, FREDRIK Columbus 40.98 35.00 5.98 0.17
HASEK, DOMINIK Detroit 33.82 27.00 6.82 0.17
THIBAULT, JOCELYN Pittsburgh 26.19 24.00 2.19 0.17
RAYCROFT, ANDREW Toronto 62.74 55.00 7.74 0.15
MASON, CHRIS Nashville 44.26 40.00 4.26 0.15
BELFOUR, ED Florida 49.00 43.00 6.00 0.15
THOMAS, TIM Boston 62.58 57.00 5.58 0.13
HUET, CRISTOBAL Montreal 41.48 38.00 3.48 0.12
KHABIBULIN, NIKOLAI Chicago 58.75 54.00 4.75 0.12
BACKSTROM, NIKLAS Minnesota 36.16 34.00 2.16 0.08
KIPRUSOFF, MIIKKA Calgary 65.53 62.00 3.53 0.07
GRAHAME, JOHN Carolina 27.94 27.00 0.94 0.05
GIGUERE, J Anaheim 45.88 44.00 1.88 0.05
GARON, MATHIEU Los Angeles 22.90 22.00 0.90 0.04
TOSKALA, VESA San Jose 30.08 29.00 1.08 0.04
NABOKOV, EVGENI San Jose 39.80 39.00 0.80 0.02
KOLZIG, OLAF Washington 46.13 46.00 0.13 0.00
BRYZGALOV, ILJA Anaheim 25.00 25.00 0.00 0.00
FLEURY, MARC-ANDRE Pittsburgh 56.29 57.00 -0.71 -0.02
BUDAJ, PETER Colorado 56.11 57.00 -0.89 -0.02
SMITH, MIKE Dallas 15.23 16.00 -0.77 -0.05
ROLOSON, DWAYNE Edmonton 64.88 68.00 -3.12 -0.06
THEODORE, JOSE Colorado 27.57 29.00 -1.43 -0.07
HOLMQVIST, JOHAN Tampa Bay 45.68 48.00 -2.32 -0.07
JOSEPH, CURTIS Phoenix 51.39 54.00 -2.61 -0.07
BIRON, MARTIN Philadelphia 33.72 36.00 -2.28 -0.09
WARD, CAM Carolina 48.86 53.00 -4.14 -0.10
FERNANDEZ, MANNY Minnesota 38.52 42.00 -3.48 -0.12
NIITTYMAKI, ANTERO Philadelphia 50.70 55.00 -4.30 -0.12
BOUCHER, BRIAN Columbus 17.38 19.00 -1.62 -0.14
LECLAIRE, PASCAL Columbus 13.87 16.00 -2.13 -0.14
JOHNSON, BRENT Washington 37.11 40.00 -2.89 -0.15
AULD, ALEXANDER Florida 30.08 33.00 -2.92 -0.16
DENIS, MARC Tampa Bay 37.94 43.00 -5.06 -0.17
AEBISCHER, DAVID Montreal 26.86 31.00 -4.14 -0.20
TELLQVIST, MIKAEL Phoenix 23.38 28.00 -4.62 -0.22
MARKKANEN, JUSSI Edmonton 16.84 20.00 -3.16 -0.25
HALAK, JAROSLAV Montreal 20.38 25.00 -4.62 -0.41
TOIVONEN, HANNU Boston 16.45 22.00 -5.55 -0.55
CLOUTIER, DAN Los Angeles 17.88 28.00 -10.12 -0.68
DUNHAM, MIKE NY Islanders 18.94 28.00 -9.06 -0.75
ESCHE, ROBERT Philadelphia 23.18 32.00 -8.82 -0.86

For the most part the table makes perfect sense. It is still surprising to see the St. Louis goalies near the top of the list (I am beginning to think this is an anomaly of some sort) but it is no surprise to see Luongo, Brodeur, Lehtonen, DiPietro, Turco, Miller, Vokoun, Lundqvist, etc. near the top of the list and Esche, Dunham, Cloutier, Toivonen, etc. at the bottom of the list. So for the most part the list passes the smell test as everything seems right.

Here are some more observations:

1. Leaf goalie Andrew Raycroft does OK here as well as he would be somewhere in the middle of the NHL regular starting goalies.

2. It is interesting that Martin Gerber ranks higher than Ray Emery.

3. While the Panthers have seemingly upgraded from Belfour to Vokoun, the same cannot be said for the Leafs as Toskala is ranked well below Raycroft. I should add that while Toskala has a pretty good save percentage it could be attributed to the quality of his opponent as he has started against the Kings 5 times, and Coyotes 4 times, St. Louis, Columbus and the weak offensive Dallas Stars 3 times. That is a pretty easy schedule. Interestingly, like Raycroft, Toskala also seemed to perform much better on the road.

4. Teams might want to consider avoiding trading for Manny Fernandez and his large contract as he ranks quite poorly.

What’s left to do?

There are still a couple of things I would like to tackle in this area of analysis. The first would be to see if I can come up with some kind of reliable method for making use of the home stats. The second thing is that while I think the above analysis does a pretty good job of accounting for shot difficulty I think the quality of the shooter is still factor that is not factored in fully and I’d like to see if I can find some kind of method for factoring that in. Problem is, I am not sure if there is enough data to properly evaluate individual shooters but I might give it a try.

This article has 4 Comments

  1. I’m not sure what you’re trying to accomplish here, but most the “shot quality” analysis has been completed with much more rigor by Alan Ryder. When you start breaking these things into little chunks you have to be careful that the groups are statistically “large enough” and whether they are statistically different than not having that group all together.

    Also, the distance groups are very large; a 15′ shot is very different than a 29′ shot for example.

  2. I have looked over some of the shot quality work Alan had done and he certainly has done some quality work. What I am trying to do is do something that the average reader can understand. Is the work Alan has done more robust and probably provide (possibly only marginally) better results than what I have done? Sure, but I suspect I am getting 90% of the way to where he is at with a much simpler method that is more readable/understandable by my audience, the average hockey fan. I write about hockey for hockey fans, not about statistics for math junkies. I understand the value of what Alan is doing but I also know that 99.99% of hockey fans probably have no interest in reading stuff like this or would really understand it if they did. But people can for the most part understand the process I took and understand that the results will be better than just looking at save percentage and that is my goal.

    And as for my shot groups, they were developed (admittedly in a somewhat adhoc manner) by looking at the success rates of shots by distance/type and developing groups that are ‘large enough’ but also to try to keep within group shooting rate variance somewhat reasonable. Could I have developed/used better groups? Probably. Do I care? Not really because I know the #1 flaw in my analysis is more likely to be the underlying data (which is questionable on many levels) and not my methodologies (which in my opinion are good enough 90% of the time).

  3. David,

    I love you analysis of these stats. I have a factor that I’m sure won’t be able to be factored in:


    Was the shot taken on a breakaway? How about a 2-on-1? Was the puck moving across the ice prior to the shot being taken, which would make the goalie move from one side to the other? Do you think a Spezza feed from the corner to Heatley at the top of the opposite side circle is less dangerous of a shot than a fourth-line winger at the bottom of a circle?

    I know – it can’t be factored in. So while numbers don’t lie, they don’t tell the whole truth either. If a team is able to protect the crease effectively (thereby forcing the shots to come from farther out), it doesn’t mean that they are doing a good job (especially if they give up room for Jagr/Ovechkin to dance in the slot). But if a team plays a pressure-style defense and the only shot a player has is right in front of the goalie who is able to challenge them correctly, are they doing a bad job?

    It is all very interesting and I’m sure that there are no correct answers, only good discussion.

    Thanks for the great article. I’m sure it was a lot of work and it is very well presented.


Comments are closed.