Before they look for goaltending again, Oilers management needs to review their previous evaluation and decision-making methods.

One of the most important parts about decision-making, especially when you’re spending significant money or assets to achieve a specific goal, is the evaluation that follows it. It’s standard business practice in the corporate world when CEO’s work to maximize the value of an entity and try to find any sort edge over their competition. Scrutinizing internal processes, finding strengths and deficiencies and re-calibrating things is a critical component when building a sustainable product.

The same approach to evaluating decision-making has to apply in professional hockey. When every corner of the roster needs to be optimized for maximum value under a salary cap system, it’s critical that teams know if their roster decisions led to positive outcomes and, just as importantly, know if their methods behind the decision process was adequate and how it can be improved going forward. With a large pool of players and a limited number of roster spots, teams are faced with recurring decisions regarding their roster construction – so refining their decision-making methods has to be top of mind.

The Edmonton Oilers are facing one of those recurring decisions, that many teams are facing this off-season, as they’ll be searching for a goaltender to potentially start a significant number of games in the upcoming year. The Oilers goalie depth chart features Mikko Koskinen, a 32 year old with 101 NHL games on his resume, and under contract for two more seasons at $4.5 million per year, along with a group of goalie prospects that have a range of potential. This list currently includes Ilya Konovalov (age 22), Olivier Rodrigue (20), Stuart Skinner (21) and Dylan Wells (22).

Koskinen has shown reasonably well as a starter at the NHL level, having now played 93 regular season games and four playoff games for the Oilers since transitioning from the KHL and SM-liiga in 2018. Among 53 goaltenders who have played at least 70 games over the last three seasons (approximately 3,000 minutes between 2017/18 and 2019/20), Koskinen’s even-strength (5v5) save percentage of 91.9% ranks 36^th overall, while his goals-saved above average of -2.52 ranks 35^th. The average team save percentage at even-strength over the same time period has been 92.04%, so it’s safe to say Koskinen has been a league-average goalie for the Oilers.

Considering how teams, especially in the west where travel is onerous, are moving away from a starter-backup approach to deploying goalies to more of a tandem where they can split ice-time more evenly, it’s important for the Oilers to continue doing the same. While Koskinen has provided decent netminding for the team, we know from a 2014 research article at Hockey Graphs that goaltender performance declines with age, especially after age 30 – and it’s beneficial for both Koskinen and the team if he has more time to rest between starts.

When the Oilers looked for a goalie last summer to split time with Koskinen, they signed Mike Smith, a 37 year old netminder at the time with 571 games of experience. How much time and effort was put into the analysis that led to this decision is unknown. But what we do know is that their approach to finding an NHL-caliber goaltender this off-season has to be considerably better.

Smith’s results as an Oiler were poor, which wasn’t surprising as the probability of goalies posting save percentages below league average levels increases after age 30 and accelerates after age 35. For more details about goalie aging curves, I’d recommend checking out this 2014 research article from Hockey Graphs.

This past season, Smith posted one of the worst save percentages in the league at even-strength (90.0%), ranking 52^nd out of 54 goalies who played at least 1,000 minutes, or about 24 games, and allowed the fourth highest rate of goals against per hour (2.92). His individual performance was a big reason why the Oilers overall team save perentage was 25^th in the league with 91.23%, allowing the sixth highest rate of goals-against, 2.72 per hour. That rate is slightly worse than what the Oilers allowed in 2017/18 when they allowed the eighth highest rate of goals (2.60 per hour) and and 2018/19 when they allowed the ninth highest rate (2.65 per hour). For context, the league average rate of goals against per hour over the last three seasons has been 2.45.

What management should hopefully be aware of is the fact that the Oilers allowed close to the league average rate of expected goals against. Expected goals measures the quality of the unblocked shots taken, and assigns a value to it depending on the probability of it becoming a goal. Key variables include the type of shot taken, where it was taken from and compares it to historical shot and goal data to determine the value. So while the actual rate of goals against the Oilers was 2.72 this past season, based on the variables from the shot data, the expected rate of goals against was 2.36 – slightly above the league average rate of 2.30.

Put another way, had the Oilers allowed the same rate and quality of shots against and received league average goaltending, they would have allowed approximately 136 goals at even-strength this past season. That’s a significant improvement of about 21 goals, (they actually allowed 157), which would translate to approximately three additional wins in the standings. And a key reason why this occurred and cost the Oilers a better spot in the league rankings is squarely on the player that management brought in last summer to shore up the goaltending.

2019/20 (5v5)	Smith	Koskinen
GP	39	38
TOI	1747.13	1702.87
Shots Against/60	29.09	33.23
Expected Goals Against/60	2.18 (19th)	2.58 (51st)
Save%	90.00% (52nd)	92.40% (23rd)
High Danger Save%	77.60% (52nd)	85.10% (9th)
Goals Against/60	2.92 (51st)	2.54 (34th)

Above is a summary of how the two netminders did for the Oilers in 2019/20 at even-strength. Included for some of the metrics is each goaltenders ranking among the 54 goaltenders who played at least 1,000 minutes this past regular season. What I found interesting was that the team’s rate of expected goals against, which again measures the quality of unblocked shot attempts, was higher when Koskinen was in net than when Smith was. Thankfully Koskinen did reasonably well, ranking 23^rd overall among his peers with a 92.40% save percentage, just slightly above league average levels.

Whatever methods that the Oilers management used to determine that Smith was going to improve the team’s chances of winning games has to be under immediate scrutiny by someone in the organization. The publicly available data and research into goaltending made it clear that the probability of Smith posting league average numbers was low, which begs the question: what data and information and process was the team basing such an important decision on? What metrics were used and how well do these metrics correlate with targeted outcomes? Did they set out benchmarks and key performance indicators to measure their decision? How did the Oilers evaluate the probability of the player’s performance? Whose opinions were involved?

There has to be some level of accountability and proper evaluation of these key decisions – and it should come from someone outside of the hockey operations department. Preferably, above the general manager’s office, if Bob Nicholson has the motivation and courage to do so, or even an external consultant to ensure whatever the findings are can be actioned upon. The reality is that the second place finish in the division could be masking Smith’s poor goaltending performance, as Holland stated in his end of season media availability that goaltending was a strength. And there doesn’t appear to be any motivation for the general manager to dissect the goaltending decision from the 2019 off-season, leaving the team at risk of repeating the same mistake.

To improve their chances of landing a reliable goaltender this off-season and contend for the playoffs, it’s imperative that the Oilers evaluate how the decision was made to sign Smith last summer. The methods that were used were clearly flawed and it’s critical that management explore new ways – using data analysis, scouting and possibly an outside perspective – to evaluate goaltenders.

Data: Natural Stat Trick, Hockey Reference

Glossary:

• Expected goals: Measures the quality of the unblocked shots taken, and assigns a value to it depending on the probability of it becoming a goal. Key variables include the type of shot taken, where it was taken from and compares it to historical shot and goal data to determine the value. (Natural Stat Trick)
• Goals saved above average – the goals this goalie prevented give his save percentage and shots faces vs. the league average save percentage on the same number of shots. Minimum four shots faced per team gamed needed to qualify (Hockey Reference)

Article also posted at The Copper & Blue.

Disappointing end to the Oilers season but it wasn’t completely unexpected. As great as their regular season was, the Oilers were a flawed team and their late season results weren’t that significantly better than Chicago’s. Combine that with the randomness that comes with playoff hockey and the fact that there was a four month layoff between the end of the regular season and the start of qualifying round it’s hard to be too upset.

Related articles:

• Re-connecting– The SuperFan (2020, June 16)
• CBC Radio Active: NHL set to return to play – The SuperFan (2020, July 10)
• Depth perception – The Copper & Blue (2020, July 19)
• Powering up in Chicago – The Copper & Blue (2020, July 23)
• CBC Radio Active: Oilers post mortem – The SuperFan (2020, August 11)

While we can’t and shouldn’t make long term projections based on the series loss to Chicago, or ridiculous characterizations of the team and individual players, we can still highlight the actual results to see what the Oilers did well and where they struggled. Long-term projections that inform the decision-making process around roster construction always needs to rely on larger sample sizes. In this case, focus on the 2019/20 regular season results rather than four games in August.

For now, I want to focus on the results from the qualifying round.

The Oilers were outscored 16-15 by the Blackhawks, scoring five goals on the powerplay and allowing four shorthanded. At even-strength, they scored 10 goals, a rate of 3.28 goals per 60, which would rank them 6^th among all teams, and 4^th among teams that were playing in a qualifying round. Unfortunately, they also allowed 12 goals at even-strength, a rate of which was second highest among all teams.

The Oilers powerplay was solid, scoring five times in just under 30 minutes of ice time, a rate of 10.38 goals per hour. That’s just a hair below where they were in the regular season when they finished with a rate of 10.64 goals per hour – an outstanding rate which had them first in the league. They generated 58.13 shots per hour against Chicago, well above the regular season league average rate of 53.62 (over the last three seasons), which is around where they were in the regular season. And they converted about the same percentage of shots into goals, posting a 17.86% shooting percentage, which is slightly below their ridiculous 20.27% shooting percentage from the regular season.

The Oilers penalty kill on the other hand, was not as good as it was in the regular season. Allowing four goals in just under 30 minutes translates to a rate of 8.56 goals against per hour, well above their regular season rate of 5.15, second best in the league. A big reason for this was the amount of shots the Oilers allowed, one of the highest among all playoff teams with 64.21. The Oilers goaltenders were fine shorthanded posting a save percentage of 86.67%, which is right around the regular season league average rate of 86.59%. This was however below the Oilers league-leading save percentage of 90.61% in the regular season – not entirely surprising considering both goalies posted league average save percentages in their careers prior to this season.

Where the goalies really let the team down was at even-strength, as the Oilers posted a team save percentage of 85.88% – worst among all teams competing in the playoffs. What’s especially frustrating is that the Oilers did a pretty good job at controlling the flow of play, posting a Corsi For% of 53.52% over the four games and out-chancing the Blackhawks posting a Fenwick For% of 52.91%. And the Oilers did a decent job limiting the shots against (28.87 per hour, 7^th best among all teams) and unblocked shot attempts against (a proxy for scoring chances) with 38.45 per hour, 6^th best among all teams. Yes there were defensive breakdowns and missed assignments in their own zone, but the goaltending was by far the biggest issue allowing goals from low-danger areas of the ice.

Worth noting too that the Oilers goaltending at even-strength was one of the worst in the league during the regular season, ranking 25^th overall with a 91.23% team save percentage. Among 54 goalies who played at least 1,000 minutes at even-strength during the regular season, Mike Smith ranked 52^nd with a 90.00% save percentage and 53^rd when it came to goals-saved above average (GSAA) with -16.26. Koskinen was much better and closer to league average levels ranking 21^st among the group with a 92.40% save percentage and 20^th in terms of GSAA with +4.53. We knew this going into the playoffs, making the decision to start Smith in game one of the series even more perplexing.

What’s also interesting is how the Oilers forward group did in the qualifying round, with coach Tippett electing to run McDavid and Nugent Hopkins as a pair on one line with rotating wingers, and Draisaitl and Yamamoto as pair on another line. Observers are fair to question why Tippett didn’t unite the trio of Draisaitl, Nugent-Hopkins and Yamamoto who posted a ridiculous 77.78% goal-share at even-strength, out-scoring opponents 28-8 in 317 minutes of ice-time in the regular season, but only played a few together against Chicago. But digging into the numbers a little more, it’s difficult to criticize Tippett when his lines actually did well against the Blackhawks, posting strong shot-share metrics, including the bottom six forwards which I was initially very skeptical about.

Metrics	Draisaitl + Yamamoto	McDavid + RNH	Bottom six
TOI	42.43	41.57	59.05
TOI/GP	10.61	10.39	14.76
Corsi For%	49.18	56.33	56.33
Fenwick For%	52.65	52.95	56.16
Expected Goals For%	60.03	57.86	59.97
Goals For%	33.33	37.50	50.00
Sh%	8.95	12.52	2.93
Sv%	81.04	78.37	95.94

With McDavid and RNH on the ice, (no Draisaitl or Yamamoto), the Oilers posted a Fenwick For percentage of 52.95% in about 42 minutes of ice time. And they scored 4.31 goals per hour, which is well above McDavid’s on-ice rate from the regular season (3.52 goals per hour). But due to the goaltending, and a 78.37% on-ice save percentage, they were outscored by Chicago posting a goal-share of 37.50%.

Similar thing happened when Draisaitl and Yamamoto were on the ice without McDavid or RNH. The Oilers posted a Fenwick For percentage of 52.65% in about 42 minutes of ice time with them on the ice, but only came away with a 33.33% goal-share. They struggled to score, posting a slightly below average on-ice shooting percentage of 8.95%, but it was really the goaltending that sunk them as their on-ice save percentage was 81.04%.

And in about 59 minutes of ice time at even-strength without McDavid, RNH, Draisailt or Yamamoto the Oilers bottom-six forwards posted a Fenwick For percentage of 56.16%, which is just outstanding. Unfortunately, the best they could do was score one goal. The key takeaway from all of this is that the Oilers were spending more time with the puck and keeping play in the offensive zone – all the things teams need to do to give themselves a chance to out-score opponents and win games. Reviewing the underlying shot-share metrics, it’s hard to criticize how Tippett constructed his line combinations.

While we can’t project much from the four game-series, it’s still important to look at the actual results and the underlying on-ice metrics to gauge what went well and what didn’t. It’s easy to point to the lack of “intensity”, and “leadership” and develop narratives about the Oilers needing to “learn how to win”. The challenge is to look past the noise, identify what the key issues were in the playoffs and the regular season, and act on actual facts based on coaching tactics, player performance and numbers when making roster decisions this off-season. This requires effort and courage and sound decision-making processes – and hopefully Oilers management is up for the challenge.

Data: Natural Stat Trick

Also posted at The Copper & Blue.