Still using completion rates as your main metric of success? The eLearning industry is fast moving to meet the technological advancements of the world, and one way that is being done is by using data-driven methods to prove ROI in L&D.
With the help of the likes of Chameleon Analytics, L&D teams across the world can understand their learners and content even more. But not only that, there becomes a plethora of tangible evidence of value to showcase to executives.
Learning Systems Manager at Specsavers, Chris Dwyer talks to Chameleon Creator about the bad rap completion rates have within the industry and whether it is uncalled for or maybe it is time for L&D teams to look beyond it to create learning solutions that positively impact the business.
Why Are Completion Rates Not Enough?
For years, completion rates have been the go-to metric for L&D teams. And that is not a bad thing at all. It is still a metric that can paint a picture but nowadays, it is just not enough.
Chris agrees with the sentiment, saying that completion rates have become the punching bag for people to measure the impact of learning programs.
“If learning of any shape or size is designed with proving capability and the right success criteria, a completion can mean so much more,” he says.
“I like to look at completion rates as helping us identify the people at the top of our funnel – the ones we need to track and understand the improvement in their overall capability, clarity or motivation… however that needs to be measured depending on the context.”
For organisations that value the learning type of ticking a box and no further questions are asked, completion rates will satisfy them. It’s easy to do what’s asked and no more. Looking at the retail sector, Chris says that the pace they move at gives learning teams no space to measure business impact over the long term and the only thing they have is completion rates.
But completion rates overall don’t paint the entire picture. And if you’re looking for positive business impact coming from eLearning, the ultimate measurement is behaviour change.
“At Specsavers, we’re lucky to have a talented team of creative individuals who understand this and create initiatives that are designed to demonstrate competence. In these instances, a completion means that a person has both displayed competence via assessments but also been judged to display competence by an expert. In this scenario, I feel great about measuring a part of impact via completions,” adds Chris.
In order to determine behaviour change, many more data-driven metrics need to come into the picture to ensure that it will happen.
The Holy Grail of eLearning Success
“Recently I was contacted by an eLearning vendor and checked out their website before I decided if I would like to hear the sales pitch. What do I see? Countless statistics about the number of people they’ve reached with their training, how quickly people completed their training and so on. My first thought? ‘So what?’. Why is this important? Did business results or behaviours change because of this?,” Chris tells.
In this day and age, completion rates can tell you the following: learners have displayed competence via assessments.
But that doesn’t tell you whether they have actually engaged with the content, whether the content stuck and if it will encourage change within them. These types of data-driven metrics can open a door to the holy grail of eLearning results.
“The holy grail is obviously some kind of business or people metric – sales, customer service results, reduction in costs or improved employee attribution,” says Chris.
Think of it like this, completion rates are your short-term leading indicators that can show how a learning program is performing. Behaviour changes such as an increase in sales are those long-term lagging indicators – what has actually changed.
“In large organisations an impact of this magnitude can take time to show so can be a little longer term,” adds Chris.
“Sometimes you need to take things into your own hands and assess a baseline capability yourself, then measure changes in the weeks or months after by reassessing capability.”
For a big retail chain such as Specsavers, Chris says that they rely on the likes of sales and customer service results, culture surveys, perception of behavioural change, increased productivity and cost reduction.
Completion rates are good, just not enough. But with data-driven metrics, you can end up mining into a gold rush, the actual tangible behaviour changes.
The Verdict?
As an industry, we may be knocking down the completion rate metric a little too hard, but there is no denying that there is more to the world of eLearning than a tick box activity.
“I wouldn’t call them outdated, I’d call them unfashionable,” says Chris.
“It’s not ‘cool’ to track completion rates anymore so it gets a bad rap, that being said, if we’re talking about measuring business impact, we do need to move to measuring more than just completion rates.”
With tools like Chameleon Analytics, you get more than just your leading indicators (your early signs of how your eLearning is performing), you’ll be able to get a glimpse into how your lagging indicators (the longer-term results of learning) are going to look.
