Your Mandatory Learning has a Completion Rate Of 99%. What Does That Actually Prove?

A completion rate tells you that people reached the last page. Knowing whether they understood the policy takes a different kind of evidence.

Tara Cull, Learning Experience Designer for the Australian Football League sat down with us to share how she went and gathered it.

Key summary:

• A high completion rate tells you people finished mandatory learning, not necessarily what they understood, retained or can apply.

• The AFL introduced pre-assessments to respect what learners already know and focus their time on the areas where they need a refresher.

• Rather than simply presenting information and testing it at the end, the AFL is designing more active learning, where people make decisions, reflect and practice what they would do in real situations.

• Tara built stakeholder buy-in by piloting the approach, gathering learner feedback and using that evidence to make the case for change.

You have built the modules, chased the last few stragglers, and the report has come back green. It comes back green every year. Nobody in the business is asking you to fix anything, which can be its own kind of problem, because you already know that finishing a module and learning something are often different events.

At the Australian Football League (AFL), that report came back at a completion rate of 99%. Learning Experience Designer Tara Cull rebuilt the training anyway. At the Sydney HR + L&D Tech Fest she explained why.

We opened her talk by asking the room two questions: who here has clicked through a mandatory compliance module as fast as they possibly could? Who here has had to build one? Most hands went up for the first. Just as many stayed up for the second.

Two things stand between that room and better training, and they pull in opposite directions.

Above: Tara Cull, Learning Experience Designer from the AFL presenting with Lucy Luo, Chief Revenue Officer at Chameleon Creator on building a learning culture. 

The Number That Can’t Tell You Anything

Completion tells you that someone finished the module. On its own, what it can't tell you is whether they understood the content, retained it, or could apply it when it matters.

The AFL's 99% makes the point. Asked about the missing 1%, Tara's answer was that it is mostly people on parental leave or who have left the business. There is no headroom in that number and nothing in it to fix. What it also cannot tell you is whether a player sitting the same refresher for the fifteenth year learned anything more each time.

“I certainly don’t retain anything if it’s information, information, information, quiz. I like to be tested along the way,” shares Tara.

The Fix Your Stakeholders May Block

The obvious repair is to stop making people sit through content they already know. Put a pre-test in front of each topic, three or four questions, and let a pass move them on while a wrong answer routes them back through the learning.

Above: The AFL’s pre-assessment let’s learners answer three to four questions at the start of each topic. If they get them all right, they can skip or choose to review the content.

Which is where the second problem starts. To a management team, skipping content often reads as ‘learning’ less. An audience member at the session described exactly that wall: they use Chameleon Creator, they want to move to using pre-assessments, and their senior leadership will not approve it. If people don’t click through the content, how will they have learnt it?

So, you are stranded. The number you report every year proves nothing, and the change that would help can’t get signed off.

What Tara Did Instead

Three decisions had to be made and, more to the point, defended. Gate the content behind a pre-test. Build bespoke rather than buy off the shelf, which on a team of one is a resourcing call as much as a design one. And put feedback inside the modules themselves — which is the decision that turned the first two from preferences into arguments.

She ran it as a pilot rather than a proposal. A cohort of learners got the first pre-test, the feedback came back through the module, and that feedback went to the executive team. With their backing, the approach moved into employee training. What learners said was consistent: the pre-test gave them their time back, and it let them find out what they didn’t know.

“There’s no point in improving the learning if you’re not gathering evidence as to whether or not learners are actually engaging in a different way with it,” says Tara.

Above: Learners can give feedback inside the module, which informs design decisions and provides evidence of engagement.

One caveat Tara states herself: “the questions have to be robust, because easy questions with easy answers prove nothing.”

The feedback that came back was not what anyone had budgeted for. Learners were thanking her for honouring their time, and in the same breath the data showed they were spending more of it in the module, not less. Committing to an answer you might get wrong takes longer than scrolling past a paragraph. What changed was where the time went — out of the content they already knew, and into the content they didn't.

Which puts average completion time in the same category as completion rate. It moved in the ‘wrong’ direction because people had stopped skimming.

What That Looks Like for the AFL

The policy refresh module covers the annual set of material staff need to get right, from gambling policy through to people policies. It runs on the topic-gated pre-test above, and the objection writes itself — aren't they just skipping? Tara's answer is that the pre-test is what makes the learning active in the first place.

People have to do something before they are told anything, and passing is the reward. The course also integrates Chameleon Collect, so learners can respond inside the module, which is usually where a genuine question about a policy surfaces.

Tara also shares there is one non-negotiable across everything she builds: “it needs to look like it’s made by the AFL, for the AFL.”. The photography and video use real staff and players, and global theming keeps every module consistent, so the people being trained recognise themselves in it rather than sitting through something generic. 

Above: The AFL use photos and videos of real staff and players to make the training authentic and relatable.

When Your Stakeholders Say No

Tara’s answer to the audience member had three parts. Flip the question back: if you hand someone the information and quiz them at the end, what evidence do you have that they engaged with any of it? Be rigorous in your choice of pre-test questions. Have three or four questions per subject, so from the learner’s perspective it feels like one test with content integrated throughout.

Then the practical part, which is how you get a first module built rather than a policy agreed.

“Start with an SME that will trust you and allow you to skip. Start with simple,” advises Tara.

When Active Participation is Now the Goal

Completion wasn’t the goal for the AFL, active participation and learning was. For Tara, Chameleon Creator is where she could put this to practice: pre-assessments, AFL-branded scenarios, reflection activities, and gathering learner feedback. The learners noticed.

The evidence layer was feedback captured inside the module, tied to the specific content it is about, in a form she was able to put in front of an executive. That is Chameleon Collect  and Chameleon Analytics.

As a frequently requested module in Chameleon Creator, users can access the pre-assesment template which provides a springboard for creating your own.

Nearly every hand in that room went up twice. The people who build compliance training are also the people who sit through it, so the room already knew what better would feel like. What it didn’t have was the evidence to ask for it.

One of those is a design problem. The other is an evidence problem, and it is the one worth solving first.

Tara joined Chameleon Creator on stage as part of the Sydney HR + L&D Tech Fest. If you want to see what this looks like in the authoring tool, book a walkthrough.