5
If Completion Rates Don’t Prove Anything, What Does? Why Security Programs Stall Without Real Behavior Change
Why “100% completion” doesn’t mean your training worked and what to measure instead to prove real behavior change.
Show Notes
In this episode, host Eliot Baker sits down with Maxime Cartier, Head of Human Risk at Hoxhunt, to unpack what organizations are getting wrong about measurement and what the most mature programs are doing instead.
Drawing from Maxime’s recent insights at the SANS Security Awareness Summit, this conversation cuts through outdated KPIs and explores what actually signals behavioral change. You’ll hear what practitioners are building in the real world, how to bring leadership along without losing them in complexity, and how to measure success beyond tick-box numbers.
This isn’t theoretical - it’s tactical guidance from a field that’s evolving fast.
What you’ll learn in this episode:
- Why 100% training completion doesn’t mean behavior has changed
- How to spot “vanity metrics” and what to replace them with
- Why security programs are borrowing measurement models from public health and road safety
- What early signals suggest real change (even before risk metrics improve)
- How to make behavioral metrics land with your board, not just your CISO
Timestamps:
(00:52) Maxime Cartier's Conference Insights
(02:16) The Problem with Training and Behavior Change
(03:40) The Misleading Nature of Completion Rates
(07:05) Advanced Metrics and Dashboards
(12:48) Behavioral Change and Public Health Parallels
(16:59) Early Indicators of Behavior Change
(19:39) Moving Beyond Compliance: Internal Buy-In
(35:43) The Power of Storytelling in Metrics
Resources:
- Our guide to the essential metrics you should be measuring: https://hoxhunt.com/blog/4-essential-phishing-metrics
- Hoxhunt's HRM playbook: https://hoxhunt.com/guide/human-risk-management-playbook
- Bird & Bird case study: https://hoxhunt.com/case-studies/bird-bird-cybersecurity-rules-in-favor-of-the-hoxhunt-human-risk-management-platform
Host links:
Eliot Baker: https://www.linkedin.com/in/eliotebaker/
Maxime Cartier: https://www.linkedin.com/in/maximecartier
Full Conversation Breakdown
In this episode of the All Things Human Risk Management Podcast, host Eliot Baker is joined by Maxime Cartier, Head of Human Risk at Hoxhunt, to tackle one of the biggest frustrations in security awareness today: “We’ve got 100% completion - but behavior hasn’t changed.”
Despite hitting compliance targets, organizations still face user-driven security incidents. Why? Because the wrong metrics are being optimized. Together, Eliot and Maxime break down how completion rates became the industry’s go-to vanity metric and what teams can do to shift toward tracking real behavioral outcomes.
The cosmetic KPI problem
Many security programs measure training completion, not capability. Maxime explains why “100% completion” is often used as a proxy for success and why it fails under pressure.
“Executives ask for numbers that look clean. Completion is clean. But it doesn’t prove you’ve reduced risk.”
Why training ≠ behavior
Just because someone finishes training doesn’t mean they can recognize or respond to threats. Maxime walks through examples where phishing simulation results stayed flat despite completion rates being perfect.
“Training needs to go beyond instruction - it needs to build detection reflexes. That’s not what most programs measure.”
Redefining impact in human risk
The episode explores how advanced programs are measuring meaningful behavior change: detection speed, error recovery, response quality, and long-term retention.
Maxime shares what Hoxhunt has learned from working with some of the world’s most complex security teams.
“We’re seeing more mature teams adopt behavioral metrics borrowed from public health and behavioral science. They’re reframing the problem.”
How to get leadership buy-in
Switching from compliance-based metrics to behavior-based metrics isn’t always easy - especially when legal, audit, or the board is used to simple dashboards.
Maxime shares tactical ways to reframe the conversation internally and present behavioral data that resonates at the executive level.
“Don’t just show numbers. Tell stories that show progress. Tie it back to incident reduction and actual resilience.”
What success really looks like
The episode wraps with practical guidance: what to stop tracking, what to start measuring, and how to align KPIs with actual human risk outcomes.
“Completion tells you who watched the video. But your risk is in who clicks the link under pressure. That’s what you need to measure.”
See Hoxhunt in action
Drastically improve your security awareness & phishing training metrics while automating the training lifecycle.