Your most valuable learning data is hiding in plain sight

A topic that strikes fear into the hearts of so many L&D professionals is data. 

Most of us went into L&D because we enjoy helping people and seeing them improve over time – but very few of us come from data backgrounds. And that’s OK! Nobody is saying we need to become statisticians overnight – but we do need to a) get smarter about the numbers we’re looking at, and b) know how those numbers translate into business impact.

We know, we know – vanity metrics are easy to collect. Every single LMS reporting dashboard will entice you with a host of tempting numbers. It feels good to tell your stakeholders that LMS logins are up 20% since the last quarter, or that 90% of salespeople have completed their sales training… but to really show the value of L&D, we need to move beyond this and start digging deeper into the numbers that really show impact.

The good news is that everything you need to measure already exists! You just need the right tools and techniques to access all that juicy data… and we might just have the solution for you.

The illusion of success

We’ve already talked about the problem with vanity metrics – primarily that they show engagement, not real performance improvement – but how do we get past the illusion of success and start measuring real success?

When we’re looking at what we’re measuring, there are two key questions to ask ourselves:

1. What changed as a result of this learning?
2. Are people doing things differently or better afterwards?

If the data we’re looking at can’t answer either of these questions, we know it’s most likely a vanity metric. For example, a 15% month-on-month increase in course completions can’t tell us what changed, and it can’t tell us how the learning has changed the way people work. In contrast, something like manager observations can indicate real behaviour change, which gives us a much more useful insight that what we’re doing is working.

The data you’re probably overlooking

Some of your richest data sources are hidden in plain sight. However, it’s probably not where you’d usually think to look, so it can be easy to miss the metrics that tell the whole story about your learning impact.

Useful learning data should be:

• Actionable – helping you iterate and personalise learning
• Linked to meaningful outcomes – such as performance, behaviour or customer experience
• Ongoing – it should show that learning is a process, not a one-off event (in other words, a continuous learning culture)

In particular, you should be looking for indicators that learners are applying knowledge, building confidence and contributing more meaningfully to their roles, all of which should result in better performance against your business objectives.

Just some of the hidden data sources you could be missing include:

• Manager feedback and observations
• Peer-to-peer sharing and conversations
• Team meetings
• Questions asked in Slack/Teams
• Repeated content visits or searches
• Self-reflection exercises
• On-the-job performance data
• Customer ratings and feedback

See what we mean about ‘hidden in plain sight’? You have access to all this contextual, behaviour-based information already – now it’s just a question of knowing how to measure it, and then actually doing it.

How AI helps you gather meaningful learning data

Measuring actual skills, not just content consumption, is a true game changer for L&D teams. Skills tell us what someone can do now, not just what they’ve engaged with.

For instance, your most prolific learner according to your vanity metrics might spend three full hours engaging with learning content a week. But how does this translate into performance? Has their performance improved alongside their learning hours, or are they still not showing any measurable growth?

What makes skills a scary thing to measure for L&D professionals is that they’re dynamic, and they don’t always fit neatly into job roles. It’s not just graphic designers who can benefit from improving their design skills, and it’s not just public-facing employees who will benefit from better communication.

It’s also about how skills are acquired too. It’s not just through formal learning – a lot of the time, skills are picked up and refined in the flow of work.

The role of AI in skills measurement

Measuring skills can be tricky for the average L&D team. After all, there’s only a finite number of hours in the working week, and you can’t be watching every employee like a hawk to monitor their skills development.

That’s why AI is L&D’s secret weapon for measuring and analysing skills development. With the right tools, AI can make skills measurement faster, smarter and easier by:

• Analysing behaviour and content interactions to infer skills, not just engagement
• Detecting patterns across tools, conversations and real outcomes
• Building evolving skill profiles for individuals, teams and departments
• Suggesting personalised next steps based on identified skills gaps

AI does all this without requiring additional manual effort from your L&D team, and brings visibility and structure to what have typically been considered intangible skills. You can’t really measure improvements in empathy from elearning course completions, but you can make observations in meetings and real conversations between colleagues, which is where AI really comes into its own. A manager can’t be closely observing all of their employees at all times… but AI sure can.

It’s time to start thinking differently about skills

If L&D is ever going to move beyond vanity metrics, it’s time for us to shift our focus to what’s real, relevant and useful. Those won’t always be the easiest things to measure, but they hold the key to L&D getting our well-deserved seat at the table and securing the budgets we need to make even bigger impacts.

That means tying learning to business impact, capturing learning as it happens (i.e. in the flow of work) and using AI to surface patterns and early improvements that we simply can’t track manually. 

There’s also a lot to be said for L&D working more closely with team managers to tap into their observations to help build up a holistic understanding of each employee as a person, not just as a series of course completions or workshop attendances.

The very good news is that we already have most of the data we need. It’s now time for us to start asking the right questions, looking in the right places and using the right tools to tap into the data that really matters.