Don’t Drink the Learning Evaluation Kool-Aid!

knowledge-retention

Learning practitioners are taught early, or should I dare say brainwashed, to believe the ‘essential’ four levels of evaluation. Many of us refer to these levels as the Kirkpatrick’s Evaluation Model and it has been a cornerstone in every learning event and also a foundation for many evaluation models that followed.

But let’s be honest, the unspoken truth is that the Kirkpatrick model is flawed. Yes, I dare say it out loud and may the learning gods, and some of my peers, strike me down. While you pick you jaw off the floor, the fact is that the evaluation method has some apparent issues.While the Kirkpatrick organization will not admit to this publicly (naturally, since it is the foundation of the revenue stream) they are attempting to ‘adjust’ it accordingly by repackaging it as the ‘New World Kirkpatrick’. This reminds us of an ‘All in the Family’ episode where Archie and Meathead ask the question about a product being new and improved asking what was wrong with the original one, was it old and lousy?new-and-improved

But I digress. Let’s review the four levels. Level one refers to learning satisfaction. Simply put, this is what learning practitioners refer to as the ‘smile sheet’. This learner feedback process asks everything from did the learning meet your needs to whether the lunch was adequate.

Level two speaks to learning retention or simply put, do you remember what you are supposed to remember? Often this is considered through some form of ‘testing’. While this is what many practitioners accept as learning success, the Kirkpatrick model assumes that if the learner remembers the knowledge they will naturally apply it to their job. I’ll revisit this logic shortly.

Level three is about changing the learner’s behavior or in layperson terms, skills application. This level is the first ‘holy grail’ for learning practitioners. The logic is that if the learner retains the knowledge from the initial learning process then their behavior will change and become more effective in their job. This sounds reasonable and correlates to Level four.

Finally, achieving level four for learning practitioners is similar to wining the Super Bowl. This level refers to the learning effort having an impact on business and performance objectives. What the Kirkpatrick model implies is that if learning practitioners are able to connect their efforts to this level the will gain the admiration of their business leaders. Essentially, this is the promise of demonstrating tangible results for your learning budget.

The Kirkpatrick methodology sounds logical and simple enough that learning practitioners are able to buy into the process but dig deeper an you will discover issues that undermine learning efforts.

To accept the premise of this post you must first accept that the role of learning in any organization is considered an internal business unit. Just like every other internal business activity whether it is accounting, marketing, or HR, learning is also held accountable to specific performance expectations for itself and how it contributes to organizational results. You don’t have to accept this premise. But if you don’t then you should also not question why your training budget gets reduced every year.

By accepting the reality that your learning efforts are part of the business and ultimately affects the business, hopefully positively, you begin to see learning from the perspective of your business leaders and business unit managers.

With that said, for any business level one and level two are essentially irrelevant. Think about it. Why would leaders care whether their employees like the learning event (level one)? It has no bearing on the business or expected results. Level one smile sheets exist for learning practitioners to prove that they are actually doing something that helps them to avoid getting fired from their job.

Level one smile sheets exist for learning practitioners to prove that they are actually doing something that helps them to avoid getting fired from their job.

Every learning practitioner has done this at least once. They wave their smile sheet results to their leaders hoping that this will validate their efforts, similar to a child seeking the admiration of their parent and trying to get their work put on the family refrigerator.

Don’t believe that Level two is any better. Like level one, your leaders could care less that employees actually can remember any of the skills they learned. Like the smile sheet learning practitioners are quick to fly their successful ‘test’ results in their leader’s faces. The problem with level two ‘learning’ retention is that, more often than not, they are inaccurate or invalid. Why? Essentially, practitioners ‘game’ results in their favor, the knowledge tested is often irrelevant to changing learner behavior, or worse, the skills tested are not applicable to their job. Whatever the reason, the practitioner’s goal is a futile attempt to prove to leaders that their efforts are close to being effective.

wrong-wayLevel two is as irrelevant for the business as is level one. What your leaders expect is that employees actually apply the skills on the job. Their logic, which many practitioners ignore, is that if an employee is applying a new skill or knowledge that improves their performance it will consequently improve the organization’s performance.

Fundamentally, leaders are concerned solely about level three and four. In reality, this all you should be concerned about as well. Regretfully for the Kirkpatrick model, there are still concerns that practitioners must be made aware. Even Kirkpatrick found flaws and hence, developed a ‘new world model’, but lets not get into that now.

At Level three the need to change behavior is not as relevant as the need for leaders to see the actual application of knowledge and skills. As any qualified psychologist will tell you changing human behavior is something that happens consistently over time and not something any type of training effort can accomplish successfully.

Simply, your leaders see level three evaluations as the vehicle to meet pre-established performance metrics and not necessarily to change employee behavior. The question we are asked from practitioners is, “how do we connect to level three expectations?” The answer is quite simple. First, don’t create new learning measures to prove your efforts are effective. Your leaders and business unit managers have their performance metrics already set. All you need to do is to partner with the business units, learn about their performance expectations, and then proactively work with them to conduct a needs assessment to determine the required skills that will help contribute to achieving their performance metrics.

Finally, level four is what every practitioner strives to achieve. Keep in mind that while level four is what your leaders expect they don’t expect every training effort to meet it. And for those initiatives that must achieve level four expectations you are not alone in your effort. You leaders don’t expect learning to be the sole hero. Recognize that when attempting to impact business results to take into account the involvement of other internal activities.

Your leaders will never believe that your ‘level 4’ achievement is only a result of your learning solution. It is a cross-functional effort so involving many internal business processes. So take credit when due but also, give credit to those that deserve it. This will build your business impact credibility and ensure sustainable leadership support for learning.

Finally, never, ever go to your leaders and refer to the Kirkpatrick four levels. They won’t understand what you are talking about and frankly don’t care about your evaluation methods. Just sayin’.

Advertisements

, , , , , , , , , , , ,

  1. #1 by Ben on November 19, 2015 - 9:37 pm

    Thanks for the well-thought out post. It does seem that the only level anyone in upper management understands or cares about is Level 1–Reaction. If they are paying for training, they want the trainees to be satisfied. Its also a chance to find out something about instructor behavior and competence and provide feedback they can use to improve their performance. Other than these hygiene-type outcomes, it doesn’t serve much purpose other than to remind participants that they are expected to apply what they learned on the job-which may be important, or may not.

    Testing mainly works to promote learning, effort to learn, or rehearsal of what was learned. It hardly evaluates the training.

    You point out that for other levels the measurements used should be based on per-established performance metrics, but it seems no one ever does this; the preferred method is to ask for follow-up feedback, not any kind of measurement. (Using Likert scales lets you pretend it is a measurement, but doesn’t make it so.)

    The major problem with evaluation is that it is done because you are supposed to. Nobody thinks about the purpose of the evaluation. A lot of times, it serves no practical purpose and there’s no reason to do it. If you can’t or don’t want to change things based on evaluation, they why evaluate? Oh, yeah, it lets you check the box.

    • #2 by Ajay M. Pangarkar CTDP, CPA, CMA on November 20, 2015 - 3:43 pm

      Hello Ben,

      Thank you for the reply of this post. Regretfully, I must agree to many of your points on because this is the reality of performance management in many companies. That said, for progressive business leaders, Level 1 is useless. The reality they really don’t care or have the ‘competitive’ time to care whether employees like the training or not. Yes, they do want them to go willingly and hope the session is conducive to learning. They do want to ensure that what employees learned is actually applied on the job (Level 3).

      One thing ‘we’ (T&D) fail to do is not focusing on the ‘end’ or rather what we refer to as Level 4. Rather than starting with L1, we need to start wit L4 which will lead to more effective job application result (L3) and allow designers and developers to build learning environments focusing on the rights elements (L2) ensuring that people actually will want to come to the training (L1).

      One note, while I agree that it is often a ‘check box’ for those in many organizations, it is not the case for many of the orgs I work with and the ones leading in their market space. They do make the connections to performance outcomes and their TD is fully integrated in the the fabric of the organization’s operational activities, rather than being an after-thought.

      Thank you again for the post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: