Replying to LO25269 --
Rick
>When I first made the career change from high-tech management to
>organizational learning consultant, I was taught by my clients this four
>level scheme for thinking about evaluation of training and development:
Your clients were tracking the Kirkpatrick's Four Levels of Evaluation.
Here's a typical discussion:
Demonstrating the Effectiveness and Value of Technical Communication
Products and Services: A Four-Level Process
by Saul Carliner
Kirkpatrick's Four Levels of Evaluation
Level I: Reaction
Assesses participants' initial reactions to a course. This, in turn,
offers insights into participants' satisfaction with a course, a
perception of value. Trainers usually assess this through a survey, often
called a "smiley sheet." Occasionally, trainers use focus groups and
similar methods to receive more specific comments (called qualitative
feedback) on the courses. According to the TRAINING magazine annual
industry survey, almost 100 percent of all trainers perform "Level I"
evaluation.
Level II: Learning
Assesses the amount of information that participants learned. Trainers
usually assess this with a criterion-referenced test. The criteria are
objectives for the course: statements developed before a course is
developed that explicitly state the skills that participants should be
able to perform after taking a course. Because the objectives are the
requirements for the course, a Level II evaluation assesses conformance to
requirements, or quality.
Level III: Transfer
Assesses the amount of material that participants actually use in everyday
work 6 weeks to 6 months (perhaps longer) after taking the course. This
assessment is based on the objectives of the course and assessed through
tests, observations, surveys, and interviews with co-workers and
supervisors. Like the Level II evaluation, Level III assesses the
requirements of the course and can be viewed as a follow-on assessment of
quality.
Level IV: Business results
Assesses the financial impact of the training course on the bottom line of
the organization 6 months to 2 years after the course (the actual time
varies depending on the context of the course).
For many reasons, Level IV is the most difficult level to measure. First,
most training courses do not have explicitly written business objectives,
such as "this course should reduce support expenses by 20 percent."
Second, the methodology for assessing business impact is not yet refined.
Some assess this measurement by tracking business measurements, others
assess by observations, some by surveys, and still others assess by
qualitative measures. Last, after six months or more, evaluators have
difficulty solely attributing changed business results to training when
changes in personnel, systems, and other factors might also have
contributed to business performance.
Despite these difficulties in obtaining a measure, over 50 percent of
organizations perform this type of evaluation on 50 percent of their
courses (TRAINING, 1995).
Level IV evaluation is assessment of quality. It does so in financial
terms, a perspective different than that of the evaluations at Level II
and Level III.
Trainers' Use of the Kirkpatrick Model
Trainers widely follow this model (although, admittedly, not every
organization performs each type of evaluation for every training course).
Nearly all organizations perform Level I and II evaluations on most of
their courses (TRAINING Industry survey). Many organizations perform Level
III and IV evaluations on some of their courses. The framework is followed
widely enough that, at conferences and in journals and magazines, trainers
refer to these evaluations in a short-hand language: Level 1, Level 2,
Level 3, and Level 4.
Regards, Gordon Housworth
Intellectual Capital Group LLC
26775 Crestwood
Franklin, MI 48025
248.626.1310
ghidra@modulor.com
--Gordon Housworth <ghidra@modulor.com>
Learning-org -- Hosted by Rick Karash <Richard@Karash.com> Public Dialog on Learning Organizations -- <http://www.learning-org.com>
"Learning-org" and the format of our message identifiers (LO1234, etc.) are trademarks of Richard Karash.