If one looks at any facet of business the concept of ´return on investment´ (ROI) is always a relevant business topic. ROI can have many connotations depending upon the users perceptions and motivations. In reality, ROI is really a measure of perceived value. Value can be different for different stakeholders. For example:
- An organization provides training to a group of participants. This person wants to know the satisfaction levels of the participants.
- A course designer creates an e-learning module. This person wants to know if the module did its job in transferring new knowledge or skill to the learner.
- A business unit manager sends two employees to training. This person wants to know the impact the training has made on the job.
- A senior executive measures performance by the business objectives that drive the company. This person wants to know the degree to which training has helped drive key business results.
- The finance group manager views benefit relative to cost on every decision. This person would want to know the benefit to cost ratio, payback period and ROI percentage from training.
-
Value is inherent in each of the aforementioned examples. So the first question one should ask when contemplating an ROI solution is ´How does my user of this information define value?´ Having said that, there is strong need to ensure that one has a balanced approach to learning measurement. A balanced approach should be able to accomplish all stakeholders perceptions of return on investment.
The best approach to accomplish this balanced scorecard is the legendary and time-tested Kirkpatrick Model, with the additional fifth level added by Dr. Jack Phillips.
Level One - Reaction
Evaluating reaction is the same thing as measuring customer satisfaction. If training is going to be effective, it is important that students react favorably to it.
Level Two - Learning
Level Two is a ´test´ to determine if the learning transfer occurred. It is important to measure learning because no change in behavior can be expected unless one or more of these learning objectives have been accomplished.
Level Three - Behavior
Level Three evaluates the job impact of training. What happens when trainees leave the classroom and return to their jobs? How much transfer of knowledge, skill, and attitudes occurs. In other words, what change in job behavior occurred because people attended a training program?
Level Four - Results
Level Four is the most important step and perhaps the most difficult of all. Level Four attempts to look at the business results that accrued because of the training:
- Use a control group if practical
- Allow time for results to be achieved
- Measure both before and after the program, if practical
- Repeat the measurement at appropriate time
- Consider costs versus benefits
- Be satisfied with evidence if proof not possible
-
Dr. Phillips outlines his approach to Level Five in his book ´Return on Investment in Training and Performance Improvement Programs´:
- Use a control group, if practical
- Allow time for results to be achieved
- Determine the direct costs of the training
- Measure a productivity or performance before the training
- Measure productivity or performance after the training
- Measure the productivity or performance increase
- Translate the increase into a dollar value benefit
- Subtract the dollar value benefit from the cost of training
- Calculate the ROI
-
Learner Based. A measurement model that captures data from training participants at two distinct points during the learning process. The first point is directly after the learning event where the main measurement focus is on Kirkpatrick´s Level I - and Level 2 to gauge satisfaction and learning effectiveness. Because a high response rate is often achieved here, it is also critical to capture indicators for advanced levels of learning such as Level 3 - Job Impact, Level 4- Business Results and Level 5 ROI. These indicators are in effect forecasting or predicting the future impact the training will have on the participant and the organization. A second data collection point is a follow up survey conducted a period of time after the participant has been back on the job. This survey is meant to true up the forecast and predictive indicators of Levels 3, 4 and 5 by gathering more realistic estimates now that the participant is back on the job. The approach is low cost if one leverages standard data collection instruments across their training and utilizes technology and automation to capture, process and report the collected data.
Manager-Based: This method has the same data collection points as the learner-based solution but adds a manager-based dimension. The manager of the participant attending training is another important data point. They can be sent an evaluation instrument timed when the participant receives a follow-up. The manager survey focuses on Levels 3, 4 and 5 of the Kirkpatrick and Phillips models therefore getting estimates surrounding job impact, business results and ROI from the manager´s perspective. Due to the increased effort it takes to conduct and analyze manager surveys the cost and time to measure at this level is higher than the Learner-Based approach. But, with automation and technology to facilitate the dissemination, collection, processing, and reporting of the data, the cost and time can be minimal.
Analyst-Based: This approach uses significantly more comprehensive post event, follow up and manager surveys it also uses other analytical tactics that go beyond surveying. For example to analytically measure Level 2 - learning effectiveness a detailed test is designed and administered to participants. Due to the time commitment of conducting a significantly detailed data collection and analytical exercise the Analyst-Based approach is only used for about 5% of all training programs in the organization. Typically these programs are the more strategic or visible and have the budget to afford a more costly and time-consuming measurement exercise.
So far we have discussed the methodologies and approaches to get reasonable data. This section discusses the output of an ROI approach.
If one buys a computer for $3,000 the expectation is that the company will get $3,000 of value out of the computer. The computer may help a salesperson increase sales or help a plant floor operator increase quality but the goal is to improve the users´ job performance through the technology.
Compare this analysis to a person. If the fully loaded salary of a newly hired employee is $50,000, the organization paying that expense expects $50,000 of value from the employee. This value could come from their contributions in one or more key business objectives such as sales, quality, productivity, cycle time, customer satisfaction, etc.
Now, say in our computer example our IT department added a $500 upgrade to it. The upgrade is intended to make the machine faster, more resistant to bugs, and more accurate in its processing. The business result is more productive employees, higher quality and reduced cycle-time for a user of the computer. The expectation is that the $500 spent on the upgrade will result in at least $500 returned in various benefits.
Compare this analysis with training. We use training to upgrade our people just as we add components to a computer to upgrade technology. Training and organizational development are proven tools to add knowledge and skills to our workforce. So, if an employee goes to a $1,000 training event over a week long period, the goal is that the employee will leverage the training to help achieve various business results back on the job. Such results include increased sales, quality, customer satisfaction, productivity etc. The expectation is that the $1,000 spent on the training will result in at least $1,000 returned in various benefits.
Phillips´ measurement principles include estimation, isolation and adjustment. These are the cornerstones to monetizing a benefit and linking it to training.
Estimation is a process commonly used in business today. Sales people will estimate their future sales, accounting people will estimate the cost of a warranty or claim that is expected in the future. Can training personnel ask that participants estimate the job performance impact that a training program will have on their job?
An estimated increase could also be related to other factors such as a competitor going out of business or new company procedures, or an incentive program, etc. So, estimates of performance change need to take into account many factors, not just training. So the next step is to isolate the estimated increase in performance to just training.
Finally, because participant estimation and isolation is participant driven one must adjust any resulting ROI calculation for the estimate for conservatism and bias Studies suggest that respondents tend to over estimate by a factor of 35%.
Taken together, the principles of estimation, isolation and adjustment form a powerful model in tabulating a systematic, replicable, and comparable ROI model for human capital.
The result of the process is a monetized benefit factor, that when multiplied by the salary (i.e. the human capital) yields a monetized benefit from training.
This article contains extracts from ´The Human Capital Return on Investment´ a paper by Knowledge Advisors and Global Learning Alliance. For a copy of the complete article please contact: Global Learning Alliance, www.glaworld.com.
This excerpt from a White Paper by Global Learning Alliance in association with Knowledge Advisors was reprinted with permission.
