Is your training program effective? How can the program be improved? Did the program achieve the desired results at the lowest possible cost?
Questions corporate trainers puzzle over all the time, as it's difficult to come up with concrete, measurable answers. The underlying point is it's tough to measure the return on investment for all of the time and money organizations spend on training.
Given the large expenditures for training in many organizations, it is important to develop tools that will help companies answer these questions and improve the measurement of training effectiveness. These tools need to provide both a methodology to measure, evaluate, and continuously improve training, as well as the organizational and technical infrastructure (systems) to implement the methodology.
Unless a training program exists simply for the sake of training, results should be measured, and measurements should include business performance data, not just training data. Including selected metrics, such as sales, customer satisfaction, workplace safety, productivity, and others into a reporting strategy can help demonstrate where training has increased revenue or decreased costs. Measurements that consider performance improvements can provide a benchmark for training effectiveness. After implementing a training initiative or changing an existing program, an organization can observe and record a change in performance. To evaluate retention rates, there should be a lag between the training and these behavior measurements.
Many organizations are unable to evaluate their training programs effectively because they lack the tools to collect the data to make higher level evaluations. Much of the data needed to bridge the gap between training and performance exists in many organizations. Individual performance data exists in performance management systems. Organizational data exists in marketing, sales, and financial systems. Bridging this gap requires a technical infrastructure and reporting strategy that minimizes the administrative effort needed to collect and analyze the training and performance data together. Business intelligence systems can connect the training data that an LMS provides with these other enterprise metrics to arrive at an indicator of how well learning transfers to business results.
Why are organizations still unlikely to comprehensively and effectively evaluate training programs? System integration, one common point of failure, is critical. Many LMS vendors with a history as product companies have limited expertise in system integration that extends beyond learning systems and databases. Successfully managing performance-based training evaluation, however, requires expertise in data management and warehousing, a variety of corporate systems and databases, analytics, and web-based application development.
However, simply measuring and reporting on metrics may not be enough to ensure that training improves performance at the enterprise level. To ensure traceable performance improvement, training professionals might borrow some ideas from the manufacturing industry, where Six Sigma - a continuous improvement strategy employed largely by the automotive and manufacturing industries - and other statistical methods are used to continually improve processes and could be used by training professionals to evaluate learning.
You might ask, "Why bother evaluating learning and knowledge transfer? If you evaluate organizational results, you will know if the training was ultimately successful. The problem with this assertion is that you don't really know if those results are due to training or to an intervening factor. Additionally, training does not always produce the desired organizational results. It is possible to gain important organizational knowledge by finding the causes of failed training.
A Six Sigma for training would use cross-functional data to build a learning model that supported valid inferences about training effectiveness. Training professionals would know where to improve and how to allocate resources and effort to maximize the impact of learning on business results. This type of model must take into account the emerging body of knowledge on transfer of training, which suggests a number of important propositions and conclusions. For example the transfer "climate can have a powerful impact on the extent to which newly acquired competencies are used back on the job. Delays between training and actual use on the job are directly related to skill decay. Social, peer, subordinate, and supervisor support all play a central role in transfer. And finally, it is possible to design intervention strategies to improve the probability of transfer. All of these intervening factors affect the results of any given training program, and there are potentially other factors.
Cross-Functional Reporting
A reporting and data management strategy that focuses on the LMS as the foundation only compounds the system integration challenges that make performance-based training evaluation unmanageable. Instead, the organization should adopt a cross-functional corporate reporting and data management strategy. A cross-functional system based on a corporate reporting strategy can address both the technical and analytical challenges of effectively evaluating training against business results. The features of a cross-functional reporting include:
" Independence from LMS
" Integration with business systems across the enterprise
" Alignment to individual and organizational performance
LMS independence
A cross-functional reporting system for training should not be locked into a single LMS platform. By utilizing a generic framework, common LMS data should map to variables in the learning intelligence system. Typically, an organization will feed performance, job code, certification, and other corporate data into the LMS reporting system. By adding a cross-functional system between the LMS and other corporate systems, the organization only needs to update one data connection if the LMS changes.
Business system integration
As a broker for business intelligence throughout the organization, the reporting system needs to aggregate the data from multiple corporate systems. If assembling information is too cumbersome and time consuming and the data is outdated or not even correct, the system cannot enhance evaluations by combining training with other business data.
If the organization has a corporate data warehouse, the learning management system can push data in its system into a consolidated source. Any corporate reporting system can then access this learning data, combine it with other business data, and make advanced ROI calculations. Although integrating multiple data sources can require significant integration effort, the organization gains greater control over its learning and business data.
Business system integration allows the organization to leverage training and business data together in a context-sensitive manner, linking data to an enterprise or departmental portal or a reporting tool used by a particular decision maker. Portal applications and reporting tools allow training professionals to make more informed decisions when designing and implementing a program. Some examples of how data can be combined for different decision-makers and purposes include:
" A training scorecard that evaluates programs on ROI and other performance metrics. The scorecard becomes a much more powerful tool to manage interdependent activities and performance if it provides an easy to-use "drill-down capability that provides supporting data, so training professionals can identify how cost and performance results contribute to a training program's score.
" Sales, manufacturing, distribution, customer service, and other scorecards that provide metrics specific to that domain, including training and other relevant operational and performance metrics
" Predictive analytical tools that allow organizations to perform what-if scenarios and make resource allocation decisions that maximize desirable organizational performance
Alignment to individual and organizational performance
What differentiates a cross-functional reporting strategy from most LMS-based reporting approaches is the ability to align training with performance objectives for the entire extended enterprise, including individuals, the organization, and its business partners. The cross-functional approach can combine the course completion, certification, and assessment scores of the LMS with the evaluation and competency data in a performance management system. Achieving this alignment depends on statistically-validated learning analytics that help an organization understand how training, individual behavior, and organizational performance are linked.
For example, one automotive dealership study contained a detailed analysis of an evaluation system that a company used to measure dealer compliance to the manufacturer's standards. The evaluation system measured over a hundred items, including training and a wide range of other factors (cleanliness, size of signs, whether the waiting room had fresh coffee). The study combined this evaluation data with performance measures, including unit sales, part sales, market share for two different vehicles, and customer satisfaction. The results of a statistical analysis identified certified training as an item that had a positive impact on the performance measures; however, non-certified training did not appear to affect performance. In this case, certification could demonstrate a measurable ROI based on the cost of the certified training programs, and on the business result improvement, as quantified by the statistical analysis.
Conclusion
In the education and training field, it is not unusual to hear criticisms of ROI measurement as an instrument of justification, rather than of performance improvement. However, by using a robust cross-functional reporting system and applying statistical methodologies, ROI calculations can support continuous improvement of instruction just as they do in manufacturing and other corporate operations. By selecting those measurements that can support valid inferences about the effectiveness of programs, learning and training professionals can know where to improve and how to allocate resources and effort. Every training program's influence on business results would improve.
