How do you choose the best survey instrument for your purpose? Here is a list of seven criteria we recommend:
1. The Tool Measures What Needs To Be Measured: What do you want the test to do? What types of decisions will it support? Hiring? Development? Organizational direction? In the simplest sense, a "good test" is one that does a good job at measuring something you are interested in measuring. How do you find a good test?
2. Reliable: There are two technical considerations in deciding whether an assessment does a good job. The first is its reliability as a performance predictor, also called its consistency. If you administer a typing test to a group of candidates on Monday, and then administer the same (or equivalent) version on Friday to the same group, you should get the same result unless the candidates brushed up on their skills during those few days. Barring the addition of training, any difference you find is the result of inconsistency in the test. Reputable test designers always report the level of consistency in their tests. There is no “magic number” to watch here, though, since there are several different methods for determining consistency. A "bad test" is one with more inconsistency than you can tolerate. (Interested readers should refer especially to the "error of measurement" section in any good statistics text.)
3. Valid: The second major consideration in determining a “good test” from a “bad test” is its validity. There are several ways to measure validity. In the job setting, validity usually means how well the test actually measures what it intends to measure. Employers use validation studies to prove test validity in their setting. Publishers can also provide validated instruments that will apply in general business settings. Reputable publishers are willing and able to provide in-depth information about their validation process.
4. Actionable: If you’re measuring what needs to be measured, another important question is whether you can use the results to take the kind of actions you hope to take. If you’re looking to determine training needs, you should use an assessment such as 360 feedback which is focused on behaviors that can be changed through training. If you’re looking to select people with good job fit, you should use personality testing. The results of personality tests are not usually actionable in a training environment, and vice versa. When it comes to assessment of the entire organization, you should ask questions that will result in directions that you’re able to do something about. For instance, if you have no flexibility with compensation due to union contracts, you shouldn’t ask compensation-related questions.
5. Normed: When you interpret survey, test, or assessment results, you should have a basis for comparison. Benchmarks or norms help you judge whether results are high or low compared to other organizations or individuals. For individual assessments, norms almost always have to be provided by the test publisher. For organizational assessments, the publisher should be able to provide norms for first-time applications. With repetition, organizations often develop their own norms to determine whether changes are taking place. In every case, the provider should be able and willing to explain how their norms are collected and maintained.
6. Respondent-Friendly: There is an “ergonomic” aspect to good surveys, tests, and assessments. Using different scales or reversing scales within the same survey, for instance, is not respondent-friendly. Respondents should be able to focus on the content of the questions, and should not have to think much about the mechanics of the survey instrument.
7. Expert-supported. In today’s business environment, organizations are called upon to justify their assessments. This often requires the endorsement or research of subject experts in the area that is being measured. When you purchase an assessment, you are really purchasing the expertise and integrity of the assessment designer. It is important that you look for the experts behind your survey selections.
Author: Kathleen Groll Connolly writes on a variety of human resources topics and is a partner in
Performance Programs, Inc., an organization specializing in human resources surveys and measurement. Click
here to contact Performance Programs or call 1-800-565-4223. Copyright Performance Programs, Inc. 2003- 2008
