A recent blog entry discussed the power of performance-based testing to assess the skills of job candidates. While there is a great deal of excellent work being done to create assessments that measure people’s performance in simulations of real-world environments (such as call centers), computer literacy remains the one area where automated performance testing is the unquestioned standard (see https://crm.hr.com/servlets/sfs?t=/blogs/blog.show&e=UTF-8&i=1116423256281&l=0&blogid=1208270968417).

Yet computer literacy as a concept has undergone as much change in the last three decades as has technology itself. In fact, the term “computer literacy” is hardly used anymore, supplanted by concepts such as “information literacy,” “digital literacy” and “fluency in information technology,” phrases which recognize that the computer is not simply a tool that enhances personal productivity, but a powerful engine for managing information and communication.

These new literacies stress the need to find, evaluate and communicate information, as well as the ability to master today’s specific technology tools. In fact, an informal consensus is developing among educators and researchers working around the world to define and measure information and communication technology (ICT) literacy. This consensus sees ICT literacy as consisting of three separate, inter-related capabilities:

(1)       Foundational knowledge of IT concepts
(2)       Skills in utilizing contemporary technology tools; and
(3)       Critical thinking skills needed to evaluate information and learn new concepts related to technology

Within this framework, it becomes clear that performance-based testing of software skills – while an excellent at measuring skills in contemporary technology - cannot measure everything needed to define an ICT literate individual. Foundational knowledge (like any knowledge) can be evaluated using objective test items (like multiple-choice questions). But higher-order thinking skills, as they relate to the use of technology for specific purposes, are still best assessed by observing people using technology in the real world (or evaluating objects – called artifacts – created using a variety of information sources). The problem is that the more complex the behavior or object being measured, the more it requires manual evaluation methods that can be costly, complex and subjective.

Australia became the first (and only) country that decided to take on the challenge of benchmarking the technology literacy of the entire country or, to be more precise the technology literacy of 6th and 10th grade students nationwide. This project (managed by the Australian Center for Educational Research or ACER) required the consistent evaluation of thousands of students in diverse environments (from major metropolitan school districts to tiny schools in remote areas of Tasmania) using a consistent assessment that could measure the complete range of ICT abilities.

The tool ACER used to perform this evaluation was a software-based evaluation that included linear test items for assessing knowledge, simulation-based test items for assessing contemporary skill and hands-on live application items which allowed candidates to create technology artifacts that were evaluated to determine how well they reflected higher-order thinking ability as it related to information and technology. These three types of items were incorporated into a single set of assessments that were delivered to a statistically significant sampling of students across the country. The results of this analysis were recently published at http://www.mceetya.edu.au/mceetya/.

For those of us heavily involved with technology assessment for students, job candidates and employees, the work of ACER represents a breakthrough, a demonstration that the complete range of ICT literacy skill can be measured consistently and accurately on a national basis. Whether or not other nations rise to this same challenge, precedent has been set for creating the types of assessments that will hopefully be finding more and more uses in the coming years.
For more information, see:

http://www.fadvassessments.com/site.pl/about/news/04/04/2005
http://www.acer.edu.au/documents/MR_080118-ICT-NAP.pdf