In the late 1970s and early 1980s, computer technology was becoming more accessible to larger numbers of people. Specifically, the personal computer was making inroads into the world of education and assessment, which led to many discussions about the impact of computers on learning and testing. Paper-and-pencil-based tests were being translated into computer-based administrations, and many issues and concerns arose. For example, people wondered how the reliability of an assessment might change when administered on a computer?
I wrote a dissertation in graduate school that evaluated the differences between both modes of administration for assessments. At the time, the personal computer was still a novel idea, and computer graphics and text were rudimentary at best. A main finding of my dissertation was that some computer-based assessments increased reliability, whereas assessments that required cognitive work involving graphics — spatial relations or sequencing of events, for example — were negatively impacted by computer-based administration. These computer-based assessments were less reliable and had an increased error of measurement compared to their paper-and-pencil counterparts. Scores were also lower for examinees.
My research found that computer-based assessments had both good and bad effects. For example, computer-based assessment had a positive impact on tests like the Minnesota Multiphasic Personality Inventory (MMPI) and the Strong Interest Inventory® that require examinees to choose responses, or preferences, quickly, thus providing assessment results quickly. For these types of tests, the structural advantages of computer-based assessments include displaying one item on the screen at a time and making examinees exert extra effort to go back and review previous answers. This test construction encourages test takers to stay focused, respond quickly, and not necessarily produce similar answers for similar items. This led to increased reliability, faster administration times, and improved validity of interpretations. For tests like the Differential Aptitude Tests®, however, the spatial reasoning or sequencing items were more difficult on a computer than on paper.
Paper-and-pencil versus computer-based comparability studies look much different today than they did when I was in graduate school. Assessments used to be exactly the same on the computer as they were on paper; the only difference was the method used to administer them. Today, online assessments are much more interactive, using simulations, games, technology enhancements, and other features of computer technology that do not necessarily have a paper-pencil counterpart.
This disparity between paper-and-pencil and online assessments may not be a bad thing. If items designed for computer-based assessments do, in fact, measure skills, knowledge, abilities, and cognitive processes in better ways than can be done with paper-pencil forms, then perhaps paper-pencil assessment does a disservice to the measurement process.
Nevertheless, the testing industry will always have a need for the paper-pencil mode of administration. Visually impaired and blind students, for example, will always need accommodations like large print and braille. Schools that lack the technological resources to accommodate online assessments, or have limited resources in relation to their number of students will also need paper-and-pencil assessments.
Having both an online and a paper-pencil version of an assessment necessitates multiple scoring tables and reporting for students, psychometric adjustments, and strong quality control procedures. It also requires both the vendor and the user to understand the strengths and limitations associated with both modes of administration and to consider how best to integrate dual mode programs into accountability systems.
I suspect that dual mode administration will be prominent for quite a while longer. However, based on Moore’s law, technology will eventually be so powerful and cost so little that paper-and-pencil administration will fade away. It just doesn’t look like that will happen in the near future.