2002

Abstract
Due to the recent availability of advanced techniques for handling missing data, data can be collected missing by design, and unobserved data can be estimated. This study deals with the application of by design missingness to psychological and educational paper and pencil tests. Test writers are usually faced with a paradox. Researchers should choose between two conflicting test goals. First, the test should be valid for predicting a given criterion. Second, the test should be a precise measurement of some attribute of an individual. When applying incomplete test designs, the testing goal should be taken into account, because techniques that handle incomplete item responses seem to optimize only one of these goals. A distinction between two kinds of model based techniques for incomplete test data can be made. One method is IRT, which allows for incomplete item response collection, estimating latent traits on the basis of available responses. Another method consists of a more general group of techniques, such as Data Augmentation (DA), that directly estimates unobserved data, using all available information. In this study, paper and pencil test were simulated, and a third part of the item responses was made unobservable. Next an IRT model, DA and two simple missing item response techniques were compared on their performance at reconstructing total scores, test reliability and predictive validity. All methods gave good reconstruction of total scores. IRT performed best at the estimation of complete data reliability; DA performed best at the estimation of complete data predictive validity.

Index terms: measurement versus prediction paradox, planned missingness, paper and pencil tests, IRT, Data Augmentation, imputation.