Test mode equivalence in a South African personality context : paper–and–pencil vs computerised testing

Abstract

The use of computers in testing has increased dramatically in recent years. Since its first introduction in the fields of education and psychological assessment, the popularity of computer–based testing (CBT) has increased to such an extent that it is likely to become the primary mode of assessment in the future. The shift towards CBT, although successful, has raised many practical, ethical and legal concerns. Due to the potential influence of differential access to computers and varying levels of computer familiarity amongst South African test–takers, it has become increasingly important to study the effect of different modes of administration on test performance. The objective of this study is to determine whether traditional paper–and–pencil and computerised assessment measures will lead to equivalent results when testing facets of personality on a dichotomous (2 point) rating scale. A cross–sectional survey design was used. A non–probability convenience sample was drawn from university students in South Africa (N = 724). The sample included undergraduate students from two higher education institutions in South Africa. A 48 item behaviour questionnaire measuring facets from the soft–heartedness personality construct was administered. Participants completed the questionnaire either in a paper–and–pencil or in a computerised format. Apart from the difference in administration mode the questionnaires were made to look as similar as possible in all other aspects, such as the number of questions per page, colour, numbering of questions, etc. to minimise the possibility of scoring differences due to other factors. The paper–and–pencil and computerised formats were then factor–analysed and subjected to correlation analysis. The two test modes were also tested for reliability using the Cronbach Alpha coefficient. The obtained results were then used to determine whether equivalence exists between the different modes of administration. Results indicated that the psychometric functioning of the traditional paper–and–pencil test mode is superior to that of the computerised version. The paper–based test consistently outperformed its computer–based counterpart in terms of mean scores, skewness, kurtosis, factor loadings, inter–item correlations and reliability. Recommendations were made for future research.