Analyzing components of work samples to evaluate performance.
A method is described in which work sample data were analyzed to reveal how aircraft maintenance technicians differed in troubleshooting performance after a training intervention. During the training phase of the study, approximately half the technicians received troubleshooting instruction from an intelligent tutoring system, and the remaining technicians received informal on-the-job training. Data from work samples, in which technicians verbally isolated simulated equipment failures, were collected from both groups before and after the training. The work sample data were then examined by subject matter experts who scored performance for overall proficiency and noted specific violations of good troubleshooting practice. These violations provided the basis for the components of performance examined in the work sample data. The results of this componential analysis allow more specific conclusions to be drawn regarding the effectiveness of the training intervention by revealing how the tutoring system improved technicians' troubleshooting (e.g., tutored technicians were more likely than untutored technicians to test components thoroughly to ensure they were faulty before replacing them).