Ten Years of Testing a
Decision Support System

Terry Bahill
Systems and Industrial Engineering
PO Box 210020
University of Arizona
Tucson, AZ 85721-0020, USA
terry@sie.arizona.edu
© 1998-2004 Bahill

We developed a decision support system to help speech clinicians diagnose small children who have begun to stutter. This paper describes how testing of the system evolved during its ten years of development. Testing included: (1) having an expert use and evaluate it, (2) running test cases, (3) developing a program to detect redundant rules, (4) using the Analytic Hierarchy Process, (5) running a program that checks a knowledge base for consistency and completeness, (6) having five experts independently critique the system, (7) obtaining diagnoses of stuttering from these five experts derived from reports of children who had been evaluated for possible stuttering problems, (8) using the system to expose missing and ambiguous information in 30 clinical reports, and (9) analyzing the dispersion and bias of six experts and the decision support system in diagnosing stuttering. When using the final system, three clinicians with widely differing backgrounds produced diagnostic opinions that evidence little variability and were indistinguishable from those of a panel of five experienced clinicians.

Reference [64]. This lecture is suitable for engineers or the general public. This talk requires an overhead projector. This talk takes one hour. This research was supported by National Institute of Child Health and Human Development grant R44 HD26209 and by Bahill Intelligent Computer Systems.