The degree of scrutiny applied to educational assessment programs appears to be growing every year. Sponsors of testing programs face increasing demands to provide evidence that their assessments are valid, reliable, and fair for all students, and yield comparable results across various administration circumstances. As a result, an increasing number of assessment programs are recruiting independent entities to evaluate many aspects of their assessment programs. These independent reviews may be conducted as a direct result of specific concerns raised in the media or state legislatures about the assessment program. In other cases, the review is embedded in the standard work for the program from the outset as an essential portion of the quality assurance program.

The presenters of this paper recently completed an independent evaluation of the statewide Florida assessment program. Drawing upon this experience, as well as their experiences in working with a number of US state departments of education and other large scale assessment programs in the US and in other countries, the presenters will outline a systematic process for conducting external, independent evaluations. The first presenter will discuss challenges that arise with disruptions in test administrations. These challenges include working with incomplete or compromised data sets and needing to analyze multiple sources of data and information, sometimes presenting conflicting findings. This first presentation will offer recommendations for how to identify critical evaluation questions and conduct analyses that will provide multi-faceted evidence in relation to these questions. The second presenter will discuss challenges associated with reviewing items and test content in relation to standards or other measurement targets and the blueprints for the assessments. The second presenter will discuss methods for establishing evidence related to test content early in the development process and how to evaluate test content evidence in ways that support subsequent improvements in alignment.