Developing and Validating Multiple-Choice Test Items: Guide for Educators & Assessment Professionals | Classroom, Online Learning & Certification Exams

Thomas Haladyna has written "...a comprehensive treatment of the development of multiple-choice (MC) test items and the study of item responses for the improvement or continued use of these items." (p. vii). The author graciously acknowledges that "...many testing specialists and educators have contributed to this book..." but he has reviewed these contributions and synthesized them into this well-written guide.The book overviews basic concepts of test validity and how to interpret test scores responsibly. Haladyna acknowledges testing standards developed by the American Educational Research Association (AERA), the National Council on Measurement in Education (NCME), and the American Psychological Association. Test validity is extended to the level of individual test questions ("items"). The validity of test items is established in two ways. First, appropriate procedures must be followed as they are written and reviewed. Then responses of test takers must be analyzed to identify and correct any remaining problems.Haladyna lays the foundation for item writing in Chapter 2 by discussing the mental processes that test developers hope to elicit. They should not only design items with appropriate content, but consider the cognitive demand of each item. Chapter 3 reviews the basic MC format of a question, a set of response options and a scoring protocol. Haladyna distinguishes between high-inference (abstract) and low-inference (concrete) items and outlines the appropriate role of each. (See the summary table on p.45.) "Not only must the item writer understand content measured by the item, but must determine whether the cognitive demand will involve recall, understanding or application." (p. 65). This is complicated by the fundamental inference problem of testing: We can never be sure whether an item elicits an intended complex mental process or simple recall of a memorized answer. The chapter closes with recommendations for choosing between MC items and open-ended, constructed response (CR) items.The book's second section presents MC item formats. Chapter 4 describes eight formats, including the familiar four-option MC and the much-dreaded true-false item. Chapter 5 reminds readers that MC development produces casualties, with only 50% of MC items crossing the finish line. Haladyna outlines practices for creating test specifications, training item writers, and editing their work. He also cites research showing that these practices produce better test items, lest we be tempted to skip a step. Chapter 6 contains examples of well-written items and ill-conceived items with interesting and instructive flaws. Chapter 7 closes the section with approaches that increase efficiency and standardization of item pools by using structured specifications for item writing.The next two chapters contain Haladyna's central messages. "The quality of test items depends on two complementary activities, the item review procedures featured in Chapter 8 and the statistical study of item responses featured in this chapter [9]." (p. 202). Readers step through item development from content definition and test specifications through producing an item writing guide and recruiting, training and managing item writers. The emphasis is on review for quality, sensitivity and fairness followed by adequate revision. Statistical methods include item difficulty and discrimination as well as analysis with item response and generalizability theories. Haladyna focuses on analyzing responses to each option of a multiple choice question, not just whether the answer was correct. He shows how to interpret graphs and tables of information about each MC option--allowing even nonstatisticians to improve their MC items. Chapter 10 describes the response patterns of cheaters, anxious test takers, and others (p. 242). The author might have also discussed using text analysis (See The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data) to detect systematic problems across items as this is an emerging and useful practice.The book closes with speculation about future influences of cognitive psychology, government education policy and ongoing testing research. He recommends increased attention to diagnostic information available from incorrect MC answers. Haladyna's book is well-documented, with 20 pages of references and indices for both key topics and contributing researchers. It also includes a suggested list of statistical software for analyzing item responses (p. 204). I highly recommend this book for anyone developing, interpreting--perhaps even taking--MC tests.