26th APS Annual Convention: Mark Your Calendar (San Francisco, CA, USA - May 22-25, 2014)

Symposium

Assessing Cognitive Biases in Judgment and Decision Making for Higher Education and Workforce Applications

Thursday, May 22, 2014, 4:00 PM - 5:20 PM
Powell Room A

Chair: Richard D. Roberts
Educational Testing Service

Although cognitive biases have been studied for decades, there have been few attempts to examine individual differences in bias measures and develop assessments that provide reliable, fair, and valid individual scores. This symposium will focus on challenges and recent advances in measuring individual differences in cognitive biases and applied research.

In recent decades, there has been growing consensus among social scientists around the view that people often rely on mental shortcuts, known as heuristics, to help them make judgments and decisions quickly and efficiently. Generally, these heuristics lead to accurate judgments and decisions. In certain circumstances, however, these heuristics can bias problem solving and decision making such that errors are produced that may have serious consequences (e.g., adverse health outcomes resulting from incorrect medical diagnoses). As such, the potential influence of biases in judgment and decision making has been a topic of interest in wide variety of fields, including economics, law enforcement, medicine, education, and intelligence analysis. Despite the widely recognized presence and influence of heuristics and biases on behavior, little attention to has been devoted to examining individual differences in biased reasoning. Several researchers have reported evidence that certain measures of bias may be related to cognitive ability, mood, motivation, and personality. In addition, there is growing demand in the public and private sectors for developing effective training interventions for mitigating biases. Given an interest in developing effective bias mitigation techniques and the likelihood that individuals differ in the extent to which they exhibit bias in judgment and decision making, a standardized assessment of cognitive bias that provides reliable, fair, and valid individual scores for educational and workforce applications could prove to be a valuable diagnostic tool. The aim of this symposium is to present a novel online assessment tool, the Assessment of Biases in Cognition (ABC), which consists of new item types measuring three cognitive biases—confirmation bias, fundamental attribution error, and the bias blind spot, and to discuss the potential for such a tool to further educational and workforce assessment. This symposium will begin with brief introductory remarks from our chair addressing the state of the literature on cognitive bias, followed by presentations that focus on 1) the composition and psychometric properties of the ABC; 2) the process and implications of translating experimental laboratory paradigms used to measure confirmation bias into assessment items; 3) the potential benefits and limitations of using item response theory to investigate individual differences in the three cognitive biases; and 4) applications of the ABC in studies evaluating the effectiveness of bias mitigation techniques in higher education and workforce populations.

 

Development and Educational Applications of the Assessment of Biases in Cognition (ABC)
Franklin M. Zaromb
Educational Testing Service
We examined the underlying structure of, and relationships between, three prominent cognitive biases using an innovative, online assessment that measures susceptibility to, and declarative knowledge (DK) of, confirmation bias (CB), fundamental attribution error (FAE), and bias blind spot (BBS). Psychometric properties and implications for educational assessment will be discussed.

Co-Author: Abigail Gertner, The MITRE Corporation

Co-Author: Robert Schneider, Research and Assessment Solutions, Ltd.

Co-Author: Dr. Jeremy Burrus, Educational Testing Service

Co-Author: Jonathan Weeks, Educational Testing Service


Confirmation Bias: Ubiquitous in Experiments, Elusive in Correlations
Gerald Matthews
University of Central Florida
Confirmation bias (CB) is ubiquitous in experimental paradigms, but individual differences in CB have been neglected. The present research investigated bias using a battery of experimental tasks. Although CB was robust in group data, individual differences in bias were largely task-specific. Experimental and psychometric perspectives on bias will be contrasted.

Co-Author: Ryan Wohleber, University of Central Florida

Co-Author: Jinchao Lin, University of Central Florida

Co-Author: Yana Weinsten, University of Massachussetts, Lowell

Co-Author: Larry Jacoby, Washington University in St. Louis


Using item response theory to investigate individual differences in three cognitive biases
Wes Bonifay
UCLA
Measures were created to assess individual differences in confirmation bias, fundamental attribution error, and bias blind spot. Responses to these measures were analyzed using item response theory (IRT). The findings demonstrated the effectiveness of IRT for measuring constructs that are often difficult to assess using traditional psychometric methods.

Co-Author: Steven Reise, UCLA

Co-Author: Jonathan Weeks, Educational Testing Service

Co-Author: Craig Wells, University of Massachussetts, Amherst

Co-Author: Amanda Soto, University of Massachussetts, Amherst


Evidence for Concept Formation within Confirmation Bias Tasks
Nathan Bos
Johns Hopkins University Applied Physics Laboratory
Naive learners not only perform poorly on tests of confirmation bias, but also do not seem to recognize the concept across different tasks as evidenced by low correlations between tasks, and test scales that consequently have low reliability. After training, however, intercorrelations and reliabilities increase, suggesting concept formation is occurring.

Co-Author: Jennifer McKneely, Johns Hopkins University Applied Physics Laboratory

Co-Author: Jonathon Kopecky, Johns Hopkins University Applied Physics Laboratory

Co-Author: Abigail Gertner, The MITRE Corporation

Co-Author: Larry Jacoby, Washington University in St. Louis


Email Bookmark and Share