Martin Pusic^1,2
Yoon Soo Park³
1 American Board of Medical Specialties
2 Harvard Medical School
3 University of Illinois Chicago School of Medicine

 

Background: 

In assessment, it can be difficult to disentangle what is uncertainty due to incomplete mastery and what is structural uncertainty where even a fully trained expert would be uncertain. Here we use item-response models to quantify diagnostic uncertainty at the clinician and case levels. 

Summary of Work: 

Forty dermatologists rated 100 images of skin lesions that might be diagnosed as malignant melanoma using a dichotomous categorization of either “no further treatment” (NFT) or “biopsy/further treatment” (Bx). We modeled the resulting fully crossed data (40 raters x100 pictures of skin lesions) using three approaches: a Rasch Model, a Signal Detection Model, and a Graded Response Model. 

Results: 

Under the Rasch model, the 100 cases demonstrated a full range of diagnostic uncertainty, from -4.18 logits (all dermatologists predicted to rate “Bx”) to +4.20 logits (all dermatologists predicted to rate “NFT”). 14 of the cases fell within 0.5 logits of the 0 mid-point where a dermatologist of average bias would be predicted to be equally likely to endorse either category, suggesting a significant proportion of cases with uncertainty for all practitioners. Several of the ultimately benign cases showed ratings consistent with malignancy. Modelling practitioners, we found that they demonstrated considerable practice variation in where they set their biopsy cut points (See Figure). Signal Detection and Graded Response Model results are found to be complementary to those of the Rasch Model. 

Discussion: 

Item response modeling, when aligned with a clinical decision such as whether to biopsy a case of potential melanoma, can be used to provide feedback as to a clinician’s tendencies and how they would be predicted to respond to individual cases. 

Conclusions: 

The presented work is an advance in that it allows case by case interpretation of an individual’s decision threshold, all taken in the context of demonstrated practice variation. 

References (maximum three) 

Baldwin P, Bernstein J, Wainer H. Hip psychometrics. Stat Med. 2009;28(17):2277-2292. doi:10.1002/sim.3616 

Pusic MV, Rapkiewicz A, Raykov T, Melamed J. "Estimating the Irreducible Uncertainty in Visual Diagnosis: Statistical Modeling of Skill Using Response Models". Accepted for publication in Medical Decision Making Feb 17, 2023. 

Pusic M, Cook D, Friedman J, et al. Modeling diagnostic expertise in cases of irreducible uncertainty: the decision aligned response model. Acad Med. 2022;98(1):88–97.