|Year : 2021 | Volume
| Issue : 1 | Page : 136-139
The unexplored value of “Normal”: A commentary on the lack of normal cases in high-stakes assessment
Andrew Logiudice1, Matthew Sibbald2, Sandra Monteiro3
1 Post-Doctoral Fellow with the MacPherson Institute for Leadership, Innovation and Excellence in Teaching, Hamilton, Canada
2 Associate Professor with the Department of Medicine, Evidence and Impact, McMaster University, Hamilton, Canada
3 Associate Professor with the Department of Health Research Methods Evidence and Impact, McMaster University, Hamilton, Canada
|Date of Submission||11-Feb-2021|
|Date of Decision||14-Mar-2021|
|Date of Acceptance||16-Mar-2021|
|Date of Web Publication||26-Jun-2021|
Dr. Sandra Monteiro
David Braley Health Sciences Centre, McMaster University, 100 Main Street, 5th Floor, Hamilton ON L8P 1H6
Source of Support: None, Conflict of Interest: None
In this article, we highlight how standard assessments in the health professions pay little attention to “normal” cases – i.e. those without pathology – and as a result may be overlooking a skill that lies at the heart of efficient health care. The issue is explored with two overarching questions in mind: What specifically might be missed by excluding these normal cases from high-stakes assessment? And what broader implications does this have for medical practice? Drawing upon a large body of research on diagnostic expertise and clinical reasoning, we argue that accurate categorization of a case as either abnormal or normal represents a key diagnostic skill, and that this skill may be neglected in many standardized assessments because they consist almost entirely of abnormal cases. Unforeseen consequences of this structure are then discussed in terms of curriculum design and trainee perceptions. If discerning “abnormal versus normal” is as critical as the literature suggests, then perhaps our typical assessment strategies need to be re-evaluated. This under explored topic warrants further research.
Keywords: Assessment, clinical reasoning, diagnostic expertise, licensing and certification
|How to cite this article:|
Logiudice A, Sibbald M, Monteiro S. The unexplored value of “Normal”: A commentary on the lack of normal cases in high-stakes assessment. Arch Med Health Sci 2021;9:136-9
|How to cite this URL:|
Logiudice A, Sibbald M, Monteiro S. The unexplored value of “Normal”: A commentary on the lack of normal cases in high-stakes assessment. Arch Med Health Sci [serial online] 2021 [cited 2021 Nov 30];9:136-9. Available from: https://www.amhsjournal.org/text.asp?2021/9/1/136/319369
| Introduction|| |
Many licensing and certification exams are chiefly focused on abnormal cases and their differentiation through more precise diagnostic labels, leaving little room for “normal” cases without pathology. For instance, the Medical Council of Canada's blueprint for qualifying examinations outlines what percentage of cases in an examination ought to reflect specific physician activities, dimensions of care, or other relevant variables (e.g, age, sex, and case complexity). The same can be said of competency examinations for the Optometry Council of Australia and New Zealand and many other organizations.,, Yet, there is typically no mention of how many normal cases ought to be included, and the default seems to be examinations almost entirely comprised abnormal cases. Consequently, a candidate taking one of these examinations hardly needs to decide whether each case is abnormal to begin with.
As a concrete example, consider how normal cases are common place when interpreting diagnostic images such as electrocardiograms, mammograms, and pelvic ultrasounds.,,, One obvious goal with these images is to identify any abnormalities that would suggest a need for additional clinical management or treatment. However, an equally important goal is to identify whether each image falls within an acceptable range of normal variation such that no further testing or management is required, thereby saving valuable time and resources. It is easy to see why an assessment strategy devoid of normal cases would fall short in this context: without testing whether the candidate can distinguish between true abnormalities and normal variation, the assessment overlooks a key role of the diagnostician.
Here, we explore this issue with two overarching questions in mind: What might we be missing by excluding normal cases from our high-stakes assessments? And what broader implications does this have for medical practice? To foreshadow, we will begin by discussing contemporary models of medical expertise, clarifying what processes ought to be trained and assessed from a theoretical standpoint. We then use this theory to speculate on the importance of distinguishing “abnormal versus normal” from an assessment standpoint, and the potential consequences of our current assessments lacking in this regard.
| Why Do Normal Cases Matter from a Theoretical Standpoint?|| |
According to a dual-process model of clinical reasoning, after a clinician has had direct experience with many cases in a specific domain and is shown a new pathological case, they undergo a fast and holistic impression of abnormality before generating more precise diagnostic hypotheses.,,, For instance, skilled physicians examining a patient for only a few seconds often report “just knowing” or “feeling” that something is amiss without yet fully deducing what prompted the feeling, leading them to propose more specific diagnostic hypotheses and pursue a course of action., Extending this logic, the absence of such a feeling in experienced clinicians might indicate that a case falls within an acceptable range of normal variation such that it does not require any further action. Such fast and holistic impressions of abnormality suggest that discerning “abnormal versus normal” is a critical diagnostic skill unto itself.
Support for the same idea comes from eye tracking studies that reveal a global–focal search pattern in experienced clinicians when they evaluate visual stimuli such as X-rays, mammograms, and Magnetic resonance imaging., The term “global–focal” is used because the eye gaze patterns of these clinicians suggest an initial holistic examination of the image followed by examination of smaller, self-selected areas of the image. More specifically, the clinician is thought to (i) receive a rapid, holistic, gestalt-like impression of visual stimuli based on input from the whole retinal image to detect possible abnormalities, then (ii) proceed to a slower, focal, search-to-find process that permits more detailed interpretation of those abnormalities or the detection of further abnormalities.,,, Consistent with a dual-process model, the presence of the initial global viewing pattern – and the resulting diagnostic accuracy – seem to depend on the clinician's experience level., These findings again suggest that diagnostic expertise is driven by a fast and holistic discernment of “abnormal versus normal.”
Additional support comes from studies on quick impressions of “well versus unwell.” This research was inspired by clinician anecdotes of an uneasy “gut feeling” or “sense of alarm” that suggests something is wrong before any specific diagnostic hypotheses are generated. Early research showed that this feeling state is pervasive among general practitioners in different countries and cultures.,, Subsequent work confirmed that experienced emergency physicians can assess the illness severity of a patient and predict their disposition with reasonable proficiency even when provided minimal information (e.g, only the patient's history, or only a minute to observe the patient without conversing) or forced to decide under tight time constraints.,,,, Once again, these findings hint at the key role of a fast and holistic discernment of “abnormal versus normal” in experienced clinicians.
Finally, some indirect support can be found in research investigating the benefits of multifaceted reasoning strategies. The main thrust of this work is that diagnostic accuracy can be improved by encouraging a clinician to embrace their holistic impression or “gut feeling” for a case before adopting a more analytic rule-based approach. For instance, medical students in one study were asked to diagnose visual dermatology cases either by engaging in a rule-based strategy, or by quickly providing the first diagnosis that came to mind before engaging in said rule-based strategy. The latter group exhibited greater diagnostic accuracy, presumably because emphasis on the early holistic approach helped them generate better diagnostic hypotheses before they could be led astray by ambiguous perceptual features. Similar findings have also been reported using less experienced participants.,, In short, these studies suggest that trainees can benefit from explicit instructions to adopt a holistic approach very early in the diagnostic process.
The theme of this section can be summarized as follows: fast and holistic impressions of abnormality play a profound role in the diagnostic process, and their accuracy seems to hinge on direct experience with both normal and abnormal cases in a specific domain. With this central idea in mind, we now discuss its implications for assessment and medical practice.
| What Implications Does this Have for Assessment?|| |
While a major goal of licensure and certification is to ensure that clinician scan safely engage in practice, we also want them to identify cases falling within an acceptable range of normal variation such that resources are not squandered on needless testing or referrals. It therefore strikes us as troubling that our standard certification and licensing exams do not usually focus on a candidate's ability to recognize when a case can be considered normal and thus requires no further action. Indeed, all the aforementioned studies strongly suggest that holistic discernment of “abnormal versus normal” plays a central role in diagnostic accuracy and clinical reasoning. So why do our assessments seem to neglect this vital skill?
Further, because our current examination structures lack normal cases, one potential consequence is that candidates may become overly focused on the later stages of clinical reasoning (e.g. differentiation or labeling of disease states, patient management) without paying enough attention to the initial decision of whether a case falls outside the range of normal variation to begin with. The end result may be candidates who excel at high-stakes assessments at the expense of skills that would serve them – and the larger health-care community – in the real world.
Having assessments comprised mainly of abnormal cases may also send the wrong message to trainees, even if only implicitly. For example, we may be falsely leading them to think as though the majority of cases in practice are abnormal. Or perhaps we are downplaying the importance of prevalence rates when thinking about sensitivity and specificity. Indeed, novice clinicians usually struggle with specificity, having a robust tendency to “overcall” diagnoses as if disease states are far more prevalent than they actually are.,, Regardless of what nuanced effects these assessments have on the perceptions of trainees, it is safe to say they do not capture the realities of practice.
| A Call to Evidence|| |
To our knowledge, we are among the first to explicitly draw attention to this issue, and so there is little existing research on the topic. One straight forward next step would be to conduct studies, in which normal variants are more extensively integrated into assessments, allowing us to investigate whether the discernment of “abnormal versus normal” truly represents a skill that is distinct from the typical differentiation of abnormal cases through diagnostic labels. Further, health professions education might benefit from more research on the cultural or patient care impact of our assessments being designed with a lack of normal variants. It stands to reason, for instance, that trainee perceptions may change substantially if some assessments are more strongly oriented toward the identification and understanding of normal variation. In closing, we argue that such studies are timely and would elucidate whether we need to rethink our assessment strategies such that they are more closely aligned with the realities of practice.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Williams J, Pearce K, Benett I. The effectiveness of a mobile ECG device in identifying AF: Sensitivity, specificity, and predictive value. Br J Cardiol 2015;22:70-2.
Kavanagh AM, Giles GG, Mitchell H, Cawson JN. The sensitivity, specificity, and positive predictive value of screening mammography and symptomatic status. J Med Screen 2000;7:105-10.
Stines J, Tristant H. The normal breast and its variations in mammography. Eur J Radiol 2005;54:26-36.
Shih CH. Effect of emergency physician-performed pelvic sonography on length of stay in the emergency department. Ann Emerg Med 1997;29:348-51.
Djulbegovic B, Hozo I, Beckstead J, Tsalatsanis A, Pauker SG. Dual processing model of medical decision-making. BMC Med Inform Decis Mak 2012;12:1-13.
Eva KW. What every teacher needs to know about clinical reasoning. Med Educ 2005;39:98-106.
Norman G, Young M, Brooks L. Non-analytical models of clinical reasoning: The role of experience. Med Educ 2007;41:1140-45.
Pelaccia T, Tardif J, Triby E, Charlin B. An analysis of clinical reasoning through a recent and comprehensive approach: The dual-process theory. Med Educ Online 2001;16:5890.
Stolper E, van Royen P, Dinant GJ. The “sense of alarm” (“gut feeling”) in clinical practice: A survey among European general practitioners on recognition and expression. Eur J Gen Pract 2010;16:72-4.
Vanstone M, Monteiro S, Colvin E, Norman G, Sherbino J, Sibbald M, et al.
Experienced physician descriptions of intuition in clinical reasoning: A typology. Diagnosis (Berl) 2019;6:259-68.
Al-Moteri MO, Symmons M, Plummer V, Cooper S. Eye tracking to investigate cue processing in medical decision-making: A scoping review. Comp Human Behav 2017;66:52-66.
van der Gijp A, Ravesloot CJ, Jarodzka H, van der Schaaf MF, van der Schaaf IC, van Schaik JP, et al.
How visual search relates to visual diagnostic performance: A narrative systematic review of eye-tracking research in radiology. Adv Health Sci Educ Theory Pract 2017;22:765-87.
Drew T, Evans K, Võ ML, Jacobson FL, Wolfe JM. Informatics in radiology: What can you see in a single glance and how might this guide visual search in medical images? Radiographics 2013;33:263-74.
Kundel HL, Nodine CF. Interpreting chest radiographs without visual search. Radiology 1975;116:527-32.
Kundel HL, Nodine CF, Conant EF, Weinstein SP. Holistic component of image perception in mammogram interpretation: Gaze-tracking study. Radiology 2007;242:396-402.
Mello-Thoms C, Mello CA, Medvedeva O, Castine M, Legowski E, Gardner G, et al.
Perceptual analysis of the reading of dermatopathology virtual slides by pathology residents. Arch Pathol Lab Med 2012;136:551-62.
Kok EM, De Bruin AB, Robben SG, Van Merrienboer JJ. Looking in the same manner but seeing it differently: Bottom-up and expertise effects in radiology. Appl Cogn Psychol 2012;26:854-62.
O'Neill EC, Kong YX, Connell PP, Ong DN, Haymes SA, Coote MA, et al.
Gaze behavior among experts and trainees during optic disc examination: Does how we look affect what we see? Invest Ophthalmol Vis Sci 2011;52:3976-83.
Stolper E, Van Royen P, Van de Wiel M, Van Bokhoven M, Houben P, Van der Weijden T, et al.
Consensus on gut feelings in general practice. BMC Fam Pract 2009;10:66.
Wiswell J, Tsao K, Bellolio MF, Hess EP, Cabrera D. “Sick” or “not-sick”: Accuracy of System 1 diagnostic reasoning for the prediction of disposition and acuity in patients presenting to an academic ED. Am J Emerg Med 2013;31:1448-52.
Cabrera D, Thomas JF, Wiswell JL, Walston JM, Anderson JR, Hess EP, et al.
Accuracy of 'My Gut Feeling:' Comparing System 1 to System 2 Decision-Making for Acuity Prediction, Disposition and Diagnosis in an Academic Emergency Department. West J Emerg Med 2015;16:653-7.
LoGiudice AB, Sherbino J, Norman G, Monteiro S, Sibbald M. Intuitive and deliberative approaches for diagnosing 'well'versus 'unwell': evidence from eye tracking, and potential implications for training. Advances in Health Sciences Education 2021:1-5.
Sibbald M, Sherbino J, Preyra I, Coffin-Simpson T, Norman G, Monteiro S. Eyeballing: The use of visual appearance to diagnose 'sick'. Med Educ 2017;51:1138-45.
Pelaccia T, Tardif J, Triby E, Ammirati C, Bertrand C, Charlin B, et al.
Insights into emergency physicians' minds in the seconds before and into a patient encounter. Intern Emerg Med 2015;10:865-73.
Kulatunga-Moruzi C, Brooks LR, Norman GR. Teaching posttraining: Influencing diagnostic strategy with instructions at test. J Exp Psychol Appl 2011;17:195-209.
Ark TK, Brooks LR, Eva KW. The benefits of flexibility: The pedagogical value of instructions to adopt multifaceted diagnostic reasoning strategies. Med Educ 2007;41:281-7.
Eva KW, Hatala RM, Leblanc VR, Brooks LR. Teaching from the clinical reasoning literature: Combined reasoning strategies help novice diagnosticians overcome misleading information. Med Educ 2007;41:1152-8.
Norman GR, Brooks LR, Colle CL, Hatala RM. The benefit of diagnostic hypotheses in clinical reasoning: Experimental study of an instructional intervention for forward and backward reasoning. Cogn Instr 2000;17:433-48.
Braun LT, Zwaan L, Kiesewetter J, Fischer MR, Schmidmaier R. Diagnostic errors by medical students: Results of a prospective qualitative study. BMC Med Educ 2017;17:191.
Newman-Toker DE. A unified conceptual model for diagnostic errors: Underdiagnosis, overdiagnosis, and misdiagnosis. Diagnosis (Berl) 2014;1:43-8.
Zwaan L, Schiff GD, Singh H. Advancing the research agenda for diagnostic error reduction. BMJ Qual Saf 2013;22 Suppl 2:i52-7.