Page 24 - JSOM Fall 2018

Your Metric Matters!
Choose Wisely to Assess User Performance
With Tourniquets in Simulated First Aid

Ning O. Zhao, BS, BA ; John F. Kragh Jr, MD* ; James K. Aden 3rd, PhD ;
1
1
3
4
5
Bryan S. Jordan, RN ; Donald L. Parsons, PA-C ; Michael A. Dubick, PhD 6

ABSTRACT
1–3
Background: Readiness to perform lifesaving interventions proficiently execute the skills required. In first aid, when a
during emergencies is based on a person’s preparation to pro­ need arises to use a limb tourniquet, it is the time to act, not to
ficiently execute the skills required. Graphically plotting the learn. That preparedness is reflected through routine tourniquet
performance of a tourniquet user in simulation has previously training programs, such as Stop the Bleed, a US public health
4,5
aided us in developing our understanding of how the user ac­ program. In the process of tourniquet teaching, we have
tually behaves. The purpose of this study was to explore per­ aimed to develop best practices so trainees become efficient us­
formance assessment and learning curves to better understand ers. In this respect, to understand the pattern of the activity of
6–8
how to develop best teaching practices. Methods: These were tourniquet users, it is helpful to document graphical plots show­
6–8
retrospective analyses of a convenience sample of data from a ing their actual performances. The information on the mea­
prior manikin study of 200 tourniquet uses among 10 users. surement of such trends has helped us assess the acquisition,
We sought to generate hypotheses about performance assess­ maintenance, or decay of skill. 7,9,10 In fact, such information
ments relevant to developing best teaching practices. The focus may facilitate instructors, classes of students, or organizations
was on different metrics of user performance. Results: When like schools or military units to undertake further action plans.
one metric was chosen over another, failure counts summed
cumulatively over 200 uses differed as much as 12­fold. That US Army Soldiers commonly apply a performance metric in
difference also indicated that the degree of challenge posed assessing skill in tourniquet use. For this metric, the user is to
11
to user performance differed by the metric chosen. When we demonstrate the skill steps to an evaluator within 60 seconds.
ranked user performance with one metric and then with an­ Locally choosing another metric is common because users may
other, most (90%; nine of 10) users changed rank: five rose vary from novice to expert (e.g., elite Soldiers may strive for
and four fell. Substantial differences in performance outcomes a more challenging threshold, such as no more than 30 sec­
resulted from the difference in metric chosen, which, in turn, onds). The choices of metrics may plausibly vary further be­
changed how the outcome was portrayed and thus interpreted. cause users may be students, caregiving practitioners, teachers,
Hypotheses generated included the following: The usefulness or course directors. To date, little evidence is available on the
of a specific metric may vary by the user’s level of skill from merits of choices.
novice to expert; demonstration of the step order in skill per­
12
formance may suffice for initial training of novices; a mechani­ In our research, we previously gathered performance data
cal metric of effectiveness, like pulse stoppage, may aid in later that we used in the present study as a sample to learn what we
training of novices; and training users how to practice on their could derive from the data. Furthermore, we aim to eventually
own and self­assess performance may aid their self­develop­ develop our abilities in helping instructors with their work.
ment. Conclusion: The outcome of the performance assess­ The purpose of the present study was to explore performance
ments varied depending on the choice of metric in this study of assessment and learning curves to better understand how to
simulated use of tourniquets. develop best teaching practices.
Keywords: education standards measures; implementation; Methods
individuality; choice behavior; first aid
This study was conducted on existing data from a manikin
12
study published in 2017 as retrospective analyses of a conve­
Introduction nience sample of data from 10 tourniquet users. Each user had
20 uses, for an overall total of 200 uses. Here we present the
Readiness to perform lifesaving interventions in emergencies is analyses topically to generate hypotheses about performance
thought to be based on an individual’s degree of preparation to assessments relevant to developing best teaching practices.
*Correspondence to Army Institute of Surgical Research, Damage Control Resuscitation, 3698 Chambers Pass, Joint Base San Antonio Fort Sam
Houston, TX 78234­7767; or john.f.kragh.civ@mail.mil
1 Ms Zhao researches bleeding control at the US Army Institute of Surgical Research, Joint Base San Antonio Fort Sam Houston, TX. Dr Kragh
2
researches bleeding control at the US Army Institute of Surgical Research, Joint Base San Antonio Fort Sam Houston, and is an associate profes­
3
sor in the Department of Surgery, Uniformed Services University of the Health Sciences, Bethesda, MD. Dr Aden is a statistician at the Brooke
4
Army Medical Center, Fort Sam Houston. Mr Jordan is a supervisory research nurse of the Multi­Organ Support Technology task area, US
5
Army Institute of Surgical Research, Joint Base San Antonio Fort Sam Houston. Mr Parsons is the deputy director of the Department of Combat
6
Medic Training at the Medical Education Training Campus at Fort Sam Houston. Dr Dubick is a resuscitation researcher and the manager of
the Damage Control Resuscitation task area, US Army Institute of Surgical Research.
22