Page 72 - JSOM Summer 2024

13
simulation remains more art than a science.” This may be organization’s training objectives. Then, we reviewed evalua-
especially true for our purposes, given the highly dynamic and tion methods used in medical simulations and military train-
variable nature of tactical medicine, and the evolution towards ing organizations and found evaluation methods that aligned
14
standardization across military services. Thus, reliability and with our stated training objectives. We then retrospectively
validity, the two basic concepts of evaluation quality, are im- compared evaluation methods to analyze their effectiveness
portant to implement when creating evaluation standards. prior to full implementation and ensure similarity in rigor. By
drawing from established criteria, we tailored our evaluation
Reliability indicates the consistency of measurement. Validity method to evaluate student competencies more appropriately.
refers to the ability to make accurate and reproducible mea-
surements using an evaluation metric based on the desired Author Contributions
learning objective. 15,16 In the context of military medical sim- IPR conceived of this study and wrote the protocol. IPR and
ulation evaluation, a highly reliable evaluation method would JMS orchestrated all approvals for field observations. IPR and
produce the same results at different times and with different JMS collected data. IPR wrote the first draft of this paper. IPR
evaluators or simulation scenarios. Validity correlates with the and BLG analyzed data and performed statistical analysis.
inferences of a student’s performance in an evaluation. For ex- BLG performed the literature review. MJL and SCR provided
ample, if a student is successful in medical simulation, we infer overall guidance and revised each draft. All authors read and
they will succeed in real-world scenarios. approved the final manuscript.

The TC method’s validity may be suboptimal due to its equal Disclosures
focus across all aspects of care. Bewley and O’Neil summa- The authors have nothing to disclose.
rized how to develop an effective evaluation for military
medicine. They recommended defining the desired training Funding
15
objectives and aligning the measures and scoring with them. No funding was received for this work.
In this framework, our training objective can be defined as
the acquisition of durable life-saving tactical medicine skills References
that can be transferred to novel environments. By making crit- 1. Issenberg SB, McGaghie WC, Hart IR, et al. Simulation technol-
ical tasks auto-failure criteria, the AWC is more effective at ogy for health care professional skills training and assessment.
emphasizing the critical assessments and treatments that align JAMA. 1999;282(9):861–866. doi:10.1001/jama.282.9.861
with our training objectives and increases the validity of our 2. Beal MD, Kinnear J, Anderson CR, Martin TD, Wamboldt R,
Hooper L. The Effectiveness of medical simulation in teaching
medical evaluations. medical students critical care medicine: a systematic review and
meta-analysis. Simul Healthc. 2017;12(2):104–116. doi:10.1097/
The difference in the TC and AWC for practice and progress SIH.0000000000000189
check scenarios was likely driven by the high quantity of er- 3. Lynagh M, Burton R, Sanson-Fisher R. A systematic review of
rors students made during practice scenarios (2.7 errors per medical skills laboratory training: Where to from here? Med Educ.
2007;41(9):879–887. doi:10.1111/j.1365-2923.2007.02821.x
scenario) compared with progress check scenarios (0.9 errors 4. Air Force Special Warfare Training Program. Air Force Manual
per scenario). For scenarios in which students make three 10-3500V1. Department of the Air Force, United States of Amer-
or more errors (as they do during many practice scenarios), ica. Updated 1 June 2022. 2024. https://static.e-publishing.af.mil/
they would fail using either the TC or AWC. As student per- production/1/af_a3/publication/afman10-3500v1/afman10-
formance improves and the quantity of errors is reduced, the 3500v1.pdf
effect of auto-failure for single critical errors using the AWC 5. Salas E, Milham LM, Bowers CA. Training evaluation in the mil-
becomes more evident. itary: misconceptions, opportunities, and challenges. Mil Psychol.
2003;15(1):3–16. doi:10.1207/s15327876mp1501_01
6. Clay DR. Biomedical Equipment Technician: Curriculum Plan.
We believe the AWC evaluation method more closely aligns Medical Education Training Campus, JBSA Fort Sam Houston,
with our training objectives than the TC evaluation method Texas. Updated January 8, 2018. Accessed 2024. https://www.
because it strongly emphasizes life-saving care, de-emphasizes metc.mil/Portals/111/Documents/Reading%20Room/(6)%
less critical care, and maintains similar overall evaluation 20BMET_CP_FY18_001R.pdf?ver=9pUXbuBM-zv_
rigor as the TC. Because it is a checklist-based evaluation, the DKHZhd-MOw%3D%3D
AWC maintains objectivity and is simple to use while provid- 7. Wivell T. Navigating through another week of Combat Con-
trol School. United States Air Force. Updated December 28,
ing numeric scores to allow for longitudinal data analysis and 2009. Accessed March 8, 2023. https://www.pope.af.mil/News/
progress tracking. Future work should examine the inter-rater Features/Display/Article/243261/navigating-through-another
reliability of the AWC, and longitudinal studies should exam- -week-of-combat-control-school/
ine the validity of the evaluation metric. 8. The United States Army Advanced Airborne School 82nd Air-
borne Division. Jumpmaster Student Study Guide. Updated July
2006. Accessed 2024. http://www.c312.com/downloads/Jump-
Limitations master%20Study%20Guide.pd_
This is a retrospective study with inherent weaknesses. Evalu- 9. Dorsch J, Rush S, Anderson S. Pararescue Medical Operations
ation data was only collected from two instructors to provide Handbooks, 8th Edition. Updated January 2021. Accessed
consistency but could also amplify their individual biases. Due 2024.
to the limitations of our training requirements, we could not 10. Sackett PR, Mullen EJ. Beyond formal experimental design: to-
analyze inter-rater reliability for the AWC. wards an expanded view of the training evaluation process. Pers
Psychol. 1993;46(3):613–627. doi:10.1111/j.1744-6570.1993.
tb00887.x
Conclusion 11. Emergency Medical Services Education Program. Emergency
Medical Technician – Basic: Field Internship Evaluation. Broome
To improve our medical simulation evaluation metric, we used County Division of Emergency Medical Services. Updated 2022.
the following framework. First, we identified and stated our Accessed 2024. https://www.gobroomecounty.com/sites/default/

70 | JSOM Volume 24, Edition 2 / Summer 2024