Learnings from a large-scale implementation pilot of the AAMC Core EPAs: A Recap (2 of 3)

Dr. Jonathan Amiel, past chair of the Association of American Medical Colleges (AAMC) Organization of Resident Representatives and Northeast Group of Educational affairs, joined MedHub to present the findings from a large-scale pilot of the Association of American Medical Colleges (AAMC) core entrustable professional activities (EPAs) in a recent webinar.

The webinar covered, with breadth and depth, the details of the findings and implications of the study. In summary, the pilot aimed to shed light on the utility and feasibility of the 13 core entrustable professional activities (EPAs) medical school graduates are expected to perform without direct supervision on the first day of residency. The webinar covered the parameters of the pilot and the results of the collaboration to implement within the medical education institutions.

Note:  In a previous post, we outlined the specifics of the pilot study. You may want to reference it as you read the text below. 

EPA pilot school data feasibility

In order to understand the findings of the data from the AAMC pilot program, we first need to outline the types of data received.

• The first column contains workplace-based assessments (WBA), which show markedly different ranges per student, with EPAs 1, 2, and 6 being the highest.
• The second column shows how often students were observed performing these EPAs and received feedback at least five times. There is also a wide range of scores, from 7% on EPA 13 through 78% on EPA 6.
• The third column, the AAMC asked pilot schools how often entrustment groups had more than four workplace-based assessments. There was a significant range of results across all EPAs. No schools reported having more than four workplace-based assessments for EPA 13, yet 86% had more than four for EPA 2.
• The fourth column measured the percentage of students who made an entrustment decision. For instance, for EPA 11 just 42% of schools thought students were ready for an entrustment decision, all way to 94% for EPA 6.

EPA pilot school data utility

When the pilot ended, the AAMC collected the data to reveal the utility of the pilot. They grouped four areas:

1. The percentage of students ready for entrustment.
2. The percentage of students ready for entrustment with medium to high confidence.
3. The percentage ready for indirect supervision (pilot school graduates were asked how often they were observed performing these EPAs that received feedback at least five times).
4. The percentage ready to perform EPAs.

The source for the first two were the pilot schools, the third was the PGY-1 survey, and the fourth was the Program Director survey.

The AAMC tracked each of the EPAs by each area listed above and uncovered some interesting clusters. The pilot found that there were some EPAs that lent themselves to workplace-based assessments and others that might require more investment in getting the data.

• The first column shows the proportion of students who were ready for entrustment ranged between 0% to almost 80%.
• The second column shows that some of the EPAs dropped off, which nods to interns not feeling confident about the entrustment decision.
• The third column from the PGI-1 Survey shows the percentage of surveyed graduates (who were interns) who thought they were ready for indirect supervision.
• The fourth column reveals the findings from the Program Director study. The average numbers are overall higher than students who were polled, meaning Program Directors had a higher level of confidence students could perform activities as compared to students themselves.

In some areas, there was a lot of alignment, yet overall, where interns were seeking additional help, Program Directors thought students were further along than they felt.

Implications on assessment

How do we interpret the data? The final post in this series dives into what the data really means, and how it applies to the modern medical school.

About Jonathan Amiel, M.D.

Jonathan Amiel, M.D. is a Professor of Psychiatry and Senior Associate Dean for Innovation in Health Professions Education at Columbia University’s Vagelos College of Physicians & Surgeons. He is also an Attending Psychiatrist at New York Presbyterian and the New York State Psychiatric Institute. Dr. Amiel obtained his bachelor’s degree in biology from Yale University and his MD from Columbia.

He is the past chair of the Association of American Medical Colleges (AAMC) Organization of Resident Representatives and Northeast Group on Educational Affairs leads its Core Entrustable Professional Activities pilot. He serves on the Advisory Committee for the Gold Humanism Honor Society and chairs its Membership Committee. Dr. Amiel’s work focuses on competency-based education and its relationship to the development of health professionals’ identities. In his work, he partners with educators across the medical school, medical center, university, and colleagues in national and international professional societies to advance the training of health professionals to optimally meet the evolving needs of the public. The overarching goal is to ensure that training is intentional, just, and aligned with public health needs – including developing our next generation of clinician-scientists, educators, and advocates.

View the Full Webinar