Keep up with our progress and learn about the concepts behind curriculum reform at the CU School of Medicine with posts from our coordinators Shanta Zimmer, MD, Chad Stickrath, MD, Tai Lockspeiser, MD, and committee members.
August 2020 Newsletter
The August 2020 issue focuses on the targeted outcomes for the School of Medicine after the implementation of the Trek Curriculum.
Clinical Presentations Video: Defining Clinical Presentations
By Tai Lockspeiser, MD
Assessment is a central component of medical education both for the learner and for the program.1 Assessment refers to gathering and providing data about how a student is performing. For the program, assessment allows us to ensure that our students are meeting standards such that they are able to provide high quality patient care – ultimately knowing what our students can and cannot do is the school’s way of upholding our commitment to the public to provide high quality patient care. In addition, assessment data becomes the basis of choosing applicants for advanced training programs such as residency and fellowship and also provides data to support the evaluation and improvement of an educational program. More importantly, from the learner’s perspective, assessment is what provides direction and motivation for future learning. Effective assessment helps students understand their strengths and weaknesses and what they need to do to move to the next level. Given the multitude of purposes of assessment, it is safe to say that thoughtful assessment drives education.
Advances in assessment in medical education are currently focused on adopting competency-based medical education or outcomes based medical education (OBME).2,3 Moving toward a truly OBME curriculum involves starting with the end in mind and defining the outcomes expected of graduates which then inform the curriculum development.4 Starting with the end in mind directs the curriculum and assessment efforts designed to assure that the goals have been accomplished. At the heart of this change, is the argument proposed by Cooke et al. that medical education needs to be individualized with standardized outcomes.5 This means that although the outcomes expected of each individual graduate may be the same, the process of getting there is unique.
If our goal is to embrace OBME, the first step in curriculum reform will be to carefully define what outcomes we expect of the CUSOM graduates. This is a challenging task and something that will require input from all stakeholders. One framework to consider when crafting these outcomes is to think of combining four different categories of outcomes that we may consider for an individual CUSOM graduate: 1) Standard outcomes for all graduates of US medical schools [such as the core entrustable professional activities for entering residency (CEPAER)], 2) Additional CUSOM specific outcomes for all students (such as those specifically related to leadership, curiosity, and commitment), 3) Outcomes unique to the field a student is matching in (the entry level milestones for a given specialty), and, 4) Individualized outcomes based on an individual student’s aspirations and goals (see Figure 1).
Figure 1: Four categories of outcomes expected of the CUSOM graduate
Here is an example of each of the four categories of outcomes:
Implementing OBME involves a substantial shift in how we think about assessment that will ultimately improve assessment for the learners, the teachers, and the school. Currently, our individual student assessments are distributed across different courses and the emphasis is on grading. For OBME, we will need a comprehensive program of assessment in which the different individual assessments are integrated and linked together to form a more complete picture of each student. We need to be able to see how a student is progressing in the history taking skills across different clerkships as opposed to just within one individual clerkship. Programmatic assessment refers to this idea of centralized and organized approach to assessment across the entire four years of medical school. Programmatic assessment has recently been described in the literature as a central feature of OBME and there are numerous articles describing the key features.6,7 I consolidated these into a mnemonic of three key aspects of programmatic assessment. In planning a program of assessment there are three P’s to consider 1) Overall Assessment Plan, 2) People involved in assessment, 3) Paper (tools used for assessment).
For programmatic assessment to be successful, the overall assessment plan must be centrally coordinated and based on the expected outcomes of the graduates.6,7 Assessment must be frequent and continuous using multiple different methods to gather data. This is crucial as it assures that data is gathered about all of the different outcomes expected, and that multiple data points are accumulated about each specific outcome. This means that the stakes for any given assessment are lower as there are multiple assessments for each outcome. This change supports the concept of assessment for learning where assessment is meant to help students grow and learn and provide formative feedback.8 Because each assessment is only one data point each individual assessment doesn’t have as much riding on it. In this model the central purpose of assessment is to drive learning and improvement and if enough data is collected that data can ultimately be combined to make high-stakes decisions. The methods used to make these high-stakes decisions must be trustworthy and evidence-based. In particular, any high-stakes decisions such as whether or not a student is ready to progress to the core clinical experience or if he/she has met the outcomes and are able to graduate should be made by a group of individuals who comprehensively review all of the data available about the student’s performance. There is evidence to suggest that using a group to make high-stakes decisions reduces bias and allows for difficult decisions to be made if necessary.9,10 To implement this, there needs to be a clear means of collecting, storing, and analyzing all of the assessment data such as an online portfolio or dashboard.11
The people involved in assessment include the teachers, the learners, as well as coaches/mentors. For the teachers, there clearly needs to be an emphasis on high quality assessment which will take training and practice. In addition, evidence supports the idea that, at least for clinical assessments, longitudinal relationships between the student and the assessor improve the quality of the assessments.12 Moving towards a model that focuses on assessment for learning encourages the learners to be more self-regulated and active participants in their education. Our students will need to become master adaptive learners who are constantly learning and growing as the world of medicine changes around them after they finish their training.13 Developing these life-long learning skills is supported by engaging our students in the learning and assessment process during their medical school years. Finally, students need mentoring/coaching to help them in processing and understanding their assessment data and translating the data into plans for learning and improvement. Self-regulated learning is not something that learners can do on their own, at least initially.14 We must put a system in place to support our students and help them learn these skills.
Finally, our approach to the tools used for assessment will also change. The central tenet here is that any single assessment point is flawed, and therefore we must accumulate multiple assessments of different types and in different contexts to truly have a complete picture of each learner.6,7 There are numerous different ways to assess learners in medical education and our program of assessment should include many of them. We must think carefully about what the best assessment methods are for each particular outcome. There is increasing evidence that supports the use of frequent testing as a means of supporting retention of knowledge in the preclinical years.15 In addition, although multiple choice questions are important as students must learn to do well on them for passing the USMLE, open-ended questions may actually better test and support deeper understanding of concepts and should be included in any assessment plan.16 In choosing an assessment of clinical skills, our emphasis must be on direct observation and specific feedback to our learners. This means that qualitative data (narrative descriptions of a student’s performance) are just as important, if not more important, as quantitative data.17,18 If scales are used, evidence suggests a strong benefit for using criterion referenced as opposed to normative referenced scales.19 This means that scales should include narrative descriptions of what is expected for each level rather than a number or a comparison with peers (i.e. below expectations, meets expectations, above expectations). Ongoing assessments of learners beyond medical school graduation will also be important to help measure the success of a new OBME program.
To truly embrace OBME and assessment for learning at CUSOM, we must engage stakeholders, start with the end product of our medical school in mind, and thoughtfully craft a comprehensive, integrated learner assessment program. While such a program may look considerably different than our current assessment structure(s), it will undoubtedly have beneficial impacts for students and our school.
By Students of Curriculum Reform
Embarking on a mission to reform curricula is a massive dedication, especially when the current curriculum is functionally great and producing wonderful physicians. Why reform? As medicine evolves, medical education must also evolve. The process of curricular reform at the University of Colorado has taken shape as 11 subcommittees with members ranging from deans to students and everyone in-between. Along with these 11 application-based committees, we have formed a committee inclusive of the entire student body to discuss and dissect ideas put forth by the 11 core committees with the help of the most crucial component: the students.
Reforming curricula without the input of students is like conducting a symphony without any instruments; this symphony might be the most amazing piece of music ever composed but no one will ever know its beauty without proof from the orchestra. Students molded by a medical institution are the proof of a functionally great curriculum; they serve as a yardstick for success. Our hope with Students of Curriculum Reform (SOCR) is to intimately involve the students currently experiencing our curriculum in the process of constructing a new one. We believe our input and perspective is invaluable in shedding-light on what parts of the curriculum actually need changing. Working with the faculty, who have an understanding of pedagogy and implementation, we can fill in the gap between knowledge and experience.
SOCR has monthly meetings where we exchange discussion points from each of the eleven subcommittees, suggest components of the curriculum that we love, components that we feel need adjustments, share topics and points that individual students find crucially vital to the reform process, and emphasize the importance of wellness and space for mental health in the new curriculum model. We are also trying out new technology platforms to work in teams and share information. Our hope is that when larger issues emerge, the faculty can use our group to survey students and create a feeling of inclusion around curricular reform.
We are grateful to our school for including us so intimately with curricular reform and look forward to future developments.”
Enhancing Medical Education through Longitudinal Relationships
By Chad R. Stickrath, MD, FACP
(February 2018) Over ten years ago, a group of medical educators proposed “continuity” as an organizing principle for educational reform efforts.1 They suggested a number of potential benefits that could be gained by having students engaged in longitudinal relationships with patients, peers, faculty, and healthcare systems. Since that time, numerous efforts have been made to build these longitudinal relationships and evidence of their beneficial effects on students, clinicians, and patients has emerged. Most efforts aimed at creating longitudinal relationships for medical students have either focused on placing students into learning communities with other students and an advisor to assist in navigating the medical school experience, placing pre-clerkship medical students into a consistent clinical setting periodically to build their clinical skills alongside their basic science classroom learning, or placing students in Longitudinal Integrated Clerkship (LIC) models for their core clerkship experience instead of traditional block models.
Several prominent learning theories support the idea that longitudinal relationships can foster learning. Cognitivism suggests that continuity in the clinical learning environment allows for a decrease in cognitive load for learners as they spend less working memory focused on a novel environment and more working memory focused on learning specific developmental knowledge and skills.2 In addition, situated learning theory proposes that knowledge has more meaning when it is acquired, or utilized, in the community of practice, or actual real-world setting with the specific rituals and norms of that setting. In this environment, students can learn to be doctors by directly applying and reflecting on knowledge and skills rather than just learning about medicine.3 Finally, transformative learning theory purports that the deep, structural shift in the basic premises of thoughts, feelings, and actions that occur as students shift from theory-based knowledge to novice clinicians is enhanced by strong, collaborative, ongoing relationships between students and their peers and their clinical educators.4 Medical students who experience these relationships develop a deeper sense of the kind of doctor they wish to become, which is reflected in their values, behaviors, and changing role in the community.5
Learning communities, or smaller groups of students and faculty engaged in a longitudinal relationship aimed at advancing a shared set of goals, principles, and values, have been used successfully in undergraduate education for many years, where they have been shown to create deeper and more integrated student learning and increased retention.5 Although experience with learning communities in medical schools is more contemporary, recent studies show that over 102 American medical schools have now adopted some form of intentional communities for students and/or faculty designed to enhance and maximize student learning.6 In most of these learning communities, students are assigned to the same small group for all four years of medical school, the community is linked to specific faculty, and includes students from many class years.6 Participation is these groups is typically mandatory and includes activities aimed to: foster communication among students and faculty; promote caring, trust and teamwork; improve professionalism, and may include problem-based learning or other core curriculum delivery.6 These medical education learning communities show benefits for learners and faculty.7 Clinical skills that are taught within the learning community context lead to better clinical skill scores in subsequent core clinical clerkships.7 Furthermore, students report increased satisfaction with wellness and counseling in learning communities versus traditional models.7 Next, learning community-based interventions aimed at patient-centeredness demonstrate decreased erosion of empathy for participants in the subsequent core clinical experience, while interventions aimed at student wellness show improvements in depression, anxiety, and stress scores.7 Finally, learning community faculty report an improvement in their physical examination and teaching skills and improvements in the job satisfaction.7
Several reviews of longitudinal student placements in the clinical setting describe the benefits of these placements on learners.8-11 Most of these longitudinal clinical placements included either students that worked with the same preceptor for one to four sessions per month during their preclinical years, or were Longitudinal Integrated Clerkship (LIC) placements.10 An LIC typically replaces some, or all, of the traditional block model of clinical education in the core clinical year.9 An LIC is characterized by “being the central element of clinical medical education whereby medical students: (1) participate in the comprehensive care of patients over time, (2) participate in continuing learning relationships with these patients and clinicians, and (3) meet the majority of the year’s core clinical competencies, across multiple disciplines simultaneously through these experiences.”9 In their Best Evidence in Medical Education (BEME) review of longitudinal placements of learners in medical education, Thisthlethwaite et. al. evaluated the impacts of placements of greater than 13 weeks on learners across the medical education continuum. The review of 53 manuscripts included learners from early medical school through residency and placements from both community and hospital-based settings. They found that learners in longitudinal placements appreciate greater continuity and deeper relationships with patients and preceptors than peers in non-longitudinal models. Furthermore, learners in these longitudinal placements display improvements in humanism, patient-centeredness, engagement with the healthcare system, and preparedness to work in interdisciplinary teams.10 Moreover, learners typically report more direct observation of their skills, better constructive feedback, and higher levels of satisfaction in these programs than traditional models. Finally, learners in these longitudinal placements accrue these benefits while consistently performing at least as well as non-longitudinally placed peers on standard assessments, including clinical evaluations, national examinations, and traditional sub-internship performance.8-11
In addition to benefiting learners, longitudinal relationships in medical education also appear to have favorable effects on teachers. Two recent reviews of longitudinal medical student placements with clinical preceptors report that preceptors believe that working with learners longitudinally enriched their lives and added new meaning to their practice.10,11 Walters et al, found that 83% of LIC clinical supervisors find their professional lives to be more satisfying when teaching in an LIC model.11 The literature also consistently reports that while teaching students in these longitudinal models initially decreased the productivity of clinical supervisors, that this was counterbalanced by increased productivity later (often after 3 months) as the learners became familiar with the preceptor, patients, and setting.11,12 Ultimately, teaching in these models has been found to be cost-neutral and has consistently led to preceptor retention rates of > 85%.12
Students in longitudinal placements demonstrate an enhanced perspective on the healthcare system, an increased sense of advocacy and idealism, and more interest in giving back to the community.10,11 Health systems that accept students in longitudinal placements may benefit from increased provider satisfaction, recruitment, and retention by providing an opportunity to teach and mentor medical students over significant periods of time.13 These placements allow for enhanced training in systems-based practice, practice-based learning, interdisciplinary team-based care, and physician leadership.14 Finally, patient care may be enhanced as students in longitudinal placements serve as patient advocates, navigators, health coaches, population health managers, and quality improvement leaders.15
Longitudinal relationships between medical students and patients, peers, faculty, and/or health systems seem to promote better continuity in medical education and are associated with numerous benefits to students, faculty, health systems, and likely patients. Notable among the benefits are improvements in student wellness and empathy, faculty satisfaction, recruitment and retention, and the ability for students to add value to patients and the healthcare system. The main burdens associated with creating these longitudinal relationships seems to be resources to support faculty in learning communities and administrative management of learning communities and longitudinal placements. The University of Colorado already has extensive experience with versions of learning communities (Advisory College Program), pre-clinical longitudinal placements (Foundations of Doctoring), and Longitudinal Integrated Clerkships (Denver Health, Integrated Longitudinal Medicine, and Colorado Springs LICs) from which to draw when deliberately considering how to potentially further utilize the principles of continuity and longitudinal relationships to enhance medical education.
1. How have longitudinal relationships benefited me as a learner and/or teacher?
2. What if the CUSOM chose to create longitudinal relationships between its learners and faculty? Learners and coaches? Learners and peers? Learners and patients?
3. How could I enhance continuity for learners in my setting?
4. How can we enhance continuity and longitudinal relationships through curriculum reform? Within the work of each curriculum reform subcommittee?
1. Hirsh DA, Ogur B, Thibault GE, Cox M. “Continuity” as an organizing principle for clinical education reform. N Engl J Med. 2007; 356:858-866.
2. Murphy G, Groeger JA, Greene CM. Twenty years of load theory – where are we now, and where should we go next? Psychon Bull Rev. 2016;23:1316-1340.
3. Lave J, Wenger E. Situated learning: legitimate peripheral participation. New York, NY: Cambridge University Press; 2007.
4. Greenhill J, Richards JR, Mahoney S, Campbell N, Walters L. Transformative learning in medical education: context matters, a South Australian longitudinal study. J Transformative Education. 2018;16:58-75.
5. Lenning OT, Ebbers LH. The powerful potential of learning communities: improving education for the future. Washington, DC: Office of Education Research and Improvement; 1999:1999.
6. Ferguson KJ, Wolter EM, Yarbrough DB, Carline JD, Krupat E. Defining and describing medical learning communities: results of a national survey. Acad Med. 2009;84:1549-1556.
7. Osterberg LG, Goldstein E, Hatem DS, Moynahan K, Shochet R. Back to the future: what learning communities offer to medical education. J Med Educ Curr Dev. 2016;3:67-70.
8. Greenhill J, Walters L. AM last page: longitudinal integrated clerkships. Acad Med. 2014;89:526.
9. Hirsh D, Walters L, Poncelet AN. Better learning, better doctors, better delivery system: possibilities from a case study of longitudinal integrated clerkships. Med Teach. 2012;34:548-554.
10. Thistlethwaite JE, Bartle E, Chong AAL, Dick M, King D, Mahoney S, Papinczak T, Tucker G. A review of longitudinal community and hospital placements in medical education: BEME Guide No. 26. Med Teach. 2013;35:e1340-e1364.
11. Walters L, Greenhill J, Richards J, Ward H, Campbell N, Ash J, Schuwirth LWT. Outcomes of longitudinal integrated clinical placements for students, clinicians and society. Med Educ. 2012;46:1028-1041.
12. Snow SC, Gong J, Adams JE. Faculty experience and engagement in longitudinal integrated clerkship. Med Teach. 2017;39:527-534.
13. Poncelet AN, Mazotti LA, Blumberg B, Wamsley MA, Grennan T, Shore WB. Creating a longitudinal integrated clerkship with mutual benefits for an academic medical center and a community health system. Perm J. 2014;18:50-56.
14. Gonzalo JD, Lucey C, Wolpaw T, Chang A. Value-added clinical systems learning roles for medical students that transform education and health: a guide for building partnerships between medical schools and health systems. Acad Med. 2017;92:602-607.
15. Gonzalo JD, Graaf D, Johannes B, Blatt B, Wolpaw DR. Adding value to the health care system: identifying value-added systems roles for medical students. Am J Med Qual. 2017;32:261-270.