Curriculum

To be successful in their research careers, trainees need Master’s level training in research methods. Unless the trainee specifically wants to work in a public health setting and needs the MPH professional degree, or enters the program already having Master’s level training in health services research, we expect that most fellows will elect to earn an MSCS. The 12-15 credit Certificate in D&I Science is not mandatory if the trainee prefers other electives, but given the importance of implementation science in primary care research, the three core D&I courses (7 credits total) are required for all APCRF trainees. Post-doctoral PhD fellows who have substantial methodological training will work with their mentors to design a Masters’ level curriculum that complements their existing knowledge base.

The MSCS program trains clinicians to be nationally competitive researchers through a structured educational program with interdisciplinary faculty. As noted above, PhDs who already have sufficient basic methodological training will select courses outside of the degree program to augment their skills. The MSCS requires at least 30 credit hours, of which 4-6 must be thesis/research hours. Applied biostatics and epidemiology are mandatory courses, along with an additional applied biostatistics course, research and critical appraisal seminars in clinical science, ethics and responsible conduct of research, grant writing, and a choice of one of the following: design and conduct of clinical trials, research methods in epidemiology, advanced data analysis, or analytical epidemiology (15-16 credits total). In addition to core courses, trainees pursuing an MSCS will complete the D&I core courses (7 credits) or the full certificate to count toward the elective courses required for this degree. Fellows interested in this degree option should contact the APCRF Program Manager prior to starting their fellowship appointment to help facilitate the D&I core courses counting toward the MSCS’s 8-10 elective course credits required.

The MPH program at the University of Colorado School of Public Health requires courses in each of the core areas of public and population health (foundations in public health; epidemiology; health care systems, management and policy; social and behavioral factors; environmental and occupational health; and biostatistics-17 credits total), at least 12 credits in a concentration area (applied biostatistics with an emphasis on global public health, community and behavioral health, environmental and occupational health, epidemiology, heath systems management and policy, maternal and child health, and population mental health and well-being), a public health setting practicum, a capstone project generally fulfilled by the trainee’s primary research project, and electives for a total of 42-45 semester credit hours. The MPH provides a broad understanding of public and population health and research skills. Fellows pursuing an MPH should contact the APCRF program manager prior to starting their fellowship appointment to help facilitate the D&I core courses counting toward the elective course credits required for their specific MPH concentration.

The D&I Certificate Program addresses a local and national need for rigorous training in D&I Science in health services research. The Program provides pragmatic training to develop competencies in D&I science and practice. The certificate program courses have been mapped to core D&I competencies including: recognizing the level of evidence required to justify disseminating a program; developing a rationale for selecting a D&I model or framework to guide one’s study design; designing pragmatic interventions; selecting appropriate process/outcome measures; engaging key stakeholders to select the appropriate implementation strategies for a given context; using study designs that assess and plan for adaptations, sustainability, and future dissemination; conducting transparent evaluations of implementation strategies that address generalizability, costs and health equity; balancing the need for site adaptations with internal validity needs; and use of mixed methods approaches for evaluation of D&I studies. The Certificate requires 12 graduate credit hours and can be completed as part of the MSCS or the MPH program or in lieu of a Master’s degree for trainees who start fellowship already having a PhD or master’s level training in health services research. The three core courses (7 credits total) are required for all APCRF trainees. 

Weekly 1-hour work-in-progress (WIP) sessions on Monday afternoons from 3-4pm in the ACCORDS office space on campus use group mentoring from core faculty plus peer-to-peer mentoring to guide trainees to conceptualize and address impactful and feasible research questions through all phases of the research process: study design and methods, approaches to stakeholder engagement, output from statistical analyses, and dissemination strategies. Trainees also critically analyze data collection instruments and manuscripts, and practice presentation of abstracts and posters in preparation for scientific meetings. WIP sessions are mandatory for trainees, program faculty, and methodologists. One trainee presents their research at each WIP session, and each trainee can expect to present roughly once per month. The WIP structure satisfies the criteria for a successful research environment; because there is a “critical mass” of trainees, mentors, and faculty from different disciplines, the trainees receive a great deal of input and support.

During WIP sessions trainees: 1) learn to effectively communicate their research objectives and findings using verbal, written, and audiovisual aids; 2) learn leadership skills by facilitating interprofessional small group seminars and managing the ebb and flow of scientific discourse; 3) learn to appreciate and incorporate critical feedback to strengthen their research; 4) gain input from program mentors who are not directly involved in their project; 5) develop critical thinking skills by assessing and critiquing the research of their peers; 6) gain exposure to a diversity of research topics, study designs, and analytic methods by hearing about the details of their peers’ projects; and 7) establish accountability for productivity. 

The APCRF program sponsors mini-course lectures shared with the SCORE Fellowship at ACCORDS in areas not otherwise identified in the School of Public Health, CCTSI MSCS and D&I curricula. These lectures take place in the ACCORDS office space on campus on Monday afternoons before WIP from 1-3pm. Mini-courses provide trainees with additional primary care research skills, especially the courses related to practice-based research networks, community-based participatory research, and economic evaluation and value. Below is a list of several mini-courses the fellowship features for our trainees:

Identifying research questions and study designs	Teaches how to specify a research question and match it to an appropriate study design. Included is an overview of common pragmatic study designs. Trainees write and critique each other’s questions and research designs.
Community engagement and community-based participatory research (CBPR) and Research in PBRNs	Covers CBPR, a collaborative research process that equitably involves all partners in the research process and community engagement. Topics include background, rationale, and methods for conducting CBPR. Trainees are eligible to obtain seed money from CCTSI to complete pilot studies that utilize a CBPR framework. Teaches about PBRNs, groups of clinical practices organized to conduct research studies relevant to the needs of the participating practices, their patients and communities. Sessions cover the importance of PBRNs in primary care research and the types of studies generally conducted.
How to Work with an Analyst, Review of Analytic Plans, and the Secondary Database curriculum	Introduces secondary data, how to ask a research question appropriate for secondary data, strengths and pitfalls in secondary data analysis, methodological issues, ethical and regulatory implications of using secondary data, and use of specific databases.  The course also touches on specific secondary databases that our recent fellows have used for their projects. Trainees develop an analytic plan using a secondary database.
The ACCORDS Qualitative Research Workshop	An intensive 2-day workshop geared toward junior investigators. Attendees are introduced to the basics of qualitative research, how to develop a qualitative research question, and the methods used to collect and analyze qualitative data. The workshop is held once or twice per year, with roughly 65 people applying each time and 12-16 accepted. Our fellows will be guaranteed a position in this popular workshop that requires potential attendees to apply for a spot.
Scientific Writing	Covers constructing clear and forceful sentences and paragraphs, avoiding passive voice and common grammatical errors, planning writing projects, written presentation of research findings (abstracts and manuscripts). Trainees provide writing samples which are used to exemplify the writing principles under discussion.
The Grant Writing Course	Consists of lectures on the types of grants available and the “how to” of obtaining grant or individual fellowship support from federal sources and foundations. Trainees write grant sections and review sections of each other’s grants. Each trainee’s grants are reviewed by another trainee, a non-primary mentor co-director, and our biostatistician. Fellows take this course for credit during the spring of their second year in the program, which consists of five lectures shared with SCORE fellows and five review sessions with the APCRF mentors and our fellows exclusively.  For more information on taking the grant writing course for credit, please contact the Program Manager.
Sustainability, Value, and Cost: Developing Enduring Health Care Interventions	Covers methods for evaluating sustainability, cost, and value of health care innovations and effectively communicating findings to all stakeholders. The economic methods include cost-benefit, cost effectiveness and return on investment analysis, as well as a focus on applying time drive activity based costing and other micro-costing approaches in each of these methods. The implementation science methods include frameworks for measuring and reporting contextually relevant sustainability factors, discovering stakeholders’ value propositions, and effectively disseminating information that are tailored to each stakeholders’ value proposition. Seminars will address seven stakeholder perspectives: (1) patients and consumers, (2) providers and staff, (3) health care organizations and health systems, (4) payers, (5) purchasers, (6) government and policy makers, and (7) society.
Equity Measurement, Reducing Health Disparities in Practice and Research and Cultural Tailoring	Focuses on designing interventions to reduce disparities in health and health care through measurement and tailoring of evidence-based interventions to improve effectiveness in disparities populations based in theoretical frameworks. The course goals will focus on: 1) how race, ethnicity, culture, geography, and SES can be used to design interventions to reduce disparities; 2) the interplay of community, state, federal, institutional, and individual level strategies to reduce disparities; and 3) active community engagement strategies to decrease disparities.
Dissemination of research findings	Discussion of dissemination theories and frameworks as well as innovative dissemination methods outside of the classic academic routes of manuscripts and research presentations.

As part of the mini-course curriculum, the program also offers didactic sessions that address the need for leadership and professional skills to prepare for careers in diverse primary care research settings, including managed care settings, federally qualified health centers, academic health centers, and government/policy environments. These lectures take place in the ACCORDS office space on campus on Monday afternoons before WIP from 1-3pm.  Below is a list of some of the career development mini-courses that the fellowship offers:

Time management	Addresses an overview of time management and balancing clinical and research responsibilities. One covers time management challenges, strategies, and tools for researchers and the other addresses real-world obstacles to achieving clinical and research success and provides tools to prioritize issues and increase productivity.
Project management	The project managers at ACCORDS lead two sessions addressing developing work plans, team communication, managing budgets, reporting to funders, and timelines.  These sessions alternate every other year with PI Management of Grant Funds lecture.
Promotion, Obtaining a position in primary care research	This is an interactive session on the diversity of primary care research positions available, letters of inquiry, CVs, giving a job talk, and negotiating a position. Senior researchers describe their career paths, settings for research, and lessons learned.
Making national contacts and taking advantage of national career development opportunities	We encourage trainees to schedule formal meetings with potential external mentors to form a national network, often done through national meeting programs such as mentoring programs through national primary care organizations. We also encourage trainees to participate in career development skills building workshops at national meetings. Participating teams must commit to attending all sessions of the program in a given year.
Transitioning out of training	Each trainee will devote one WIP session to discussing skills and support needed for an effective transition from trainee to their first research position.

In order for trainees to learn and apply research skills, they design and complete at least one major research project, present their findings at a national meeting, and write a manuscript for a peer-reviewed journal. Many trainees in our past fellowships have completed two or more projects during the program.

At the beginning of training, each trainee is assigned one of the core mentorship faculty as a program mentor. Throughout their training, trainees meet weekly or bi-weekly with a core faculty mentor to develop and execute their primary research project. provides programmatic mentorship, intended to ensure understanding of program requirements, achievement of program milestones, achievement of individual learning objectives, and productive relationships with the sponsoring academic unit. Subject mentorship in a specific clinical topic of research is provided by the fellowship mentor or a separate mentor. All fellowship mentors provide methodological mentorship for the trainee because of their broad background in primary care research. If the specific methods related to the trainee’s research are not within the expertise of the co-directors, methodological mentors are identified from ACCORDS, primary care departments or divisions and other departments of CU-AMC, the Colorado School of Public Health, or CU Denver. Through individual meetings with the primary mentor, any additional content or methods mentors, and the work-in-progress (WIP) sessions, each trainee forms a mentorship team. The primary mentor assures that the trainee is meeting with the mentorship team regularly.

Within this mentoring structure, trainees spend the first summer of the program discussing research ideas and conducting directed literature reviews. During the late summer and early fall, trainees focus on developing an idea for their primary research study and begin to design a research protocol. This evolving protocol is discussed at weekly WIP sessions and is formally reviewed in the spring of the first year. Following protocol development and approval, trainees are supported in conducting their research projects including collecting data, supervising analyses, and interpreting and presenting results. At project completion, trainees submit abstracts to national meetings and write a manuscript.