---

Conduct a Pragmatic Intervention Pilot - Detailed Guidance

Why it Matters
Even effective interventions fail if they are too burdensome, misaligned with workflows, or difficult to sustain. Pragmatic pilots emphasize fit-to-context to improve generalizability to routine settings rather than in controlled study settings.

The intervention (the "thing" being delivered) and the implementation strategies (how to do the "thing") should be feasible, efficient, adaptable, and aligned with everyday practice.

Pragmatic trials save time and resources over the long run by implementing the trial in real-world settings. They also outlast typical trials because they are more likely to be sustained if they are implemented with real-world settings and with real-world users.

Example
The intervention is a brief decision aid focused on core benefits and risks of defibrillators. The implementation strategy relies on existing clinic staff introducing the aid before the clinician visit, rather than adding new personnel or visit time. Pilot testing revealed that this original plan took too long, prompting streamlining to preserve feasibility. The pragmatic pilot explicitly tests whether this simplified approach can be delivered consistently during routine care.

Actions Steps

Intervention - define the problem: Clearly Define the Problem, cite supporting evidence for your intervention’s previous effectiveness, and describe its key features to assess its fit for addressing the health gap.

Intervention - streamline: Consider if there are components that might be removed or streamlined without losing effectiveness.

Identify which components of the intervention can be modified or simplified without losing effectiveness in a pragmatic setting.

Implementation strategies – review: We recommend reviewing previous studies through a rapid literature review or your own assessments to determine which strategy best fits your intervention in the context you are working in.  

Implementations strategies – specify: Try to clearly specify the details of your strategy, including how you plan to implement the intervention, using strategy specification tools. Below are links to commonly used strategies and tools for specifying them.

Consider efficiency, burden, adaptability, and alignment with existing workflows (in the intervention delivery setting) – this helps plan ahead for sustainability. (see Implementation Strategies Detailed Guidance)

Impact Summary
Clearly outlining the intervention and implementation strategies will help you better align your project with the specific context in which it is intended to be used, ultimately improving outcomes, feasibility, adoption, and sustainability.

Tools and Resources

• The Implementation Strategies tip sheet and detailed guidance are resources and tools related to intervention mapping and strategies
• Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide
• Proctor et. al. classic reference on specifying and reporting implementation strategies
  

Why it Matters
Interventions that are developed or tested in highly controlled environments often do not translate well to routine practice because they fail to reflect the realities of the settings and people responsible for delivering them. Context strongly influences feasibility, effectiveness, implementation success, and sustainability. Designing and testing interventions in real-world settings with the people who would normally deliver them helps ensure that results are relevant, actionable, and more easily sustained over time.

At the same time, contexts are not static. Organizational priorities, workflows, staffing, resources, and patient needs change. Interventions that cannot adapt to these changes often fail to sustain, even when they are initially successful. Building flexibility while maintaining core intervention components allows programs to remain effective and relevant across diverse settings and over time.

Example
A pilot is conducted in three routine cardiology clinics that vary in size and patient demographics. Usual clinic staff, including medical assistants and clinicians, deliver the intervention as part of standard visits, without adding research-specific personnel or workflows.

During implementation, some clinics request a digital version of the decision aid for patient portals, while others prefer printed handouts. The team identifies the intervention's core components, key messages and visual risk communication, and allows flexibility in delivery format. Adaptations are documented along with the reasons for the changes and relevant contextual factors. This approach helps ensure that findings reflect real-world practice while supporting local fit and cross-site learning.

Action Steps
Select and Understand the Context

• Choose settings that closely resemble those where the intervention is intended to be used. 
• Ensure participants are similar to the populations expected to benefit and reflect the groups typically served in those settings. 
• Use usual staff whenever possible rather than research personnel to deliver the intervention. 
• Review the physical environment, available resources, organizational characteristics, and prior experience with similar interventions.

Assess Local Workflows and Readiness

• Conduct a landscape analysis to identify local strengths, needs, resources, organizations, and community assets that may influence implementation.
• Assess existing workflows, staff relationships, leadership engagement, and implementation climate.
• Identify anticipated barriers and facilitators, including staff capacity and available resources.
• Determine how the intervention can fit within existing workflows rather than creating additional burdens whenever possible.
• Create process maps or workflow diagrams to clarify staff roles, responsibilities, interactions, and key implementation time points.

Impact Summary
By designing for context, you can better understand how an intervention performs across diverse settings and populations. Staff are more likely to support and continue programs that fit their workflows and local needs, and organizations are better positioned to sustain interventions as contexts evolve.

Tools and Resources

• Chapters in Evidence-Based Public Health provide practical guidance on how contextual factors influence implementation success and how dissemination and implementation strategies can be planned, adapted, and evaluated across public health settings. 
• Weiner et. al. work (including organizational readiness for change and implementation climate frameworks) offers foundational tools and concepts for assessing how leadership, culture, resources, and staff readiness shape implementation outcomes.
• The Implementation in Context (ICON) Framework synthesizes evidence into a comprehensive framework describing context at micro-, meso-, and macro-levels, including six domains, 22 attributes, and 108 contextual features that can facilitate or hinder implementation.
• Recent context-assessment studies expand on frameworks such as ICON and CFIR by developing measures and methods to systematically assess contextual determinants that affect implementation effectiveness and long-term sustainability. Example resource: ICON Framework article and supplementary materials.
• Designing for Dissemination and Sustainability to Promote Equitable Impacts on Health – This resource outlines principles and methods for designing interventions that are easier to disseminate, sustain, and adapt across diverse settings while promoting health equity through improved fit with implementation contexts.
• Iterative Practical, Robust Implementation and Sustainability Model (iPRISM) paper and webtool – iPRISM is a pragmatic framework and accompanying web-based tool that helps teams assess and improve the fit between an intervention, its recipients, and its organizational and external contexts throughout planning, implementation, evaluation, and sustainment. 
  

Why it Matters
Contexts are not static. Organizational priorities, workflows, staffing, resources, and patient needs change. Interventions that cannot adapt to these changes often fail to sustain, even when they are initially successful. Building flexibility while maintaining core intervention components allows programs to remain effective and relevant across diverse settings and over time.

Action Steps
Plan for Adaptation

• Identify which intervention components and implementation strategies must remain consistent (core components) and which can be adapted to fit local needs.
• Modify aspects such as delivery personnel, timing, format, modality, or implementation supports based on contextual findings.
• Allow flexibility while preserving the essential functions of the intervention.

Monitor and Learn Over Time

• Monitor contextual changes throughout implementation and sustainment.
• Document all adaptations, including what changed, why the change was made, and the contextual factors that prompted it.
• Update implementation strategies as needed to maintain fit with the evolving setting.
• When feasible, study multiple settings, staff groups, and populations to enhance generalizability and broader applicability.

Impact Summary
Thoughtful adaptation increases feasibility, relevance, cost-effectiveness, and sustainability while generating practical guidance for future implementation. 

Why it Matters
Models help teams align on goals, make design choices explicitly, and communicate decisions clearly to partners and funders. Some teams may choose to use an existing theory, model or framework (TMF) that they know fits well with their project plan. Other teams may find it valuable to start with a simple logic model, add complexity and then decide on whether to use their own logic model or choose/adapt an existing TMF. Using a model, like a logic model or other TMF, to clarify assumptions, pathways, and outcomes can help you decide upon outcomes and key intervention functions. Also, it may help you later to generalize your findings to other context with some potential adaptation. We do not suggest developing a new TMF for the field. There are many. Your logic model may suffice or adapting/combining existing TMFs should fit your needs. 

Example
1. The team starts with a simple implementation logic model linking the decision aid, limited visit time, staff delivery, and patient engagement. As the study evolves, they map this logic model to RE-AIM constructs to clarify outcomes and reporting. The model helps the team explain why uptake varies across clinics. It also guides outcome selection and interpretation for partners and funders.
This more basic logic model template, in this case, sped up the project by quickly identifying and targeting the problem with pointed action: 

2. Below is an example of a more detailed logic model that may fit your needs without a more formal framework. Alternatively, the concepts in this logic model can be used to help you select a framework.

Actions Steps

Start simple. Use the simpler logic model and add complexity only if needed.

Consider the partners involved and develop logic models with appropriate complexity for them. You may need several versions ranging from simple to very complex.

For complex projects or if you want to submit implementation science papers or a grant, you very likely need a more formal framework such as CFIR, PRISM, EPIS or RE-AIM

Your initial logic model should be amended iteratively to fit your project needs and periodically as your project proceeds.

Use models to clarify outcomes, mechanisms, and assumptions.

Revise the model as the project evolves.

Impact Summary
Stronger alignment across the team and clearer justification for design and measurement choices

Tools and Resources

• Gomes R et al. provide A Guidebook to the Pragmatic and Iterative Use of PRISM and RE-AIM for Planning, Implementation, and Sustainment that provides guidance to apply the widely used frameworks and use them iteratively. 
• The Nilsen book, Implementation Science: Theory and application provides a summary of the leading theories, models and frameworks in implementation science along with chapters on applying each.
• Logic Model Templates – simple, more complex, and very detailed templates to use    Implementation Research Logic Model website provides examples of the use of the IRLM (very detailed version) and template PDFs and PPTs. 
  
• Most Strongly Recommended Resource: The D&I Models Webtool is an interactive resource for selecting, combining, or adapting D&I TMFs. It is particularly useful for reviewing framework-specific assessment methods and checking whether your logic model is well aligned – or if an alternative or complementary framework is needed. The tool also provides clear guidance on how to combine or adapt frameworks. In addition, its expanded Assess section offers a strong repository of outcome measures linked to different constructs and TMFs, making it especially valuable for outcome assessment.
  

 

Why it Matters
Recruitment is the first step to ensuring a project that has strong reach and representation. Multi-level recruitment means considering all the steps including deciding what health systems clinicians, staff and patients to recruit, and how to do this in ways that maximize participation and representativeness of participants at these various levels. By strategically deciding who to include, recruiting and engaging participants, you are much more likely to reach the intended audiences and have a bigger impact.  

Example
Clinics are recruited through existing health system leadership channels. Clinicians are invited during routine staff meetings, and patients are identified through the electronic health record. Eligible patients are both invited as part of pre-visit materials and approached during regular visits using standard clinic communication practices. This multi-level approach improves reach while minimizing burden.

Actions to Consider
Think about multiple levels. Which settings need to or can be involved? Who is involved from all levels from system leaders and delivery staff to patients and caregivers. Things may be constrained by your resources available and level of trial you are attempting to design. 

Minimize exclusion criteria where possible (e.g., low-resource clinics; anyone having comorbid conditions) and remove barriers to participation (e.g., requiring lots of staff time or in person visits). 

Consider using recruitment strategies that are already in place. Learn from who is already being reached and expand or modify as needed.

Identify who is not being reached and adjust strategies accordingly. Engaging partners can help enhance representation.

Meet participants and partners where they are. Select methods, with your partners, that allow you to track progress (See Engagement Tips). This will allow for future adaptations.

Impact Summary
Ending up with recruitment strategies that are broadly applicable, feasible, and sustainable in your partner settings.

Tools and Resources

• The Expanded Consort diagram paper and fillable worksheet can guide recruitment efforts by helping focus on eligibility and representativeness at all levels from community to patients
• RE-AIM Planning Tool (Reach) provides planning questions and downloadable worksheets to help teams design recruitment strategies that maximize participation, improve representativeness, and track who is and is not being reached across multiple levels. 
• PCORI Tools and Resources to Support Engagement in Research offers practical guidance, tools, and examples for engaging patients, caregivers, clinicians, and health system partners throughout recruitment and study implementation to improve participation and representation. 
• Pragmatic Applications of RE-AIM for Health Care Initiatives in Community and Clinical Settings (Glasgow & Estabrooks, 2018) provides practical planning questions ("who, what, where, how, when, and why") that help researchers design recruitment and implementation strategies that are feasible, representative, and relevant in real-world clinical and community settings.
  

What This Focuses on
Selecting a study design that balances speed and rigor with feasibility in real-world conditions. Your experimental design is the basis for evaluating the effects of your intervention and/or implementation strategies. There are many designs to choose from, which depend on the questions you want to answer:

• Experimental designs can be as simple as testing an intervention in a setting for a short period of time to comparing different interventions in a randomized design in multiple sites over years.  
• Some common designs might be a randomized trial (randomized at the patient, clinician, clinic, or system level); A case series (better for less common events); A pre-post design where you try the intervention on everyone for a short period of time.  
• Many experimental designs include a comparator condition (sometimes called a “control group”) to ensure conclusions are meaningful and relevant to real world decision-making

Why it Matters
Traditional, highly controlled, randomized designs may be impractical or unethical in routine care settings. Pragmatic pilots often require more flexible approaches

Example
The team selects a stepped-wedge design, so all clinics eventually receive the decision aid. Clinics transition from control (no intervention) to the intervention at staggered time points based on operational readiness. This design balances rigor with feasibility and helps understand replication (or lack of such) across settings. It also aligns with leadership priorities to avoid withholding a potentially beneficial tool.

Actions Steps

Consider Pragmatic Designs: such as cluster randomization, stepped wedge, time-series, or quasi-experimental approaches.

Optimize Standardization: Decide which elements must be standardized and which can vary.

Define Comparators: Ensure comparator conditions are meaningful to real-world decision-makers.

Impact Summary
Designs that produce credible, actionable evidence without disrupting care delivery.

Tools and Resources

• Prioritization Check-in Tool
• Use the PRECIS-2-PS (article) and associated PRECIS-2 website to determine which design elements are most relevant to the study
• Curran, G.M., Smith, J.D., Landsverk, J., Vermeer, W., Miech, C.H., Kim, B., Cruden, G., Fernandez, M.F.,   & Brown, C.H. (2023).Design and analysis in dissemination and implementation research. In RC Brownson, G Colditz, EK Proctor (Eds.) Dissemination and Implementation Research in Health. (3rd Ed.) New York: Oxford. Pages 291-326
• This University of Washington website provides a nice UW Overview of Study Designs in Implementation Science
• Designs that emphasize longitudinal assessment and iteration: time series, roll-out designs, repeated measures
  

Why it Matters
Measuring pragmatic outcomes that matter to patients, teams, and decision-makers in everyday practice increases the likelihood that findings are used, sustained, and scaled. 

Pragmatic Measures*

Required Criteria	Additional Criteria
Important to key partners- helps them address priority issues	Low probability of harm
Brief, burden is low to moderate	Address public health or quality of care goal(s)
Broadly applicable	Related to a theory or model
Sensitive to change- can show intervention effects	“Maps” to “gold standard” metric or measure
Actionable: key partners see it as feasible and helpful

*Note- these criteria are adapted from Glasgow and Riley. There are also other pragmatic criteria (e.g., Stanick & Lewis et. al.) and we encourage you to think creatively about what best fits your project

Key Pragmatic Outcome Types

Outcome Type	Definition	Example
A. Implementation Outcomes	Observable effects of integrating interventions, explaining contextual success or failure	reach, adoption, fidelity, cost, maintenance
B. Patient-centered Outcomes	To ensure that interventions are meaningful to patients and support personalized care, enhancing engagement and uptake by aligning with what patients value most	quality of life, goal alignment, experience of care
C. Clinical Effectiveness Outcomes	Evidence that interventions improve health under real-world care conditions	health status, symptoms, disease status
D. Mechanisms (Other)	Processes and contextual factors explaining how and why outcomes occur	approval workflows, existing resources

Example
Implementation outcomes include reach, adoption by clinicians, time required to deliver the aid, adaptations made, costs of implementation, and sustainment (or intent to sustain or adapt at study end). Patient-centered outcomes focus on perceived alignment with patients’ goals and satisfaction with the decision process. Clinical effectiveness outcomes assess decision quality and downstream treatment alignment as well as traditional biomedical outcomes. Mechanisms such as workflow fit and increased confidence in addressing the problem are examined to explain variation in outcomes.

Actions Steps
Include both implementation and patient-centered outcomes in all pragmatic pilots.

Use brief, low-burden measures when possible.

Rely on routine data sources (e.g., EHRs) where feasible.

Use mixed methods to understand what worked, how, and why.

Impact Summary
Results that are actionable, generalizable, and useful for future implementation decisions.

7A. Implementation Outcomes

Example
A project team uses the RE-AIM framework to prioritize implementation outcomes in a study in the VA. Since the intervention is mandatory, Adoption is irrelevant, and Cost reporting is unnecessary because the health system will cover expenses indefinitely. The team evaluates Reach and Representativeness, ongoing Effectiveness, fidelity and adaptations to the Implementation plan, and long-term Maintenance. Findings show clinic staff did not implement with fidelity, requiring adaptations for successful implementation.

Actions Steps
Select an implementation framework (e.g., RE-AIM; PRISM; CFIR; EPIS; NPT – more on D&I Models Webtool and section on models above for help) and outcome measures that align with your study goals and context.

Measure feasibility or acceptability in planning, but actual reach, adoption and implementation during the trial. 

Use mixed methods to capture both quantitative and qualitative insights into implementation processes.

Pragmatic Use of RE-AIM Implementation Outcomes

RE-AIM Dimension	Key pragmatic questions to consider and answer
Reach (Individual level)	WHO is intended to benefit and who actually participates or is exposed to the intervention?  (Participation rate and representativeness)
Effectiveness (Individual level)	WHAT is the most important benefits you are trying to achieve and what is the likelihood of negative outcomes? (Main and subgroup (equity) effects on multiple outcomes and unintended consequences)
Adoption (Setting and Staff levels)	WHERE is the program or policy applied and WHO applied it? (Beginning participation rate and representativeness of settings and staff)
Implementation (Setting and Staff levels)	1-HOW consistently is the program or policy delivered,  2-HOW is it adapted,  3-HOW much will (did) it cost, and  4-WHY will the results come about?
Maintenance (Individual and Setting levels)	HOW LONG will it be sustained (Setting level); and how long are the results sustained (Individual level)?

Most of the issues in the table above are straightforward, but there are a few issues related to collecting RE-AIM (and other implementation outcomes) that are often overlooked. See the table below for more information on these issues.

Potential Pitfalls when Collecting Implementation Outcomes and How to Address Them

Often Overlooked	Actions to Address
Narrowly defined or undocumented reach or representativeness	Assessing Representativeness or similarity of those participating or benefitting compared to those who do not is important for all RE-AIM dimensions
Using single method approaches	Multiple and mixed methods are encouraged for all dimensions to understand what, how, and why your results come about. Think creatively- there are tons of other qualitative assessment methods besides focus groups; AI can increasingly help with evaluation.
Implementation is the most complex and multi-faceted RE-AIM dimension	It consists of three parts: How consistently are the key functions of the intervention delivered (Fidelity) What Adaptations are made to make the intervention work in different settings? How much and whose time and money does it take to deliver the intervention (Cost)
Limited time to observe the long-term impact of your intervention or its continued delivery	Assess if it can continue to be delivered after completing the study, and if implementing clinics and staff are intending to do so if your study is not long enough to measure actual maintenance.

Tools and Resources

• RE-AIM Website: Describes and provides framework details, measures, and guides.
• PRISM & RE-AIM Guidebook: Gomes et al. (Re). Provides practical guidance for applying RE-AM and PRISM.
• iPRISM Webtool: Web-based assessment with automated feedback on RE-AIM outcomes and contextual factors.
• Proctor et al. (2023): Scoping review of implementation outcomes research in Implementation Science. Summarizes research on implementation outcomes

Effectiveness Outcomes
In good pragmatic trials, effectiveness outcomes measure what matters to patients and decision-makers in everyday settings. There are 1) patient-centered effectiveness; and 2) more traditional clinical effectiveness outcomes. You might also consider 3) mechanisms if you are interested in the theoretical reasons or processes through which results are obtained.

7B. Patient-Centered Outcomes

Example
A project included patient-centered outcomes on satisfaction with care and alignment of treatment with personal goals. Assessing what mattered most to the patients allowed the team to ensure optimal patient success in the program. 

Actions Steps
Use validated patient-reported outcome measures (PROMs) such as those recommended in the SPIRIT-PRO extension (Ref).

Engage patients and clinicians early to identify outcomes important to them.

Some examples include decisional regret/satisfaction, knowledge questions about the condition/intervention, value concordance, health outcomes

Include outcomes that are sensitive to change and actionable by care teams.

Tools and Resources

• Ottawa Decision Support Framework provides evidence-based guidance for assessing decisional needs and engaging patients and other stakeholders in shared decision-making, with tools to identify barriers, clarify values, and support implementation of patient-centered interventions. 
• SPIRIT-PRO (SPIRIT Patient-Reported Outcome Extension) recommendations provide guidance for incorporating patient-reported outcomes into clinical trial protocols, helping researchers plan participant-centered outcome collection, minimize missing data, and improve the quality and relevance of PRO measurement.
• D&I Models Webtool – Assess helps researchers identify implementation determinants, assess barriers and facilitators, and select appropriate implementation strategies and measures to inform study design and adaptation throughout a project.

7C. Clinical Effectiveness Outcomes

What We Mean
These are traditional health outcomes such as symptom improvement, disease progression, hospitalization rates, or mortality, assessed in real-world settings.

Example
A project focused on physical activity used a pragmatic trial to assess cardiovascular outcomes including improved resting heart rate and VO2 max scores. These markers showed substantial improvement in a relatively short period of time, suggesting that the intervention was working as intended across diverse populations and settings.

Tools and Resources

• PRIME-9 initiative (Pragmatic Research and Innovation through Multinational Experimentation) describes an international collaboration to strengthen pragmatic randomized clinical trials by promoting representative recruitment, embedding research in routine care, engaging patients and health systems, and generating real-world evidence that is broadly applicable across diverse healthcare settings.

7D. Mechanisms (Other)
Optional; primarily for research grants planning an NIH proposal or strong interest in theoretical issues

Example
An expanded logic model can illustrate how specific strategies are expected to influence outcomes via underlying mechanisms. See the resources below on logic models that provide examples of comprehensive logic models that include mechanisms.

Realist analysis is a leading approach to studying mechanisms, focusing on Context-Mechanism-Outcome (C-M-O) relationships. (See Pawson book reference below).

Realist analysis asks complex questions about timing, context, delivery, and population to uncover causal pathways. A realist approach attempts to answer questions of the form:

“What intervention and implementation strategy functions produce what outcomes under what conditions for what populations in what contexts when delivered by what persons/modalities at what time frames, and how to these results come about?”

Tools and Resources

• Realist Review Methods: Guidance on conducting realist evaluations and reviews (e.g., Pawson & Tilley frameworks). Provides an overview of realist evaluations featuring mechanisms.
  • Realistic Evaluation (Pawson & Tilley, 1997) introduces the realist evaluation framework and the core Context–Mechanism–Outcome (CMO) configuration for understanding how, why, for whom, and under what circumstances interventions work. 
  • RAMESES Project: Methodological Guidance for Realist Reviews provides comprehensive guidance, publication standards, training materials, and best practices for conducting and reporting realist reviews and realist evaluations. 
• Logic Model Templates: Visual tools to map hypothesized mechanisms and link strategies to outcomes. Examples are provided in the section on Defining Your Problem and the figure above.
• Smith et.al. detailed the approach to use the Implementation Research Logic Model here
• Pawson’s book The Science of Evaluation: A realist manifesto on the science of evaluation discusses C-O-M frameworks and methods: Structured approaches to analyze context, outcomes, and mechanisms in implementation science.