For any organization, the ability to demonstrate impact — to funders, board members, and other stakeholders — is crucial. Demonstrating the value of an organization, project, or service is important to access financial support, engage stakeholders, and achieve organizational objectives (Fedorciow, 2012). Within the Canadian sport system, sport administrators are increasingly required to design and evaluate initiatives (e.g. new resources, programs and policies) in a way that determines if the right people know about them, if they work, and if they are being used in an appropriate manner — all while advancing the organization’s mission and objectives (Lawrason et al., 2020).

One tool that offers insight into the processes and outcomes that contribute to long-term impact is the RE-AIM Framework (Glasgow, Voigt & Boles, 1999). To help sport administrators develop an evaluation plan to assess the impact of new or ongoing initiatives, this article introduces the Framework, and provides two “real-world” examples of how it has been used to evaluate impact in the Canadian sport sector.

The RE-AIM Framework

RE-AIM was originally developed to evaluate public health interventions, such as community-based physical activity programs (e.g., Estabrooks, Bradshaw, Dzewaltowski, & Smith-Ray, 2008). More recently, it has attracted the attention of sport-related researchers because it provides a systematic approach for capturing and organizing the information needed to evaluate an initiative’s impact across multiple levels of the sport system.

For an initiative to have impact, it needs to be implemented and sustained in a way that produces the desired outcomes for its target population in the long-term (Glasgow et al., 1999). RE-AIM is an acronym for five key dimensions of impact — reach, effectiveness, adoption, implementation, and maintenance. As a sport administrator, consideration of these five dimensions can provide you with a starting point for WHAT to evaluate when assessing impact.

RE-AIM Dimension	Definition	Example
Reach	The number, proportion and representativeness of people in the target population who engage in, receive or are affected by the initiative.	The number of NSO administrators that have registered for a workshop on concussion policy, and the percentage of all NSOs represented.
Effectiveness	The positive and negative outcomes that result from the initiative.	Participants in a communications webinar complete an audience poll before and after the webinar to assess changes in their level of confidence when engaging in stakeholder communications.
Adoption	The number, proportion and representativeness of possible settings and staff that are participating in the initiative.	The total number and percentage of all community sport organizations (in a particular sport) that have developed LTAD plans based on guidance from their NSO.
Implementation	The cost and extent to which the initiative was delivered as planned.	The number of planned versus actual workshops delivered as part of a regional coach engagement strategy.
Maintenance	The extent to which the initiative is sustained over time, which includes an assessment of participant outcomes and organizational delivery beyond six months.	An annual survey sent to current and past participants of a sport for development program to assess participation in and outcomes of the program over time.

Developing an Evaluation Plan using RE-AIM

While the RE-AIM dimensions offer a tool for identifying WHAT you need to evaluate to assess impact, your RE-AIM evaluation plan should also consider WHO you need to engage in the evaluation (e.g., which stakeholders and/or levels of the sport system are implicated in or affected by your initiative?) and HOW the evaluation will be conducted (e.g., What capacity does your organization have for evaluation? What information/data exists or what data do you need to generate?).

WHO? RE-AIM is a practical tool for the sport system because of its ability to accommodate evaluation across multiple levels of administration and stakeholder groups (Evans, McGuckin, Gainforth, Bruner, & Côté, 2015; Finch & Donaldson, 2010; Lawrason et al., 2020) — from athletes, parents, coaches and administrators at the grassroots level to national and international sport organizations. For example, if a national sport organization is interested in evaluating the impact of a new gender equity policy that would affect the coaching staff of clubs and teams at all levels of the sport, the evaluation should likely include the national team, provincial/territorial teams, and community sport clubs or teams. Alternatively, if a community sport club is interested in evaluating its new youth program, the perspectives of program participants, their parents, and their coaches are needed. 

HOW? Another important consideration is the extent to which an organization can leverage existing data to inform the design and evaluation of an initiative (e.g., registration information, feedback forms, social media analytics), and whether or not new data needs to be collected (Evans et al., 2015). Capacity for evaluation can be limited by a number of factors, including the availability and expertise of partners and staff, time, and financial resources (e.g., Millar & Doherty, 2018). For this reason, access to existing data may be the most pragmatic option for some organizations (Lawrason et al., 2020).

What follows are two examples of how the RE-AIM Framework was used to evaluate two initiatives: (1) The Canadian Centre for Ethics in Sport (CCES) True Sport initiative, and (2) Ringette Canada’s guidelines for structuring Children’s Ringette. Knowing that capacity is a significant consideration for many sport organizations, these two examples were chosen, in part, because of their difference in HOW the evaluation was conducted.

Leveraging existing data: A RE-AIM evaluation of True Sport

The CCES is a registered charity and multisport service organization that runs the True Sport initiative through truesportpur.ca to promote value-based sport for Canadians. True Sport relies on its website as the primary delivery mechanism for achieving its objective of promoting values-based sport, which is conceptualized through seven principles for positive sport participation (see truesportpur.ca/true-sport-principles). Canadians interested in the promotion of positive sport can register as a True Sport Member through various member-types (e.g., communities, teams, citizens) free of charge, giving them access to resources and updates via email distribution.

To evaluate the impact of the website through which True Sport is administered, True Sport staff worked collaboratively with a group of researchers at Queen’s University. The decision to use the RE-AIM Framework was based on two main factors: (1) RE-AIM can be adapted to assess multiple end-user outcomes (e.g., athlete, coach, administrator; Finch & Donaldson, 2010), and (2) the RE-AIM dimensions provide administrators with an understanding of how and where to intervene in the initiative to improve end-user outcomes. Of note, a detailed description of this project is published in the Journal of Sport Psychology in Action (see Lawrason et al., 2020).

The first step of the evaluation involved a comprehensive review of the True Sport website to map the objectives of the True Sport initiative onto the RE-AIM dimensions. The types of data that were either publicly available (e.g., Canadian population statistics) or internally collected (e.g., number of visitors to the True Sport website) were then identified. Through discussions between True Sport staff and the research team, a template was developed to categorize and analyze existing data relative to the RE-AIM dimensions. Based on the available data, it was decided to focus the impact evaluation on reach and adoption as sufficient data were not available for the effectiveness, implementation, and maintenance dimensions.

Based primarily on website analytics, several indicators of reach were available. These included the number and demographics (e.g. age, gender) of visitors to the True Sport website, and the type and number of resources downloaded from the website. Website analytics were also the dominant source of data for adoption, which included the total number of True Sport members in Canada. Adoption was further broken down based on member type (e.g., citizen, organization, facility, community) and geographic location (i.e., province or territory). When possible, Canadian population statistics were used to calculate reach and adoption indicators as a proportion of the broader target population (e.g., the number of True Sport members as a proportion of all possible members in the Canadian population).

Takeaway: The fact that data was readily available only for indicators of reach and adoption shows that while existing data can be leveraged for evaluations, there are likely to be gaps. A focus on these gaps, however, provides an important place to start if opportunities (and resources) for new data collection arise.

Collecting new data: A RE-AIM evaluation of small-area games in children’s ringette

In 2019, Ringette Canada introduced guidelines for structuring its new children’s ringette program. The guidelines recommend a shift toward small-area games — such as half-ice or cross-ice — for children in the FUNdamentals and early Learn to Train stages of development (e.g. under the age of 10). Although evidence from other sports suggest that small-area games provide benefits for player development (e.g., Aguiar, Botelho, Lago, Maças, & Sampaio, 2012; USA Hockey, 2019), the guidelines were met with pushback from some members of the ringette community.

With funding from a SIRC Researcher-Practitioner Match Grant and a SSHRC Partnership Engage Grant, Ringette Canada is partnering with researchers at York University and Queen’s University to evaluate the impact of the guidelines and the practice of small-area games at the provincial, community and participant levels. The goal of this research is to provide the first ringette-specific evidence for the benefits of small-area games, while also assessing the uptake of Ringette Canada’s new guidelines across Canada. The RE-AIM Framework was selected for this project because it provides a comprehensive assessment of process- and outcome-related indicators of impact across multiple levels of the sport system that will allow for evidence-informed updates to the content and dissemination of the guidelines in the future.

Phases 1 and 2 of the evaluation targeted the provincial and community levels. In Phase 1, technical directors in all provincial ringette associations were interviewed. In Phase 2, administrators in community ringette associations across Canada were asked to complete an online survey. Although different methods were used to collect the data, the questions asked in the interviews and surveys corresponded to the same set of RE-AIM indicators. Examples of the questions include:

• Reach: Are you aware of Ringette Canada’s recommendation that all games for players in children’s ringette programs be played on an area that does not exceed half the ice surface?

• Effectiveness: Do you think that small-area games are a good idea for children? In your opinion, what are some of the benefits of small-area games?

• Adoption: Does your association have a formal policy in place to guide the implementation of small-area games? Please describe any documents or resources (e.g., strategic plan) that address small-area games in your association.

• Implementation: Please describe how, if at all, small-area games have been implemented in your organization (e.g., size of the ice surface, equipment used, age or skill level of players, rule modifications). During the 2019-2020 season, approximately what proportion of the total games played in the children’s ringette program within your association were on ice surfaces that did not exceed half the ice?

• Maintenance: Does your organization plan to begin or continue supporting Ringette Canada’s recommendation that all games for players in children’s ringette programs be played on an area that does not exceed half the ice surface?

While Phases 1 and 2 are complete, Phase 3 is currently on hold due to the COVID-19 pandemic. Phase 3 will evaluate the effectiveness of small-area games with respect to player’s physical (e.g., passing, shooting, skating) and psychosocial (e.g., enjoyment, quality of peer interactions) development. To do so, games from two matched samples of children’s ringette participants — one group playing small-area games and one group playing full-ice games — will be video-recorded, analyzed and compared on measures of player development.

Takeaway: Collecting data from multiple sources helps to improve the quality of the dataset and provides insight into areas of alignment or disconnect from the perspectives of stakeholders. New data also allows for an evaluation to be tailored and expanded in line with the needs and objectives of the organization. The trade-off is that new data often requires time and resources (e.g., human, financial) to collect. Nonetheless, a number of simple and easy-to-use tools exist to help collect meaningful data — ranging from online surveys, feedback forms, and interactive software (e.g., mentimeter) to in-person or phone check-ins with relevant staff members or stakeholders. Partnerships between researchers and practitioners can leverage resources, skills and expertise that benefit everyone involved.

Tips for implementing the RE-AIM Framework

RE-AIM provides a comprehensive framework for understanding the benefits, challenges, and complexities of implementing programs, policies, and other initiatives in the “real world” context of the Canadian sport system. It also provides a useful tool for the design and evalution of initatives with impact. Nonetheless, for a RE-AIM evaluation to produce meaningful and impactful data, careful planning and forethought are necessary. To increase the likelihood of a successful evaluation, consider the following questions before getting started: 

• For each RE-AIM dimension, which data is most important? In other words, which data will help demonstrate the value of the initiative to funders, board members, or other stakeholders?
• Does this data already exist, or will new data need to be collected?
• How can new data be collected given internal resources that are available? What external resources (e.g., technology, funding) are needed to collect new data?
• Are experts (e.g., researchers) needed to design or execute any part of this evaluation?

Answering these questions will contribute to a well-designed evaluation that demonstrates the impact of organizational initiatives, while capturing valuable information for future plans and improvements.

---

About the Author(s)

Veronica Allan is SIRC’s Research and Innovation Specialist and a Postdoctoral Fellow in the School of Kinesiology and Health Science at York University. Her education and training to date—which includes a PhD in Sport Psychology from Queen’s University and a Munk Fellowship in Global Journalism at the University of Toronto—has equipped her with a unique skillset grounded in innovative research design and knowledge translation, as well as a specialized focus on research communications.

References

Aguiar, M., Botelho, G., Lago, C., Maças, V., & Sampaio, J. (2012). A review on the effects of soccer small-sided games. Journal of Human Kinetics, 33, 103-113.

Estabrooks, P. A., Bradshaw, M., Dzewaltowski, D. A., & Smith-Ray, R. L. (2008). Determining the impact of Walk Kansas: Applying a team-building approach to community physical activity promotion. Annals of Behavioral Medicine, 36, 1-12. doi:10.1007/s12160-008-9040-0

Evans, M. B., McGuckin, M., Gainforth, H. L., Bruner, M. W., & Côté, J. (2015). Coach development programmes to improve interpersonal coach behaviours: A systematic review using the RE-AIM framework. British Journal of Sports Medicine, 49, 871-877. doi:10.1136/bjsports-2015-094634

Fedorciow, L. (2012, November 20). Why measuring impact is essential to any social enterprise. The Guardian. Retrieved from https://www.theguardian.com/social-enterprise-network/2012/nov/20/measuring-impact-social-enterprise-essential

Finch, C. F., & Donaldson, A. (2010). A sports setting matrix for understanding the implementation context for community sport. British Journal of Sports Medicine, 44(13), 973-978. doi:10.1136/bjsm.2008.056069

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327. doi:10.2105/AJPH.89.9.1322

Lawrason, S., Turnnidge, J., Tomasone, J. R., Allan, V., Côté, J., Dawson, K., & Martin, L. J. (2020). Employing the RE-AIM framework to evaluate multisport service organization initiatives. Journal of Sport Psychology in Action. Advance online publication. doi: 10.1080/21520704.2020.1773592

Millar, P., & Doherty, A. (2018). “You can’t just start and expect it to work”: An investigation of strategic capacity building in community sport organizations. Journal of Sport Management, 32(4), 348-361.

USA Hockey (2015, January 29). NHL Analytics Tracking of 8U Hockey Players. Retrieved from https://www.youtube.com/watch?v=CB_Ygapyl7c

---

The information presented in SIRC blogs and SIRCuit articles is accurate and reliable as of the date of publication. Developments that occur after the date of publication may impact the current accuracy of the information presented in a previously published blog or article.