Blogs are an important tool for delivering evidence-based research and information to the sport community. In the SIRC blog, read how course instructors at Brock University used blog writing as a powerful teaching tool to reinforce student learning and mobilize sport and recreation research.
Use double quotes to find documents that include the exact phrase: "aerodynamic AND testing"
Use double quotes to find documents that include the exact phrase: "aerodynamic AND testing"
This blog is part of a series in collaboration with Brock University. Written by a student in the ‘Program Evaluation in Professional Practice’ course, this blog draws on a student’s first-hand experience conducting a program evaluation during a placement with the Bounce Back League. In this blog, Ashley Romano, a 4th year undergraduate student, offers practical tips to engage children in program evaluations, and make evaluation fun!
Designing activities that cater to the children’s skills and interest is a routine part of any children’s sport program. In the context of program evaluation, choosing data collection methods that work for kids is just as important.
Research has shown children are capable, accurate, and valuable communicators, yet many data collection methods come with considerable limitations related to effectively engaging and understanding children’s experiences (Driessnack & Furukawa, 2011). To overcome these limitations, creative approaches to data collection account for children’s skills and abilities while making evaluation engaging and fun.
Using an evaluation of a children’s sport program—the Bounce Back League—as an example, this blog explores creative approaches to data collection, including movement-based methods, and offers tips and tricks for implementing movement-based methods with children.
Limitations of traditional data collection methods
Traditional approaches to data collection, such as surveys or assessments, may not be suitable for children for several reasons. First, methods that rely on adults to observe and/or report on children’s progress or experiences (e.g., physical literacy assessments, progress reports) diminish what children have to offer. Children think differently than adults, but not less (Carter & Ford, 2013). Second, many self-report techniques, such as surveys, were developed for adults and adapted to be “child friendly” (Driessnack & Furukawa, 2011), but these adaptations do not always ensure sensitivity to the childhood experience. Often, these methods receive low response rates, have poor reliability, and are influenced by social desirability (i.e., wanting to give the “right” answer; Soland et al., 2019). In contrast, creative approaches to data collection can be developmentally tailored, provide valuable insight directly from children’s experiences, and save time by doubling as a program activity.
Creative approaches to data collection
The term “creative” fits nicely within the scope of methods for collecting data with children. Creative approaches incorporate developmentally appropriate strategies, including but not limited to, individual or group interviews, creative play, or thinking (Christian et al., 2010). Creative approaches resonate with the childhood experience and cater to children’s skills and interests (Carter & Ford, 2013). Movement-based methods are one example of a creative approach to data collection that is particularly relevant for evaluations of sport and physical programs. These methods tend to be low-cost, easy to implement, are typically well-received by children due to the active nature of involvement, and can help improve on-task behaviour and performance when engaging in these methods (Savina et al., 2016).
Movement-based methods can be understood as integrating physical activity into data collection and can take many forms, such as cooperative games or team sport in which children use the structure or rules of the game to provide feedback about a program or their experiences. For example, in Ship, Shore, Anchor, children run to the wall labelled “yes/agree” or “no/disagree” when questions are called out about the program. This method is explained in more detail below.
Movement-based methods can be used to capture how and why sport fosters development through children’s perspectives. Using movement is an engaging process that helps children ‘buy in’—and it is fun!
Movement-based methods in the Bounce Back League
The Bounce Back League (BBL) is a nationally-run trauma-sensitive sport program for children aged eight to 12 years old. BBL uses a proactive approach to build children’s skills, including resiliency, as many who attend BBL are at risk of experiencing trauma or adversity. In the BBL, program evaluations are crucial to understand children’s program experiences, and given its sport-focused nature, movement-based methods are a natural fit. While several movement-based methods have been used to inform BBL evaluation, Ship, Shore, Anchor is a notable example.
Ship, Shore, Anchor uses space and movement to elicit children’s thoughts and ideas about their program experience and can be run by program staff or an external evaluator. To prepare, staff label one wall “yes/agree” and the opposite wall “no/disagree.” Another space or object (e.g., a hula-hoop or mat) is labelled the anchor—a neutral space for children who are unsure or do not feel comfortable answering. A staff member then calls out “I-statements” that relate to children’s experiences (e.g., “I feel safe at BBL”) or outcomes (e.g., “I learned to not give up when things are difficult”). How children respond to the questions can be documented by a facilitator. At the end of the activity, staff can foster a reflective discussion in a debrief with the children that is aligned with the evaluation purpose. This discussion can include questions such as “What skills did you use today?” or “What did you learn about yourself through this activity?”. If feasible, the debrief can be audio-recorded or notes can be taken and analyzed as part of the evaluation data.
Tips and tricks for implementing movement-based methods
- Use activities that children know and love: It can be difficult to create a movement-based activity from scratch, so start with an activity that program participants already know and love. Making slight modifications for data collection will be easier knowing the activity is familiar and well-received.
- Demonstrate directions: Making modifications to well-known activities, even if minimal, can be confusing for children who are used to playing an activity a certain way. When explaining instructions, it can be beneficial for facilitators to physically or visually demonstrate changes to the structure or rules.
- Keep it simple: Children have short attention spans and benefit from clear instructions (Carter & Ford, 2013). Keeping language short and simple is important for both the instructions and evaluation questions. “I-statements” and signs/labels are easy ways to keep evaluation questions and instructions clear and simple.
- Check in: It is important to continuously check in with children to ensure they understand what is going on, what is being asked of them, and why. A check in could be as simple as seeking a thumbs up for “I get it,” thumbs down for “I don’t get it,” or a thumbs sideways for “I kind of get it, but not completely.”
Can standardized drills designed to measure strength, speed and explosiveness predict success in complex sports such as football? Analysis from sport scientists at the University of Rhode Island found the NFL Scouting Combine can predict only 20-25 percent of a player’s future success, and fails to capture intangible traits such as leadership, drive and “mental makeup.” The researchers suggest past performance on the field is the best predictor of future performance.
In October 2019, Kenyan athlete Eliud Kipchoge ran a sub-two-hour marathon. Research from the University of Exeter on a previous attempt in 2017 discovered elite runners require a “perfect balance” of VO2 max (rate of oxygen uptake), efficiency of movement, and a high “lactate turn point” (above which the body experiences more fatigue) for success.
“The reality is there simply aren’t nearly enough qualified physicians to provide timely assessments for athletes hoping to return-to-play after a sports concussion…For athletes and parents frustrated with long waiting lists to see a doctor or nurse practitioner, [a role for team therapists in return-to-play decisions] is a potential game changer.” Dr. Pierre Frémont from Université Laval talks about his new research in the SIRC blog.
Data Privacy Day is an international effort held annually on Jan. 28. The purpose is to create awareness about the importance of privacy and protecting personal information and encourages dialogue for organizations about which best practices should be implemented.
The COVID-19 pandemic has brought about many changes for the Canadian sport sector including a shift to remote work and working from home. Unfortunately, this digital response to the COVID-19 crisis has also created new security vulnerabilities. Many sectors are seeing an increase in cybercrime and cyber attacks, not only in frequency but in size, sophistication, and cost to victims. A recent survey of Canadian organizations found an increase in both cyberattack volume and breaches during the past 12 months in Canada. It is a mistake to think that hackers are only interested in the large databases of big companies – work-from-home systems can be ripe targets for cyber criminals simply because they are often much easier to infiltrate.
The risk
As modern society leans into an era of revolutionary technological advancement, awareness of the inherent risk of our devices and networks falls short. A lack of understanding around online tools, apps, and technology could lead to catastrophic results for your teams, organization, and public reputation. During a cyber attack, all types of data – employees’ personal information, corporate data, customer information, intellectual property, and key infrastructure – are at risk.
Human error is the leading cause of data and security breaches. For cyber criminals, people are the most exploited vulnerability or “vector of compromise.” A 2019 study by the UK’s Information Commissioner’s Office revealed that human error caused 90% of data breaches. These come from weak password management; using old, non-updated software; careless handling of data; and, as previously mentioned, lack of knowledge in the cyber security field.
So how can sport organizations take a proactive approach to mitigating their data risks before the unthinkable occurs? The answer is building a culture of security centered around digital security awareness training.
How to protect your data
Security awareness training is a fundamental step in the process of modernizing your organization and better enabling it to deal with cyber threats. However, despite its growing importance, keeping employees engaged in these programs remains a major challenge for management teams. To encourage developing a “security-first” culture amongst staff:
- Make data security a priority for everyone, from staff to volunteers.
- Develop policies and training opportunities to educate your team about the best practices to minimize data-related risk.
- Ensure staff have a clear understanding of what actions to take in the event your organization’s data is compromised.
Choosing to invest in a security awareness training program will yield immediate benefits for your organization. These benefits include strengthening organizational resilience against cyber threat risks, developing a “security-first” culture amongst staff, generating buy-in towards greater data security initiatives, and reducing the odds of human error – all contributing factors to mitigating the modern cyber risk.
Read about Ringette Canada’s experience with a ransomware attack, and their management tips if your computer network is compromised.
For any organization, the ability to demonstrate impact — to funders, board members, and other stakeholders — is crucial. Demonstrating the value of an organization, project, or service is important to access financial support, engage stakeholders, and achieve organizational objectives (Fedorciow, 2012). Within the Canadian sport system, sport administrators are increasingly required to design and evaluate initiatives (e.g. new resources, programs and policies) in a way that determines if the right people know about them, if they work, and if they are being used in an appropriate manner — all while advancing the organization’s mission and objectives (Lawrason et al., 2020).
One tool that offers insight into the processes and outcomes that contribute to long-term impact is the RE-AIM Framework (Glasgow, Voigt & Boles, 1999). To help sport administrators develop an evaluation plan to assess the impact of new or ongoing initiatives, this article introduces the Framework, and provides two “real-world” examples of how it has been used to evaluate impact in the Canadian sport sector.
The RE-AIM Framework
RE-AIM was originally developed to evaluate public health interventions, such as community-based physical activity programs (e.g., Estabrooks, Bradshaw, Dzewaltowski, & Smith-Ray, 2008). More recently, it has attracted the attention of sport-related researchers because it provides a systematic approach for capturing and organizing the information needed to evaluate an initiative’s impact across multiple levels of the sport system.
For an initiative to have impact, it needs to be implemented and sustained in a way that produces the desired outcomes for its target population in the long-term (Glasgow et al., 1999). RE-AIM is an acronym for five key dimensions of impact — reach, effectiveness, adoption, implementation, and maintenance. As a sport administrator, consideration of these five dimensions can provide you with a starting point for WHAT to evaluate when assessing impact.
| RE-AIM Dimension | Definition | Example |
| Reach | The number, proportion and representativeness of people in the target population who engage in, receive or are affected by the initiative. | The number of NSO administrators that have registered for a workshop on concussion policy, and the percentage of all NSOs represented. |
| Effectiveness | The positive and negative outcomes that result from the initiative. | Participants in a communications webinar complete an audience poll before and after the webinar to assess changes in their level of confidence when engaging in stakeholder communications. |
| Adoption | The number, proportion and representativeness of possible settings and staff that are participating in the initiative. | The total number and percentage of all community sport organizations (in a particular sport) that have developed LTAD plans based on guidance from their NSO. |
| Implementation | The cost and extent to which the initiative was delivered as planned. | The number of planned versus actual workshops delivered as part of a regional coach engagement strategy. |
| Maintenance | The extent to which the initiative is sustained over time, which includes an assessment of participant outcomes and organizational delivery beyond six months. | An annual survey sent to current and past participants of a sport for development program to assess participation in and outcomes of the program over time. |
Developing an Evaluation Plan using RE-AIM
While the RE-AIM dimensions offer a tool for identifying WHAT you need to evaluate to assess impact, your RE-AIM evaluation plan should also consider WHO you need to engage in the evaluation (e.g., which stakeholders and/or levels of the sport system are implicated in or affected by your initiative?) and HOW the evaluation will be conducted (e.g., What capacity does your organization have for evaluation? What information/data exists or what data do you need to generate?).
WHO? RE-AIM is a practical tool for the sport system because of its ability to accommodate evaluation across multiple levels of administration and stakeholder groups (Evans, McGuckin, Gainforth, Bruner, & Côté, 2015; Finch & Donaldson, 2010; Lawrason et al., 2020) — from athletes, parents, coaches and administrators at the grassroots level to national and international sport organizations. For example, if a national sport organization is interested in evaluating the impact of a new gender equity policy that would affect the coaching staff of clubs and teams at all levels of the sport, the evaluation should likely include the national team, provincial/territorial teams, and community sport clubs or teams. Alternatively, if a community sport club is interested in evaluating its new youth program, the perspectives of program participants, their parents, and their coaches are needed.
HOW? Another important consideration is the extent to which an organization can leverage existing data to inform the design and evaluation of an initiative (e.g., registration information, feedback forms, social media analytics), and whether or not new data needs to be collected (Evans et al., 2015). Capacity for evaluation can be limited by a number of factors, including the availability and expertise of partners and staff, time, and financial resources (e.g., Millar & Doherty, 2018). For this reason, access to existing data may be the most pragmatic option for some organizations (Lawrason et al., 2020).
What follows are two examples of how the RE-AIM Framework was used to evaluate two initiatives: (1) The Canadian Centre for Ethics in Sport (CCES) True Sport initiative, and (2) Ringette Canada’s guidelines for structuring Children’s Ringette. Knowing that capacity is a significant consideration for many sport organizations, these two examples were chosen, in part, because of their difference in HOW the evaluation was conducted.
Leveraging existing data: A RE-AIM evaluation of True Sport
The CCES is a registered charity and multisport service organization that runs the True Sport initiative through truesportpur.ca to promote value-based sport for Canadians. True Sport relies on its website as the primary delivery mechanism for achieving its objective of promoting values-based sport, which is conceptualized through seven principles for positive sport participation (see truesportpur.ca/true-sport-principles). Canadians interested in the promotion of positive sport can register as a True Sport Member through various member-types (e.g., communities, teams, citizens) free of charge, giving them access to resources and updates via email distribution.
To evaluate the impact of the website through which True Sport is administered, True Sport staff worked collaboratively with a group of researchers at Queen’s University. The decision to use the RE-AIM Framework was based on two main factors: (1) RE-AIM can be adapted to assess multiple end-user outcomes (e.g., athlete, coach, administrator; Finch & Donaldson, 2010), and (2) the RE-AIM dimensions provide administrators with an understanding of how and where to intervene in the initiative to improve end-user outcomes. Of note, a detailed description of this project is published in the Journal of Sport Psychology in Action (see Lawrason et al., 2020).
The first step of the evaluation involved a comprehensive review of the True Sport website to map the objectives of the True Sport initiative onto the RE-AIM dimensions. The types of data that were either publicly available (e.g., Canadian population statistics) or internally collected (e.g., number of visitors to the True Sport website) were then identified. Through discussions between True Sport staff and the research team, a template was developed to categorize and analyze existing data relative to the RE-AIM dimensions. Based on the available data, it was decided to focus the impact evaluation on reach and adoption as sufficient data were not available for the effectiveness, implementation, and maintenance dimensions.
Based primarily on website analytics, several indicators of reach were available. These included the number and demographics (e.g. age, gender) of visitors to the True Sport website, and the type and number of resources downloaded from the website. Website analytics were also the dominant source of data for adoption, which included the total number of True Sport members in Canada. Adoption was further broken down based on member type (e.g., citizen, organization, facility, community) and geographic location (i.e., province or territory). When possible, Canadian population statistics were used to calculate reach and adoption indicators as a proportion of the broader target population (e.g., the number of True Sport members as a proportion of all possible members in the Canadian population).
Takeaway: The fact that data was readily available only for indicators of reach and adoption shows that while existing data can be leveraged for evaluations, there are likely to be gaps. A focus on these gaps, however, provides an important place to start if opportunities (and resources) for new data collection arise.
Collecting new data: A RE-AIM evaluation of small-area games in children’s ringette
In 2019, Ringette Canada introduced guidelines for structuring its new children’s ringette program. The guidelines recommend a shift toward small-area games — such as half-ice or cross-ice — for children in the FUNdamentals and early Learn to Train stages of development (e.g. under the age of 10). Although evidence from other sports suggest that small-area games provide benefits for player development (e.g., Aguiar, Botelho, Lago, Maças, & Sampaio, 2012; USA Hockey, 2019), the guidelines were met with pushback from some members of the ringette community.
With funding from a SIRC Researcher-Practitioner Match Grant and a SSHRC Partnership Engage Grant, Ringette Canada is partnering with researchers at York University and Queen’s University to evaluate the impact of the guidelines and the practice of small-area games at the provincial, community and participant levels. 
The goal of this research is to provide the first ringette-specific evidence for the benefits of small-area games, while also assessing the uptake of Ringette Canada’s new guidelines across Canada. The RE-AIM Framework was selected for this project because it provides a comprehensive assessment of process- and outcome-related indicators of impact across multiple levels of the sport system that will allow for evidence-informed updates to the content and dissemination of the guidelines in the future.
Phases 1 and 2 of the evaluation targeted the provincial and community levels. In Phase 1, technical directors in all provincial ringette associations were interviewed. In Phase 2, administrators in community ringette associations across Canada were asked to complete an online survey. Although different methods were used to collect the data, the questions asked in the interviews and surveys corresponded to the same set of RE-AIM indicators. Examples of the questions include:
- Reach: Are you aware of Ringette Canada’s recommendation that all games for players in children’s ringette programs be played on an area that does not exceed half the ice surface?
- Effectiveness: Do you think that small-area games are a good idea for children? In your opinion, what are some of the benefits of small-area games?
- Adoption: Does your association have a formal policy in place to guide the implementation of small-area games? Please describe any documents or resources (e.g., strategic plan) that address small-area games in your association.
- Implementation: Please describe how, if at all, small-area games have been implemented in your organization (e.g., size of the ice surface, equipment used, age or skill level of players, rule modifications). During the 2019-2020 season, approximately what proportion of the total games played in the children’s ringette program within your association were on ice surfaces that did not exceed half the ice?
- Maintenance: Does your organization plan to begin or continue supporting Ringette Canada’s recommendation that all games for players in children’s ringette programs be played on an area that does not exceed half the ice surface?
While Phases 1 and 2 are complete, Phase 3 is currently on hold due to the COVID-19 pandemic. Phase 3 will evaluate the effectiveness of small-area games with respect to player’s physical (e.g., passing, shooting, skating) and psychosocial (e.g., enjoyment, quality of peer interactions) development. To do so, games from two matched samples of children’s ringette participants — one group playing small-area games and one group playing full-ice games — will be video-recorded, analyzed and compared on measures of player development.
Takeaway: Collecting data from multiple sources helps to improve the quality of the dataset and provides insight into areas of alignment or disconnect from the perspectives of stakeholders. New data also allows for an evaluation to be tailored and expanded in line with the needs and objectives of the organization. The trade-off is that new data often requires time and resources (e.g., human, financial) to collect. Nonetheless, a number of simple and easy-to-use tools exist to help collect meaningful data — ranging from online surveys, feedback forms, and interactive software (e.g., mentimeter) to in-person or phone check-ins with relevant staff members or stakeholders. Partnerships between researchers and practitioners can leverage resources, skills and expertise that benefit everyone involved.
Tips for implementing the RE-AIM Framework
RE-AIM provides a comprehensive framework for understanding the benefits, challenges, and complexities of implementing programs, policies, and other initiatives in the “real world” context of the Canadian sport system. It also provides a useful tool for the design and evalution of initatives with impact. Nonetheless, for a RE-AIM evaluation to produce meaningful and impactful data, careful planning and forethought are necessary. To increase the likelihood of a successful evaluation, consider the following questions before getting started:
- For each RE-AIM dimension, which data is most important? In other words, which data will help demonstrate the value of the initiative to funders, board members, or other stakeholders?
- Does this data already exist, or will new data need to be collected?
- How can new data be collected given internal resources that are available? What external resources (e.g., technology, funding) are needed to collect new data?
- Are experts (e.g., researchers) needed to design or execute any part of this evaluation?
Answering these questions will contribute to a well-designed evaluation that demonstrates the impact of organizational initiatives, while capturing valuable information for future plans and improvements.
A new mathematical model developed by researchers at the University of Alberta reveals how figure skaters move across the ice in three dimensions. Their findings could help skaters hone their technique or avoid common injuries like spiral fractures in the lower leg.
Longitudinal studies allow us to observe behavioural changes and identify patterns over time, providing unique insight on how behaviours are affected by life events. New research using data from the Monitoring Activities of Teenagers to Comprehend their Habits (MATCH) study, which followed nearly 1,000 children for eight years, describes how Canadian youth follow different trajectories of specific health-related behaviours during the transition from childhood to adolescence. Learn more.
Competitive swimmers use an undulatory underwater swimming technique that until now, was little understood. Scientists from the University of Tsukuba in Japan used underwater 3D motion capture and stereo particle image velocimetry to discover how swimmers gain thrust by creating vortices around the foot during the downward kick, which collide to form a jet.