Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Osborne, D., & Ross, D. (2006). The Stay on Track program evaluation. Final report of the longitudinal study. Prepared for the National Center for Prevention and Research Solutions. Study 2Rivera, M., & Rabby, M. (2009). Quasi study of the Stay on Track program. Volusia County quasi study report. Prepared for the National Guard Bureau and Volusia County Schools.
Outcomes
| Outcome 1: Knowledge and attitudes about substance use |
|
Description of Measures
|
Knowledge of the effects of substances was measured by a scale of true/false items, such as "smoking marijuana helps you to concentrate and be more alert," "sniffing or huffing inhalants makes food taste better," and "drinking coffee or taking a cold shower are good ways of helping someone who is drunk sober up."
Awareness of the risk of harm from drug use was measured by asking how much people risk harming themselves (physically or in other ways) if they smoke or use smokeless tobacco, try marijuana, drink one or two drinks nearly every day, use prescription or over-the-counter drugs in ways they were not intended to be used, or sniff or huff inhalants. Response options were "great risk," "moderate risk," "slight risk," "no risk," and "don't know."
Attitudes about peer use of substances were measured by asking students to rate their agreement, from "strongly agree" to "strongly disagree," with four statements regarding their belief that it is wrong for someone their age to smoke cigarettes; drink beer, wine, or hard liquor regularly; use methamphetamines; and use LSD, cocaine, or any illegal drug.
|
|
Key Findings
|
In one study, students who received the intervention each year in 6th through 8th grade were compared with students in a control group. Over the 3-year study period, intervention students demonstrated a significantly greater awareness of the effects of drug use compared with students in the control group (p < .01). From grade 6 pretest to grade 6 posttest, intervention students had a significant increase in their reported awareness of the risk of harm of substance use compared with students in the control group (p = .019), but this effect was not sustained over the 3-year study period.
In a study conducted with 7th-grade students, students who received the intervention for 1 year demonstrated significant improvement in attitudes about peer use of substances (p = .012) from pre- to posttest compared with students in the control group. In addition, intervention students showed significant improvement in knowledge of drug effects from pre- to posttest (p = .016), while students in the control group showed no significant change.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
| Outcome 2: Personal competence skills and self-esteem |
|
Description of Measures
|
Personal competence skills included self-reported goal-setting and decisionmaking skills.
- Goal-setting skills were measured in one study with 6 items, each rated on a 4-point scale. Examples include "Once I set a goal, I don't give up until I achieve it," "Whenever I do something, I always give it my best," and "How often do you work on goals that you have set for yourself?" In another study, students were asked whether they typically set their goals by themselves or based on what other people (e.g., peers, parents) want them to do.
- Decisionmaking skills were measured by asking students several questions about how they make decisions (e.g., who or what they rely on when they make decisions in an uncomfortable situation).
Self-esteem was measured by asking students how satisfied they are with themselves ("all the time," "sometimes," or "never") and asking them to rate their level of agreement, from "strongly agree" to "strongly disagree," with the statement, "I take a positive attitude toward myself."
|
|
Key Findings
|
In one study, students who received the intervention each year in 6th through 8th grade were compared with students in a control group. Intervention students demonstrated significant short-term improvement in goal-setting skills from grade 6 pretest to grade 6 posttest compared with students in the control group (p = .008). This effect was sustained through grade 7 posttest (p < .005) but not grade 8 posttest.
In a study conducted with 7th-grade students, students who received the intervention for 1 year demonstrated significant improvement in goal-setting skills (p = .003), decisionmaking skills (p = .001), and self-esteem (p = .034) from pretest to posttest compared with students in the control group.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.4
(0.0-4.0 scale)
|
| Outcome 3: Social skills/intentions |
|
Description of Measures
|
In one study, resistance skills were measured by presenting students with hypothetical scenarios in which they may feel unwanted pressure from their best friend to drink or take drugs. Students were asked to rate how hard it would be, from "not hard at all" to "very hard," to resist. In another study, students were asked to rate their level of agreement, from "strongly agree" to "strongly disagree," with the following statements: "I plan to accomplish my goals by resisting drugs," "I have the skills and knowledge about drugs and alcohol to make good choices," and "I plan on using my skills and knowledge to refuse drugs."
Communication skills were measured by asking students to rate, on a scale from "never" to "always," statements regarding communication (e.g., "When I want people to understand me, I talk in a way that is clear and specific," "When I want to understand other people, I repeat what they said back to them in my own words to be sure I understood them correctly").
Social skills measures comprised three scenario items, each of which had multiple-choice responses representing what the student would do. In the first scenario, the student is at someone's house and a friend offers the student an alcoholic drink. In the second scenario, the student is looking at CDs in a music store with a friend, and the friend encourages the student to take a CD. In the third scenario, the student wants to go over to a friend's home, but the student's mother tells him or her to stay home to avoid getting into trouble by going out.
As a measure of advocacy skills, students were asked to rate their agreement, from "strongly agree" to "strongly disagree," with the statement, "I am committed to helping my friends resist drugs."
|
|
Key Findings
|
In one study, students who received the intervention each year in 6th through 8th grade were compared with students in a control group. Intervention students demonstrated significant short-term improvement in resistance skills (p < .001) and communication skills (p = .002) from grade 6 pretest to grade 6 posttest compared with students in the control group. These effects were not sustained by grade 7 posttest.
In a study conducted with 7th-grade students, students who received the intervention for 1 year demonstrated significant improvement in resistance skills (p = .024), communication skills (p = .004), social skills (p = .001), and advocacy skills (p = .038) from pretest to posttest compared with students in the control group.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.4
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
13-17 (Adolescent)
|
50.9% Male
49.1% Female
|
79.8% White
8.5% Black or African American
6.4% Hispanic or Latino
5.3% Race/ethnicity unspecified
|
|
Study 2
|
6-12 (Childhood)
13-17 (Adolescent)
|
51.5% Male
48.5% Female
|
48.5% White
33.9% Hispanic or Latino
12.1% Black or African American
2.4% American Indian or Alaska Native
2.4% Race/ethnicity unspecified
0.6% Asian
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Knowledge and attitudes about substance use
|
3.0
|
2.5
|
3.0
|
2.0
|
2.3
|
2.5
|
2.5
|
|
2: Personal competence skills and self-esteem
|
3.0
|
2.5
|
2.9
|
2.0
|
2.0
|
2.1
|
2.4
|
|
3: Social skills/intentions
|
3.0
|
2.5
|
2.9
|
2.0
|
2.0
|
2.1
|
2.4
|
Study Strengths The survey used for all outcome measures was developed using a compilation of items from well-known scales with modifications based on feedback from a panel of experts and confirmatory factor analysis. Adequate internal consistency was established in the studies, and validity of the individual scales had been demonstrated in previous research. Teachers were trained in implementation of the program, and in the second study, oversight was provided via phone or meetings. The first study used a fidelity checklist and observations by an outside evaluator, and researchers computed intraclass correlation coefficients to determine the level of agreement between teacher and process evaluator reports of activity completion. In this study, the percentage of completion of unit activities was high. Six schools were randomly assigned to study condition in the first study, while seven classrooms from one school were randomly assigned to study condition in the second study. In the first study, missing values were imputed for students who remained in their respective group throughout the study, and the data for 8th-graders were analyzed using growth curve modeling with socioeconomic status included as a covariate.
Study Weaknesses For the type of survey used in these studies, additional analyses (e.g., test-retest) for the whole survey would have strengthened reliability. For the second study, researchers did not provide documentation of fidelity, such as number of minutes spent on lessons and number of classrooms that completed the program. Attrition analysis was not conducted in either study; it was not possible in the second study due to the confidentiality of student identities. In the second study, the researchers examined missing values and found no trends, but no data from this analysis were reported. In the first study, one significant baseline difference between students attending intervention and control schools was not taken into account in the analysis, posing an issue with potential confounding variables. In addition, because the first study was conducted in six schools within a small geographic area and the second study was conducted in only one school, students in intervention and control groups may have interacted, resulting in contamination. Although one of the studies was longitudinal in nature, the students in the intervention group continued to receive the curriculum during each year of the study, so there was no follow-up to determine outcomes months after intervention termination.
|
|
Readiness for Dissemination
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
National Center for Prevention and Research Solutions. (2009). Stay on Track familiarization and certification handbook. South Daytona, FL: Author.
National Center for Prevention and Research Solutions. (2009). Stay on Track level 1 toolkit. South Daytona, FL: Author.
National Center for Prevention and Research Solutions. (2009). Stay on Track level 2 toolkit. South Daytona, FL: Author.
National Center for Prevention and Research Solutions. (2009). Stay on Track level 3 toolkit. South Daytona, FL: Author.
National Center for Prevention and Research Solutions. (2009). Stay on Track program guide. South Daytona, FL: Author.
National Center for Prevention and Research Solutions. (2010). Stay on Track. South Daytona, FL: Author.
Program Web site, http://www.ncprs.org/sotHome.htm
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
3.5
|
3.8
|
3.5
|
3.6
|
Dissemination Strengths Program materials are detailed and well sequenced, with clear personnel and organizational requisites for successful implementation and suggestions for handling implementation obstacles. Guidance for assessing organizational readiness is also provided. Leader training and certification in the model is available through the program developer, with recertification of trained leaders required annually. The program Web site includes a calendar of upcoming meetings. Multiple resources are available to support implementation, including Web-based, developer, and peer support. The implementation manual is clearly scripted to support fidelity to the model. Checklists and student surveys for fidelity and outcome monitoring are available to support quality assurance.
Dissemination Weaknesses Some program materials may not be engaging to the target audience, such as videos using graphics and still slides with voiceover (rather than live action and speakers), lesson slides and student materials with dense text, and extended lectures from program leaders. Basic training information, such as how to register for available trainings or how to identify appropriate individuals to participate in the training of trainers, is not available in program materials or on the program Web site. A fee is associated with obtaining the outcome measurement tool and receiving a report of results, which may deter use.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
Toolkit with Crew Chief's Handbook
|
$245.69-$375.14 each depending on grade level
|
Yes
|
|
Driver's Manual
|
$5 each
|
Yes
|
|
Your Turn at the Wheel (take-home book)
|
$5 each
|
Yes
|
|
Introductory online training course (for certified teachers only)
|
Free
|
No
|
|
1-day implementer training in Daytona Beach, FL (includes the Toolkit with Crew Chief's Handbook, the Driver's Manual, and Your Turn at the Wheel)
|
$350 per participant
|
No
|
|
1-day, on-site implementer training (includes the Toolkit with Crew Chief's Handbook, the Driver's Manual, and Your Turn at the Wheel)
|
$3,500 for 5-15 participants ($100 for each additional participant), plus travel expenses
|
No
|
|
2-day training of trainers (for certified teachers only), or 3-day training of trainers (for all other implementers); both in Daytona Beach, FL
|
$650 per participant
|
No
|
|
On-site training of trainers
|
Varies depending on class size and facilities
|
No
|
|
Phone and email consultation
|
Free
|
No
|
|
Web forum for implementer support
|
Free
|
No
|
|
Quality assurance tools
|
Included in implementation materials
|
No
|
Additional Information Stay on Track materials are updated regularly. Updates for the Crew Chief's Handbook and related toolkit materials are provided at no charge to organizations purchasing more than 400 updated Driver's Manuals in either English or Spanish.
|
Replications
Selected citations are presented below. An asterisk indicates that the document
was reviewed for Quality of Research.
* Osborne, D., & Ross, D. (2006). The Stay on Track program evaluation. Final report of the longitudinal study. Prepared for the National Center for Prevention and Research Solutions.
Rivera, M. (2007). Stay on Track program evaluation. Report prepared for the National Guard Bureau.
* Rivera, M., & Rabby, M. (2009). Quasi study of the Stay on Track program. Volusia County quasi study report. Prepared for the National Guard Bureau and Volusia County Schools.
|
|
|
