Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Hepler, N., Williams, C., Butler, D., & Cook, P. (1998, October). Community Based-TADPOLE agency evaluations--Annual reports to agencies: Fiscal year 97/98. Memphis, TN: University of Memphis. Study 2Hepler, N., & Williams, C. (2002). Interpreting and applying the Community Based-TADPOLE report: Fiscal year 2001-2002. Memphis, TN: University of Memphis. Study 3Williams, C. (2003). Community Based-TADPOLE control group study report: 2003 (Technical report prepared for STARS). Memphis, TN: University of Memphis.
Supplementary Materials Dinwiddie, R., Davis, S., Diffenderfer, T., & Hepler, N. (2003). Excerpts from Students Taking A Right Stand Student Assistance Program STARS SAP fidelity of implementation report (Report submitted to the Substance Abuse and Mental Health Services Administration). Nashville, TN: STARS Nashville.
Hepler, N., & Kanu, M. (2011). A summary of research methods for the University of Memphis Tennessee Alcohol and Drug Prevention Outcome Longitudinal Evaluation (TADPOLE).
Hepler, N., Kanu, M., & Williams, C. (2009). Assessing effectiveness of Students Taking A Right Stand (STARS) programs: A case study of 14 years (SY94-95 through SY0809) of evaluations of the STARS program in Nashville, TN.
Okerson, B. B., & Quaranta, P. C. (1996, March). TADPOLE: An application for evaluation of the state of Tennessee's alcohol and drug prevention program using the SAS system. Poster presented at the 21st annual SAS Users Group International Conference, Chicago, IL.
Outcomes
| Outcome 1: Substance use and abuse |
|
Description of Measures
|
Substance use and abuse were assessed using the Student Attitudinal Inventory, a self-administered, paper-and-pencil survey that includes both attitudinal and behavioral dimensions:
- The attitudinal dimensions measure risk and protective factors: positive social attitude and social bonding; self-esteem; value attachment to school and school bonding; rebelliousness (violence); and attitudes unfavorable toward drug use and experimentation.
- The behavioral dimensions measure alcohol, tobacco, and other drug use: gateway drug use; smokeless tobacco use; and drug abuse, including getting drunk on alcohol.
|
|
Key Findings
|
Participants in all three studies were middle and high school students in Tennessee.
In one study, pre- and posttest data from the students who received the STARS Nashville SAP were compared with mean student data from all State-funded drug prevention programs that provided services to students in the same region as the SAP (norm comparative group). From pre- to posttest, SAP participants had a greater decrease in gateway drug use (p < .001) and drug abuse (p < .05) relative to students in the norm comparative group. In addition, data from SAP participants were compared with a growth curve model. The growth curve model is a statistical control group that uses matched group pretest data to construct age-related posttest data, which can be used to predict what the no-treatment effect would have been for participants if they had not received the intervention. From pre- to posttest, SAP participants had a greater decrease in gateway drug use (p < .05) and drug abuse (p < .05) relative to the growth curve model.
In a second study, data from STARS Nashville SAP participants were compared with a growth curve model. From pre- to posttest, SAP participants had a decrease in gateway drug use relative to the growth curve model (p < .05).
In a third study, participants were randomly assigned to the intervention group, which received the STARS Nashville SAP, or the wait-list control group. From pre- to posttest, participants in the intervention group had a decrease in drug abuse (p < .05), and those in the control group had an increase in drug abuse (p < .05); the between-group difference was statistically significant (p = .03). In addition, data from the SAP participants were compared with a growth curve model. From pre- to posttest, participants in the intervention group had a decrease in drug abuse relative to the growth curve model (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
| Outcome 2: Attitudes toward drugs |
|
Description of Measures
|
Attitudes toward drugs were assessed using the Student Attitudinal Inventory, a self-administered, paper-and-pencil survey that includes both attitudinal and behavioral dimensions:
- The attitudinal dimensions measure risk and protective factors: positive social attitude and social bonding; self-esteem; value attachment to school and school bonding; rebelliousness (violence); and attitudes unfavorable toward drug use and experimentation.
- The behavioral dimensions measure alcohol, tobacco, and other drug use: gateway drug use; smokeless tobacco use; and drug abuse, including getting drunk on alcohol.
|
|
Key Findings
|
Participants in all three studies were middle and high school students in Tennessee.
In one study, pre- and posttest data from the students who received the STARS Nashville SAP were compared with mean student data from all State-funded drug prevention programs that provided services to students in the same region as the SAP (norm comparative group). From pre- to posttest, SAP participants and students in the norm comparative group had an increase in healthy attitudes toward drug use (i.e., attitudes unfavorable toward drug use and experimentation) (p < .05 for each group). In addition, data from SAP participants were compared with a growth curve model. The growth curve model is a statistical control group that uses matched group pretest data to construct age-related posttest data, which can be used to predict what the no-treatment effect would have been for participants if they had not received the intervention. From pre- to posttest, SAP participants had an increase in healthy attitudes toward drugs relative to the growth curve model (p < .05).
In a second study, data from STARS Nashville SAP participants were compared with a growth curve model. From pre- to posttest, SAP participants had an increase in healthy attitudes toward drugs relative to the growth curve model (p < .05).
In a third study, participants were randomly assigned to the intervention group, which received the STARS Nashville SAP, or the wait-list control group. From pre- to posttest, participants in the intervention group had an increase in healthy attitudes toward drugs relative to those in the control group (p < .05). In addition, data from SAP participants were compared with a growth curve model. From pre- to posttest, participants in the intervention group had an increase in healthy attitudes toward drugs relative to the growth curve model (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
| Outcome 3: School values |
|
Description of Measures
|
School values were assessed using the Student Attitudinal Inventory, a self-administered, paper-and-pencil survey that includes both attitudinal and behavioral dimensions:
- The attitudinal dimensions measure risk and protective factors: positive social attitude and social bonding; self-esteem; value attachment to school and school bonding; rebelliousness (violence); and attitudes unfavorable toward drug use and experimentation.
- The behavioral dimensions measure alcohol, tobacco, and other drug use: gateway drug use; smokeless tobacco use; and drug abuse, including getting drunk on alcohol.
|
|
Key Findings
|
Participants in all three studies were middle and high school students in Tennessee.
In one study, pre- and posttest data from the students who received the STARS Nashville SAP were compared with mean student data from all State-funded drug prevention programs that provided services to students in the same region as the SAP (norm comparative group). From pre- to posttest, SAP participants had a greater increase in school values (i.e., value attachment to school and school bonding) relative to students in the norm comparative group (p < .05). In addition, data from SAP participants were compared with a growth curve model. The growth curve model is a statistical control group that uses matched group pretest data to construct age-related posttest data, which can be used to predict what the no-treatment effect would have been for participants if they had not received the intervention. From pre- to posttest, SAP participants had an increase in school values relative to the growth curve model (p < .05).
In a second study, data from STARS Nashville SAP participants were compared with a growth curve model. From pre- to posttest, SAP participants had an increase in school values relative to the growth curve model (p < .05).
In a third study, participants were randomly assigned to the intervention group, which received the STARS Nashville SAP, or the wait-list control group. From pre- to posttest, participants in the intervention group had an increase in school values relative to the control group, which had a decrease in school values (p < .05). In addition, data from SAP participants were compared with a growth curve model. From pre- to posttest, participants in the intervention group had an increase in school values relative to the growth curve model (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
| Outcome 4: Social attitude and social bonding |
|
Description of Measures
|
Social attitude and social bonding were assessed using the Student Attitudinal Inventory, a self-administered, paper-and-pencil survey that includes both attitudinal and behavioral dimensions:
- The attitudinal dimensions measure risk and protective factors: positive social attitude and social bonding; self-esteem; value attachment to school and school bonding; rebelliousness (violence); and attitudes unfavorable toward drug use and experimentation.
- The behavioral dimensions measure alcohol, tobacco, and other drug use: gateway drug use; smokeless tobacco use; and drug abuse, including getting drunk on alcohol.
|
|
Key Findings
|
Participants in all three studies were middle and high school students in Tennessee.
In one study, pre- and posttest data from the students who received the STARS Nashville SAP were compared with mean student data from all State-funded drug prevention programs that provided services to students in the same region as the SAP (norm comparative group). From pre- to posttest, SAP participants and students in the norm comparative group had an increase in positive social attitude and social bonding (p < .05). In addition, data from SAP participants were compared with a growth curve model. The growth curve model is a statistical control group that uses matched group pretest data to construct age-related posttest data, which can be used to predict what the no-treatment effect would have been for participants if they had not received the intervention. From pre- to posttest, SAP participants had an increase in positive social attitude and social bonding relative to the growth curve model (p < .05).
In a second study, data from STARS Nashville SAP participants were compared with a growth curve model. From pre- to posttest, SAP participants had an increase in positive social attitude and social bonding relative to the growth curve model (p < .05).
In a third study, participants were randomly assigned to the intervention group, which received the STARS Nashville SAP, or the wait-list control group. From pre- to posttest, SAP participants and those in the wait-list control group had an increase in positive social attitude and social bonding (p < .05). In addition, data from SAP participants were compared with a growth curve model. From pre- to posttest, participants in the intervention group had an increase in positive social attitude and social bonding relative to the growth curve model (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
| Outcome 5: Rebellious and violent attitudes |
|
Description of Measures
|
Rebellious and violent attitudes were assessed using the Student Attitudinal Inventory, a self-administered, paper-and-pencil survey that includes both attitudinal and behavioral dimensions:
- The attitudinal dimensions measure risk and protective factors: positive social attitude and social bonding; self-esteem; value attachment to school and school bonding; rebelliousness (violence); and attitudes unfavorable toward drug use and experimentation.
- The behavioral dimensions measure alcohol, tobacco, and other drug use: gateway drug use; smokeless tobacco use; and drug abuse, including getting drunk on alcohol.
|
|
Key Findings
|
Participants in all three studies were middle and high school students in Tennessee.
In one study, pre- and posttest data from the students who received the STARS Nashville SAP were compared with mean student data from all State-funded drug prevention programs that provided services to students in the same region as the SAP (norm comparative group). From pre- to posttest, SAP participants had a greater decrease in rebellious and violent attitudes relative to students in the norm comparative group (p < .02). In addition, data from SAP participants were compared with a growth curve model. The growth curve model is a statistical control group that uses matched group pretest data to construct age-related posttest data, which can be used to predict what the no-treatment effect would have been for participants if they had not received the intervention. From pre- to posttest, SAP participants had a decrease in rebellious and violent attitudes relative to the growth curve model (p < .05).
In a second study, data from STARS Nashville SAP participants were compared with a growth curve model. From pre- to posttest, SAP participants had a decrease in rebellious and violent attitudes relative to the growth curve model (p < .05).
In a third study, participants were randomly assigned to the intervention group, which received the STARS Nashville SAP, or the wait-list control group. From pre- to posttest, participants in the intervention group and those in the wait-list control group had a decrease in rebellious and violent attitudes (p < .05); the between-group difference was statistically significant (p = .03). In addition, data from SAP participants were compared with a growth curve model. From pre- to posttest, participants in the intervention group had a decrease in rebellious and violent attitudes relative to the growth curve model (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
13-17 (Adolescent)
|
60% Female
40% Male
|
74% White
21% Black or African American
5% Hispanic or Latino
|
|
Study 2
|
6-12 (Childhood)
13-17 (Adolescent)
|
58% Female
42% Male
|
74% White
20% Black or African American
6% Hispanic or Latino
|
|
Study 3
|
6-12 (Childhood)
13-17 (Adolescent)
|
58% Female
42% Male
|
71% White
22% Black or African American
7% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Substance use and abuse
|
3.5
|
3.0
|
2.0
|
3.0
|
2.0
|
2.0
|
2.6
|
|
2: Attitudes toward drugs
|
3.5
|
3.0
|
2.0
|
3.0
|
2.0
|
2.0
|
2.6
|
|
3: School values
|
3.5
|
3.0
|
2.0
|
3.0
|
2.0
|
2.0
|
2.6
|
|
4: Social attitude and social bonding
|
3.5
|
3.0
|
2.0
|
3.0
|
2.0
|
2.0
|
2.6
|
|
5: Rebellious and violent attitudes
|
3.5
|
3.0
|
2.0
|
3.0
|
2.0
|
2.0
|
2.6
|
Study Strengths The Student Attitudinal Inventory has good reliability, with Cronbach's alpha coefficients ranging from .80 to .88 and a mean alpha coefficient of .83. Field research in Canada, the United Kingdom, and the United States has shown the instrument to have acceptable validity when used with school-aged children and adolescents. Study documents indicate that annual training was provided to the STARS Nashville SAP specialists, training was provided to new personnel working with the STARS Nashville SAP, the STARS Nashville SAP was delivered by counselors, and referrals were provided by school personnel, all of which help to ensure intervention fidelity. In addition, program managers conducted site visits for fidelity of implementation. The studies used fidelity instruments with acceptable psychometric properties. The researchers accounted for missing data through the use of a mean imputation technique for survey questions not answered by individual students. Pre- and posttest matching was based on demographics, and quality control of surveys was conducted with a computer and manually to minimize the loss of pre- and posttest matches. Attrition rates across the three studies were low to moderate (17%, 22%, and 7%) and were accounted for in the matching process.
Study Weaknesses Although study documents mentioned items that help to ensure intervention fidelity, it was difficult to determine the level and quality of intervention fidelity; for example, the documents did not mention evidence of judgments by experts, systematic collection of data for dosage, time spent in training, or adherence to guidelines. Across the studies, potential confounding variables are a concern (e.g., data were collected by school personnel, not the researchers; differences in the study population were not adequately taken into account). In addition, two of the studies did not use an adequate control group. Study documentation did not provide sufficient evidence to rule out the possibility of a type III error (i.e., incorrectly inferring the direction of the effect). Analyses did not account for a number of psychosocial variables correlated with adolescent substance use or abuse (e.g., violent and sexual risk-taking behavior), resulting in concerns regarding the control of type I errors (i.e., false-positive results).
|
|
