Quality of Research
Review Date: October 2007
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Carter, S. (with Letts, D., Tom, G., Tallant, A., Soce, B., Cleveland, R., et al.). (2005). FY 2005 final evaluation progress narrative report on Project Venture Middle School. Submitted to the New Mexico Department of Health Behavioral Health Services Division. Gallup, NM: National Indian Youth Leadership Project. Study 2Carter, S., Straits, K. J. E., & Hall, M. (2006). Project Venture: Evaluation of a positive, culture-based approach to substance abuse prevention with American Indian youth. Manuscript in preparation.
Carter, S., Straits, K. J. E., & Hall, M. (2007). Project Venture: Evaluation of an experiential, culturally-based approach to substance abuse prevention with American Indian youth. Journal of Experiential Education, 29(3), 397-400.
Outcomes
| Outcome 1: Use of alcohol, tobacco, marijuana, and other illicit drugs |
|
Description of Measures
|
Respondents indicated the number of days (0, 1-2, 3-5, 6-9, 10-19, 20-31) in the past month they used alcohol, tobacco, marijuana, and other illicit drugs.
|
|
Key Findings
|
In one study, alcohol use increased for both intervention and control students from baseline to the 6-month follow-up, then leveled off among intervention students and continued to increase among control students from the 6- to 18-month follow-up (p < .05). In a second study, alcohol use from pre- to posttest remained the same among students in the intervention group and increased among students in the control group (p < .01). This difference represents a large effect size (partial eta-squared = 0.189).
From pre- to posttest, use of marijuana (p < .01) and other illicit drugs (cocaine, other stimulants, heroin, tranquilizers, hallucinogens, inhalants, steroids, and club drugs; p < .05) remained the same among students in the intervention group and increased among students in the control group. The effect sizes were large for marijuana (partial eta-squared = 0.162) and medium for other illicit drugs (partial eta-squared = 0.097).
Tobacco use remained the same over time among students in the intervention and control groups.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.2
(0.0-4.0 scale)
|
| Outcome 2: Substance abuse risk and protective factors |
|
Description of Measures
|
Four subscales from the Strategies for Success Survey were used to measure perceptions of alcohol and other drug use by peers, the availability of alcohol and other drugs, the intent to abstain from substance use, and perceptions of adult alcohol and other drug use.
|
|
Key Findings
|
From pre- to posttest, intervention students showed no change in their risk and protective factors. During the same time period, students in the control group showed decreases in their intent to abstain from substance use (p < .05) and increases in perceptions of alcohol and other drug use by peers (p < .01), availability of alcohol and other drugs (p < .05), and perceptions of adult alcohol and other drug use (p < .001). Effect sizes ranged from medium (partial eta-squared = 0.116 for availability of alcohol and other drugs; partial eta-squared = 0.101 for intention to abstain from substance use) to large (partial eta-squared = 0.256 for perceptions of adult alcohol and other drug use; partial eta-squared = 0.226 for perceptions of alcohol and other drug use by peers).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.1
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
13-17 (Adolescent)
|
51.6% Male
48.4% Female
|
61% American Indian or Alaska Native
27% Hispanic or Latino
12% White
|
|
Study 2
|
6-12 (Childhood)
13-17 (Adolescent)
|
50.3% Female
49.7% Male
|
75.5% American Indian or Alaska Native
15.6% Hispanic or Latino
5.3% White
3.3% Race/ethnicity unspecified
0.3% Native Hawaiian or other Pacific Islander
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Use of alcohol, tobacco, marijuana, and other illicit drugs
|
2.5
|
2.5
|
2.5
|
1.5
|
1.5
|
2.5
|
2.2
|
|
2: Substance abuse risk and protective factors
|
2.3
|
2.5
|
2.5
|
1.5
|
1.5
|
2.5
|
2.1
|
Study Strengths The evaluation studies used measures with acceptable psychometric properties, and the data were analyzed using appropriate statistical techniques. Program implementation was assessed qualitatively by observing the implementation sessions, documenting meeting attendance, and interviewing both participants and staff members. In addition, a fidelity checklist was developed and used.
Study Weaknesses The lack of random assignment to groups weakened the impact of the intervention and created the potential for confounding variables. It appears that participant attrition was present and increased over time. Because attrition was not addressed in the evaluation, however, it is difficult to evaluate its potential impact on the findings. In addition, the impact of the after-school activities on the findings is not clear.
|
|
Readiness for Dissemination
Review Date: October 2007
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Hall, M., Carter, S., Flesher, J., & Pilz, A. (2005). Project Venture replication guide. Gallup, NM: National Indian Youth Leadership Project.
National Indian Youth Leadership Project. (2006). Office of Juvenile Justice and Delinquency Prevention Field Initiated Research and Evaluation Program grant proposal [Excerpt]. Gallup, NM: Author.
National Indian Youth Leadership Project Replication Site Survey, 2007
Positive Youth Development Approaches for American Indian Youth [pamphlet]
Program Web site, http://www.niylp.org
Project Venture Replication/Adaptation Assessment Scale
Training materials:
- National Indian Youth Leadership Project Decision Workshop [PowerPoint slides]
- Project Venture sample training agenda
- Project Venture sample training workshop evaluation results
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
2.3
|
3.5
|
2.5
|
2.8
|
Dissemination Strengths A well-written replication guide with a valuable organizational readiness component is provided. The developer requires a consultation and customized initial training before allowing the purchase of implementation materials. The assessment scale serves as a fidelity tool to support quality assurance.
Dissemination Weaknesses While the lack of step-by-step instructions offers flexibility for the delivery of this complex program, it may be a significant barrier for some implementers. Further implementation guidance is needed for the school-based and youth recruitment components. Guidance on outcome and process evaluation is limited to the need to hire an outside evaluator.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
Curriculum guide
|
$250 each
|
Yes
|
|
2.5- to 3-day, on-site training (recommended for sites with six or more trainees)
|
$3,000 per site plus travel expenses
|
Yes, one training option is required
|
|
Off-site training at National Indian Youth Leadership Project workshop
|
$500 per person
|
Yes, one training option is required
|
|
Advanced Project Venture Programming workshop
|
$500 per person
|
No
|
|
Phone consultation
|
Free
|
Yes
|
|
Consultation for assessment of resources for readiness, programming, sustainability, and evaluation
|
$1,000 per day plus travel expenses
|
No
|
|
10 hours of follow-up consultation via phone or Web
|
$2,000 per year
|
Yes
|
|
On-site follow-up consultation, coaching, and support
|
$1,000 per day plus travel expenses
|
Yes
|
|
Fidelity self-study instrument
|
Free
|
No
|
|
Replications
No replications were identified by the developer.
|
|
|
