Quality of Research
Review Date: February 2008
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Holder, H. D., Gruenewald, P. J., Ponicki, W. R., Treno, A. J., Grube, J. W., Saltz, R. F., et al. (2000). Effect of community-based interventions on high-risk drinking and alcohol-related injuries. Journal of the American Medical Association, 284(18), 2341-2347.  Study 2Treno, A. J., Gruenewald, P. J., Lee, J. P., & Remer, L. G. (2007). The Sacramento Neighborhood Alcohol Prevention Project: Outcomes from a community prevention trial. Journal of Studies on Alcohol and Drugs, 68(2), 197-207. 
Supplementary Materials Gruenewald, P. J., & Johnson, F. W. (2006). The stability and reliability of self-reported drinking measures. Journal of Studies on Alcohol, 67(5), 738-745. 
Holder, H. D., Saltz, R. F., Grube, J. W., Treno, A. J., Reynolds, R. I., Voas, R. B., et al. (1997). Summing up: Lessons from a comprehensive community prevention trial. Addiction, 92(Suppl. 2), S293-S301. 
Holder, H. D., Saltz, R. F., Grube, J. W., Voas, R. B., Gruenewald, P. J., & Treno, A. J. (1997). A community prevention trial to reduce alcohol-involved accidental injury and death: Overview. Addiction, 92(Suppl. 2), S155-S171. 
Holder, H. D., Saltz, R. F., Treno, A. J., Grube, J. W., & Voas, R. B. (1997). Evaluation design for a community prevention trial. An environmental approach to reduce alcohol-involved trauma. Evaluation Review, 21(2), 140-165. 
Information on the reliability and validity of measures used in the studies
Treno, A. J., & Holder, H. D. (2001). Prevention at the local level. In N. Heather, T. J. Peters, & T. R. Stockwell (Eds.), International handbook of alcohol dependence and problems (pp. 771-783). New York: John Wiley & Sons.
Treno, A. J., Lee, J. P., Freisthler, B., Remer, L. G., & Gruenewald, P. J. (2005). Application of evidence-based approaches to community interventions. In T. Stockwell, P. J. Gruenewald, J. W. Toumbourou, & W. Loxley (Eds.), Preventing harmful substance use: The evidence base for policy and practice (pp. 177-189). New York: John Wiley & Sons.
Outcomes
| Outcome 1: Alcohol consumption patterns and related problems |
|
Description of Measures
|
Alcohol consumption patterns were assessed using self-report measures administered via telephone surveys to randomly selected individuals from households in each intervention and comparison site. Among persons who drank, average frequencies of drinking were obtained from responses to a series of drinking questions (i.e., numbers of occasions drinking 2, 3, 6, and 9 or more drinks). These data were used to estimate quantities consumed per drinking occasion as well as variances in drinking quantities over time, known to be related to probabilities of heavy drinking and alcohol-related problems. Frequencies of driving while intoxicated were obtained from answers to a question about the number of days in the past 6 months a person had driven after having "too much to drink." Frequencies of drunk driving were obtained from responses to a question asking when the respondent felt he or she had driven when "over the legal limit." Responses to all these items were averaged across respondents within each 3-month period and community.
|
|
Key Findings
|
Individuals living in intervention sites had significant reductions in drinking quantities (p = .008), variances in drinking quantities (p < .001), rates of driving when having had too much to drink (p = .009), and rates of driving over the legal limit (p = .002) relative to individuals living in comparison sites (communities that were similar to intervention sites in sociodemographics but were not exposed to the intervention). Differences between intervention and comparison sites in average frequency of drinking were not statistically significant.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 2: Alcohol-related traffic crashes |
|
Description of Measures
|
Alcohol-related traffic crashes were assessed using several sources:
- Archival crash data from the California Statewide Integrated Traffic Reporting System and South Carolina Department of Public Safety traffic records. Monthly aggregate crash rates were computed for each community for three types of crashes: nighttime injury crashes (8:00 p.m. to 4:00 a.m.), which tend to be alcohol related; all crashes in which the driver was cited for driving under the influence (DUI) of alcohol; and, as a control, daytime crashes (4:00 a.m. to 8:00 p.m.), which are rarely alcohol related. Each crash series was adjusted for change in population size using annual estimates from the California Department of Finance and NPA Data Services, Inc.
- Local police department data on alcohol-related traffic crashes. These data were aggregated on a monthly basis, which allowed for examination of changes in the neighborhoods over time.
- Local emergency medical services (EMS) data, which consisted of EMS responses to injuries in which incident reports identified the source of injury as an assault, motor vehicle accident, alcohol or other drug problem, or suicide.
|
|
Key Findings
|
In one study, intervention sites had a significant decrease in the number of nighttime crashes per month (p = .009) relative to comparison sites (communities that were similar to the intervention sites in sociodemographics but were not exposed to the intervention). In addition, during and after the active phase of the intervention, intervention sites had a significant decrease in monthly rates of DUI crashes relative to comparison sites (p = .001). In contrast, no decline in daytime motor vehicle crashes was observed.
In another study, intervention sites (two neighborhoods in Sacramento, California) had a significant reduction in motor vehicle accidents (p = .28) relative to the comparison site (Sacramento at large).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
3.4
(0.0-4.0 scale)
|
| Outcome 3: Alcohol-related assaults |
|
Description of Measures
|
Alcohol-related assaults were assessed using several sources:
- Archival hospital discharge data, which were used to measure the observed number of alcohol-related assault cases admitted into hospitals every month from each community. Assault cases included in the analysis were limited to those that were serious and resulted in at least one overnight stay at the hospital.
- Emergency department surveys conducted in one matched intervention and comparison site and in one additional intervention site. Interviewers were present in emergency departments on a weekly or biweekly basis on Friday and Saturday evenings from 9:00 p.m. to 2:00 a.m. Completed interviews included a breath test of blood alcohol content and a self-report of the cause of the injury.
- Local police department data on incidents related to assault and public drunkenness. These data were aggregated on a monthly basis, which allowed for examination of changes in the neighborhoods over time.
- Local emergency medical services data, which consisted of EMS responses to injuries in which incident reports identified the source of injury as an assault, motor vehicle accident, alcohol or other drug problem, or suicide.
|
|
Key Findings
|
In one study, intervention sites had a significant decline in alcohol-related assault cases observed in emergency departments (p = .05) relative to comparison sites (communities that were similar in sociodemographics to intervention sites but were not exposed to the intervention). Intervention sites also had a significant decline in serious alcohol-related assault cases relative to comparison sites (p < .001).
In another study, intervention sites (two neighborhoods in Sacramento, California) had a significant reduction in alcohol-related assaults as reported by police data (p < .001) and EMS data (p = .019) relative to the comparison site (Sacramento at large).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
13-17 (Adolescent) 18-25 (Young adult) 26-55 (Adult) 55+ (Older adult)
|
Data not reported/available
|
Data not reported/available
|
|
Study 2
|
13-17 (Adolescent) 18-25 (Young adult) 26-55 (Adult) 55+ (Older adult)
|
Data not reported/available
|
62% Race/ethnicity unspecified 21% Hispanic or Latino 17% Black or African American
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Alcohol consumption patterns and related problems
|
3.5
|
3.5
|
2.5
|
4.0
|
2.5
|
4.0
|
3.3
|
|
2: Alcohol-related traffic crashes
|
4.0
|
4.0
|
2.0
|
4.0
|
2.5
|
4.0
|
3.4
|
|
3: Alcohol-related assaults
|
3.8
|
3.8
|
2.0
|
3.5
|
2.8
|
4.0
|
3.3
|
Study Strengths The reliability and validity of measures used in the studies are well established. Missing data and attrition were taken into account in the analyses, and threats to internal validity were minimal. The analyses conducted across the studies were in concert with the intended outcomes and support the statistical significance of the results. The evaluation of the intervention was well conducted, and data collection methods were appropriate.
Study Weaknesses There is little if any solid information indicating whether the five components of the intervention were delivered with fidelity to the intervention plan. In addition, it is not clear whether adjustments were made to account for potential confounding variables resulting from, for example, nonrandom selection of intervention and comparison sites and unsolicited responses on the telephone survey.
|
|
Readiness for Dissemination
Review Date: February 2008
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
National Highway Traffic Safety Administration, U.S. Department of Transportation. (2001). Community how to guides: Underage drinking prevention. Retrieved from http://www.nhtsa.dot.gov/people/injury/alcohol/community%20guides%20html/guides_index.html
Pacific Institute for Research and Evaluation. (n.d.). Center for the Dissemination of Environmental Strategies To Reduce High Risk Drinking [Pamphlet]. Berkeley, CA: Author.
Prevention Research Center. (1992). Interviewers' manual for the Community Trials Project. Berkeley, CA: Author.
Prevention Research Center. (1994). Blood alcohol concentration and breathalyzers. Berkeley, CA: Author.
Prevention Research Center. (1996). Guide to responsible alcohol sales: Off-premise clerk, licensee, and manager training. Berkeley, CA: Author. Retrieved from http://www.pire.org/communitytrials/manuals/off-premise.pdf
Prevention Research Center. (1996). Responsible alcohol service (RAS) on-premise licensee/manager training: Trainer's guide. Berkeley, CA: Author. Retrieved from http://www.pire.org/communitytrials/manuals/on-premise.pdf
Prevention Research Center. (n.d.). SNAPP management information system manual. Berkeley, CA: Author.
Prevention Research Center. (n.d.). Survey training manual: Oceanside roadside and emergency room survey. Berkeley, CA: Author.
Prevention Research Center, Pacific Institute for Research and Evaluation (Producer). (2002). Community Prevention Trials [Video]. Retrieved from http://www.prev.org/prc/outcomes.wmv
Program Web site, http://www.pire.org/communitytrials/index.htm
Remer, L., & Treno, A. (1999). SNAPP data manager's manual. Berkeley, CA: Prevention Research Center.
Roeper, P., Remer, L., & Holder, H. (1998). Community data manager's manual: Community Trials Program to prevent alcohol-involved trauma. Berkeley, CA: Prevention Research Center.
Treno, A. (n.d.). Community Trials Intervention To Reduce High Risk Drinking [PowerPoint slides]. Berkeley, CA: Prevention Research Center.
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
3.0
|
2.9
|
2.5
|
2.8
|
Dissemination Strengths Program "how to" guides are clearly written with broad topical coverage. Initial consultation and assistance in selecting program components is provided through the developer. Many training and implementation materials are available at no charge through the developer's Web site. Training and technical assistance are available to implementers. Data management and evaluation tools are provided to support quality assurance.
Dissemination Weaknesses Program materials provide a conceptual framework but do not describe the step-by-step implementation of program components in detail. It is unclear whether step-by-step implementation is addressed during training because training is not standardized or required for implementers. More information is needed on how to evaluate and sustain each program component using the quality assurance tools provided.
|
|