Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Bell, M. L., Padget, A., Kelley-Baker, T., & Rider, R. (2007). Can first and second grade students benefit from an alcohol use prevention program? Journal of Child and Adolescent Substance Use, 16(3), 89-107. Study 2Bohman, T. M., Barker, E. D., Bell, M., Lewis, C. M., Hollerman, L., & Pomeroy, E. (2004). Early intervention for alcohol use prevention and vehicle safety skills: Evaluating the Protecting You/Protecting Me curriculum. Journal of Child and Adolescent Substance Abuse, 14(1), 17-40. Study 3Bell, M., Kelley-Baker, T., Rider, R., & Ringwalt, C. (2005). Protecting You/Protecting Me: Effects of an alcohol prevention and vehicle safety program on elementary students. Journal of School Health, 75(5), 171-177. 
Padget, A., Bell, M. L., Shamblen, S. R., & Ringwalt, C. L. (2006). Does learning about the effects of alcohol on the developing brain affect children's alcohol use? Prevention Science, 7, 293-302. Study 4Padget, A., Bell, M., Shamblen, S. R., & Ringwalt, C. (2005). Effects on high school students of teaching a cross-age alcohol prevention program. Journal of Drug Education, 35(3), 201-216.
Supplementary Materials Background information on reliability and validity of PY/PM measures
Bell, M. L., Bliss, K., Padget, A., & Bonsu, A. (n.d.). The experience of MADD's Protecting You/Protecting Me: Using evaluation to enhance program development. Rockville, MD: Substance Abuse and Mental Health Services Administration.
Bell, M., Kelley-Baker, T., Bliss, K., Jones, K., & Falb, T. (2004). Protecting You/Protecting Me: An alcohol prevention program for elementary students taught by peer helpers. Peer Facilitator Quarterly, 19(3), 126-136.
Bell, M., Kelley-Baker, T., Falb, T., & Roberts-Gray, C. (2005). Protecting You/Protecting Me: Evaluation of a student-led alcohol prevention and traffic safety program for elementary students. Journal of Alcohol and Drug Education, 49(1), 33-53.
Pankratz, M. M., Jackson-Newson, J., Giles, S. M., Ringwalt, C. L., Bliss, K., & Bell, M. (2006). Implementation fidelity in a teacher-led alcohol use prevention curriculum. Journal of Drug Education, 36(4), 317-333.
Simmons, J., & Bratkovich, R. (2003). Protecting You/Protecting Me: Teacher feedback, Year III. Internal report.
Steiner, J. C. (2006). MADD New Mexico's Protecting You/Protecting Me program at the Albuquerque Public Schools. Final report for fiscal year 2006. Prepared for the Bernalillo County DWI program.
Outcomes
| Outcome 1: Media awareness and literacy |
|
Description of Measures
|
Media awareness and literacy were measured using survey items developed for the study, such as:
- "Do commercials on TV tell us what we need to know?"
- "Do commercials on TV ever leave out information that could hurt people?"
- "Do commercials on TV always tell the truth?"
- "Do beer commercials on TV show what happens when people drink?"
For younger participants (grades 1 and 2), pictures of gesturing hands (thumb up, thumb down, or open hands) were used for items with yes, no, and maybe responses, and appropriate pictures with text were used to depict answers to other types of questions. For students in grades 3-5, responses were given on a 4-point scale from strongly agree to strongly disagree.
|
|
Key Findings
|
Students in grades 1 and 2 who received the youth-led PY/PM intervention showed a statistically significant increase in media awareness and literacy at posttest relative to the assessment-only comparison group (p < .01).
Students in grades 3-5 who received the youth-led PY/PM intervention showed statistically significant improvements in media awareness and literacy at posttest and at 6-week follow-up relative to the assessment-only comparison group (all p values < .05).
Students in grades 4 and 5 who received the teacher-led PY/PM intervention did not show any statistically significant change in media awareness and literacy at posttest relative to the assessment-only comparison group.
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.8
(0.0-4.0 scale)
|
| Outcome 2: Alcohol use risk and protective factors |
|
Description of Measures
|
Alcohol use risk and protective factors were measured using a combination of items developed for the study, items from the Monitoring the Future survey, and items from a scale developed by Williams, Toomey, McGovern, Wagenaar, and Perry.
- Items developed for the study. Students in grades 3-5 and 11 and 12 were asked about underage alcohol use and their intentions to use alcohol; for example, "Is it OK for older teenagers to drink beer?," "Do you think you might drink beer when you are a teenager?," and "In the future, do you think you will: drink beer; drink wine (other than at religious services); drink flavored alcohol drinks or wine coolers; drink liquor (whisky, vodka, tequila, etc.) straight or in mixed drinks." Students also were asked to indicate their agreement or disagreement with the following statements: "Drinking alcohol is OK for people under age 21," "I can't wait to be old enough to drink alcohol legally," "I might drink alcohol when I am older," and "I will probably drink alcohol when I am old enough."
- Items adapted from the Monitoring the Future survey. Students in grades 1-5 and 11 and 12 were asked about the perceived harm of alcohol use; for example, "Is it harmful for teenagers to drink beer?" and "How much do you think people under 21 years of age harm themselves if they: try one or two drinks of beer; try one or two drinks of wine; try one or two drinks of liquor." Students also were asked about the perceived risks of high and low levels of alcohol use, rating the following as being associated with no risk, slight risk, moderate risk, or great risk: "Have five or more drinks at one time," "Drive after drinking one or two drinks," "Drive after drinking three or more drinks," "Try one or two drinks of an alcoholic beverage," and "Have one or two drinks once or twice a month."
- Items developed by Williams, Toomey, McGovern, Wagenaar, and Perry. Students in grades 11 and 12 were asked about their self-efficacy to refuse alcohol; for example, "How sure are you that you could say no if you were offered alcohol in these situations: at a friend's house; by an older brother or sister; at a party or dance; by a boyfriend or girlfriend."
|
|
Key Findings
|
Students in grades 1 and 2 who received the youth-led PY/PM intervention showed a statistically significant increase in their knowledge of the dangers of underage alcohol use at posttest relative to the assessment-only comparison group (p = .05).
Students in grades 3-5 who received the youth-led PY/PM intervention did not show any statistically significant increase in their knowledge of the risks of underage drinking at posttest and at 6-week follow-up relative to the assessment-only comparison group.
Students in grades 4 and 5 who received the teacher-led PY/PM intervention showed statistically significant increases in negative attitudes toward underage alcohol use (p = .02), perception of the potential harm of alcohol use (p < .01), and intentions not to use alcohol (p < .05) at posttest relative to the assessment-only comparison group (p < .01).
High school students (grades 11 and 12) who taught PY/PM showed statistically significant increases in their perceptions of the potential harm of underage alcohol use (p < .001) and of the risks associated with high levels of alcohol use (p = .022) at posttest relative to the assessment-only comparison group. There were no statistically significant differences between the intervention and comparison groups in self-efficacy to refuse alcohol.
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3, Study 4
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
3.2
(0.0-4.0 scale)
|
| Outcome 3: Knowledge of brain growth and development |
|
Description of Measures
|
Knowledge of brain growth and development was measured using a combination of survey items developed for the study, including:
- "When are people grown up?" (18 years old, 21 years old, or 25 years old)
- "What is the boss of your body?" (heart or brain)
- "What is the brain in charge of?" (thinking, feelings, breathing, or all of the above)
- "Drinking alcohol: messes up how the brain and body communicate; changes the brain's chemistry; changes how the brain works."
- "Drinking alcohol affects people the same regardless of their age."
- "Are people grown up when they are 18 years old?"
- "Drinking alcohol is more dangerous for people under 21 than for people 21 and older."
- "I know what alcohol does to the brain of people under 21."
- "Drinking alcohol distorts messages to and from the brain."
- "Drinking alcohol harms the development of people under 21."
For younger participants (grades 1 and 2), pictures of gesturing hands (thumb up, thumb down, or open hands) were used for items with yes, no, and maybe responses, and appropriate pictures with text were used to depict answers to other types of questions. For students in grades 3-5, responses were given on a 4-point scale from strongly agree to strongly disagree.
|
|
Key Findings
|
Students in grades 1 and 2 who received the youth-led PY/PM intervention showed a statistically significant increase in their knowledge of brain growth and development at posttest relative to the assessment-only comparison group (all p values = .05).
Students in grades 3-5 who received the youth-led PY/PM intervention showed statistically significant improvements in their knowledge of brain growth and development at posttest and at 6-week follow-up relative to the assessment-only comparison group (all p values < .001).
Students in grades 4 and 5 who received the teacher-led PY/PM intervention showed a statistically significant increase in their knowledge of alcohol's effect on brain development at posttest relative to the assessment-only comparison group (p < .01).
High school students (grades 11 and 12) who taught PY/PM showed a statistically significant increase in their knowledge of brain growth and development at posttest relative to the assessment-only comparison group (p < .001).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3, Study 4
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.9
(0.0-4.0 scale)
|
| Outcome 4: Vehicle safety knowledge/skills |
|
Description of Measures
|
Vehicle safety knowledge/skills were measured using a combination of items developed for the study, items from the Monitoring the Future survey, and items from a scale developed by Williams, Toomey, McGovern, Wagenaar, and Perry.
- Items developed for the study. Students in grades 1-5 were asked what they would do if they had to ride with a driver under the influence of alcohol; for example, "If you ever had to ride in a car with a driver who had been drinking alcohol, what would you do?" (talk to the driver; sit in the front seat to be near the driver in case he or she needed help; sit in the back seat). Other questions asked students what they would do in specific scenarios (e.g., riding with a friend's college-age brother who had been drinking beer; riding with a friend's mother who may have had a drink of beer or wine). Students also were asked about their attitudes toward drinking and driving (e.g., "Should people who drive cars always be alcohol-free?" and "Is it okay to get in a car with a driver who has had a beer?").
- Items adapted from the Monitoring the Future survey. Students in grades 11 and 12 were asked, "How many times have you ridden in a car with a driver who had recently been drinking alcohol?" and "How many times have you driven a car within two hours after drinking alcohol?" Response choices included not in the past 30 days, 1 or 2 times, 3-10 times, 11-19 times, and 20+ times.
- Items developed by Williams, Toomey, McGovern, Wagenaar, and Perry. Students in grades 11 and 12 were asked about their efficacy to refuse to ride with an impaired driver: "How sure are you that you could avoid getting into a car driven by someone who has been drinking?" Response options ranged from 1 (I could say no) to 5 (I could not say no).
|
|
Key Findings
|
Students in grades 1 and 2 who received the youth-led PY/PM intervention showed a statistically significant increase in vehicle safety knowledge/skills at posttest relative to the assessment-only comparison students (all p values < .05).
Students in grades 3-5 who received the youth-led PY/PM intervention showed statistically significant improvements in vehicle safety knowledge/skills at posttest and at 6-week follow-up relative to the assessment-only comparison group (all p values < .05).
Students in grades 4 and 5 who received the teacher-led PY/PM intervention showed a statistically significant increase in vehicle safety knowledge/skills at posttest relative to students in the assessment-only comparison group (p < .01).
High school students (grades 11 and 12) who taught PY/PM did not show any statistically significant change in their vehicle safety knowledge/skills relative to the assessment-only comparison group.
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3, Study 4
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 5: Alcohol use |
|
Description of Measures
|
Alcohol use was assessed using items from the Monitoring the Future survey. Students were asked "How recently, if ever, have you done the following: had beer; had wine (other than at religious services); had flavored alcohol drinks or wine coolers; had liquor (whisky, vodka, tequila, etc.) straight or in mixed drinks." Response options included never, at least once in the past month (past 30 days), at least once since school began in the fall, and at least once during lifetime.
To assess binge drinking, students were asked "How many times have you had five or more alcohol drinks in one sitting?" Response options included not in the past 30 days, 1 or 2 times, 3-10 times, 11-19 times, and 20+ times.
|
|
Key Findings
|
Students in grades 4 and 5 who received the teacher-led PY/PM intervention showed a statistically significant decrease in current alcohol use at posttest relative to the assessment-only comparison group (p < .05). This effect was mediated by improved knowledge on the effects of alcohol in the brain and a decrease in intentions to use alcohol.
High school students (grades 11 and 12) who taught PY/PM showed a statistically significant decrease in their frequency of binge drinking at posttest relative to the assessment-only comparison group (p = .014). No statistically significant group difference was found in the overall frequency of alcohol use.
|
|
Studies Measuring Outcome
|
Study 3, Study 4
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
|
Data not reported/available
|
44% White
34% Hispanic or Latino
15% Black or African American
4% Race/ethnicity unspecified
2% Asian
1% American Indian or Alaska Native
|
|
Study 2
|
6-12 (Childhood)
|
52% Female
48% Male
|
54% White
20% Hispanic or Latino
15% Black or African American
11% Race/ethnicity unspecified
|
|
Study 3
|
6-12 (Childhood)
|
51% Female
49% Male
|
56% White
22% Hispanic or Latino
10% Black or African American
9% American Indian or Alaska Native
2% Race/ethnicity unspecified
1% Asian
|
|
Study 4
|
13-17 (Adolescent)
18-25 (Young adult)
|
70% Female
30% Male
|
66% White
13% Black or African American
12% Hispanic or Latino
9% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Media awareness and literacy
|
2.2
|
2.3
|
2.7
|
3.2
|
2.8
|
3.9
|
2.8
|
|
2: Alcohol use risk and protective factors
|
3.1
|
3.3
|
2.5
|
3.4
|
2.8
|
3.9
|
3.2
|
|
3: Knowledge of brain growth and development
|
2.4
|
2.4
|
2.5
|
3.4
|
2.8
|
3.9
|
2.9
|
|
4: Vehicle safety knowledge/skills
|
2.9
|
2.6
|
2.5
|
3.4
|
2.8
|
3.9
|
3.0
|
|
5: Alcohol use
|
3.1
|
3.6
|
2.5
|
3.8
|
2.8
|
4.0
|
3.3
|
Study Strengths The studies addressed teacher training and fidelity and had high retention rates. The analyses were appropriate and took into account attrition and other confounding variables. The psychometric properties of the measures used were generally strong for the older participants.
Study Weaknesses One weakness of these studies is related to the survey items and the difficulty of surveying young children. Specifically, while the Monitoring the Future survey is a reliable instrument, it is designed for secondary students, not primary students. In these studies, it is not known whether the elementary students understood all the survey items, particularly those related to brain growth and development and driving safety. The individuals administering the questionnaire were not blind to the study design, increasing the possibility of bias. Intervention fidelity in some of the studies was not adequately addressed.
|
|
Readiness for Dissemination
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Bell, M. L., Bliss, K., Padget, A., & Bonsu, A. (n.d.). The experience of MADD's Protecting You/Protecting Me: Using evaluation to enhance program development. Rockville, MD: Substance Abuse and Mental Health Services Administration.
MADD. (2000). I don't trust my life to luck: Reasons for rules and laws. Center City, MN: Hazelden.
MADD. (2000). Our brain is the boss: When will I be a grown up? Center City, MN: Hazelden.
MADD. (2000). Stop and think it through: Maggie and the medicine. Center City, MN: Hazelden.
MADD. (2006). Protecting You, Protecting Me: Adult-led training manual and materials. Center City, MN: Hazelden.
MADD. (2006). Protecting You, Protecting Me: An alcohol use prevention curriculum--Grade 1 teaching guide. Center City, MN: Hazelden.
MADD. (2006). Protecting You, Protecting Me: An alcohol use prevention curriculum--Grade 2 teaching guide. Center City, MN: Hazelden.
MADD. (2006). Protecting You, Protecting Me: An alcohol use prevention curriculum--Grade 3 teaching guide. Center City, MN: Hazelden.
MADD. (2006). Protecting You, Protecting Me: An alcohol use prevention curriculum--Grade 4 teaching guide. Center City, MN: Hazelden.
MADD. (2006). Protecting You, Protecting Me: An alcohol use prevention curriculum--Grade 5 teaching guide. Center City, MN: Hazelden.
MADD. (n.d.). Protecting You Protecting Me: Curriculum evaluation form. Center City, MN: Hazelden.
MADD. (n.d.). Protecting You Protecting Me: Directions for administering elementary student surveys. Center City, MN: Hazelden.
MADD. (n.d.). Protecting You Protecting Me: Fidelity checklist. Center City, MN: Hazelden.
MADD. (n.d.). Protecting You Protecting Me Survey: Grades 1-3. Center City, MN: Hazelden.
MADD. (n.d.). Protecting You Protecting Me Survey: Grades 4-5. Center City, MN: Hazelden.
MADD. (n.d.). Protecting You, Protecting Me: Why 21? It's all about the underage brain [VHS]. Center City, MN: Hazelden.
MADD. (n.d.). Protecting You, Protecting Me: Youth-led training manual and materials. Center City, MN: Hazelden.
Program Web site, http://www.hazelden.org/pypm
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
4.0
|
4.0
|
3.3
|
3.8
|
Dissemination Strengths Program materials are comprehensive, easy to follow, engaging, and age appropriate. The online training for teachers provides a flexible alternative for those who cannot attend the national training. The evaluation guide includes a fidelity checklist, outcome measurement guidance, and a number of evaluation forms and surveys to support quality assurance.
Dissemination Weaknesses Additional quality assurance tools are needed to measure, monitor, and improve implementation fidelity.
|
|
