Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Kupersmidt, J. B., Scull, T. M., & Austin, E. W. (2010). Media literacy education for elementary school substance use prevention: Study of Media Detective. Pediatrics, 126(3), 525-531. 
Supplementary Materials Kupersmidt, J. B., & Barrett, T. M. (2010). Media Detective. Durham, NC: innovation Research and Training.
Outcomes
| Outcome 1: Media deconstruction skills for alcohol |
|
Description of Measures
|
Project staff administered a paper-and-pencil questionnaire to students at pre- and posttest. Each question was read aloud to students to account for differences in reading levels. The questionnaire included 6 open-ended items to measure each student's ability to deconstruct an alcohol advertisement:
- "What is being sold in this ad?"
- "What type of person do you think would like this ad (male or female, kid or adult)?"
- "What is the purpose of this ad?"
- "What did the people who made this ad do to make people stop and look at this ad?"
- "What do they want you to think about this product? Finish this sentence: If I get this product, then…"
- "Is there anything this ad is not telling you about the product that you would need to know before buying or using it?"
The resulting qualitative data were coded by three trained coders, producing scores in seven categories:
- The product score, which ranged from 0 to 3, captured a student's ability to recognize the product being advertised.
- The target audience score, which ranged from 0 to 3, assessed a student's ability to recognize the target audience of a particular advertisement.
- The purpose score, which ranged from 0 to 2, assessed a student's understanding that the purpose of the advertisement is to sell products.
- The ad hook score, which was 0 or 1, assessed a student's understanding of how advertisements attract attention.
- The hidden message score, which ranged from 0 to 2, assessed a student's ability to recognize implied messages in advertisements.
- The missing information score, which ranged from 0 to 2, assessed a student's ability to recognize information purposefully omitted from an advertisement.
- The visual elements score, which ranged from 0 to 2, assessed a student's understanding of how advertisers use graphic elements (e.g., font, color, placement of items such as warning labels) to capture attention or to make the product seem more appealing.
The seven scores were summed to create the media deconstruction skills score, which had a possible range of 0-15, with higher scores indicating greater ability to deconstruct the advertisement. Each student's score was calculated as the average of the three coders' scores for that individual. Pretest scores for the outcome variable were included as covariate prediction variables; therefore, outcome variable means were reported as adjusted posttest scores.
|
|
Key Findings
|
At posttest, students in the intervention group had higher mean scores for media deconstruction skills compared with students in the wait-list control group (6.31 vs. 4.59; p < .0001).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.4
(0.0-4.0 scale)
|
| Outcome 2: Understanding of persuasive intent of advertising |
|
Description of Measures
|
Project staff administered a paper-and-pencil questionnaire to students at pre- and posttest. Each question was read aloud to students to account for differences in reading levels. Three items measured students' understanding of the persuasive intent of advertising by asking "how often is this true" for the following statements:
- "Advertisers want you to buy the product even if it isn't good for you to have."
- "Advertisers care more about making money than about what is good for you."
- "Advertising doesn't change the way I think."
Students responded using a 6-point Likert scale that ranged from 0 (never) to 5 (always), with higher scores indicating a stronger understanding of the persuasive intent of advertising. Pretest scores for the outcome variable were included as covariate prediction variables; therefore, outcome variable means were reported as adjusted posttest scores.
|
|
Key Findings
|
At posttest, students in the intervention group had higher mean scores for understanding of the persuasive intent of advertising compared with students in the wait-list control group (3.93 vs. 3.58; p < .05).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
| Outcome 3: Interest in alcohol-branded merchandise |
|
Description of Measures
|
Project staff administered a paper-and-pencil questionnaire to students at pre- and posttest. Each question was read aloud to students to account for differences in reading levels. Six items measured each student's interest in alcohol-branded merchandise. Two pictures, one alcohol themed and one soda themed, were presented for each of the 6 items. Each picture was labeled A or B, and students were asked to indicate which one they preferred. Students responded using a 5-point scale that ranged from 1 (I like B a lot more) to 5 (I like A a lot more), with higher scores indicating greater interest in alcohol-branded merchandise. Each student's score for interest in alcohol-branded merchandise was derived from the average of responses across the 6 items. Pretest scores for the outcome variable were included as covariate prediction variables; therefore, outcome variable means were reported as adjusted posttest scores.
|
|
Key Findings
|
At posttest, boys in the intervention group had lower mean scores for interest in alcohol-branded merchandise compared with boys in the wait-list control group (1.56 vs. 1.76; p < .05). The scores for girls did not differ significantly between groups.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
| Outcome 4: Intentions to use alcohol and tobacco |
|
Description of Measures
|
Project staff administered a paper-and-pencil questionnaire to students at pre- and posttest. Each question was read aloud to students to account for differences in reading levels. Eight items assessed each student's intention to use alcohol and tobacco:
- "Before you are 21 years old, do you think you will drink beer, wine, or hard liquor (more than just a few sips)?"
- "Before you are 21, do you think you will get drunk or drink a lot of alcohol at one time?"
- "Before you are 18 years old, do you think you will smoke cigarettes?"
- "Before you are 18 years old, do you think you will chew tobacco or use snuff?"
- "During the next year, do you think you will drink beer, wine, or hard liquor (more than just a few sips)?"
- "During the next year, do you think you will get drunk or drink a lot of alcohol at one time?"
- "During the next year, do you think you will smoke cigarettes?"
- "During the next year, do you think you will chew tobacco or use snuff?"
Students responded using a 4-point Likert scale that ranged from 0 (I definitely will not) to 3 (I definitely will). Scores for intentions to use alcohol and tobacco were derived from the average of responses across the 8 items, with higher scores indicating a stronger intention to use alcohol and tobacco. Pretest scores for the outcome variable were included as covariate prediction variables; therefore, outcome variable means were reported as adjusted posttest scores.
|
|
Key Findings
|
At posttest, among students who had previously used alcohol and tobacco, those in the intervention group had lower mean scores for intentions to use alcohol and tobacco compared with those in the wait-list control group (0.30 vs. 0.41; p < .05). The scores for students who had not previously used alcohol or tobacco did not differ significantly between groups.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
| Outcome 5: Self-efficacy related to drinking and smoking behaviors |
|
Description of Measures
|
Project staff administered a paper-and-pencil questionnaire to students at pre- and posttest. Each question was read aloud to students to account for differences in reading levels. Four items assessed each student's self-efficacy and feelings of personal control in regard to drinking and smoking behaviors by asking "how often is this true" for the following statements:
- "I feel like I have to drink beer or other alcoholic drinks."
- "I feel like I have to smoke cigarettes."
- "I would feel like I had to drink beer or other alcoholic drinks if my friends were drinking."
- "I would feel like I had to smoke cigarettes if my friends were smoking."
Students responded using a 6-point Likert scale that ranged from 0 (never) to 5 (always). Scores were then reverse coded so that higher scores indicated higher levels of self-efficacy. Scores for self-efficacy in regard to the use of alcohol and tobacco were derived from the average of responses across the 4 items. Pretest scores for the outcome variable were included as covariate prediction variables; therefore, outcome variable means were reported as adjusted posttest scores.
|
|
Key Findings
|
At posttest, students in the intervention group had higher mean scores for self-efficacy compared with students in the wait-list control group (4.79 vs. 4.69; p < .05). Among students who previously used alcohol or tobacco, those in the intervention group had higher mean scores for self-efficacy compared with those in the wait-list control group (4.73 vs. 4.50; p < .05). The scores for students who had not previously used alcohol or tobacco did not differ significantly between groups.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.7
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
|
50.7% Female
49.3% Male
|
Data not reported/available
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Media deconstruction skills for alcohol
|
1.5
|
2.5
|
2.8
|
2.8
|
2.4
|
2.5
|
2.4
|
|
2: Understanding of persuasive intent of advertising
|
2.3
|
2.3
|
2.8
|
2.8
|
2.4
|
2.5
|
2.5
|
|
3: Interest in alcohol-branded merchandise
|
2.3
|
2.8
|
2.8
|
2.8
|
2.4
|
2.5
|
2.6
|
|
4: Intentions to use alcohol and tobacco
|
2.3
|
2.3
|
2.8
|
2.8
|
2.4
|
2.5
|
2.5
|
|
5: Self-efficacy related to drinking and smoking behaviors
|
2.8
|
3.3
|
2.8
|
2.8
|
2.4
|
2.5
|
2.7
|
Study Strengths The internal consistency and face validity of the scales were acceptable, and there was some evidence of concurrent validity. Teachers completed a fidelity checklist after each lesson, and program staff provided fidelity ratings for the lessons they observed, which indicated a relatively high level of fidelity. Attrition and missing data were minimal. The analytic strategy of using a hierarchical linear model was appropriate.
Study Weaknesses Test-retest and interrater reliabilities were not provided, and no evidence of construct or predictive validity was given. Only 20% of the lessons were observed, and no information was provided about teachers' self-reported ratings for the unobserved lessons. It is unclear whether there was differential attrition across treatment groups or across classrooms. The analysis did not account for nesting at the school level.
|
|
Readiness for Dissemination
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
innovation Research and Training. (2006). Media Detective notebook: Grade 3. Durham, NC: Author.
innovation Research and Training. (2006). Media Detective notebook: Grades 4 and 5. Durham, NC: Author.
innovation Research and Training. (2006). Media Detective poster flipchart. Durham, NC: Author.
innovation Research and Training. (2006). Media Detective teacher manual: An evidence-based media literacy substance abuse prevention program, elementary school, grades 3-5. Durham, NC: Author.
Media Detective Web site, http://www.irtinc.us/products/mediadetective/index.html
Web-based teacher training program, http://irtprevention.com/about_mdtt.htm
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
3.5
|
3.5
|
3.4
|
3.5
|
Dissemination Strengths The teacher manual is well organized and follows a clear, logical sequence. The manual's pages include a sidebar that lists the lesson components and highlights where the user is within the lesson. Implementation materials include optional homework assignments to enhance students' learning. The Web-based training is well aligned with the content of the teacher manual and includes session-by-session assessments of what has been learned. At the end of the training, participants who successfully complete the assessment tests are certified as program teachers. A fidelity checklist with a rating scale helps teachers assess how well they followed the fidelity instructions, which are provided for each session.
Dissemination Weaknesses Little guidance is provided on optimal class size and methods for incorporating the program into existing classroom curricula. It is unclear how implementers access support, if needed, for the Web-based training. Little information is provided on how an implementer should administer the fidelity checklist and interpret its results.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
Curriculum kit (includes teacher manual, poster flipchart, CD of media examples, notebook for grade 3, notebook for grades 4 and 5, and one sample bookmark)
|
$210 each
|
Yes
|
|
CD of media examples
|
$5 each
|
No
|
|
Overhead transparencies of media examples
|
$58 per set
|
No
|
|
Notebooks (grade 3 or grades 4 and 5)
|
$24 for 10
|
Yes
|
|
Bookmarks
|
$3.75 for 30
|
No
|
|
1-year license for on-demand, Web-based teacher training (includes teacher certification test)
|
$75 per participant
|
No
|
|
1-day, on-site teacher training workshop (includes teacher certification test)
|
$2,800 for up to 25 participants, plus travel expenses
|
No
|
|
Limited phone and email consultation
|
Free
|
No
|
|
Pre- and posttest outcome assessment instruments
|
Free
|
No
|
|
Fidelity checklists
|
Free
|
No
|
|
Implementation design and monitoring consultation
|
$175 per hour
|
No
|
|
Evaluation services consultation
|
$175 per hour
|
No
|
|
Contractual evaluation services
|
Varies depending on the number of participants, types of services, and number of evaluation reports needed
|
No
|
Additional Information The ongoing costs of program implementation are optional and include the purchase of a notebook and a bookmark for each student.
|
Replications
No replications were identified by the developer.
|
|
|
