Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Metcalf, M. P., Hennis, M., Kelley, L., Sealy, S., & Workman, K. L. (2007). BrainTrain4Kids.com: An online substance abuse prevention and science education program for children. HHSN271200455234C. N44DA-4-5523. September 15, 2003 to July 15, 2007 (Final report submitted to the National Institute on Drug Abuse). Chapel Hill, NC: Clinical Tools.
Supplementary Materials Metcalf, M. P. (2009). BrainTrain4Kids: Assessment of an interactive Internet program to educate children aged 7-9 about science, the brain and drugs. Manuscript submitted for publication.
Outcomes
| Outcome 1: Attitudes toward science and drugs |
|
Description of Measures
|
The 16-item BrainTrain4Kids Attitude Assessment Instrument was used to assess each child's attitude toward science (6 items), health (5 items), and related health behavior issues (5 items). For example, items asked about science attitudes in general ("I think doing science is fun"), health and drug prevention ("Drinking alcohol can be harmful to your brain"), and health behavior ("I think exercise is fun"). Participants answered 13 items with a 5-point Likert scale ranging from "NO!" to "YES!," with the available response of "I'm not sure what this question means." In addition, participants responded to 3 items that were related to the health behaviors of others (e.g., "If someone I knew was smoking cigarettes, I would feel..."). These items had four possible answers: "not at all worried about them," "a little worried," "very worried," or "I don't understand this sentence." A higher score indicated greater interest in science and greater concern about drug use.
|
|
Key Findings
|
Participants were randomly assigned to the intervention group or the wait-list control group, which had a 6-week waiting period to match the length of participation by the intervention group. The intervention group completed a pretest (initial assessment), an immediate posttest (second assessment), and a 1-week follow-up assessment (third assessment). The wait-list control group completed a pretest (initial assessment), an assessment after the 6-week waiting period (second assessment), and an immediate posttest (third assessment).
Study results included the following:
- There was no significant difference in attitudinal scores between the intervention group and the wait-list control group at the initial assessment.
- Results of the second assessment indicated that the two groups had significantly different attitudinal scores (p = .012). From the initial assessment to the second assessment, the intervention group's attitudinal score increased significantly (86.58 to 90.32; p < .001), but the wait-list control group's score did not (85.31 to 85.87).
- From the second assessment to the third assessment, the wait-list control group's attitudinal score increased significantly (85.87 to 91.00; p = .001), and the intervention group's score decreased (90.32 to 88.60). From the initial assessment to the third assessment, the intervention group's attitudinal score did not have a significant increase (86.58 to 88.60).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
1.7
(0.0-4.0 scale)
|
| Outcome 2: Knowledge of science and drugs |
|
Description of Measures
|
The BrainTrain4Kids Knowledge Assessment Instrument was used to determine each child's knowledge of the program content, which included the basic components of scientific inquiry, the names and functions of four major parts of the brain, basic information on nerve cells, the differences between drugs and medicines, and the effects of nicotine and alcohol on the brain and lungs. The instrument was made up of 16 multiple-choice items and 5 true-false items, which also included "I don't know" as a possible response. A higher score indicated greater knowledge of the program content.
|
|
Key Findings
|
Participants were randomly assigned to the intervention group or the wait-list control group, which had a 6-week waiting period to match the length of participation by the intervention group. The intervention group completed a pretest (initial assessment), an immediate posttest (second assessment), and a 1-week follow-up (third assessment). The wait-list control group completed a pretest (initial assessment), an assessment after the 6-week waiting period (second assessment), and an immediate posttest (third assessment).
Study results included the following:
- There was no significant difference in knowledge scores between the intervention group and the wait-list control group at the initial assessment.
- Results of the second assessment indicated that the two groups had significantly different knowledge scores (p < .001). From the initial assessment to the second assessment, the intervention group's knowledge score increased significantly (52.67 to 69.06; p < .001), and the wait-list control group's score decreased (53.75 to 49.18).
- From the second assessment to the third assessment, the wait-list control group's knowledge score increased significantly (49.18 to 70.95; p < .001), and the intervention group's score decreased (69.06 to 68.33). From the initial assessment to the third assessment, the intervention group's knowledge score increased significantly (52.67 to 68.33; p < .001).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
1.5
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
|
50.8% Female
49.2% Male
|
73.7% White
15.3% Black or African American
5.1% Asian
4.2% American Indian or Alaska Native
1.7% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Attitudes toward science and drugs
|
2.0
|
2.0
|
1.5
|
1.3
|
1.8
|
1.5
|
1.7
|
|
2: Knowledge of science and drugs
|
1.0
|
2.0
|
1.5
|
1.3
|
1.8
|
1.5
|
1.5
|
Study Strengths Use of a randomized wait-list control group reduced some threats to internal validity. The measures used had face validity.
Study Weaknesses The documentation reviewed did not provide adequate information to support the psychometric properties of the measures used (e.g., Cronbach's alpha values were not reported for the knowledge items, items were created by the developer, there was no documentation of reliability and validity at acceptable levels by independent investigators, only face validity and content validity of the measures were assessed through consultant review and not documented by independent researchers). Intervention fidelity was assessed through usage logs, but it is unclear how they were analyzed and what the analysis indicated (e.g., no data were provided on dosage, the number of minutes children spent on each lesson, or the number of children who completed some or all of the modules). Differential attrition between the experimental and wait-list control groups was not analyzed. Although certain variables, such as age and gender, were compared, other key outcome-related variables were not used for comparison of those who did and did not complete the assessments. Missing data for the knowledge scale were treated as incorrect instead of missing. The analyses used were not strong enough to support the stated outcomes of the study; that is, it was not clear whether the analyses accounted for participants' pretest scores and any potentially confounding variables.
|
|
Readiness for Dissemination
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Program Web site, http://www.BrainTrain4Kids.com
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
2.5
|
1.5
|
2.5
|
2.2
|
Dissemination Strengths The program's Web site is engaging and easy to navigate. Several hands-on, offline activities are available for users who want to extend and practice the skills learned online. Information provided in the Grown-Up Guide can help parents, teachers, and other adults assist children participating in the program. Pre- and posttest measures for attitudes and knowledge are available for parents or other adults to monitor children's outcomes.
Dissemination Weaknesses No guidance is provided for implementing the intervention with users who have limited reading skills or who do not have access to a computer and a high-speed Internet connection. Some materials include typographical errors and content that does not seem logical. No guidance is provided to ensure that participants can read and comprehend the questions being asked in the self-report quality assurance tools provided. No guidance is provided to adults for addressing situations where the intervention or questionnaire content creates anxiety in young children, a possible occurrence when a family member has issues with alcohol or drugs. Although users can submit queries through a Web-based contact form, no direct access to developer staff (e.g., in-person, phone, or email contact information) is readily available as a source for technical assistance to new implementers.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
BrainTrain4Kids Web site (includes Grown-Up Guide, supplemental activity sheets, and assessment tools)
|
Free
|
Yes
|
|
Materials for group usage (includes hard copies of Grown-Up Guide and assessment tools, as well as aggregated data on group participants' use of the site)
|
$25 per group
|
No
|
Additional Information Access to a computer with an Internet connection is required. In addition, access to a printer is required if the user opts to print the supplemental activities (e.g., crossword puzzles, coloring sheets).
|
Replications
No replications were identified by the developer.
|
|
|
