Quality of Research
Review Date: February 2007
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Weissberg, R. P., Cowen, E. L., Lotyczewski, B. S., & Gesten, E. L. (1983). The Primary Mental Health Project: Seven consecutive years of program outcome research. Journal of Consulting and Clinical Psychology, 51, 100-107. Study 2Chandler, C. L., Weissberg, R. P., Cowen, E. L., & Guare, J. (1984). Long-term effects of a school-based secondary prevention program for young maladapting children. Journal of Consulting and Clinical Psychology, 52, 165-170. Study 3Nafpaktitis, M., & Perlmutter, B. F. (1998). School-based early mental health intervention with at-risk students. School Psychology Review, 27, 420-432. Study 4Pollard, J. A., & Thomas, C. F. (1989). An evaluation of the effectiveness of the Primary Intervention Program in improving the school and social adjustment of primary grade students. Report submitted to the California Department of Mental Health. Study 5Crean, H. F., & Lotyczewski, B. S. (1995). An evaluation of the Early Mental Health Initiative's Primary Intervention Program and Enhanced Primary Intervention Program for the 1994-95 academic year. Technical Report T95-428.1. Rochester, NY: Children's Institute.
Supplementary Materials Cowen, E. L., & Hightower, A. D. (1989). The Primary Mental Health Project: Thirty years after. Prevention in Human Services, 6, 225-257.
Cowen, E. L., Weissberg, R. P., Lotyczewski, B. S., Bromley, M. L., Gilliland-Mallo, G., DeMeis, J. L., et al. (1983). Validity generalization of a school-based preventive mental health program. Professional Psychology: Research and Practice, 14, 613-623.
Crean, H. F., & Lotyczewski, B. S. (1996). An evaluation of the Early Mental Health Initiative's Primary Intervention Program and Enhanced Primary Intervention Program for the 1995-96 academic year. Technical Report T96-431.1. Rochester, NY: Children's Institute.
Duerr, M. (1993). Early Mental Health Initiative: Year-end evaluation report. Chico, CA: Duerr Evaluation Resources. Report submitted to the California Department of Mental Health.
Johnson, D. (2002) Primary Project program development manual. Rochester, NY: Children's Institute.
Primary Mental Health Project. (1994). An evaluation of the State of California Early Mental Health Initiative's PIP and PIP Plus Programs for the 1993-1994 school year. Report submitted to the California Department of Mental Health.
Winer-Elkin, J. I., Weissberg, R. P., & Cowen, E. L. (1988). Evaluation of a planned short-term intervention for school children with focal adjustment problems. Journal of Clinical Child Psychology, 17(2), 106-115.
Outcomes
| Outcome 1: Task orientation |
|
Description of Measures
|
Task orientation incorporated such factors as learning difficulty, tolerance for frustration, willingness to follow school rules, and disruptive behavior. This outcome was measured using seven subscales constructed from various instruments. The instruments included: (1) the Classroom Adjustment Scale (CARS) and Health Resources Inventory (HRI), which was completed by teachers and includes subscales for learning difficulty (academic motivation and performance problems) and "good student" (effective learning skills); (2) the Teacher Child Rating Scale (T-CRS), a potential substitute for teacher completion of the CARS and HRI that includes scales for learning problems and task orientation; (3) the Professional Termination Report (PTR), which was completed by a school mental health professional at the end of student participation in the program and includes an item for academic skills; (4) the Professional Summary Report (PSR), a substitute for the PTR completed by mental health professionals that includes an item for task orientation; and (5) the Aides Status Evaluation Form (ASEF), completed by the school aide after four program contacts (preintervention) and at program termination (postintervention); this instrument parallels the CARS and includes subscales for learning difficulty.
|
|
Key Findings
|
Across multiple evaluations, data consistently indicated that the Primary Project is effective in improving task orientation. One evaluation of program effects combining seven consecutive annual cohorts of children indicated that children significantly improved on four factors of task orientation measured by the CARS, HRI, PTR, and ASEF (p < .001). Statistically significant improvement occurred during at least 4 of the 7 years for each of the four factors (p < .05). A second evaluation found overall improvement in task orientation, as measured by the CARS and HRI, at follow-up 2 to 5 years after the program (p < .01).
In a third evaluation, a sample of students who participated in the Primary Project program showed better task orientation as measured by the learning difficulty and good student variables (p < .01) than a group of least-adjusted students who did not participate in the program.
Similarly positive findings were found using the T-CRS and PSR scales among more than 27,000 students in 185 schools during the 1994-95 academic year. In other evaluations using the T-CRS measures, participation in Primary Project was associated with improvements in task orientation, even after accounting for student gender and variation in baseline scores (p < .001). Students who participated had greater average improvement in both the learning problems and task orientation subscales compared with students who did not participate in the program (p < .01).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3, Study 4, Study 5
|
|
Study Designs
|
Experimental, Quasi-experimental, Preexperimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 2: Behavior control |
|
Description of Measures
|
Behavior control incorporated such factors as "acting out," aggression, tolerance for frustration, willingness to follow school rules, and disruptive behavior. This outcome was measured by nine subscales constructed from various instruments (see Outcome 1 for additional details about when and by whom each instrument was completed). The instruments included: (1) the Classroom Adjustment Scale (CARS) and Health Resources Inventory (HRI), which include subscales for acting out, frustration tolerance, and follows rules; (2) the Teacher Child Rating Scale (T-CRS), a substitute for teacher completion of the CARS and HRI that includes scales for acting out and frustration tolerance; (3) the [Mental Health] Professional Termination Report (PTR), which includes an item for disruptive behavior; (4) the [Mental Health] Professional Summary Report (PSR), which includes items for frustration tolerance and acting out/aggressive; and (5) the Aides Status Evaluation Form (ASEF), which includes a factor for behavior control described as acting out (10 items).
|
|
Key Findings
|
Results for this outcome were mixed across multiple evaluations. However, most results showed that the intervention improved most elements of behavior control.
One evaluation that combined seven consecutive annual cohorts of children found that children significantly improved on five subscales of behavior control measured by the CARS, HRI, PTR, and ASEF (p < .05). In addition, statistically significant improvement occurred in at least 5 of the 7 years for three of the five subscales (p < .05), the exception being the acting out subscales of the CARS and ASEF. Two other evaluations found overall improvement in behavior control, as measured by the frustration tolerance subscale of the HRI, at follow-up 2 to 5 years after the program (p < .01). Similar results were not obtained for the acting out and follows rules subscales.
In a separate evaluation, a sample of students who participated in the Primary Project program showed better behavior control than a group of least-adjusted students who did not participate in the program as measured by the acting out, follows rules, and frustration tolerance subscales of the CARS and HRI (p < .001).
Similarly positive findings using different measures of behavior control (T-CRS and PSR scales) were reported among more than 27,000 students in 185 schools during the 1994-95 academic year. An evaluation conducted simultaneously in three school districts found that participation in Primary Project was found to be associated in greater frustration tolerance as measured by the T-CRS, even after accounting for student gender and variation in baseline scores (p < .001). Change in the values for the acting out item of the T-CRS did not achieve statistical significance.
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3, Study 4, Study 5
|
|
Study Designs
|
Experimental, Quasi-experimental, Preexperimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 3: Adaptive assertiveness |
|
Description of Measures
|
Adaptive assertiveness incorporated assertiveness in social situations (including sharing opinions) and in comparison with shyness and anxiety. This outcome was measured using nine subscales constructed from various instruments (see Outcome 1 for additional details about when and by whom each instrument was completed). The instruments included: (1) the Classroom Adjustment Scale (CARS) and Health Resources Inventory (HRI), which include subscales for shy/anxious (12 items) and adaptive assertiveness (7 items); (2) the Teacher Child Rating Scale (T-CRS), a substitute for teacher completion of the CARS and HRI that includes scales for shy/anxious and assertiveness social skills; (3) the [Mental Health] Professional Summary Report (PSR), which includes items for assertive social skills and shy, withdrawn, anxious; and (4) the Aides Status Evaluation Form (ASEF), which includes a factor for adaptive assertiveness described as shy/anxious (12 items).
|
|
Key Findings
|
Across several evaluations, the intervention was shown to be effective in improving adaptive assertiveness.
One evaluation of program effects that combined seven consecutive annual cohorts of children found that children improved on all three subscales of adaptive assertiveness measured by the CARS, HRI, and ASEF (p < .001). In addition, significant improvement occurred in at least 6 of the 7 years for each of the three subscales (p < .05). Two other evaluations found overall improvement in adaptive assertiveness, as measured by the subscales of the HRI and CARS at follow-up 2 to 5 years after the program (p < .01).
Similarly positive findings using different measures of adaptive assertiveness (T-CRS and PSR scales) were reported among more than 27,000 students in 185 schools during the 1994-95 academic year. An evaluation conducted simultaneously in three school districts found that participation in Primary Project was associated with student improvement in adaptive assertiveness as measured by the T-CRS, even after accounting for student gender and variation in baseline scores (p < .001). Two additional evaluations found statistically significant improvement using the T-CRS subscales of adaptive assertiveness (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3, Study 4, Study 5
|
|
Study Designs
|
Experimental, Quasi-experimental, Preexperimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 4: Peer sociability |
|
Description of Measures
|
Peer sociability was measured using four subscales constructed from four instruments (see Outcome 1 for additional details about when and by whom each instrument was completed). The instruments included: (1) the Health Resources Inventory (HRI), which includes a 10-item subscale for peer sociability; (2) the Teacher Child Rating Scale (T-CRS), a substitute for teacher completion of the CARS and HRI that includes a subscale for peer sociability; (3) the [Mental Health] Professional Termination Report (PTR), which includes an item devoted to social skills with peers; and (4) the [Mental Health] Professional Summary Report (PSR), which includes a specific component on peer social skills.
|
|
Key Findings
|
Across multiple evaluation designs, the Primary Project was found to be effective in improving peer sociability.
One evaluation of program effects that combined seven consecutive annual cohorts of children indicated that children improved on the peer sociability subscale of the HRI (p < .001). Statistically significant improvement occurred individually in each of the 7 years (p < .001). Results at follow-up 2 to 5 years after the program were mixed: one analysis found no statistically significant difference in peer sociability between former participants in the Primary Project and other students, while another analysis found retention of the peer sociability benefits (p < .001).
Positive findings using different measures of peer sociability (T-CRS and PSR scales) were reported among more than 27,000 students in 185 schools during the 1994-95 academic year. An evaluation conducted simultaneously in three school districts found that participation in Primary Project was associated with student improvement in peer sociability as measured by the T-CRS, even after accounting for student gender and variation in baseline scores (p < .001). Two additional evaluations found statistically significant improvement using the T-CRS subscales of peer sociability (p < .01).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3, Study 4, Study 5
|
|
Study Designs
|
Experimental, Quasi-experimental, Preexperimental
|
|
Quality of Research Rating
|
3.2
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
0-5 (Early childhood)
6-12 (Childhood)
|
65.5% Male
34.5% Female
|
Data not reported/available
|
|
Study 2
|
6-12 (Childhood)
|
Data not reported/available
|
Data not reported/available
|
|
Study 3
|
0-5 (Early childhood)
6-12 (Childhood)
|
56.4% Female
43.6% Male
|
69.2% White
23.1% Hispanic or Latino
5.1% Black or African American
2.6% Race/ethnicity unspecified
|
|
Study 4
|
6-12 (Childhood)
|
58.1% Male
41.9% Female
|
Data not reported/available
|
|
Study 5
|
0-5 (Early childhood)
6-12 (Childhood)
|
56% Male
44% Female
|
47% White
35% Hispanic or Latino
10% Black or African American
4% Asian
3% Race/ethnicity unspecified
1% American Indian or Alaska Native
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Task orientation
|
3.0
|
3.5
|
2.5
|
3.5
|
3.5
|
4.0
|
3.3
|
|
2: Behavior control
|
3.0
|
3.5
|
3.0
|
3.5
|
3.0
|
4.0
|
3.3
|
|
3: Adaptive assertiveness
|
3.0
|
3.5
|
2.5
|
3.5
|
3.0
|
4.0
|
3.3
|
|
4: Peer sociability
|
3.0
|
3.5
|
2.5
|
3.3
|
2.8
|
4.0
|
3.2
|
Study Strengths Multiple studies provide ongoing documentation of the program's evolution. Most psychometric measures were well established and well described, with acceptable reliability and validity. The use of common outcome measures across studies, although not always common instruments, lends credibility to the view that the intervention is effective in promoting outcomes. Some studies relied on small samples, but attrition did not appear to be an issue, and missing data were sufficiently addressed in the studies with larger samples. Analyses were generally appropriate across the different study designs.
Study Weaknesses Designs used were occasionally weak. The studies lack an estimation of statistical power and reporting of effect sizes. Additional observations of actual play behavior in concert with some qualitative feedback from parents and teachers would better support this type of intervention. Fidelity was measured and reported variously over the studies, although the properties of the measures and procedures are not always identified. There was no documentation of interrater reliability or documentation regarding follow-up training for paraprofessionals, teachers, or counselors who performed the behavioral and other observations.
|
|
Readiness for Dissemination
Review Date: February 2007
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Children's Institute, Inc. (1999). Teacher-Child Rating Scale (T-CRS) 2.1. Rochester, NY: Author.
Children's Institute, Inc. (2001). Background information form. Rochester, NY: Author.
Children's Institute, Inc. (2002). AML Behavior Rating Scale: Revised (AML-R). Rochester, NY: Author.
Children's Institute, Inc. (2002). Associate-Child Rating Scale (A-CRS). Rochester, NY: Author.
Children's Institute, Inc. (2002). Screening and evaluation guidelines. Rochester, NY: Author.
Children's Institute, Inc. (2002). T-CRS 2.1: Teacher-Child Rating Scale examiner's manual. Rochester, NY: Author.
Children's Institute, Inc. (2003). Primary Project: Helping children grow [VHS]. Rochester, NY: Author.
Children's Institute, Inc. (2006). Primary Project: The intervention/Basic skills. DVD and companion resource guide. Rochester, NY: Author.
Cowen, E., Hightower, A., Pedro-Carroll, J., Work, W., Wyman, P., & Haffey, W. (1996). School-based prevention for children at risk: The Primary Mental Health Project. Washington, DC: American Psychological Association.
Johnson, D. (2002). Primary Project program development manual. Rochester, NY: Children's Institute.
Mijangos, L., & Farie, A. (2002). Primary Project: Supervision of paraprofessionals in school-based programs. Rochester, NY: Children's Institute.
Primary Project program Web site, http://www.childrensinstitute.net/programs/primaryproject/
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
3.3
|
3.5
|
3.0
|
3.3
|
Dissemination Strengths The program development manual provides an accessible, step-by-step resource for schools to implement the Primary Project. The DVD is an effective tool for training child associates. Training and program consultation are offered to schools implementing this program. Evaluation is identified as a core component of implementation, and recommended outcome measures and screening protocol are included to support quality assurance.
Dissemination Weaknesses Program materials would be enhanced with more descriptions of activities that take place in the program playroom. It is unclear whether ongoing technical assistance and coaching are offered to support successful implementation. No instrument for assessing adherence to practice standards is provided.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
Program development manual
|
$50 each
|
Yes
|
|
Supervision DVD
|
$90 each
|
Yes
|
|
Basic Helping Skills DVD
|
$90 each
|
Yes
|
|
Helping Children Grow DVD
|
$25 each
|
Yes
|
|
Online program overview
|
Free
|
Yes
|
|
2-day new project training
|
$2,000-$4,000 depending on the number of participants, plus travel expenses
|
Yes
|
|
Additional customized training
|
$1,000 per trainer per day for up to 24 participants, plus travel expenses
|
No
|
|
In-person, email, phone, or Webinar consultation
|
$800 per day plus travel expenses
|
No
|
|
COMET data system and data scoring services
|
About $16 per participant depending on number of participants and measures selected
|
Yes
|
Additional Information The cost of a single contact session with a child served through Primary Project is typically about $36. The average annual cost of seeing a single child is less than $600 per year, including evaluation.
|
Replications
Selected citations are presented below. An asterisk indicates that the document
was reviewed for Quality of Research.
* Chandler, C. L., Weissberg, R. P., Cowen, E. L., & Guare, J. (1984). Long-term effects of a school-based secondary prevention program for young maladapting children. Journal of Consulting and Clinical Psychology, 52, 165-170.
Cowen, E. L., Weissberg, R. P., Lotyczewski, B. S., Bromley, M. L., Gilliland-Mallo, G., DeMeis, J. L., et al. (1983). Validity generalization of a school-based preventive mental health program. Professional Psychology: Research and Practice, 14, 613-623.
Cowen, E. L., Dorr, D. A., Trost, M., & Izzo, L. D. (1972). Follow-up study of maladapting school children seen by nonprofessionals. Journal of Consulting and Clinical Psychology, 39(2), 235-238.
* Crean, H. F., & Lotyczewski, B. S. (1995). An evaluation of the Early Mental Health Initiative's Primary Intervention Program and Enhanced Primary Intervention Program for the 1994-95 academic year. Technical Report T95-428.1. Rochester, NY: Children's Institute.
Crean, H. F., & Lotyczewski, B. S. (1996). An evaluation of the Early Mental Health Initiative's Primary Intervention Program and Enhanced Primary Intervention Program for the 1995-96 academic year. Technical Report T96-431.1. Rochester, NY: Children's Institute.
* Nafpaktitis, M., & Perlmutter, B. F. (1998). School-based early mental health intervention with at-risk students. School Psychology Review, 27, 420-432.
* Pollard, J. A., & Thomas, C. F. (1989). An evaluation of the effectiveness of the Primary Intervention Program in improving the school and social adjustment of primary grade students. Report submitted to the California Department of Mental Health.
Primary Mental Health Project. (1994). An evaluation of the State of California Early Mental Health Initiative's PIP and PIP Plus Programs for the 1993-1994 school year. Report submitted to California Department of Mental Health.
Winer-Elkin, J. I., Weissberg, R. P., & Cowen, E. L. (1988). Evaluation of a planned short-term intervention for school children with focal adjustment problems. Journal of Clinical Child Psychology, 17(2), 106-115.
|
|
|
