Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Lutra Group. (2006). Year One (FY 05-06) evaluation report for the Celebrating Families! grant. Unpublished report. Salt Lake City, UT: Author. Study 2Quittan, G. A. (2004). An evaluation of the impact of the Celebrating Families Program and Family Drug Treatment Court on parents receiving Family Reunification Services. Unpublished master's thesis, San Jose State University, San Jose, California.
Supplementary Materials Center for Applied Local Research. (2005). Family Treatment Drug Court Head Start Program. Annual report: October 1, 2003, to September 30, 2004.
Clarification of the description of measures appearing on p. 10 of the 2006 Lutra Group evaluation report
Coleman, K. D. (2006). The success rate of Hispanic vs. non-Hispanic participants in Celebrating Families! Unpublished master's thesis, San Jose State University, San Jose, California.
Hill, L. G., & Betz, D. L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26(4), 501-517.
Kumpfer, K. (2008). NIDA WDC Scale reliabilities (parent and child questionnaires).
Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3), 341-349.
Outcomes
| Outcome 1: Parenting skills |
|
Description of Measures
|
Parenting skills were measured by the Kumpfer Parenting Skills scale, a 40-item instrument with questions such as "I praise my child when he/she has behaved well," "I use clear directions with my child," and "We talk as a family about issues/problems or we hold family meetings." Responses are given on 5-point rating scale ranging from 1 (never) to 5 (almost always).
|
|
Key Findings
|
Staff in two community-based organizations and one women's residential treatment facility were trained to administer CF! to parents in early recovery from substance dependence and their children. The CF! group met once weekly for 16 weeks in addition to receiving the standard substance abuse treatment(s) offered by the facility. Following participation in CF!, parents reported more positive parenting (p < .0001), greater involvement with their children (p < .0001), better parenting skills (p < .009), more effective parenting (p < .0001), and increased supervision of their children (p < .0001). These parenting improvements were associated with a range of large effect sizes (Cohen's d = 0.95 to 2.81).
Self-reports from parents in the CF! evaluation described above were compared with self-reports from an independent sample of parents in the Strengthening Families Program (SFP), a nationally administered program measuring the same outcomes. Parents who participated in CF! reported more positive parenting (p < .001) and greater involvement with their children (p < .001) compared with parents who participated in SFP. These group differences were associated with small effect sizes (Cohen's d = 0.21 and 0.24, respectively).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Preexperimental
|
|
Quality of Research Rating
|
2.3
(0.0-4.0 scale)
|
| Outcome 2: Parent tobacco and substance use |
|
Description of Measures
|
Self-reported use of tobacco and other substances was measured using the 6-item Center for Substance Abuse Prevention (CSAP) Government Performance and Results Act (GPRA) scale. Parents were asked "In the past 30 days, on how many days have you used the following?" for each of the following substances: alcohol (both in general and used to the point of intoxication), tobacco, marijuana/hashish/pot, other illegal drugs, and prescription drugs not prescribed by a doctor. For illegal and prescription drugs, parents were asked to specify the type of drug used.
|
|
Key Findings
|
Staff in two community-based organizations and one women's residential treatment facility were trained to administer CF! to parents in early recovery from substance dependence and their children. The CF! group met once weekly for 16 weeks in addition to receiving the standard substance abuse treatment(s) offered by the facility. Following participation in CF!, parents reported less alcohol and drug use (p < .0001), a finding that was associated with a large effect size (Cohen's d = 2.00).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Preexperimental
|
|
Quality of Research Rating
|
2.4
(0.0-4.0 scale)
|
| Outcome 3: Parent depressive symptoms |
|
Description of Measures
|
Parent depression was measured using the Beck Depression Inventory (BDI). The BDI is a 21-item self-report instrument for measuring depressive symptoms in the past week. Total scores range from 0 to 63 and indicate whether depression is minimal (0-13), mild (14-19), moderate (20-28), or severe (29-63).
|
|
Key Findings
|
Staff in two community-based organizations and one women's residential treatment facility were trained to administer CF! to parents in early recovery from substance dependence and their children. The CF! group met once weekly for 16 weeks in addition to receiving the standard substance abuse treatment(s) offered by the facility. Following participation in CF!, parents reported less depression (p < .0001), a finding that was associated with a large effect size (Cohen's d = 1.50).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Preexperimental
|
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
| Outcome 4: Family environment |
|
Description of Measures
|
Family environment was measured by the Family Strengths Assessment, a parent-completed 12-item questionnaire that gauges the strength of 12 family factors: supportiveness, communication, parenting, discipline style, family organization, family unity, positive mental health, physical health, emotional strength, knowledge and education, social network, and spiritual strength. Responses are given on a scale from 1 (none) to 5 (very strong).
In addition, family conflict, organization, communication, and cohesion were measured by four modified subscales (3 to 5 items each) of the Moos Family Environment Scale (FES). The FES consists of 10 subscales that measure the social and environmental characteristics of family systems: family cohesion, expressiveness, conflict, organization, control, independence, achievement orientation, intellectual-cultural orientation, active-recreational orientation, and moral-religious emphasis.
|
|
Key Findings
|
Staff in two community-based organizations and one women's residential treatment facility were trained to administer CF! to parents in early recovery from substance dependence and their children. The CF! group met once weekly for 16 weeks in addition to receiving the standard substance abuse treatment(s) offered by the facility. Following participation in CF!, parents reported better family organization (p < .0001), cohesion (p < .0001), and communication (p < .0001) with greater family strengths/resilience overall (p < .00001). These improvements in family environment were associated with a range of large effect sizes (Cohen's d = 2.27 to 3.12).
Self-reports from parents in the CF! evaluation described above were compared with self-reports from an independent sample of parents in the Strengthening Families Program (SFP), a nationally administered program measuring the same outcomes. Parents who participated in CF! reported greater family cohesion (p < .001), better communication (p < .001), less family conflict (p < .001), and greater overall family strengths/resilience (p < .001) compared with parents who participated in SFP. These group differences were associated with a range of small effect sizes (Cohen's d = 0.20 to 0.33).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Preexperimental
|
|
Quality of Research Rating
|
2.1
(0.0-4.0 scale)
|
| Outcome 5: Child behaviors |
|
Description of Measures
|
Child aggression, conduct disorders, and depression were measured by parent report on 62 items from the revised Kellam Parent Observation of Children's Activities (POCA-R), such as "runs away from home overnight," "starts physical fights," "has lots of friends," "loses temper," "fidgets or moves a lot," "gets distracted easily," and "lies to parents or teachers." Responses were given on a 5-point scale ranging from 1 (never) to 5 (always).
|
|
Key Findings
|
Staff in two community-based organizations and one women's residential treatment facility were trained to administer CF! to parents in early recovery from substance dependence and their children. The CF! group met once weekly for 16 weeks in addition to receiving the standard substance abuse treatment(s) offered by the facility. Following participation in CF!, parents reported more overt aggression or acting out behavior (p < .04) and hyperactivity (p < .004) in their children, despite a significant improvement in concentration (p < .0001). The effect sizes for acting out behavior, hyperactivity, and concentration ranged from medium to large (Cohen's d = 0.74 to 2.49). The increase in acting out behavior was limited to the children of parents in the women's residential facility, while the increase in hyperactivity was limited to the children of parents who were either recently released from prison or to children of families with an incarcerated family member. Parents in the women's residential facility reported significantly less depressive symptoms in their children (p < .001) following participation in CF!, a finding that was associated with a large effect size (Cohen's d = 3.22).
Self-reports from parents in the CF! evaluation described above were compared with self-reports from an independent sample of parents in the Strengthening Families Program (SFP), a nationally administered program measuring the same outcomes. Parents who participated in CF! reported more acting out behavior (p < .001) and hyperactivity (p < .001) in their children compared with parents participating in SFP. These group differences were associated with small and medium effect sizes (Cohen's d = 0.53 and 0.23, respectively). Parents who participated in CF! reported less depression in their children compared with parents participating in SFP (p < .001); this group difference was associated with a small effect size (Cohen's d = 0.31).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Preexperimental
|
|
Quality of Research Rating
|
2.1
(0.0-4.0 scale)
|
| Outcome 6: Family reunification |
|
Description of Measures
|
Family reunification was measured as the time between the date of a court-issued order for parent-child reunification and actual reunification of a parent with at least 50 percent of his or her children previously in child welfare services. This amount of time was also calculated as a rate over a 24-month follow-up period (corresponding with four successive 6-month case review hearings for juvenile dependency as mandated by the Adoption and Safe Families Act).
|
|
Key Findings
|
Data on parents with a history of participation in one of three treatment programs--CF!, Family Treatment Drug Court (FTDC), or the Traditional Child Welfare Case Plan--were obtained from the Santa Clara County, California, child welfare system database. Each record included 18 to 24 months of follow-up data and was selected based on a one-to-one match across treatment programs by gender (all female), age, ethnicity, number of children in the child welfare system, type of abuse, removal date, and reunification date. Family reunification occurred sooner for parents participating in CF! and FTDC compared with parents participating in the Traditional Child Welfare Case Plan program (in months 7-12 and 13-18, respectively, vs. in months 19-24; p = .01). CF! and FTDC parents also had higher family reunification rates compared with parents in the Traditional Child Welfare Case Plan program (73% and 72%, respectively, vs. 37%; p = .01).
|
|
Studies Measuring Outcome
|
Study 2
|
|
Study Designs
|
Preexperimental
|
|
Quality of Research Rating
|
2.1
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
13-17 (Adolescent)
26-55 (Adult)
|
85.7% Female
14.3% Male
|
42.9% White
37.1% Hispanic or Latino
20% Race/ethnicity unspecified
|
|
Study 2
|
18-25 (Young adult)
26-55 (Adult)
|
100% Female
|
44.9% Hispanic or Latino
39.7% White
7.7% Black or African American
6.4% Race/ethnicity unspecified
1.3% American Indian or Alaska Native
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Parenting skills
|
3.0
|
2.5
|
3.0
|
1.0
|
2.0
|
2.5
|
2.3
|
|
2: Parent tobacco and substance use
|
3.5
|
2.5
|
3.0
|
1.0
|
2.0
|
2.5
|
2.4
|
|
3: Parent depressive symptoms
|
3.5
|
3.5
|
3.0
|
1.0
|
2.0
|
2.5
|
2.6
|
|
4: Family environment
|
2.0
|
2.0
|
3.0
|
1.0
|
2.0
|
2.5
|
2.1
|
|
5: Child behaviors
|
2.0
|
2.0
|
3.0
|
1.0
|
2.0
|
2.5
|
2.1
|
|
6: Family reunification
|
1.0
|
1.5
|
1.5
|
4.0
|
1.5
|
3.0
|
2.1
|
Study Strengths The Beck Depression Inventory is a gold standard instrument. Adequate to good reliability is present for some measures. Considerable attention was paid to intervention fidelity, and fidelity checklists were used by two observers during randomly selected sessions.
Study Weaknesses Too little information is available to assess most of the measures used; the measures represent selected items from larger scales or were developed by the principal evaluator without any independent testing of validity and reliability. Assessment instruments were changed to a retrospective pre/post assessment due to a lack of confidence in the truthful response rate of the pretest data. While fidelity checks were in place, the authors acknowledged that not all content was delivered; it is unclear what content was not delivered as planned and why. Both studies had weak designs with multiple threats to internal validity. The sample size in one study was small, and the authors were aware that cases had missing values, yet no analytic strategies were used to address missing data. The retrospective chart review, which was conducted for one gender only, also reflected a small sample size with a limited matching scheme for group assignment.
|
|
