Quality of Research
Review Date: December 2011
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Morris, E., Suarez, L., & Reid, J. C. (1997). Behavioral outcomes of home-based services for children and adolescents with serious emotional disorders. Family Preservation Journal, 2(2), 21-32. Study 2Kirk, R. S., & Griffith, D. P. (2002). Annual report to the Governor and the General Assembly of the State of North Carolina on the Intensive Family Preservation Services program for the 2001-2002 State fiscal year. Raleigh: North Carolina Department of Health and Human Services, Division of Social Services.
Kirk, R. S., & Griffith, D. P. (2004). Intensive family preservation services: Demonstrating placement prevention using event history analysis. Social Work Research, 28(1), 5-18. Study 3Blythe, B. J., & Jayaratne, S. (1999). Michigan Families First effectiveness study: A summary of findings. Report submitted to the State of Michigan, Family Independence Agency.
Supplementary Materials Fidelity Checklist for Michigan
Fidelity Checklist for Missouri
Fidelity Checklist for North Carolina
Outcomes
| Outcome 1: Child behavior problems |
|
Description of Measures
|
Child behavior problems were assessed with the Child Behavior Checklist (CBCL), a 113-item measure of a child's externalizing and internalizing behaviors, as reported by a parent or other caretaker. Using a 3-point scale ranging from 0 (not true) to 2 (very true or often true), respondents rate each item. From these responses, a total score, internalizing and externalizing scale scores, and eight subscale scores (withdrawn, somatic, anxious/depressed, social, thought, attention, delinquent, and aggressive) are derived.
|
|
Key Findings
|
Children ages 4-17 who were at risk for out-of-home placement, demonstrated internalizing or externalizing behavior problems, and met the diagnosis for at least one DSM-III-R psychiatric disorder received intensive in-home services through Missouri's Families First program, which used the HOMEBUILDERS model. The CBCL was administered to one parent, usually the mother, at the initial intake screening (pretest) and at the end of the 4- to 6-week intervention period (posttest).
From pre- to posttest, children had an improvement in behavior problems, as indicated by the CBCL total score (p = .0001), the internalizing scale score (p = .0001), the externalizing scale score (p = .0001), and scores for all eight subscales (withdrawn, p = .0002; somatic, p = .0078; anxious/depressed, p = .0001; social, p = .0001; thought, p = .0031; attention, p = .0001; delinquent, p = .0001; and aggressive, p = .0001).
Analyses also were conducted with data from children with oppositional defiant disorder, mood disorders, or conduct disorder. Results from pre- to posttest indicated improvements in behavior problems:
- Children with oppositional defiant disorder had an improvement in the CBCL total score (p = .0002), the internalizing scale score (p = .0027), the externalizing scale score (p = .0019), and scores for seven subscales (withdrawn, p = .0038; anxious/depressed, p = .0065; social, p = .0022; thought, p = .0062; attention, p = .0053; delinquent, p = .0074; and aggressive, p = .0040).
- Children with mood disorders had an improvement in the CBCL total score (p = .0015), the internalizing scale score (p = .0001), the externalizing scale score (p = .0174), and scores for five subscales (somatic, p = .0083; anxious/depressed, p = .0002; social, p = .0168; attention, p = .0119; and delinquent, p = .0086).
- Children with conduct disorder had an improvement in the CBCL total score (p = .0098), the externalizing scale score (p = .0044), and scores for two subscales (delinquent, p = .0002, and aggressive, p = .0040).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Preexperimental
|
|
Quality of Research Rating
|
3.2
(0.0-4.0 scale)
|
| Outcome 2: Out-of-home placement |
|
Description of Measures
|
In one study, this outcome was assessed by the number of days that children spent in out-of-home placement over a period of 1 year from the beginning of services for families who received the intervention or over a period of 1 year from the date of a substantiated report of abuse and/or neglect for families who received traditional child welfare services. These data were obtained from several of North Carolina's statewide information systems.
In another study, this outcome was assessed by the number of days that children spent in out-of-home placement, which includes State-supported foster homes, private agency foster homes, private child care institutions, public institutions, public shelters, mental health facilities, and detention facilities. These data were obtained from Michigan's Children's Services Management Information Systems.
|
|
Key Findings
|
A 6-year retrospective study analyzed data from 51 counties that offered North Carolina's Intensive Family Preservation Services (IFPS) program (based on the HOMEBUILDERS model) to families who had a substantiated report of abuse and/or neglect and were referred by county departments of social services to receive county-based child welfare services. These families were screened with a standardized child protective services risk assessment, and those determined to be at high risk for the out-of-home placement of children were included in the study. Families who were eligible for IFPS program services but did not receive them owing to the program's limited resources received traditional services through public and contract agencies (e.g., counseling, parent skill training, mental health referral and services, protective services, day care, foster care). Findings included the following:
- Compared with children in high-risk families who received traditional child welfare services, those in high-risk families who received services through the IFPS program had reduced out-of-home placement rates or delayed out-of-home placement (p < .01), after controlling for risk factors such as prior placement authority and prior substantiated reports of abuse and/or neglect.
- Among high-risk families without a new substantiated report of abuse and/or neglect within 12 months of the start of services, children in high-risk families who received services through the IFPS program were 32% less likely than children in high-risk families who received traditional child welfare services were to experience an out-of-home placement (p < .001).
In another study, families were randomly assigned to the intervention group, which received intensive family preservation services from Michigan's Families First program (based on the HOMEBUILDERS model) or the comparison group, which received foster care services. During the 12 months after the intervention, children in families who received services from the Families First program spent fewer days in out-of-home care than did children in the comparison group (p < .0001).
|
|
Studies Measuring Outcome
|
Study 2, Study 3
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.9
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
0-5 (Early childhood)
6-12 (Childhood)
13-17 (Adolescent)
|
57.6% Male
42.4% Female
|
75% White
25% Black or African American
|
|
Study 2
|
0-5 (Early childhood)
6-12 (Childhood)
13-17 (Adolescent)
|
50.4% Female
49.6% Male
|
54.4% White
45.6% Race/ethnicity unspecified
|
|
Study 3
|
0-5 (Early childhood)
6-12 (Childhood)
13-17 (Adolescent)
|
52% Male
48% Female
|
71.3% Black or African American
22% White
6.7% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Child behavior problems
|
4.0
|
3.0
|
2.8
|
3.6
|
2.3
|
3.4
|
3.2
|
|
2: Out-of-home placement
|
2.8
|
2.9
|
3.3
|
3.6
|
2.5
|
2.4
|
2.9
|
Study Strengths The CBCL, included in one study, is a widely used and validated measure of children's internalizing and externalizing behaviors. In each of two studies, the number of days of out-of-home placement was extracted from State administrative databases; because these data are from official State records, they should be fairly reliable. In one State, the database had recently undergone improvement efforts, which may have enhanced reliability and validity. The validity of out-of-home placement based on official State records is high, as the measure is actual days rather than a proxy. In all three studies, intervention fidelity was addressed through staff training, an intervention manual, and team meetings. A fidelity checklist was provided for each study. In one study, researchers reported high intervention fidelity on the basis of such factors as response time (i.e., time from referral to first home visit), length of service, and number and distribution of contact hours; cases that did not meet strict fidelity requirements were removed from analyses. In another study, researchers noted that agency workers completed a fidelity questionnaire at the end of each case and that a questionnaire adapted for foster care workers was administered to them via telephone; the results of this assessment demonstrate that a relatively high level of treatment fidelity was obtained. Missing data and attrition were not issues in any of the three studies. The use of a longitudinal retrospective analysis with a comparison group in one study was strong. Another study used a randomized group design, which helped control for a number of potential confounding variables. The retrospective study also used a more sophisticated survival analysis to assess the out-of-home placement of children.
Study Weaknesses Only the parent-report version of the CBCL was used, although the measure's developer recommends that data be collected from multiple informants. Standardized assessments of fidelity were not used. It is not clear who completed the fidelity review checklists, when they were completed, on how many families they were completed, and the level of adherence to the intervention. In one study, participants who received services through the IFPS program but did not have a high level of treatment fidelity were removed from analyses; however, the researchers do not explain how this may have impacted the findings. Another study used a single-group pre- and posttest design, which is subject to a number of potential confounding variables, including selection bias and history; in addition, the types and amount of other services were not taken into consideration in the analysis. One of the studies had a serious design flaw, which led to a major concern with confounding variables: children in the intervention group remained in the home and received services to prevent out-of-home placement, and children in the comparison group were removed from the home and placed in foster care; thus, the outcome regarding out-of-home placement did not have an equivalent likelihood of occurring between intervention and comparison groups. Two of the studies did not apply corrections for conducting multiple tests, thus increasing the probability of inflating the experiment-wide error rate. The analyses conducted in one study were overly simplistic and did not include multivariate analyses of some key factors that probably impacted the outcome.
|
|
Readiness for Dissemination
Review Date: December 2011
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Behavioral Science Institute--HOMEBUILDERS. (1992). Skills for family and community living [DVD and leader's guide]. Federal Way, WA: Author.
Institute for Family Development. (2003). Job description: QUEST consultant. Federal Way, WA: Author.
Institute for Family Development. (2003). Program consultation and quality assurance skills for HOMEBUILDERS supervisors. Federal Way, WA: Author.
Institute for Family Development. (2007). HOMEBUILDERS quality enhancement system. Federal Way, WA: Author.
Institute for Family Development. (2009). HOMEBUILDERS clinical paperwork guidelines. Federal Way, WA: Author.
Institute for Family Development. (2009). HOMEBUILDERS sample site development plans. Federal Way, WA: Author.
Institute for Family Development. (2009). HOMEBUILDERS standards (version 3.0). Federal Way, WA: Author.
Institute for Family Development. (2010). Assessment, goal setting and service planning. Federal Way, WA: Author.
Institute for Family Development. (2010). Drug-affected clients: Motivational interviewing and relapse prevention. Federal Way, WA: Author.
Institute for Family Development. (2010). Fundamentals of supervising HOMEBUILDERS: Intensive family preservation services. Federal Way, WA: Author.
Institute for Family Development. (2011). HOMEBUILDERS core curriculum. Federal Way, WA: Author.
Institute for Family Development. (2011). Improving decision making through critical thinking. Federal Way, WA: Author.
Institute for Family Development. (2011). Teaching skills to families. Federal Way, WA: Author.
Institute for Family Development. (2011). Utilizing behavioral principles and strategies with families. Federal Way, WA: Author.
Institute for Family Development. (2011). Utilizing cognitive strategies with families. Federal Way, WA: Author.
Institute for Family Development. (n.d.). Active listening: John [DVD]. Federal Way, WA: Author.
Institute for Family Development. (n.d.). Client family session: Jack [DVD]. Federal Way, WA: Author.
Institute for Family Development. (n.d.). Crisis card demo [DVD]. Federal Way, WA: Author.
Institute for Family Development. (n.d.). Your deal [Card game]. Federal Way, WA: Author.
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
3.5
|
3.5
|
3.5
|
3.5
|
Dissemination Strengths The implementation and standardized training materials are comprehensive, detailed, and very well organized. Training materials include DVDs and supplemental resources, which present guidance on how to integrate motivational interviewing into the intervention. Optional trainings for specific populations are easy to identify, and the materials for these trainings are well outlined. Tools and resources for measuring fidelity and replicating the program are available and include performance benchmarks, standardized forms, and guided reviews of the program processes.
Dissemination Weaknesses There is no overview document that clearly defines the scope of the program. Implementing staff must attend time-intensive trainings and follow complex quality assurance procedures, which may be barriers to implementation for some agencies.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
HOMEBUILDERS site development and implementation readiness consultation (in person or by phone)
|
Varies depending on site needs and travel expenses
|
Yes
|
|
4-day, on-site HOMEBUILDERS Core Curriculum Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $120 per participant for materials
|
Yes
|
|
1-day, on-site Goal Setting and Paperwork Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $20 per participant for materials
|
Yes
|
|
2-day, on-site Motivational Interviewing Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $40 per participant for materials
|
Yes
|
|
1-day, on-site Relapse Prevention Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $20 per participant for materials
|
Yes
|
|
1-day, on-site Utilizing Cognitive Strategies With Families Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $20 per participant for materials
|
Yes
|
|
1-day, on-site Utilizing Behavioral Principles and Strategies With Families Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $20 per participant for materials
|
Yes
|
|
1-day, on-site Teaching Skills to Families Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $15 per participant for materials
|
Yes
|
|
2-day, on-site Improving Decision Making Through Critical Thinking Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $25 per participant for materials
|
Yes
|
|
2- to 5-day, on-site Fundamentals of Supervising HOMEBUILDERS: Intensive Family Preservation Services Training (2-part training)
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $275 per participant for materials
|
Yes
|
|
2- to 3-day, on-site Program Consultation and Quality Assurance Skills for HOMEBUILDERS Supervisors Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $75 per participant for materials
|
Yes
|
|
1.5- to 2-day, on-site Online Data Manager Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $15 per participant for materials
|
Yes
|
|
2-day, on-site Addressing Domestic Violence: Strategies for In-Home Practitioners Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $45 per participant for materials
|
No
|
|
1-day, on-site Ethical Issues in In-Home Services Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $15 per participant for materials
|
No
|
|
1-day, on-site Self-Advocacy Skills for Families: A Territorial Model of Assertiveness Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $10 per participant for materials
|
No
|
|
1-day, on-site Working With Parents With Cognitive Limitations Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $15 per participant for materials
|
No
|
|
1-day, on-site Substance Exposed Newborns Training
|
- $1,250 per day for 1 trainer (up to 15 participants) or $2,500 per day for 2 trainers (up to 30 participants), plus travel expenses
- $15 per participant for materials
|
No
|
|
Phone consultations (held weekly in the first 2 years of implementation, monthly in year 3, and quarterly thereafter)
|
$100 per hour
|
Yes
|
|
3- to 4-day, on-site visits (2 times per year)
|
$1,250 per day, plus travel expenses
|
Yes
|
|
Technical assistance as needed via phone or email
|
$100 per hour
|
Yes
|
|
File and fidelity reviews (2 times per year)
|
$100 per hour
|
Yes
|
|
Access to the Online Data Manager, a Web-based client information and data system that includes assessments, service plans, service summaries, contact logs, referral information, and other quantitative data tools for program fidelity
|
- $4,900 activation fee (year 1 only)
- $350 monthly fee
- $980 annual upgrade fee
|
Yes
|
|
Replications
Selected citations are presented below. An asterisk indicates that the document
was reviewed for Quality of Research.
* Blythe, B. J., & Jayaratne, S. (1999). Michigan Families First effectiveness study: A summary of findings. Report submitted to the State of Michigan, Family Independence Agency.
Fraser, M. W., Pecora, P. J., & Haapala, D. A. (1991). Families in crisis. New York, NY: Aldine de Gruyter.
Fraser, M., Walton, E., Lewis, R., Pecora, P., & Walton, W. (1996). An experiment in family reunification services: Correlates of outcomes at one year follow up. Children and Youth Services Review, 18(4/5), 335-361.
Kentucky Department for Community Based Services. (2008, January). Kentucky's Family Preservation program: Comprehensive program evaluation. Frankfort, KY: Author.
* Kirk, R. S., & Griffith, D. P. (2004). Intensive family preservation services: Demonstrating placement prevention using event history analysis. Social Work Research, 28(1), 5-18.
* Morris, E., Suarez, L., & Reid, J. C. (1997). Behavioral outcomes of home-based services for children and adolescents with serious emotional disorders. Family Preservation Journal, 2(2), 21-32.
Rodenhiser, R. W., Chandy, J., & Ahmed, K. (1995). Intensive family preservation services: Do they have any impact on family functioning? Family Preservation Journal, 1(1), 69-85.
Wood, S., Barton, K., & Schroeder, C. (1988). In-home treatment of abusive families: Cost and placement at one year. Psychotherapy, 25, 409-414.
|
|
|
