Quality of Research
Review Date: December 2011
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Early, K. W., Hand, G. A., Blankenship, J. L., & Chapman, S. F. (2011). Experiential community-based interventions for delinquent youth: An evaluation of recidivism and cost-effectiveness. Unpublished manuscript. Study 2Early, K. W., Blankenship, J. L., & Hand, G. A. (2011, June). Evaluation of AMIkids alternative school and juvenile justice program educational outcomes: An examination of pre/post test academic change. Tallahassee, FL: Justice Research Center.
Supplementary Materials 2010 AMIkids Outputs and Outcomes Report
AMIkids Behavior Modification Fidelity and Integrity Workbook
AMIkids Personal Growth Model Evaluation Manual
AMIkids Personal Growth Model Fidelity Indicators
Outcomes
| Outcome 1: Recidivism |
|
Description of Measures
|
Recidivism was assessed on the basis of the following factors:
- Readjudication/reconviction, defined as any subsequent offense that occurred within 1 year of the completion of program services and that resulted in either a juvenile adjudication or adult conviction
- Rearrest, defined as an arrest for any subsequent offense, whether adjudicated, convicted, or not, that occurred within 1 year of the completion of program services
- Felony rearrest, defined as an arrest for any subsequent felony offense (as defined statutorily by the State), whether adjudicated, convicted, or not, that occurred within 1 year of the completion of program services
- Felony readjudication/reconviction, defined as any subsequent felony offense (as defined statutorily by the State) that occurred within 1 year of the completion of program services and that resulted in either a juvenile adjudication or adult conviction
- Recommitment, adult probation, or prison incarceration, defined as any subsequent offense that occurred within 1 year of the completion of program services and that resulted in either a juvenile adjudication or adult conviction where the juvenile was sentenced to a juvenile commitment facility, adult probation, or an adult prison
These factors were determined from juvenile outcomes calculated for the Florida Department of Juvenile Justice (FDJJ) Comprehensive Accountability Report, which contains data extracted from the following sources:
- The Juvenile Justice Information System of FDJJ, which contains all statewide official delinquency records for the youth of the State
- The Criminal Justice Information Services repository of the Florida Department of Law Enforcement, which contains all statewide official arrest records for adults charged with felonies and for those juveniles charged with felonies who have been direct filed, waived, or transferred to the adult system for the State
- The Offender Based Information System of the Florida Department of Corrections, which contains all statewide official corrections records for adults and for those juveniles who have been direct filed, waived, or transferred to the adult system for the State
|
|
Key Findings
|
Youth who completed AMIkids PGM in a day treatment program were compared with a matched sample of youth who completed non-AMIkids PGM secure residential programming. Results indicated that relative to youth who completed residential programming, those who completed AMIkids PGM had lower rates of the following factors:
- Readjudication/reconviction (38% for AMIkids PGM vs. 43% for non-AMIkids PGM residential programming; p = .01)
- Rearrest (54% for AMIkids PGM vs. 59% for non-AMIkids PGM residential programming; p = .02)
- Felony rearrest (30% for AMIkids PGM vs. 42% for non-AMIkids PGM residential programming; p < .01)
- Felony readjudication/reconviction (30% for AMIkids PGM vs. 42% for non-AMIkids PGM residential programming; p < .01)
- Recommitment, adult probation, or prison incarceration (23% for AMIkids PGM vs. 29% for non-AMIkids PGM residential programming; p < .01)
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 2: Academic achievement |
|
Description of Measures
|
Academic achievement was assessed using the following measures:
- The Basic Achievement Skills Inventory (BASI), a norm-referenced achievement test that includes Math Computation, Math Application, Reading Comprehension, Vocabulary, Spelling, and Language Mechanics subtests. Trained staff administered the BASI on a computer to participants over approximately 2 hours and acted as proctors to ensure completion of the assessment in accordance with the developer's guidelines. Growth scale value (GSV) scores were calculated for each subtest and used to indicate academic achievement.
- The Brief Battery Assessment (BBA) and the Mini-Battery of Achievement (MBA), instruments that assess academic achievement through Reading, Writing, and Math subscales. Both instruments are designed for use with children and adults and require approximately 5-10 minutes to administer. Staff administered the BBA (also referred to as the Woodcock-Johnson III Normative Update Brief Battery) and the MBA, which was used until a transition was made to the BBA. Scoring was based on grade level equivalents (GEs).
|
|
Key Findings
|
A study examined academic achievement of youth receiving AMIkids PGM in an alternative school or juvenile justice setting. Participants were assessed at pre- and posttest with the BASI (youth who received AMIkids PGM in Florida) or the MBA or BBA (youth who received AMIkids PGM in Georgia, Illinois, Louisiana, New Mexico, North Carolina, South Carolina, or Virginia). Results were calculated for those participants who completed both pre- and posttests.
Youth who received AMIkids PGM in Florida had improvements in academic achievement from pre- to posttest, as indicated by the following results (presented in order of greatest improvement):
- For the Reading Comprehension subtest, average GSV scores increased by a factor of 2.52 (p < .001).
- For the Spelling subtest, average GSV scores increased by a factor of 1.98 (p < .001).
- For the Math Computation subtest, average GSV scores increased by a factor of 1.55 (p < .001).
- For the Vocabulary subtest, average GSV scores increased by a factor of 1.52 (p < .001).
- For the Math Application subtest, average GSV scores increased by a factor of 1.51 (p < .001).
- For the Language Mechanics subtest, average GSV scores increased by a factor of 1.51 (p < .001).
Youth who received AMIkids PGM in States other than Florida had improvements in academic achievement from pre- to posttest, as indicated by the following results (presented in order of greatest improvement):
- For the Reading subscale, the average increase in GE was 1.60 for the BBA (p < .001) and 1.37 for the MBA (p < .001).
- For the Math subscale, the average increase in GE was 1.61 for the BBA (p < .001) and 1.16 for the MBA (p < .001).
- For the Writing subscale, the average increase in GE was 1.54 for the BBA (p < .001) and 1.01 for the MBA (p < .001).
|
|
Studies Measuring Outcome
|
Study 2
|
|
Study Designs
|
Preexperimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
13-17 (Adolescent)
|
84% Male
16% Female
|
52% Black or African American
37% Race/ethnicity unspecified
11% Hispanic or Latino
|
|
Study 2
|
6-12 (Childhood)
13-17 (Adolescent)
|
Data not reported/available
|
Data not reported/available
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Recidivism
|
3.3
|
3.3
|
2.8
|
4.0
|
3.8
|
2.8
|
3.3
|
|
2: Academic achievement
|
3.8
|
3.8
|
2.8
|
2.8
|
2.8
|
3.0
|
3.1
|
Study Strengths The studies used standardized measures that had strong, well-documented reliability and validity. In one study, data were collected and verified from several State agencies (i.e., records of law enforcement, department of corrections, and juvenile justice), and there were no known missing records for participants. Matching the intervention and comparison groups on several variables at baseline controlled for potential confounding variables in one study. Both studies had large sample sizes.
Study Weaknesses Neither study showed evidence (i.e., reporting of data) of acceptable fidelity for adherence to intervention guidelines. There is no indication that study protocols were used. In one study, participants without pre- and posttest data were not discussed, and participants with and without pre- and posttest data were not compared. Data analysis did not adequately account for variance, thereby limiting confidence in both studies' results not having type I errors (i.e., false-positive results).
|
|
Readiness for Dissemination
Review Date: December 2011
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
AMIkids. (2006). Program excellence programming. Tampa, FL: Author.
AMIkids. (2009). Personal Growth Model: A unified approach to behavior modification, education and treatment [Training handouts]. Tampa, FL: Author.
AMIkids. (2009). Personal Growth Model: Service philosophy manual. Tampa, FL: Author.
AMIkids. (2010). Common issues with administering behavior modification system in AMIkids. Tampa, FL: Author.
AMIkids. (2010). Personal Growth Model: Evaluation manual July 1, 2010-June 30, 2011. Tampa, FL: Author.
AMIkids. (2010). Personal Growth Model: Fidelity indicators July 1, 2010-June 30, 2011. Tampa, FL: Author.
AMIkids. (2011). Behavior modification operations manual. Tampa, FL: Author.
AMIkids. (2011). Behavior modification workbook #2: Implementing behavior modification with integrity and fidelity. Tampa, FL: Author.
AMIkids. (2011). Personal Growth Model: A unified approach to behavior modification, education and treatment. Tampa, FL: Author.
AMIkids. (2011). Utilizing behavior modification in AMIkids. Tampa, FL: Author.
Other program materials:
- AMIkids Philosophy of Education
- Day Treatment Mental Health and Substance Abuse Services
- Program Excellence Plan
- Program Excellence Plan (PEP) Guidelines
Program Web site, http://www.amikids.org/
Training Web site, https://moodle.amikids.org/login
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
3.0
|
3.3
|
3.3
|
3.2
|
Dissemination Strengths The program materials are comprehensive and well written, and they cover all aspects of the intervention, from theory to practice. Practical solutions, based on years of experience in implementing the program at a variety of sites, are shared for potential implementation barriers. The program's training is highly structured and includes modules with quizzes to test the user's understanding of the content. Within the training materials, program-specific terms are reviewed to ensure their consistent use in implementation. Timetables and expectations for reporting quality assurance data to the developer are clearly addressed. Several manuals and tools are provided to aid new implementers in the development of a plan to monitor implementation fidelity. The program developer completes annual fidelity checks at all implementation sites, and designated regional directors are responsible for ensuring quality assurance.
Dissemination Weaknesses The administrative, programmatic, and fiscal relationships between the program developer and the licensed implementation sites are not clearly outlined in the program materials. Some implementation materials are not well organized for ease of use. It is unclear how new implementers learn about training events or requirements. Substantial amounts of staff time are needed for quality assurance reporting requirements.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
Annual licensing fee
|
$5,000 per site
|
Yes
|
|
5-day, on-site implementation training (includes all implementation materials, Behavior Modification System training, and pretraining readiness assessment)
|
$20,000 per site for up to 10 individuals, plus travel expenses
|
Yes
|
|
Annual on-site booster training
|
$7,000 per site, plus travel expenses
|
Yes, each year after implementation training
|
|
First-year, on-site follow-up consulting and videoconferencing (2 days per site per quarter)
|
$8,000 per site, plus travel expenses
|
Yes
|
|
Subsequent follow-up consulting and videoconferencing
|
$200 per hour
|
No
|
|
Fidelity evaluation tools, annual on-site assessment, and data collection
|
$8,000 per site, plus travel expenses
|
Yes
|
|
Monthly and annual analytical reports
|
$6,000 per site
|
Yes
|
|
Replications
No replications were identified by the developer.
|
|
|
