Quality of Research
Review Date: January 2012
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Anker, M. G., Duncan, B. L., & Sparks, J. A. (2009). Using client feedback to improve couple therapy outcomes: A randomized clinical trial in a naturalistic setting. Journal of Consulting and Clinical Psychology, 77(4), 693-704.  Study 2Reese, R. J., Norsworthy, L. A., & Rowlands, S. R. (2009). Does a continuous feedback system improve psychotherapy outcome? Psychotherapy: Theory, Research, Practice, Training, 46(4), 418-431.  Study 3Miller, S. D., Duncan, B. L., Brown, J., Sorrell, R., & Chalk, M. B. (2006). Using formal client feedback to improve retention and outcome: Making ongoing, real-time assessment feasible. Journal of Brief Therapy, 5(1), 5-22. Study 4Reese, R. J., Toland, M. D., Slone, N. C., & Norsworthy, L. A. (2010). Effect of client feedback on couple psychotherapy outcomes. Psychotherapy: Theory, Research, Practice, Training, 47(4), 616-630. 
Supplementary Materials Campbell, A., & Hemsley, S. (2009). Outcome Rating Scale and Session Rating Scale in psychological practice: Clinical utility of ultra-brief measures. Clinical Psychologist, 13(1), 1-9.
Miller, S. D., Duncan, B. L., Brown, J., Sparks, J. A., & Claud, D. A. (2003). The Outcome Rating Scale: A preliminary study of the reliability, validity, and feasibility of a brief visual analog measure. Journal of Brief Therapy, 2(2), 91-100.
Outcomes
| Outcome 1: Therapeutic progress |
|
Description of Measures
|
Therapeutic progress was assessed using the Outcome Rating Scale. The ORS is a 4-item self-report scale that measures a participant's psychological functioning and distress by asking how the participant is doing individually (personal well-being), interpersonally (family, close relationships), socially (work, school, friendships), and overall (general sense of well-being). Depending on the method of ORS administration, participants rate each item either by marking a line to reflect the strength of their opinions or by stating the appropriate ratings. Scores for each item range from 1 to 10, with a total score ranging from 0 to 40. Lower scores indicate more severe distress, and a comparison of the change in scores over time indicates the trajectory of therapeutic progress.
|
|
Key Findings
|
In one study, participating couples were randomly assigned to the intervention group, which received couple therapy with PCOMS, or the comparison group, which received couple therapy only. Couples in the intervention group had higher ORS scores than those in the comparison group at posttreatment (p < .001) and 6-month follow-up (p < .01), even after adjusting for pretreatment ORS score and therapist.
In another study, participants were randomly assigned to the intervention group, which received individual therapy with PCOMS, or the comparison group, which received individual therapy only. Half of the participants received services from faculty members or practicum students at a university counseling center, and half received services from practicum students at a graduate training clinic. Among participants who received services at a university counseling center, from pre- to posttest, those in the intervention group had a greater increase in ORS scores than those in the comparison group (p < .05). Among participants who received services at a graduate training clinic, from pre- to posttest, those in the intervention group had a greater increase in ORS scores than those in the comparison group (p < .01), even after adjusting for pretest ORS scores.
In a third study, participants in an employee assistance program received at least two phone-based counseling sessions that included PCOMS. The study had multiple phases, including a 6-month baseline period, when PCOMS was administered during the counseling sessions; a subsequent 6-month period, when a computer program was introduced to aid therapists in administering PCOMS and interpreting a participant's ORS and SRS scores during counseling sessions; and a 12-month period, after use of the computer program had been fully integrated into the counseling sessions. The average change in participants' ORS scores between counseling sessions was compared over these three phases. The mean increase in participants' ORS scores was larger for the subsequent 6-month period (p < .001) and 12-month period (p < .001) than it was for the baseline period.
In a fourth study, participating couples were randomly assigned to the intervention group, which received couple therapy with PCOMS, or the comparison group, which received couple therapy only. From pre- to posttest, couples in the intervention group had a greater increase in ORS scores than those in the comparison group (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3, Study 4
|
|
Study Designs
|
Experimental, Preexperimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
| Outcome 2: Marital status |
|
Description of Measures
|
Marital status was assessed by client self-report. Couples were categorized as intact (i.e., not divorced or separated) or not intact.
|
|
Key Findings
|
Couples participating in the study were randomly assigned to the intervention group, which received couple therapy with PCOMS, or the comparison group, which received couple therapy only. At the 6-month follow-up, a greater proportion of couples in the intervention group were intact relative to couples in the comparison group (p = .014).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
18-25 (Young adult) 26-55 (Adult) 55+ (Older adult)
|
50% Female 50% Male
|
100% Non-U.S. population
|
|
Study 2
|
18-25 (Young adult) 26-55 (Adult) 55+ (Older adult)
|
70.3% Female 26.4% Male
|
79.1% White 10.8% Hispanic or Latino 5.4% Race/ethnicity unspecified 3.4% Black or African American 1.4% Asian
|
|
Study 3
|
18-25 (Young adult) 26-55 (Adult)
|
66.7% Female 33.3% Male
|
Data not reported/available
|
|
Study 4
|
18-25 (Young adult) 26-55 (Adult) 55+ (Older adult)
|
50% Female 50% Male
|
73.9% White 16.3% Hispanic or Latino 5.4% Race/ethnicity unspecified 4.3% Black or African American
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Therapeutic progress
|
4.0
|
3.5
|
2.5
|
2.3
|
3.0
|
3.3
|
3.1
|
|
2: Marital status
|
3.5
|
3.5
|
2.5
|
1.5
|
3.0
|
4.0
|
3.0
|
Study Strengths The reliability and validity of the ORS are well supported. The studies were undertaken in a variety of real-world settings and included therapists with a variety of professional qualifications. Three of the four studies included random assignment, which helped mitigate the effect of potential confounds. The use of advanced analytical techniques in the same three studies produced strong and dependable findings.
Study Weaknesses Although all studies appear to have used mechanisms to ensure intervention fidelity, there was no formal assessment of whether or to what extent the intervention was delivered as intended. Attrition was substantial in three of the four studies and difficult to assess in the fourth. Issues arising from the use of the ORS both as part of the intervention and as the outcome measure are not addressed.
|
|
Readiness for Dissemination
Review Date: January 2012
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Implementation materials:
- Bargmann, S., & Robinson, B. (2011). International Center for Clinical Excellence manuals on feedback-informed treatment (FIT): Manual 2--Feedback-informed clinical work: The basics. Chicago, IL: International Center for Clinical Excellence.
- Bertolino, B., & Babbins-Wagner, R. (2011). International Center for Clinical Excellence manuals on feedback-informed treatment (FIT): Manual 6--Implementing feedback-informed work in agencies and systems of care. Chicago, IL: International Center for Clinical Excellence.
- Bertolino, B., Bargmann, S., & Miller, S. (2011). International Center for Clinical Excellence manuals on feedback-informed treatment (FIT): Manual 1--What works in therapy: A primer. Chicago, IL: International Center for Clinical Excellence.
- Duncan, B., & Miller, S. (2007). Group Session Rating Scale (GSRS).
- Duncan, B., Miller, S., Huggins, A., & Sparks, J. (2003). Young Child Outcome Rating Scale (YCORS).
- Duncan, B., Miller, S., Huggins, A., & Sparks, J. (2003). Young Child Session Rating Scale (YCSRS).
- Duncan, B., Miller, S., & Sparks, J. (2003). Child Outcome Rating Scale (CORS).
- Duncan, B., Miller, S., & Sparks, J. (2003). Child Session Rating Scale (CSRS).
- Duncan, B., Miller, S., Sparks, J., & Murphy, J. (2011). Child Group Session Rating Scale (CGSRS).
- International Center for Clinical Excellence. (2011). Overview of ICCE training & treatment manuals. Chicago, IL: Author.
- International Center for Clinical Excellence. (2011). The FIT implementation process. Chicago, IL: Author.
- Maeschalck, C., & Babbins-Wagner, R. (2011). International Center for Clinical Excellence manuals on feedback-informed treatment (FIT): Manual 5--Feedback-informed clinical work: Advanced applications. Chicago, IL: International Center for Clinical Excellence.
- Maeschalck, C., & Miller, S. (2011). International Center for Clinical Excellence manuals on feedback-informed treatment (FIT): Manual 3--Feedback-informed supervision. Chicago, IL: International Center for Clinical Excellence.
- Miller, S. (2010). Workshop and consultation packet.
- Miller, S. (2011). Psychometrics of the SRS and ORS: Results from meta-analyses of routine outcome monitoring and feedback: The available evidence [PowerPoint slides].
- Miller, S., & Duncan, B. (2000). Outcome Rating Scale (ORS).
- Miller, S., & Duncan, B. (2004). The Outcome and Session Rating Scales: Administration and scoring manual. Chicago, IL: Institute for the Study of Therapeutic Change.
- Miller, S., Duncan, B., & Johnson, L. (2002). Session Rating Scale (SRS v.3.0).
- Seidel, J., & Miller, S. (2011). International Center for Clinical Excellence manuals on feedback-informed treatment (FIT): Manual 4--Documenting change: A primer on measurement, analysis and reporting. Chicago, IL: International Center for Clinical Excellence.
Training materials:
- Advanced FIT Training materials:
- Advanced intensive jeopardy game [PowerPoint slides]
- International Center for Clinical Excellence. (2010). Curriculum for the advanced 4-day in feedback-informed treatment. Chicago, IL: Author.
- International Center for Clinical Excellence. (2011). ICCE core competency exam: Version A. Chicago, IL: Author.
- Miller, S. (2011). Transforming behavioral health care: Integrating feedback-informed services into behavioral healthcare [PowerPoint slides].
- Miller, S., & Bargmann, S. (2011). Advanced intensive: Day 2 [PowerPoint slides and handouts]. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., & Bargmann, S. (2011). Feedback informed treatment: The advanced intensive [PowerPoint slides and handouts]. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., & Bargmann, S. (2011). Putting data to work: Applying research concepts to the management of outcome in treatment [PowerPoint slides and handouts]. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., Axsen, R., & Seidel, J. (2011). Core competencies for the ICCE clinician. Chicago, IL: International Center for Clinical Excellence.
- Mindset questionnaire
- Basic FIT Training Workshop materials:
- Axsen, R., & Maeschalck, C. (2011). Feedback informed treatment in practice: A summary. Chicago, IL: International Center for Clinical Excellence.
- Axsen, R., & Maeschalck, C. (2011). Feedback informed treatment: Reading list. Chicago, IL: International Center for Clinical Excellence.
- Axsen, R., & Maeschalck, C. (2011). Key research underpinnings for feedback informed work. Chicago, IL: International Center for Clinical Excellence.
- Axsen, R., & Maeschalck, C. (2011). Training evaluation form. Chicago, IL: International Center for Clinical Excellence.
- International Center for Clinical Excellence. (n.d.). Feedback-informed treatment (FIT): Improving effectiveness by doing what counts [PowerPoint slides]. Chicago, IL: Author.
- Maeschalck, C., & Axsen, R. (2011). Feedback-informed treatment (FIT): Curriculum for one day basic training. Chicago, IL: International Center for Clinical Excellence.
- Miller, S. (n.d.). Feedback-informed treatment: Making services FIT consumers.
- FIT Supervision Training materials:
- Axsen, R., & Maeschalck, C. (2011). Key tasks for FIT supervisors: A primer. Chicago, IL: International Center for Clinical Excellence.
- Axsen, R., & Maeschalck, C. (2011). Practical application of FIT in clinical supervision. Chicago, IL: International Center for Clinical Excellence.
- Axsen, R., & Maeschalck, C. (2011). Supervision tips for cases of concern. Chicago, IL: International Center for Clinical Excellence.
- International Center for Clinical Excellence. (n.d.). Feedback informed treatment (FIT): Clinical supervision [PowerPoint presentation]. Chicago, IL: Author.
- Maeschalck, C., & Axsen, R. (2011). Feedback-informed supervision training curriculum. Chicago, IL: International Center for Clinical Excellence.
- Maeschalck, C., & Axsen, R. (2011). Foundations and principles of FIT supervision. Chicago, IL: International Center for Clinical Excellence.
- Maeschalck, C., & Axsen, R. (2011). Targeting cases for supervision. Chicago, IL: International Center for Clinical Excellence.
- Maeschalck, C., & Axsen, R. (2011). The skills of feedback informed treatment (FIT). Chicago, IL: International Center for Clinical Excellence.
- FIT Training of Trainers Workshop materials:
- Miller, S., Maeschalck, C., & Axsen, R. (2010). Agency implementation. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., & Axsen, R. (2010). Assessing agency readiness. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., & Axsen, R. (2010). FIT training of trainers: Core curriculum. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., & Axsen, R. (2010). Integrating FIT practices into clinical supervision. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., & Axsen, R. (2010). Key content for training supervisors. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., & Axsen, R. (2010). One day FIT training: Sample outline. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., & Axsen, R. (2010). Preparation tips for successful training events. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., & Axsen, R. (2010). Tips for trainers. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., & Axsen, R. (2010). 2010 training of trainers: Official agenda. Chicago, IL: International Center for Clinical Excellence.
- Miller, S., Maeschalck, C., & Axsen, R. (n.d.). Training of trainers [PowerPoint slides]. Chicago, IL: International Center for Clinical Excellence.
- Other training and support materials:
- International Center for Clinical Excellence. (2011). FIT training resources. Chicago, IL: Author.
- International Center for Clinical Excellence. (2011). ICCE support resources. Chicago, IL: Author.
- Miller, S. D. (2005). Client-directed outcome-focused psychotherapy [DVD]. In J. Carlson (Host), Series 1--Systems of psychotherapy. Washington, DC: American Psychological Association.
- Miller, S. (n.d.). Brief therapy inside out: Client-directed interaction: Adjusting the therapy not the person [DVD]. Phoenix, AZ: Zeig, Tucker, & Theisen.
- Miller, S. (n.d.). Outcome informed clinical work [CD-ROM]. Chicago, IL: Institute for the Study of Therapeutic Change.
- Miller, S. (n.d.). Therapeutic alliance: What works in therapy [CD-ROM]. Chicago, IL: Institute for the Study of Therapeutic Change.
- Miller, S., & Tilsen, J. (n.d.). Feedback informed treatment [DVD]. Hanover, MA: Microtraining and Multicultural Development.
- Miller, S., & Wyatt, R. (2009). What works in psychotherapy: Instructor's manual. San Francisco, CA: Psychotherapy.net.
- Miller, S., & Wyatt, R. (n.d.). What works in psychotherapy [Institutional and instructor's version DVD]. San Francisco, CA: Psychotherapy.net.
Quality assurance materials:
- Feedback Readiness Index and Fidelity Measure (FRIFM). (2005).
- International Center for Clinical Excellence. (2011). Feedback-informed concurrent service delivery agreement. Chicago, IL: Author.
- International Center for Clinical Excellence. (2011). Feedback-informed progress note. Chicago, IL: Author.
- International Center for Clinical Excellence. (2011). Feedback Readiness Index and Fidelity Measure instructions. Chicago, IL: Author.
- International Center for Clinical Excellence. (2011). FIT quality assurance procedures. Chicago, IL: Author.
- International Center for Clinical Excellence. (2011). Instructions for using the FIT service delivery plan & progress note. Chicago, IL: Author.
- Maeschalck, C., & Axsen, R. (2011). MyOutcomes data management system: User guide. Chicago, IL: International Center for Clinical Excellence.
Program Web site, http://www.centerforclinicalexcellence.com
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
4.0
|
4.0
|
4.0
|
4.0
|
Dissemination Strengths ICCE, an initiative to integrate PCOMS into clinical practice through FIT, has an array of comprehensive, well-organized, and high-quality materials to support the implementation of PCOMS. The steps for successful implementation are clear and accompanied by tools and guidance to support the entire process, from the determination of organizational readiness through evaluation. Key intervention tools are easily accessible through the program Web site. Training manuals address core clinical competencies and assist clinicians in using the intervention. Many training and consultation opportunities are available to support all stages of the implementation process, and the process for accessing these services is clear. A Web forum provides clinicians with opportunities to share program news and discuss successes and obstacles. Cost and staff burden are considered to encourage the use of quality assurance materials and procedures. Along with the ORS and SRS for outcome monitoring, client process monitoring tools and fidelity checklists are provided to support quality assurance. Measures are accompanied by clear guidance for administration and use.
Dissemination Weaknesses No weaknesses were identified by reviewers.
|
|