image_print

Michael J. Lambert, PhD

Continuing Education Information

Routine outcome monitoring (ROM) has become a recommended clinical practice. Evidence for the positive effects of monitoring client treatment response is presented, along with those aspects of monitoring that are needed to make it feasible and effective.  The monitoring of the outcome of therapy can easily be implemented into private practice routines as well as agency procedures.

During your Fourth of July break you got to sit on the beach alone several times. It was quite relaxing, and it gave you a chance to reflect on your patients and practice over the last six months. It was a tough period, as you had a number of difficult cases (with several crises). And you felt you did quite good work and helped most of your patients. However, one patient surprised you when he abruptly terminated in June, remarking that four months of therapy had done nothing for him—and, in fact, he felt worse. Moreover, one of your major referral sources recently informed you that two of the patients you are currently seeing mentioned that their therapy was not going too well. Could this really be true? Are your self-assessment skills and outcomes that bad? If so, what might you do differently in your practice is order to better predict when treatment is going awry and correct for it?

Actually, your experience reflects the common case for all psychotherapists—a self-assessment bias. The general perception of providers is that the success rate within their own caseload is close to 85% (i.e., their clinical impression is that nearly all of their clients improve). Each individual provider also regards themselves as having better outcomes than their peers, a finding that holds true across professions, craftsmen, and others (Walfish et al., 2012). Practitioners rarely note worsening in the client’s case record, even when it is dramatic. And even more importantly, they fail to predict which clients will leave treatment with a negative outcome even when asked to do so. Psychotherapists are optimistic about the potential for change, even in the face of great difficulties. Therapists offer their patients “hope” for a better future as well as problem solving and other skills to achieve it. Therapists always see the silver lining. This is a necessary stance in psychotherapy, but it also cuts the therapist off to some extent from the reality of some patients’ lack of positive change.

The harsh reality of clinical outcome research is that psychotherapy is not universally effective. In tightly controlled clinical trials, psychotherapy, of all kinds, has been shown to help a simple majority of treated patients. But it is also true that a significant percentage (between 40% and 60% of patients, depending on the severity of initial dysfunction) do not benefit—with about eight percent actually worsening. This appears to be true even when the “right” psychotherapy is provided to the “right” patient in high-quality clinical trials. In routine care outcomes are generally worse, likely because treatment length is shorter (five sessions rather than 12–20) and because clinicians offering routine care in community settings cannot use exclusion criteria or otherwise be selective about the patients they treat. Thus, one possible way of improving the general outcomes of psychotherapy is to reduce treatment failure (broadly defined as “no change as well as deterioration”).

“Routine outcome monitoring with feedback” is proposed as a method of alerting clinicians to potential treatment failure and likely causes—which can improve the general effects of psychotherapy as well. The American Psychological Association (APA, 2006) has recommended routine outcome monitoring (ROM) be a part of effective psychological services because certain methods of monitoring have been shown to enhance client outcome. However, the APA has not specified any particular practices or necessary characteristics of feedback practices. There are a growing number of feedback systems emerging, but the Outcome Questionnaire System (OQ-45, Lambert et al., 2013) and the Partners for Change System (PCOMS, Duncan & Miller, 2008) will be emphasized here in order to describe what providers need to consider if they implement feedback into their clinical practices.

The Effects of Feedback

Why Feedback Helps

Feedback theory suggests human performance is improved when feedback is provided that illuminates actual success and failure. In the case of psychotherapy—measuring, monitoring, and providing feedback are hypothesized to improve patient outcomes. More specifically, it assumes that when measured treatment response is added to a clinician’s subjective impression of success and there is a discrepancy, feedback will enable a psychotherapist to be more responsive to the patient. Feedback will be helpful to the extent that there is a discrepancy between a clinician’s overly optimistic judgment of patient progress and measured progress. This will occur if psychotherapists are aware of a discrepancy between the goal and “reality” (particularly if the goal is attractive and the clinician believes it can be accomplished); the feedback source is credible; and if feedback is immediate, frequent, systematic, cognitively simple (such as graphic in nature), unambiguous, and provides clinicians with concrete suggestions of how to improve. According to theory, it is not enough to formally measure and monitor a patient’s mental health status in order to improve the outcomes of treatment for an individual case. The feedback must also alert a clinician to the emerging failure to facilitate client progress.

Can Final Outcome be Accurately Predicted?

Many attempts have been made to predict the effects of psychotherapy. But much of this research relates to broad characteristics and group differences which cannot be used by the clinician to help the client they are currently facing (e.g., co-morbidity reduces likelihood of a positive outcome). A more precise question is needed. Can treatment failure (and, more specifically, outright deterioration) be predicted for a given patient while they are still in treatment? The answer is a resounding “yes,” especially if psychotherapy lasts five months or less.

We have found that two simple pieces of information are critical to know: 1) the degree of disturbance that characterizes the client at the time they enter treatment, and 2) the degree of disturbance that characterizes the client at each session of treatment following its inception. If we know how disturbed a person is and how they respond to our interventions, especially early on in treatment, this is the critical information. When we monitor client progress, we need to do so with an assessment tool that gives a good estimate of degree of disturbance, and we need to possess normative data from psychotherapy patients who have taken that measure over the entire course of psychotherapy.

With such information and statistical modeling methods, we can establish an expected (average) course of recovery and critical negative deviations from such a course. Essentially the further a client deviates negatively from expected recovery, the more likely they are to be a person who will be classified as “deteriorated” at termination. Should a client pass a critical threshold at any given session of care after their initial session they are considered “at-risk” (either not-on-track, or an alarm case) for deterioration. The therapist is alerted to this information each time the client completes the measure.


Feedback theory suggests human performance is improved when feedback is provided that illuminates actual success and failure.


Psychotherapists do not learn how to better predict treatment failure by using such actuarial systems. Rather, they gain information that something atypical is occurring and something may need to change in the treatment if it is to be successful. The metaphor would be a physician using a lab test to manage diabetes. Blood sugar level is measured, compared to normative data, and a feedback report generated with critical cut-off scores produced. Conditions outside of the normative range require new actions to be taken by doctor and patient to better manage the disease. The routine measurement of outcome in psychotherapy can provide the same type information on the status and condition of the patient’s psychological distress and functioning for the psychotherapist.

Based on precise measurement of mental health vital signs and statistical modeling, for that around eight percent of patients who will leave psychotherapy having “deteriorated,” we are able to accurately identify between 85% and 100%. Predictions using statistical models also produce “false alarms” (clients who are predicted to deteriorate but do not). However, it is rare that these incorrectly identified individuals ultimately achieve a positive outcome. Rather, they are likely to be in the “unchanged” outcome group—and thus warrant additional attention regarding the intervention and relationship anyway. Thus, these so-called false alarms are not true errors of prediction, as they still represent cases that are not moving in a clearly positive direction.

How Large is The Benefit of Using ROM with the OQ-45?

Sixteen clinical trials examining the impact of OQ-45 feedback on clients have been conducted to date (Lambert & Whipple, in press). Most of these studies have been undertaken in the USA (five in Western Europe). Patient samples have included community mental health, university counseling centers, substance abuse, specialty inpatient eating disorder, psychosomatic inpatient, and outpatient clinics. The clients are representative of mixed Axis I disorders treated in individual and group psychotherapy mainly with CBT and other eclectic mixtures. The usual design called for clients to be randomly assigned within therapists to receive either treatment as usual (TAU) or TAU+feedback. About 2/3 of studies found a statistically significant difference between the feedback-assisted psychotherapy and psychotherapy practiced without feedback among clients who went off-track in the course of treatment. Much to therapists’ surprise, their own off-track clients had superior outcomes when the therapists were given feedback.


The most powerful intervention appeared to be formal feedback to therapists and clients plus Clinical Support Tools that helped clinicians problem solve with off-track cases.


Among the sixteen studies, a variety of feedback methods were examined—feedback to therapists which they were free to discuss with their clients, feedback to therapists and directly to clients through graphs and written reports, and feedback to therapists plus problem-solving tools. The most powerful intervention appeared to be formal feedback to therapists and clients plus Clinical Support Tools that helped clinicians problem solve with off-track cases. Shimokawa, Lambert, and Smart (2010) provided an analysis of individual patient success with and without feedback of this kind. Client deterioration was reduced from 1 in 5 clients (that is, 20%) in treatment as usual to 5.5% in feedback assisted treatment that included the use of Clinical Support Tools. The reader is reminded that these outstanding results apply only to “at-risk” (either not-on-track, or off-track) clients. For clients who progress as expected, feedback does not have such dramatic effects.

The number of clients identified as “at-risk” during treatment varies from clinic to clinic and is dependent on how fragile the patients are. In community mental health centers that serve clients with long histories of poor functioning, early mistreatment, and serious lack of resources in their current life, the rate of going off-track can be expected to be quite high (perhaps 40%). With clients being seen in a college counseling center and who in general are much less disturbed and who have much better external resources, the rate of going off-track will be much lower (closer to 20%). But, even in such college counseling centers, it will be the most disturbed (and most vulnerable) who are most prone to have a rough and bumpy road to recovery or leave treatment worse off than they started. After 20 years of research and development, the OQ-45 provides an effective method for preventing treatment failure with the most vulnerable and at-risk patients within a psychotherapist’s caseload.

How Effective is Using ROM with PCOMS Outcomes?

There are now nine clinical trials of feedback effects using Partners for Change System (PCOMS, Duncan & Miller, 2008) methodology. This methodology grew out of the OQ-45 system, with which it has numerous commonalities. For example, it is aimed at tracking patients’ progress on a session-by-session basis. However, the developers felt the 45-item OQ-45 was too long and developed an ultra-brief alternative (the Outcome Rating Scale; ORS), while still trying to capture patients’ symptomatic distress, interpersonal relationships, social role functioning, and well-being. They reduced the 22 items in the OQ-45 symptom distress subscale (e.g., looking back over the last week I have had trouble with headaches, tiredness, sadness) to a single item rated on a scale from 1–10. While the OQ is completed at a computer kiosk, on a cellphone, or online, the PCOMS is most often completed in the presence of the therapist as a hard copy at the beginning of each session.

In addition to the four outcome items the developers created an ultra-brief measure of the therapeutic alliance, the Session Rating Scale (SRS), which is administered to the client near the end of each session. This procedure was put in place because the alliance, as reported by the client, correlates with psychotherapy outcome, whereas therapist judged alliance does not. These procedures when used together insure that client and therapist will quantify and discuss, at each session of care, progress and relationship issues (and thereby increase collaborative efforts and client engagement in treatment). PCOMS also has a method of identifying off-track cases on both progress and relationship, alerting therapist to possible problems with progress and relationship, and thus prompting actions on the part of the therapist to resolve problems in the therapy.

Lambert and Whipple (in press) reported that six of the nine PCOMS studies found the difference between feedback-assisted treatment using the PCOMS methods compared to TAU reached statistical significance. In the cases where it did not reach significance, some benefits still accrued. Like the OQ-System, PCOMS has been studied across a wide variety of patient populations and problems, and in studies of individual, group, and couple treatment. It often achieves its effects across all clients, rather than just with at-risk clients, although its alarm system identifies the majority of cases as at-risk. The size of treatment effects suggests a possible doubling of the number of successful (reliably improved or recovered) cases and a definite reduction of cases regarded as having not changed by the time therapy is terminated. Although the average effect size between TAU and TAU plus feedback of .30 is considered small, it is certainly larger than that produced by many medical interventions and those obtained when comparing an evidence-based psychotherapy compared to a TAU control (d = .10–.20; Lambert, 2013).

Implementation

From a practical perspective, the logistics of maintaining an ongoing treatment-monitoring system are easier to manage when assessment becomes a routine part of practice applied with all clients at every session. Both systems are relatively easy to implement. Both can be used in a paper-pencil format, but this alternative, although practically free, takes clinician time and eliminates the application of statistical models for predicting treatment failure. Since the effective component of ROM is alerting the clinician to off-track status, use of hard copy administration cannot be recommended as a best practice.

Instead clients can complete the OQ-45 between sessions at home or online via the Internet or smart phones. The PCOMS is completed within the session, with the ORS (treatment progress) discussed at the beginning of the session and the SRS (relationship) discussed at the end. If the client completes the OQ-45 in the office they should be instructed to come 10 minutes early. At the time the instruments are introduced to the client the therapist indicates that she will ask the clients to complete a measure of mental health functioning for the purpose of tracking how things are going (a kind of mental health lab test), as a means of capturing what is going on in areas that may not be talked about during sessions (e.g., sleep problems, sexual satisfaction, suicidal thinking, substance abuse). The process of administration of ROM has become increasingly automated and can easily be handled as part of routine administrative tasks. Both measures (and related child measures) can be demonstrated online, installed on the clinician’s desktop, which can be provided with IT support if needed (www.centerforclinicalexcellence.com; www.oqmeasures.com). For a review and critique of the system and their costs, the interested provider can visit The National Registry of Evidence-Based Programs and Practices (www.nrepp.samhsa.gov/) and search for empirically supported interventions under either “PCOMS” or “OQ-Analyst.” For example, NREPP recently rated the OQ-Analyst a 3.9 (out of 4) and the PCOMS 4 out of 4 as meeting their criteria for “readiness for dissemination” based on user guides and step-by-step instructions for implementation.

The OQ-Analyst software is licensed on a yearly basis and the cost (about $200 per year) is per clinician rather than per client or per administration. PCOMS (ORS and SRS) is free as long as it is used in the paper/pencil versions. The PCOMS System can also be installed by visiting their website. In contrast to the OQ System, it is typically expected that the 4-item ORS will be completed in-session and discussed immediately afterwards at the start of the session. If it is completed in the waiting room and through the use of computer or smart phone, a report is quickly available to the clinician. The SRS (Session Rating Scale) is completed by the client at the end of each session and scored by therapist or client and then briefly discussed. The amount of session time used for the ORS and SRS is variable across clients and across sessions but intentionally brief.


The process of administration of ROM has become increasingly automated and can easily be handled as part of routine administrative tasks.


In the typical case a clinician will visit the system’s website and view the available options for administration. The most preferred method in a solo practice for using the OQ system is to have the OQ-Analyst software installed on the therapist’s computer by OQMeasures’ IT Department. At that time therapists decide how they want clients to enter data, i.e., via a computer kiosk or tablet in the office, or at home through any device that has connectivity, such as an iPad, cell phone, or computer.

Once in use, after a client enters their data, a therapist report is rapidly generated on the therapist’s computer. This report includes treatment outcome predictions, a graph of progress, and a written message interpreting the graph. For example, based on the client’s initial score and score at the current session, the therapist would get a one-page report that would indicate progress is satisfactory (or problematic); a graph documenting progress at all previous sessions in relation to a line indicating healthy functioning, and a line indicating expected treatment response for individuals entering treatment at the same level of initial disturbance. Affirmation of critical items by the client (such as suicidal ideation) would be highlighted; and a written message interpreting the progress report are also provided for the therapist. In the case that the therapist wishes to provide the client with a written report, one with language more suitable for clients, can be generated with a click. If the client meets criteria for being “at risk” for treatment failure they are asked to complete a 40-item problem-solving measure that is used to speed up identification of factors that may be responsible for client worsening such as problems with the therapeutic alliance or problems with social supports. In the PCOMS system, the SRS feedback is discussed in session and the client and therapist work out problems together.

If the therapeutic alliance is identified as problematic, the OQ-Analyst highlights the general area that is problematic (bond, task, or goal disagreement) as well as providing specific problematic-item feedback (e.g., a low rating on an item such as “My therapist seems glad to see me”) and concrete suggestions are made based on the work of Safran, Muran and Eubanks-Carter (2011) on alliance rupture repair:

  • Pay careful attention to the amount of agreement between you and your client concerning the overall goals of treatment and the tasks necessary to achieve those goals
  • Reframe the meaning of tasks or goals and/or modify tasks and goals
  • Work with resistance by retreating when necessary and
    being supportive
  • Provide a therapeutic rationale for your techniques,
    actions, and/or behaviors and repeat the rationale throughout
    treatment
  • Discuss the here-and-now therapeutic relationship with your client
  • Give and ask for feedback on the therapeutic relationship
  • Spend more time exploring your client’s experiences
  • Pay attention to subtle cues that there may be a problem with the alliance
  • Allow the client to assert their negative feelings about the relationship
  • Explore with your client their fears about asserting their negative feelings about the relationship
  • Accept responsibility for your part in alliance ruptures
  • Give more positive feedback
  • Process transference and be aware of counter transference
  • Discuss therapist and therapeutic style match
  • Discuss shared experiences
  • Clarify misunderstandings at a surface level
  • Explore links between the alliance rupture and common patterns in the client’s life

The OQ system and PCOMS were designed to be tools to aid clinical decision making rather than to dictate or prescribe to clinicians. Decisions regarding the continued provision of treatment, the modification of ongoing treatment, obtaining case consultation or supervision, or the application of clinical support tools cannot be made on the basis of a single questionnaire or independent from clinical judgment. Thus, we envision these ROM systems as analogous to a “lab test” in medical treatment, which can supplement and inform clinical decision making rather than replacing the clinician’s judgment.

Some Limitations

Within routine practice several limitations have been identified. Some individuals do not provide valid self-reports of their functioning. Highly disturbed individuals such as those who are currently psychotic and individuals who cannot read have great difficulties providing valid data. At times this may require a therapist to ask clients about their understanding of the measure and its purpose. It is easy to discern the purpose of self-report scales, and clients who want to provide distorted information can do so easily. For example, those individuals who are coerced into treatment may believe it is in their best interest to understate their problems all along the way. There are also instances where overstating problems may distort the intake level of problematic states and throw off the algorithms used for identification of at-risk cases. Trust issues may result in some clients using the measures to distort scores in order to find out if there will be any negative consequence that will come to them.

A problem is sometimes raised about the degree to which the ORS and SRS scores are affected by pressure to please the therapist, especially because the measures are completed in the presence of the therapist. Even if therapists are careful to indicate the importance of honest responding and true interest in understanding the client’s experience, it is not uncommon for clients to want to please their therapist by overstating their gains as a way to reward therapists for their efforts. Therapists may avoid or minimize this by saying, “in order for you to get better, your honesty is essential, don’t worry about hurting my feelings.”

While, one must be alert to the above possibilities, the vast majority of clients understand the measures are being used to track their mental health functioning and report honestly. After all, they want help and want to get better. When they understand that honest feedback is important, they provide it.

Wrapping Up

Routine outcome monitoring (ROM) is highly recommended, and it is becoming a practice expectation. The OQ-45 and PCOMS represent two empirically validated systems for ROM in psychological services settings, both private practice and agency practice. They can be cost-efficiently and time-efficiently incorporated in routine clinical processes and practice.

Michael J. Lambert, PhD, is a former Professor in the Department of Psychology at Brigham Young University, and holds the Susa Gates University Professorship. He is coauthor of the Outcome Questionnaire and Youth Outcome Questionnaires (measures of patient change used in private practice, public mental health, and managed care settings).He edited Bergin & Garfield’s Handbook of Psychotherapy and Behavior Change (5th & 6th Eds.) Wiley, 2004, 2013. He has been engaged in private practice as a humanistically-oriented psychotherapist throughout his 43 year career.

References

American Psychological Association Presidential Task Force on Evidence-Based Practice. (2006). Evidence based practice in psychology. American Psychologist, 61, 271-285. doi:
10.1037/0003-066X.61.4.271

Duncan, B. L., & Miller, S. D. (2008). The Outcome and Session Rating Scales: The revised administration and scoring manual, including the Child Outcome Rating Scale. Chicago: Institute for the Study of Therapeutic Change.

Lambert, M. J. (2013). The efficacy and effectiveness of psychotherapy. In M. J. Lambert (Editor) Bergin & Garfield’s Handbook of Psychotherapy and Behavior Change (6th Ed., Pp. 169-218). New York: Wiley.

Lambert, M. J., Kahler, M., Harmon, C., Burlingame, G. M., Shimokawa, White, M. M. (2013). Administration and Scoring Manual: Outcome Questionnaire OQ®-45.2. Salt Lake City: OQMeasures.

Lambert, M. J. & Whipple, J. M. (in Press). Client and therapist feedback. (in press). In J. C. Norcross & M. J. Lambert (Eds.) Relationships that work (3rd Edition) Volume I. Washington, DC: APA Press. Check this – I believe wrong publisher. CHECK – I think Oxford Univ Press.

Safran, J. D., Muran, J. C., & Eubanks-Carter, C., (2011). Repairing alliance ruptures. Psychotherapy 48(1), 80-87.

Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of psychotherapy quality assurance systems. Journal of Consulting and Clinical Psychology,78, 298-311. doi:10.1037/a0019247

Walfish, S., McAlister, B., O’Donnell, P., & Lambert, M. J. (2012). An investigation of self-assessment bias in mental health providers. Psychological Reports, 110, (2), 639-644. DOI 10.2466/02.07.17.