Michael J. Lambert, PhD
Psychotherapy of various orientations and formats has been found to be effective across a variety of patient disorders. The extent and richness of this finding extends over decades of research, thousands of treated individuals, hundreds of settings, and multiple cultures. Psychotherapists should be encouraged by the mass and breadth of empirical results that clearly demonstrate that the treatments they provide reduce distressing symptoms, resolve interpersonal problems, restore work performance, and improve life quality for the majority of those who seek treatment (Lambert & Ogles, 2004).
Nevertheless, it is also clear that psychotherapy can occasionally be harmful or result in no detectable progress in a minority of patients. Estimates of the number of patients who deteriorate while in treatment are difficult to obtain, but a fair estimate is between 5% and 10% (Lambert & Ogles, 2004). Just as positive therapy outcomes depend largely on patient characteristics, so
too do the negative changes that occur in patients who are undergoing psychological treatments. Even so, positive as well as negative patient change can be affected by therapist actions and inactions.
Despite the relatively small proportion of treatment failures, preventing negative outcomes is a topic of considerable importance. The current climate of enhancing patient outcomes has placed a primary emphasis on studying and documenting effective treatments for specific disorders and increasing the likelihood that an empirically supported or evidence-based treatment will be offered to the patient. Unfortunately, offering the right treatment for the right disorder is not a remedy that has a proven track record at reducing patient deterioration. Even in clinical trials that support the value of a particular treatment with a particular disorder, a sizable minority of patients find no benefit (some-where around 40%; Hansen, Lambert, & Forman, 2002) and a small group of patients deteriorate.
The problems of non-response and negative response are central issues for health care providers and policy makers as economic pressures force reductions in services. External pressures for cost containment (economic pressures) immediately expose two important provider vulnerabilities: (1) psychotherapy service delivery is largely driven by theory-based practices rather than empirical research, and (2) the inability of clinicians to make accurate judgments about patient worsening. These vulnerabilities lead to a paradox. In general, the judgments by psychotherapists about how to help and about how treatment is progressing, tend to be overly positive and rarely indicate that a patient is having an unfavorable treatment response (worsening while under care). In some ways, this is a positive phenomenon. It suggests considerable optimism, confidence, and hope within the therapist about his/her own treatment methods and the effects and attitudes that have been shown to be related to positive patient outcome. This optimismallows therapists to work hard in the face of difficult patient behaviors and severe dysfunction and to remain determined in the face of minimal or small improvement. But it also allows therapists to ignore, to some degree, patient worsening. For example, considerable clinical lore has built up around the idea that it is to be expected that patients will get worse before they get better, despite the fact that this is rarely the road to recovery and, in fact, is an indicator that portends a final negative outcome. The positive traits of theory allegiance and confidence may make it difficult for therapists to appreciate the importance of negative treatment response and take appropriate steps to ward off treatment failure.
Research is needed to investigate the extent of the problem. To investigate therapist accuracy in predicting negative treatment outcomes, Hannan, et al. (2005) asked a group of 48 therapists (26 trainees and 22 licensed psychologists) at a university outpatient clinic to predict which of their clients were likely to end treatment worse off than when they started treatment. The specific question was: In your clinical judgment alone, predict this client’s end of treatment outcome. This client will (choose one prediction): Recover, Improve but not recover, Make no progress in treatment, Get worse. Therapists were informed both verbally and in written form previous to the study that deterioration rates in the clinic had remained relatively constant at 8% over the preceding years, and that the primary interest in administering the questionnaire was to see if therapists could indeed predict that important percentage of clients who worsen during psycho-therapy. Therapists made predictions over a three week time period with predictions made for a client on either one, two, or three occasions.
Therapist prediction of negative outcome was compared with outcome based on client self-reported mental health status as measured by the Outcome Questionnaire-45. The Outcome Questionnaire (OQ-45) is a 45-item measure developed specifically for the purpose of tracking and assessing clients symptomatic states, degree of interpersonal problems, and role functioning. It was created to be brief (take 5 minutes) and to be administered repeatedly in a therapeutic setting (Lambert, et al., 2004). The OQ-45 is a well-established instrument that has been validated across the country and across a broad range of normal and client populations from various ethnic groups. It has excellent reliability and correlates highly with other measures of client disturbance such as the Symptom Checklist-90, MMPI, Beck Depression Inventory, the State-Trait Anxiety Inventory, and similar measures of psychological disturbance.
Actual OQ-45 data collected by Hannan, et al. indicated that 40 clients (7.3%) out of a total of 550 clients deteriorated by the end of therapy but that therapists rarely predicted deterioration. In fact, they predicted only three (less than 1%) of 550 clients would deteriorate, and only one of the three clients predicted to deteriorate did so, a hit rate of 1/40 (2.5%). In contrast, actuarial methods based on the OQ-45 applied for the purpose of making the same predictions were able to accurately identify 31 of the 40 patients (hit rate of 78%) who deteriorated. These sobering results reinforce the notion that therapists need assistance from independent methods to alert them to the fact that treatment is not having its intended effects and that deterioration may be forthcoming for some clients.
Developing Actuarial Methods for Predicting Treatment Failure
The first step aimed at improving outcomes for poorly responding patients involved the development of a signal-alarm system that could identify the failing patient before termination occurred. In order to accomplish this goal, Outcome Questionnaire-45 data from over 11,000 patients whose progress was measured after each session of psychotherapy was subjected to statistical analysis that led to the development of average improvement on a session-by-session basis. Recovery of clients undergoing psychotherapy could then be modeled. Such modeling techniques resulted in recovery curves that varied mainly as a function of the initial level of patient disturbance. That is, highly disturbed patients, although making considerable gains while in treatment, take longer to recover than less disturbed patients. This finding is consistent with clinical experience and holds up across most diagnostic categories. In general, more disturbed depressed patients take longer to recover than less disturbed depressed patients, more disturbed anxiety disordered patients take more time to recover than less disturbed anxiety disordered patients, and patients with both Axis I and Axis II disorders take longer to recover than mono-symptomatic patients.
Given the large number of patients whose treatment response was known, it was possible to model recovery based on at least 220 patients at each level of initial disturbance. Thus, 50 recovery curves were developed representing the diversity of initial levels of disturbance shown by patients who enter psychotherapy. Such curves allow one to plot 220 patients treatment response after each treatment session and compare a given patient’s progress with patients who begin treatment at the same level of disturbance. Average recovery curves can also be used to see how far a given patient’s response to treatment deviates for the typical patient. As a consequence, confidence bands can be set up around the average response in order to flag a patient whose treatment response deviates too far from the typical patient. Decision rules can then be used to classify patient treatment response at every session of therapy as being within the range of expected response, better than expected, or, most importantly, at an alarmingly slow rate. As the first step in reducing patient deterioration rates, we classify each patient’s treatment response as on track (giving either a green or white signal) or as not on track (giving either a yellow or red signal), depending on how far away the patient is from the expected positive response.
These decision rules proved capable of identifying patients who eventually deteriorated by the time they left treatment. In a study of 492 consecutive clients who received psychotherapy, 36 (7.3%) were reliably worse/deteriorated at termination. The algorithms correctly identified all 36 (100%) during the course of treatment (86% of whom were identified by the third treatment session; Lambert, Whipple, Bishop, et al., 2002). In this study, any patient who deviated far enough from average recovery at any session of treatment as to warrant a yellow or red signal was predicted to have a final negative outcome. At the same time, this method misidentified 83 (18%) clients as likely to have a negative outcome when they did not. The outcome of these misidentified cases (false alarms/positives) was further studied and contrasted with the outcome of clients who were not identified as signal-alarm cases (predicted positive outcome). Of the 83 misclassified signal-alarm cases, 18% improved or recovered at termination, while 74% showed no reliable change.
In contrast, of the 373 cases that were not identified as signal-alarm cases, 50% recovered or improved and 50% showed no reliable change. These later findings offer further support for the signal-alarm method in that they suggest that even the false alarms have a poorer outcome than cases that are never identified as likely treatment failures. That is, if an alarm is given, the client has a one in five chance of having a positive outcome, compared to a 50/50 chance if no alarm is given by the algorithms. Unlike some medical decisions where the cost of over identification of signal cases may result in intrusive and even health threatening interventions, the signal-alarm in psychotherapy merely alerts the therapist to the possible need for reconsidering the value of ongoing treatment, rather than mandating specific changes. Since the signal-alarm alerts therapists to the possible need for action, rather than triggering a negative chain of events such as termination or referral, the current level of misidentification would seem to be tolerable.
Further analyses explored the difference between red and yellow warnings: What was the relative outcome for clients receiving a red versus a yellow signal? Outcome for these clients was classified into three categories: Reliably improved/recovered, no reliable change, and deteriorated. Of the 36 deteriorated cases, the empirical method’s red alarm picked up 34 of the 36 deteriorated cases, while the yellow signal picked up the remaining two deteriorated cases. The red alarm is indeed a more serious indicator for deterioration, one that should generate greater cause for concern to clinicians than the yellow signal.
As already noted, the ability of the algorithms to identify failing cases in the 550 clients studied by Hannan, et al. (2005) was less than 100%. In that study, clinicians made predictions based on their clinical judgment during a three week period and the algorithms were applied during the same three weeks rather than across the course of therapy. Even so, the algorithms far surpassed clinical judgment for identifying treatment failures.
Giving Feedback to Therapists
The next step that we took to reduce treatment failure was to conduct controlled experiments to test the consequences of supplementing clinical practice with feedback. There is little point in predicting an event unless this information can help to alter the course of the event. Feedback information from the decision rules was given to psychotherapists prior to each session of treatment along with a graph of patient progress and a written message.
The current method of providing feedback is displayed in Figure 1. This screen is from the OQ-Analyst computer software. It depicts the clinician feedback report on a fictional patient, John Doe, provided to the therapist prior to the 4th treatment session. The horizontal line at a score of 64 represents the cut-off for entering the ranks of normal functioning. The dark, slightly vertical line is the line of average recovery for patients with an initial score of 90 (John Doe’s intake score). The dark line with the dia-monds, that drops at the second and third session points and then goes up to a score of 117, is John Doe’s progress line. Within parenthesis at session two and three is a “G” indicating that at these points in time the client was on track but not recovered. At the session 4 point the alert status changed to red, indicating the patient is a signal-alarm case. Below the graph is the written feedback message for this case at this point in time, a message that suggests concern about eventual outcome for this case.
Five large-scale studies aimed at evaluating the effects of providing such research-based feedback on patient progress have been conducted in the United States (Harmon, et al. 2005; Hawkins, Lambert, Vermeersch, Slade, & Tuttle, 2004; Lambert, Whipple, Smart, et al., 2001; Lambert, Whipple, Vermeersch, et al., 2002; Whipple et al., 2003). Each of the studies required about one year of data collection and evaluated the effects of providing therapists with feedback about a patient’s improvement through such progress graphs and warnings about patients who were not demonstrating expected treatment responses (signal-alarm cases). Our primary question was: Does formal feedback to therapists about likely treatment failure improve psychotherapy outcomes? We hypothesized that: Patients identified as signal-alarm cases (those predicted to have a poor final treatment response) whose therapist received feedback would show better outcomes than similar patients whose therapist did not receive feedback.
The studies shared many things in common: 1) Each included consecutive cases seen in routine care regardless of patient diagnosis or co-morbid conditions (rather than being disorder specific); 2) random assignment of patient to experimental (feedback) and treatment as usual conditions (no feedback); 3) psychotherapists provided a variety of theoretically guided treatments with more clinicians adhering to cognitive behavioral and eclectic orientations followed by psychodynamic and experiential orientations; 4) the same therapists saw both experimental (feedback) and treatment as usual (no feedback) cases, thus limiting the likelihood that outcome differences between conditions could be due to therapist effects; 5) the length of therapy (dosage) was determined by patient and therapist rather than by research design or arbitrary insurance limits.
The results of giving feedback to therapists versus treatment as usual (no formal feedback) reached statistical significance in each study, had an effect size of about .40, and resulted in clinically meaningful changes as summarized in Table 1.
Given the large samples and replications of the individual studies in this summary, the current findings seem compelling. Providing feedback to therapists about patients who are failing to have a positive response to therapy has an important positive impact on patient well being, including cutting the rate of deterioration in half. We do not fully understand why feedback is so powerful. We believe that a fundamental reason is that the actuarial information provided by feedback is not available to the therapist through intuition. This supposition is supported by the fact that feedback indicating that a patient is having a positive treatment response (a view generally held by therapists) does not bolster outcomes for these on track cases. Evidence across studies suggests that therapists tend to keep signal-alarm cases in treatment for more sessions when they receive feedback, reinforcing the notion that feedback increases therapist interest and investment in a patient.
Since each of the studies that have been undertaken delivered feedback just as would be done in routine practice, we believe that this type of intervention is ready for implementation in routine clinical practice. Employing such a feedback system in routine care will hopefully become an essential aspect of clinical practice. We have also experimented with adding additional clinical sup-port tools, including a decision tree for problem solving with the failing case. In addition, three of the completed studies examined the effects of these added interventions. The practical difficulties of adding monitoring and feedback activities to busy practices can be an important barrier to implementation. Fortunately, recent developments in software programs make the possibility of instantaneous feedback to clinicians and clients easy to implement. A software program that is well suited to this task is the OQ-Analyst which incorporates the feedback system used in our research program. With the use of this software, patients take the tracking/outcome measure via a hand held computer. After completing the questions the device is placed in its sync or used with a wireless connection and feedback is instantaneously presented on the therapist’s computer prior to the patient’s scheduled appointment. A feedback report for discussion with the client can also be generated with the click of a button. Similar methods have been employed by other researchers within the United States (Brown, & Jones, 2005; Lueger, et al., 2001; Miller, et al., 2005) and abroad (Barkham, et al. 2001; Kordy, et al. 2001) although none have published research on the consequence of their feedback methods for enhancing patient outcomes. We are aware of only one managed health care organization (PacifiCare Behavioral Health) that routinely encourages therapists to monitor patient treatment response and provides them with feedback (Brown, et al. 2001).
The results of the research summarized in this review suggest the value of implementing monitoring of treatment response, applying statistical algorithms for identifying problematic response, and providing timely feedback to therapists as promising methods for enhancing patient outcome and improving the quality of care. Such interventions can be easily implemented by practitioners.
Michael J. Lambert, Ph.D. is Professor at the Department of Psychology at Brigham Young University.