In the previous post, we looked at the latest meta-analysis on the effects of feedback tools. 

In this article, we will look at some of the critique and other studies that came after. 

Get ready, as this is a deep dive into three studies. 

I. Danish Study on the Effectiveness and Moderators of Feedback Informed Treatment

In 2018, Ole Østergård did his doctoral work in Denmark based on three publications examining the effectiveness and moderators of FIT.  

See this ​PDF​ for his dissertation. 

One of his paper (Study 2), is a meta-analysis on the Partners forChange Outcome Management System (PCOMS) feedback tool, Key Graf: 

• Overall PCOMS Effect: The meta-analysis, including 18 studies, found a small overall effect of using PCOMS on general symptoms (Hedge’s g = 0.27)
• Different Setting as a Moderator: PCOMS showed no significant effect in psychiatric settings (g = 0.10) across 10 studies. In contrast, a positive effect was observed in counseling settings (g = 0.45) across eight studies, though this was deemed potentially biased due to researcher allegiance and the exclusive use of ORS as the outcome measure
• No Effect on Deterioration or Dropout: No significant effect of PCOMS was found on the number of deteriorated clients or dropout rates

Differences in Meta-Analyses Between Østergård’s 2018 and de Jong’s 2021 

If you read last week’s email, you might be wondering how is it that Østergård et al. (2018) arrived at a somewhat different conclusion than Kim de Jong and colleagues meta-analysis (2021).

For the curious minded, with the help of NotebookLM and ChatGPT (strangely some hallucinations from ChatGPT), here’s a comparison table between Østergård’s 2018 and de Jong’s 2021 meta-analyses:

Østergård’s main conclusion in his dissertation questions the reliability of the reported positive effects of PCOMS, citing reasons sure as being ineffective in psychiatric settings and inflation of effect sizes due to allegiance bias and the ORS measurement sensitivity. 

As I’ve been using outcome measures in my clinical practice since 2004, Østergård’s findings piqued my interest. 

In part, around 2008-2009, my colleagues and I did a small naturalistic year-long study in Singapore, trying to understanding the impact of using FIT in an outpatient psychiatric setting. 

In the Treatment-As-Usual (TAU) control group, therapists saw a group of clients for six months. We had research assistants, aka unpaid student labour, administer the Outcome Rating Scale (ORS), and Session Rating Scale (SRS) before and after each appointment. 

In the Feedback Group, with the same group of therapists (i.e., serving as their own controls), but with a new batch of clients, they now tracked their outcomes and alliance, session-by-session. (This study design differed from Østergård’s Paper 3, in case you are interested). 

All we did was to encourage therapists to have an open-dialogue with each of their clients about their progress. 

Here’s what we found:

To be clear, this was just a poster presentation. 

But the results were striking enough to give me pause. 

Could this be a demand characteristic, that clients just scored higher when asked by therapists instead of RAs lurking outside clinic rooms? 

Maybe. 

But take a look at the results for the working alliance. 

One thing that is consistent in our field, is the importance of the working alliance impacting outcome. More crucially, evidence suggest that it’s not just good alliance that influences outcome, but alliance that improves over time that has a significant impact on effectiveness of treatment. 

Here’s the comparison of working alliance between the control and feedback group:

Note: The alliance had no significant difference at the first session. But engagement improved over time with the feedback group, and not so in the control group. 

I reckon that giving therapists and clients an explicit opportunity to discuss about the alliance helps them to recalibrate, thereby increasing the odds of improvement. 

Still, it would be wrong to conclude that “FIT works.” 

If we began with the question “Does FIT work?” we are likely to be misled, even if the results are positive. 

No one expects a stopwatch to make you run faster. We wouldn’t be asking the question, “Does the stopwatch work?” 

A stopwatch doesn’t make Usain Bolt.

Any tool or technology is only as good as the therapist who uses it. 

The tool doesn’t effect change; the person does. 

This is why one shouldn’t expect using measures in therapy to “work,” especially when it is implemented from the top-down, where therapists are mandated to do so without proper implementation and guidance.

And what if our attitudes towards feedback are less than open? Does that affect the outcome?

II. Norwegian Study

In the same year as Østergård’s publication, their Norwegian counterparts ​published a RCT in a hospital-based mental health clinic​, examining the effects of outcome monitoring on therapy outcomes during an implementation process. This is somewhat a similar setting to the placed I worked in and conducted the research in Singapore. 

In 2018, Heidi Brattland and colleagues found that the feedback group were 2.5 times more likely to achieve improvement, in their symptoms and functioning, as measured by the Behavior and Symptoms Identification Scale (BASIS-32), than those in the TAU condition, compared to the no-feedback group. 

Of note, the superiority of ROM over TAU increased significantly over the duration of the four-year study. 

Brattland and colleagues suggested that this progressive increase in effectiveness might be due to PCOMS being used more effectively over time, potentially as a result of sustained implementation efforts, including regular training and supervision for therapists. 

Unlike Østergård et al.’s Paper 3, this study was highlighted as the first to demonstrate PCOMS effects using an independent main outcome measure (BASIS-32), rather than solely relying on the PCOMS’s own Outcome Rating Scale (ORS).

Brattland and team was also the first clinical trial to show PCOMS benefit in a psychiatric setting. This speaks more about the implementation process than the use of measures itself. 

By the way, in case you are wondering, it typically takes 4-5 years for an agency to fully implement these ideas. So when an organisation thinks they can just roll this out in six months, simply but demanding their therapists to use measures, I say to them, think again. 

Read the ​full paper​ to get a feel or what’s involved. 

III. Vermont Study

Finally, here’s a recent 2025 study worth highlighting. 

Joanna Drinane and team from Vermont, US published a study looking at the attitudes among therapists who do and do not implement Feedback Informed Treatment (FIT). 

The title is:

​Attitudes among Therapists who Do (or Do Not) Implement Feedback-Informed Treatment​

(click ​here​ to read the full paper)

Despite efforts to facilitate therapist engagement, including training, supplemental group supervision, consultation, and additional compensation, two distinct groups emerged: current FIT users (n=30) and past FIT users (n=19)

Here’s what they found: 

• Perceived Validity: Current FIT users perceived the FIT data as significantly more valid than past users.
• Perceived Utility: Therapists who continued using FIT perceived the measures as more useful to themselves and their clients. This indicates that finding the measures helpful for understanding client progress is a significant factor in sustained use.
• Antagonism towards Predictive Data: Past FIT users reported significantly more antagonism towards actuarial predictions compared to current users.
• Openness to Outside Information: Current FIT users reported significantly greater openness to considering outside information to support their clinical judgments. This suggests a willingness to question their own clinical sense and integrate external data.
• Validity of Intuition: Clinicians who stopped using FIT reported a greater trust in their ability to track clinical progress and outcomes based on their intuition without the need for FIT data. 

You might recall I’ve mentioned this three emails ago: 

We shouldn’t outsource decision-making entirely to data. And neither should we reply solely on our own intuition.

We need both in order to make better decisions. 

We have seen in two of our studies that we are highly inaccurate in of our own effectiveness. (see the studies ​here​ and ​here​). 

Conclusion

There is quite a lot to chew on with the three studies referenced above. 

My hope for you is to appreciate that it’s not about the measures per se, but what you do with it that makes the difference. 

Not only does implementation science matter, your attitude towards the use of measures to change you and your decision-making impacts as well. 

If anything, the greatest playbook of FIT is that is a “pause-book.” 

It should at least give you pause.