The Tyranny of Metrics

For all this talk about the importance of measuring outcomes, we must resist the other side, the tyranny of metrics that can undermine the dignity of our human experience.

A lack of appreciation and a mishandling of metrics can have unintended consequences.

Here are some examples of how different sectors “game” the numbers:

• Surgeons choosing not to operate on people with co-morbidity for fear of making their clinical outcomes look bad;
  
  
• Academics are incentivised by the number of citations of their work. You can game the system by asking/paying people to cite your work, or worse, you get cited because your study is referenced as negative or lousy study. Likewise, measuring the number of publications can be gamed. Instead of publishing one big study, split it into a few publications;
  
  
• The police can arrest 1 man who has committed 100 crime and counting it as 1 arrest, or counting it as a 100 cases of crime solved;
  
  
• Schools can create easier exams that increase the pass rates.

Returning to therapy-land, here are some pitfalls to watch out for when you employ outcome measurement in your therapeutic work:

1. Do not simply value whatever you are asked to measure; measure what is of value to your clients.
   
   
2. Be wary of proceeding without dialogue when some desk jockey without functional knowledge of the field, insists on a battery of measures that you need to employ in your clinical work. Begin by creating a pilot team to discuss on point #1, and open up the dialogue top-down and buttom-up.
   
   
3. Resist the objectivistic approach of turning outcome measures to an assessment tool. Instead, learn to use measures as a conversational tool. Use numbers and then “leave” numbers.
   
   
4. Employing routine outcome monitoring is not necessarily a form of feedback. True feedback must feed-forward the information both to the therapist and the client in real-time. Make the data visible for both parties; what’s visible becomes visceral.
   
   
5. Researchers put the different kinds of outcome measures on the following 2 spectrum: Reactivity and Specificity.
   A measure that is considered high in reactivity is rated by an “expert”, whereas a measure that is low in reactivity is rated by the client. A measure that is considered high in specificity is typically a symptom-based measure, whereas a measure that is low in specificity is one that is a measure of global wellbeing.[1]
   Take the time to decide what measures to employ based these 2 spectrum of reactivity and specificity in your organisation.[2]
   Remember to consult point #1.
   
   
6. The moment you start to punish and reward people for their performance based on some kind of outcome indicator, you’ve now caused a new problem of (see the examples above). And you’ve also inadvertently destroyed intrinsic motivation. Resist any behavioristic approach here.
   
   
7. Divorcing client’s lived experience and the use of measures is one way to lead clinicians to burnout and leave clients feeling judged.
   
   
8. Simply tracking client’s pre/post outcome scores is an example that outcome measurement is more for the bean counters and less so for the client. Use measures to guide your work session-by-session, not just to evaluate your work.
   
   
9. Outsourcing decision making purely to outcome metrics indicates that sound decision making has left the building. Instead, our clinical data must combine with our clinical intuition to help us make better decisions.
   
   
10. When we fail to use localised aggregated data to meaningfully help us get better at what we do, we now have created data exhaust. (For more on how to go beyond using measure and to achieve better results, check out our latest book, Better Results, co-written with Scott Miller and Mark Hubble).
    
    
11. An overemphasis on performance can actually impede deep learning (listen to this Frontiers Radio podcast episode #5).
    
    
12. Finally, we think that simply providing statistics of our outcomes to funders is the way to create “accountability.” Here we run the risk of creating a “data monologue.” Data can only come to life by the way humans help shape the story (by this I do not mean fudging the data). In other words, we need to learn how to tell better stories with our data, especially to our funders. Because facts never speak for themselves.
    As we become more accountable, we need to disarm our seduction to measuring “competence,” and emphasise more on measuring growth over time.
    

When in doubt about the tyranny of metrics, do not add more metrics to your work. Strip it down and figure out what is of true value, because the measure must match the mission.

Whose mission? Start with the client.

Footnotes:
[1] Read Tak Minami and colleagues paper for more details about Reactivity and Sensitivity.
Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S. (2007). Benchmarks for Psychotherapy Efficacy in Adult Major Depression. Journal of Consulting & Clinical Psychology April, 75(2), 232-243.
[2] Here’s a hint from Minami and colleagues (2007) study:
High- reactivity/high-specificity measures produced the largest Effect sizes (ES; a measure of the magnitude of change)
Low-reactivity/high-specificity measures produced intermediate ESs.
Low-reactivity/low-specificity measures produced the smallest ESs

Take a conservative view here. See Point #1 above.