What is the evidence for evidence-based treatments? A hard look at our soft underbelly

Peter S. Jensen, Robin Weersing, Kimberly Eaton Hoagwood, Eliot Goldman

Research output: Contribution to journalArticle

87 Citations (Scopus)

Abstract

In the rising quest for evidence-based interventions, recent research often does not give adequate attention to "nonspecific therapeutic factors," including the effects of attention, positive regard, and therapeutic alliance, as well as the effects of treatment dose, intensity and actual processes mediating therapeutic change. To determine the extent to which recent clinical trial designs fully this problem, the authors conducted a systematic review of PsychLit/Medline of all controlled child psychotherapy treatment studies from 1995 to 2004. A total of 52 studies were identified that met review criteria: Two or more therapy conditions and random assignment of participants to intervention groups. Of the 52 studies, one group (n = 29) compared a presumably active treatment with 1 or more similarly intensive treatments (often an attention control group) presumably not containing the active therapeutic ingredients. Of these, 14 studies found evidence of consistent differences between the two groups, whereas 15 did not. An additional group of studies (n = 27) compared therapy groups with different levels of intensity and "dose" of the putatively active treatment; 13 of these found evidence of the effects of different levels of treatment dose/intensity on outcomes and 14 did not. Four studies met criteria for inclusion in both groups. Across both groups of studies, when positive effects were found, few studies systematically explored whether the presumed active therapeutic ingredients actually accounted for the degree of change, nor did they often address plausible alternative explanations, such as nonspecific therapeutic factors of positive expectancies, therapeutic alliance, or attention. Findings suggest that many child psychotherapy treatment studies have not inadequately controlled for nonspecific factors such as attention and treatment intensity and have failed to assess specific mediators of change. Specific recommendations for future studies are offered, specifically: 1. Initial specification in study design how investigators will test if the intervention is efficacious over and above the effects of positive expectancies, positive regard, or attention; 2. Planned (rather than post hoc) analyses to explore whether and how a given treatment's specific ingredients or the overall intensity of its "dose" are related to treatment outcomes; 3. More studies cond ucting head-to-head tests of different types but equally credible forms of treatment, with planned analyses testing different mechanisms of change 4. When positive effects of a treatment vs. a control are found, systematic elimination of specific, competing hypotheses of reasons for treatment efficacy (e.g., attention, therapeutic alliance, face validity of treatment and client's treatment expectations, changes in self-efficacy), and 5. When negative effects are reported, appropriate analyses to address alternative explanations (lack of power, floor or ceiling effects, mediator analyses to address possible fidelity or adherence problems, therapist effects, absence of main effects but possible subgroup/moderator effects, etc.).

Original languageEnglish (US)
Pages (from-to)53-74
Number of pages22
JournalMental Health Services Research
Volume7
Issue number1
DOIs
StatePublished - Mar 2005
Externally publishedYes

Fingerprint

Therapeutics
Psychotherapy
Self Efficacy
Group Psychotherapy
Reproducibility of Results
Research Personnel
Clinical Trials
Control Groups
Research

Keywords

  • Attention controls
  • Controlled clinical trials
  • Efficacy
  • Psychotherapy

ASJC Scopus subject areas

  • Health Policy

Cite this

What is the evidence for evidence-based treatments? A hard look at our soft underbelly. / Jensen, Peter S.; Weersing, Robin; Hoagwood, Kimberly Eaton; Goldman, Eliot.

In: Mental Health Services Research, Vol. 7, No. 1, 03.2005, p. 53-74.

Research output: Contribution to journalArticle

Jensen, Peter S. ; Weersing, Robin ; Hoagwood, Kimberly Eaton ; Goldman, Eliot. / What is the evidence for evidence-based treatments? A hard look at our soft underbelly. In: Mental Health Services Research. 2005 ; Vol. 7, No. 1. pp. 53-74.
@article{cbb3feacf58541b892b092b47c9c0a46,
title = "What is the evidence for evidence-based treatments? A hard look at our soft underbelly",
abstract = "In the rising quest for evidence-based interventions, recent research often does not give adequate attention to {"}nonspecific therapeutic factors,{"} including the effects of attention, positive regard, and therapeutic alliance, as well as the effects of treatment dose, intensity and actual processes mediating therapeutic change. To determine the extent to which recent clinical trial designs fully this problem, the authors conducted a systematic review of PsychLit/Medline of all controlled child psychotherapy treatment studies from 1995 to 2004. A total of 52 studies were identified that met review criteria: Two or more therapy conditions and random assignment of participants to intervention groups. Of the 52 studies, one group (n = 29) compared a presumably active treatment with 1 or more similarly intensive treatments (often an attention control group) presumably not containing the active therapeutic ingredients. Of these, 14 studies found evidence of consistent differences between the two groups, whereas 15 did not. An additional group of studies (n = 27) compared therapy groups with different levels of intensity and {"}dose{"} of the putatively active treatment; 13 of these found evidence of the effects of different levels of treatment dose/intensity on outcomes and 14 did not. Four studies met criteria for inclusion in both groups. Across both groups of studies, when positive effects were found, few studies systematically explored whether the presumed active therapeutic ingredients actually accounted for the degree of change, nor did they often address plausible alternative explanations, such as nonspecific therapeutic factors of positive expectancies, therapeutic alliance, or attention. Findings suggest that many child psychotherapy treatment studies have not inadequately controlled for nonspecific factors such as attention and treatment intensity and have failed to assess specific mediators of change. Specific recommendations for future studies are offered, specifically: 1. Initial specification in study design how investigators will test if the intervention is efficacious over and above the effects of positive expectancies, positive regard, or attention; 2. Planned (rather than post hoc) analyses to explore whether and how a given treatment's specific ingredients or the overall intensity of its {"}dose{"} are related to treatment outcomes; 3. More studies cond ucting head-to-head tests of different types but equally credible forms of treatment, with planned analyses testing different mechanisms of change 4. When positive effects of a treatment vs. a control are found, systematic elimination of specific, competing hypotheses of reasons for treatment efficacy (e.g., attention, therapeutic alliance, face validity of treatment and client's treatment expectations, changes in self-efficacy), and 5. When negative effects are reported, appropriate analyses to address alternative explanations (lack of power, floor or ceiling effects, mediator analyses to address possible fidelity or adherence problems, therapist effects, absence of main effects but possible subgroup/moderator effects, etc.).",
keywords = "Attention controls, Controlled clinical trials, Efficacy, Psychotherapy",
author = "Jensen, {Peter S.} and Robin Weersing and Hoagwood, {Kimberly Eaton} and Eliot Goldman",
year = "2005",
month = "3",
doi = "10.1007/s11020-005-1965-3",
language = "English (US)",
volume = "7",
pages = "53--74",
journal = "Mental Health Services Research",
issn = "1522-3434",
publisher = "Springer GmbH & Co, Auslieferungs-Gesellschaf",
number = "1",

}

TY - JOUR

T1 - What is the evidence for evidence-based treatments? A hard look at our soft underbelly

AU - Jensen, Peter S.

AU - Weersing, Robin

AU - Hoagwood, Kimberly Eaton

AU - Goldman, Eliot

PY - 2005/3

Y1 - 2005/3

N2 - In the rising quest for evidence-based interventions, recent research often does not give adequate attention to "nonspecific therapeutic factors," including the effects of attention, positive regard, and therapeutic alliance, as well as the effects of treatment dose, intensity and actual processes mediating therapeutic change. To determine the extent to which recent clinical trial designs fully this problem, the authors conducted a systematic review of PsychLit/Medline of all controlled child psychotherapy treatment studies from 1995 to 2004. A total of 52 studies were identified that met review criteria: Two or more therapy conditions and random assignment of participants to intervention groups. Of the 52 studies, one group (n = 29) compared a presumably active treatment with 1 or more similarly intensive treatments (often an attention control group) presumably not containing the active therapeutic ingredients. Of these, 14 studies found evidence of consistent differences between the two groups, whereas 15 did not. An additional group of studies (n = 27) compared therapy groups with different levels of intensity and "dose" of the putatively active treatment; 13 of these found evidence of the effects of different levels of treatment dose/intensity on outcomes and 14 did not. Four studies met criteria for inclusion in both groups. Across both groups of studies, when positive effects were found, few studies systematically explored whether the presumed active therapeutic ingredients actually accounted for the degree of change, nor did they often address plausible alternative explanations, such as nonspecific therapeutic factors of positive expectancies, therapeutic alliance, or attention. Findings suggest that many child psychotherapy treatment studies have not inadequately controlled for nonspecific factors such as attention and treatment intensity and have failed to assess specific mediators of change. Specific recommendations for future studies are offered, specifically: 1. Initial specification in study design how investigators will test if the intervention is efficacious over and above the effects of positive expectancies, positive regard, or attention; 2. Planned (rather than post hoc) analyses to explore whether and how a given treatment's specific ingredients or the overall intensity of its "dose" are related to treatment outcomes; 3. More studies cond ucting head-to-head tests of different types but equally credible forms of treatment, with planned analyses testing different mechanisms of change 4. When positive effects of a treatment vs. a control are found, systematic elimination of specific, competing hypotheses of reasons for treatment efficacy (e.g., attention, therapeutic alliance, face validity of treatment and client's treatment expectations, changes in self-efficacy), and 5. When negative effects are reported, appropriate analyses to address alternative explanations (lack of power, floor or ceiling effects, mediator analyses to address possible fidelity or adherence problems, therapist effects, absence of main effects but possible subgroup/moderator effects, etc.).

AB - In the rising quest for evidence-based interventions, recent research often does not give adequate attention to "nonspecific therapeutic factors," including the effects of attention, positive regard, and therapeutic alliance, as well as the effects of treatment dose, intensity and actual processes mediating therapeutic change. To determine the extent to which recent clinical trial designs fully this problem, the authors conducted a systematic review of PsychLit/Medline of all controlled child psychotherapy treatment studies from 1995 to 2004. A total of 52 studies were identified that met review criteria: Two or more therapy conditions and random assignment of participants to intervention groups. Of the 52 studies, one group (n = 29) compared a presumably active treatment with 1 or more similarly intensive treatments (often an attention control group) presumably not containing the active therapeutic ingredients. Of these, 14 studies found evidence of consistent differences between the two groups, whereas 15 did not. An additional group of studies (n = 27) compared therapy groups with different levels of intensity and "dose" of the putatively active treatment; 13 of these found evidence of the effects of different levels of treatment dose/intensity on outcomes and 14 did not. Four studies met criteria for inclusion in both groups. Across both groups of studies, when positive effects were found, few studies systematically explored whether the presumed active therapeutic ingredients actually accounted for the degree of change, nor did they often address plausible alternative explanations, such as nonspecific therapeutic factors of positive expectancies, therapeutic alliance, or attention. Findings suggest that many child psychotherapy treatment studies have not inadequately controlled for nonspecific factors such as attention and treatment intensity and have failed to assess specific mediators of change. Specific recommendations for future studies are offered, specifically: 1. Initial specification in study design how investigators will test if the intervention is efficacious over and above the effects of positive expectancies, positive regard, or attention; 2. Planned (rather than post hoc) analyses to explore whether and how a given treatment's specific ingredients or the overall intensity of its "dose" are related to treatment outcomes; 3. More studies cond ucting head-to-head tests of different types but equally credible forms of treatment, with planned analyses testing different mechanisms of change 4. When positive effects of a treatment vs. a control are found, systematic elimination of specific, competing hypotheses of reasons for treatment efficacy (e.g., attention, therapeutic alliance, face validity of treatment and client's treatment expectations, changes in self-efficacy), and 5. When negative effects are reported, appropriate analyses to address alternative explanations (lack of power, floor or ceiling effects, mediator analyses to address possible fidelity or adherence problems, therapist effects, absence of main effects but possible subgroup/moderator effects, etc.).

KW - Attention controls

KW - Controlled clinical trials

KW - Efficacy

KW - Psychotherapy

UR - http://www.scopus.com/inward/record.url?scp=14944339656&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=14944339656&partnerID=8YFLogxK

U2 - 10.1007/s11020-005-1965-3

DO - 10.1007/s11020-005-1965-3

M3 - Article

C2 - 15832693

AN - SCOPUS:14944339656

VL - 7

SP - 53

EP - 74

JO - Mental Health Services Research

JF - Mental Health Services Research

SN - 1522-3434

IS - 1

ER -