Last week we spoke about Evidence Based Practices (EBP) and how their use has helped create more effective interventions. However, we also mentioned that EBP are difficult to implement. We spoke about how part of the problem is that they can be costly and can go against what most people in the field are used to doing in their practice. This time, I want to explain why most times, these interventions are costly and difficult to move into real-world practice, not only because they may go against what the field is used to do, but also for some other practical reasons.
EBP are usually tested under very rigorous conditions: The most stringent criteria for calling something an Evidence-Based Practice requires the use of a randomized control trial approach. That means that participating individuals may be assigned to one of two (or maybe more) groups: One that receives the treatment or one that receives nothing. Now the justification for doing something like this is because we want to be able to demonstrate that the reason we see change after the treatment, is due to the treatment and not other reason (for example: just the passing of time or in some cases, due to some developmental reasons, when developmental changes make sense). Now, even in those conditions, there may be potential confounding variables that may affect the final outcome.
One of the biggest problems facing many treatments is the fact that many times, individuals show improvement just because they are told (or they believe) that they are receiving some wonder-therapy (or drug). This is so prevalent in clinical trials that people speak about the “
placebo effect”. Therefore, a way to control for the potential effect of placebos is to include a treatment condition which is a placebo (when testing medications, people speak about “sugar pills”) or what may be considered the “normal treatment” (which sometimes is labeled as “business as usual”), where those who did not go into the treatment being tested are receiving the treatment that they might have received had there not been this treatment under testing. Placebo is a very powerful effect, and most of the therapies that sometimes are advertised on TV may work, because of this effect (quiz: how many times have you seen in those late TV ads a comparison group? Or comparisons against a placebo control?).
There are multiple ways to try to prove that a specific intervention is working, but as explained, most people tend to agree that the best approach is to use what is known as the “gold standard” or random assignment to different clinical conditions. The reason random assignment is considered the “gold standard” is that for the most part, it balances out many variables that could potentially affect the outcomes in unexpected ways. Things like age, gender, ethnicity, length of time with the illness, type of treatments received in the past and so forth. How will random assignment control for all of that? Because every individual with any potential combination of these variables has the chance of being assigned to one of the treatments in the study. Therefore, it is expected that individuals with many if not all the potential combinations that may affect the final outcomes end up in one of the groups in the study, and therefore the effect of all those variables cancels out.
All this dancing is so scientists and the public in general can make informed decisions about the effectiveness of a treatment (i.e., are my outcomes better when I use treatment “A” as opposed to treatment “B”), as well as being able to generalize to a larger group of people than those included in the study. After all, if you were not included in the study, what good will it do to you to know that a program may work if you are not sure that the treatment will work in people like you?
Doing this work means time and money. People involved in testing the treatment needs to conduct multiple studies so they can get some assurance that the results are sound and can withstand multiple tests, under different conditions. They also need to be closely monitored so researchers can be alert if something is not going well. If the new treatment under scrutiny has the potential for being harmful, then they may want to stop the study before too long. On the other hand, if the results are going very well, perhaps it is time to stop the study with confidence that the new treatment will work as expected (though when treating human lives, you don’t want to take any chances).
There are multiple institutions that have created databases where evidence for or against Evidence Based Practices can be found.
The Substance Abuse and Mental Health Services Administration (SAMHSA) maintains a website with links to several organizations where such information can be found.
Creating and documenting the effectiveness of a specific intervention is not enough. In a country as diverse as the U.S., there are many instances where an intervention that has been proven to work for a specific group of people (say African American), may not necessarily work for another ethnic group (e.g., Latino). The reasons can be associated with genetic makeup as well as with ethnic background (customs and traditions, for example, can be a big impulse or deterrent for some interventions). Therefore sometimes interventions that have been proven to work in an ethnic group (or in a research setting) need to be tested under different conditions (e.g., a different ethnic group or on a community-based environment). This is no easy task, which once more affects how quickly an intervention can be used outside the testing grounds.
This is a very active area of research which is known as validity. People speak about internal or external validity, and if you ever took a “research methods” class in college, then you may recognize many of these ideas or even terms. One book that describes the rationale an many specific examples is
Shadish, Cook and Campbell. However, be warned that this book can be hard to read without some introduction to research methods
One final note: Evidence Based Practices are the top of the pyramid, but there are some interventions/programs that have not been able to prove their worth using the most restrictive criteria (the gold standard) and yet are considered worth more research.
A ‘Promising model/practice’ is defined as “one with at least preliminary evidence of effectiveness in small-scale interventions or for which there is potential for generating data that will be useful for making decisions about taking the intervention to scale and generalizing the results to diverse populations and settings.”
Department of Health and Human Services Administration for Children and Families Program Announcement. Federal Register, Vol. 68, No. 131, (July 2003), p. 40974. These are interventions where some initial testing has been done, and the outcomes observed so far seem to indicate that the intervention may be effective. However, more and more strict testing is needed to endorse it as an EBP.
Emerging practices, on the other hand, are “practices that have very specific approaches to problems or ways of working with particular people that receive high marks from consumers and/or clinicians but which are too new or used by too few practitioners to have received general, much less scientific attention.” We took this definition from the
Oakland County Community mental health authority. In this case, it is argued just like in the case of the promising practices, that the intervention being described has produced effective outcomes, but much more testing is still necessary.
