I have been working in social policy research and evaluation a number of years and tend to use standard quasi-experimental evaluation designs when we are faced with this problem, which occurs frequently. I am really interested though in keeping up to date at looking at how others are dealing with it and whether there are any new or emerging robust evaluation designs which anyone might have used to deal with this.
Many thanks.
Mark
If you have data on those site prior to intervention, you can do a time series. You can also try to find control/comparison sites outside of your study sites.
here http://www.marketsforgood.org/an-alternative-to-rcts/ you can find author arguing that observational cohort study (OCS) is a good and not less valid alternative to RCT as experiments are in reality not that controlable.
regards
http://www.marketsforgood.org/an-alternative-to-rcts/
You could employ a simple two point time series design and use the correspondence rule of the variable of interest before and after the intervention. In this instance if (Time 2 -Time 1) / Time 1 = a 20% or greater difference, then it is deemed a statistically significant amount of change. Just ensure that you utilize a sensitive interval measure for your dependent variable.
What about a pre=post evaluation with a baseline for each group to see how the intervention affect each group, then make comparisons?
Thank you Mary-Helen and Ian for your contributions, I do appreciate it.
I am very interested in all of the approaches people have been kind enough to describe - does anyone have any citations of practice examples by any chance?
I'm keen on Pawson and Tilley's realistic evaluation approach as a theoretical framework for developing evaluations where controls aren't possible/appropriate. They use a theory based approach of developing mechanisms, contexts and outcome patterns to assess what intervention might work, for whom, and in what circumstances.
I've attached a link to the book, though Googling realistic evaluation will bring up a number of summary descriptions too.
http://www.uk.sagepub.com/books/Book205276
You can find some comparison sites and measure the same variables you are measuring in your intervention sites, but you need to establish the contextual factors including any similar programmes implemented in both the comparison and the intervention site. You can then assess the differences and follow up with a qualitative study to seek explanations for the observed differences and this is particularly necessary where you see the comparison sites performing much better than the intervention sites on the measured/desired outcomes. Maybe you can search for literature for this approach I am suggesting.
Most of what I'd like to suggest have been raised here, but I thought I'd offer something here based on my experience from my current project:
We're working on a 5 year implementation general practice project within a study community. Due to limited resources, we can't roll it out to all GP practices at once, but will recruit several at a time and roll it out over a few years. Our evaluation strategies include: (1) annual surveys over 4 years to show a change in practice within the study community (using the RE-AIM framework), (2) qualitative interviews to evaluate our implementation processes (using normalization process theory), and (3) social network analysis to show a change in professional network influenced by the implementation. We conducting a RCT but it's more like a implementation project because we're rolling out the intervention as subjects are being recruited, we're rolling out the intervention to the control group after they complete their follow-up period, and I think the concept of the above evaluation strategies have been used without RCT.
We're also referring to this paper:
Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact.
Curran GM1, Bauer M, Mittman B, Pyne JM, Stetler C.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3731143/
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3731143/
I don't know, but I am disappointed in the answers here. It seems with medicine especially, there is often a ceiling, a 6-sigma limit for example, on the known efficacy for all inteventions used to treat a given condition. Any single-study using only 1 intervention and no control group should be able to be measured against that standard, but I'm not aware of any conventional statistical math whereby that is normally done. Unfortunately when that does happen, and I've seen it happen, the results of the study are generally ignored when in fact they should be deemed as very substantial. It's a huge flaw in the way medical studies are often reviewed, published, and received.
yes, there is synthetic control method especially for this case. Here is an article explaining the same : https://medium.com/analytics-vidhya/synthetic-control-method-5c01f72da4e
i designed a multi-site evaluation of a program for teen mothers many may years ago-- and i remember why we could not use, actually why a control group cannot be used in social policy research and evaluation. it is unethical and no IRB would ever approve a study whereby one group of people is denied an intervention. in other words, a group would have to be selected for withholding an intervention. has this changed??
On one study, we were able to do it because all clients were receiving an intervention-- so the control group became, "intervention as usual," while the "experimental" group/treatment group would receive the intervention under study.
- Badges
- Science topic
- Similar topics
- Social Science
- Sociology