we are looking at a series of pilot studies - which we know have very severe selection bias. However we know from the literature and experience what drives the selection bias. We have been asked to examine the outcomes and costs of rolling out the pilots to the total population. We therefore need to somehow move from the pilot to inferring something about the whole population.
I am coming to believe that if we gather a non-probability sample of the whole population which takes account of the selection bias we can estimate through PSA we can find the effects of the outcome and then we can weight these to reflect the total population. Is this right?
I should add the literature from around the world shows that this intervention is always and everywhere better however the literature is very quiet on the costs. So we need to quantify the outcome improvement and compare it to the cost (increase or decrease).
one other consideration - we know that some groups are very expensive under the old regime and remain so in the new regime - likewise a group who are relatively cheap. However we know of one group who are expensive in the old regime but have very variable costs in the new regime (very, very high and relatively low) in the new regime. We intend to over-sample this group.
As background - the pilots are looking at people with a disability in Ireland who live in institutions - Government policy Is to de-congregate over the next few years. Some pilots have been run and these have been very successful. But we are unclear what the transition costs are and how much the new system will need. The group who have variable outcomes are people with behaviour that challenges. Sometimes their challenging behaviour declines and sometimes they need a staff of 8 so there is a loss of economies of scale.
Using propensity scores and matching are different processes. Matching is choosing sample from the population that most closely align with the attributes relevant to your study. Propensity is running a logistic (or probit) model on your sample with a binary dependent variable of your variable of interest (the disabled yes/no). Then you use the the likelihood (propensity) scores as weights. See my paper on its application to real estate and economics in general if it helps.
Thanks Spenser - so it principle there is no objection to recruiting a matched sample? I was concerned that unidentified selection bias would be exacerbated.
I don't want to oversimplify a complex issue. First, you can use the highest propensity scores as a matching criteria, of course.
Second, I believe the question is more concerned around your sample. If your population sample is representative, then using a matching technique to more accurately measure the ceterus paribus effect of a treatment on a subset of the population more likely to be impacted by this treatment shouldn't cause issues. There is a large literature on this to support this technique.
If your initial sample is not representative of the population, there are other concerns.
Thanks again Spenser - my initial sample is not representative - but isn't that what PSA was designed to correct for? I am working my way through Guo and Fraser's book on PSA at the moment - could you recommend something else?
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis