Thanks Erica.. You are right.. The design will be with these limitations. Since the current research problem is for my Ph.D. Research and as my Research is not sponsored and since most of the variables are uncontrollable, and also it becomes very difficult rather impossible to find a comparison group with similar characteristics as that of control group. Please read my whole research problem in detail in this comment later so as to offer your valuable inputs.

Milindo: I am explaining my actual research problem in a bit more detail below. Please go through it and offer your valuable inputs regarding the Research design and the other Research Techniques.

Herman Ader: Thanks for your explanatory input. Thanks for making your comment available inspite of some technical problems with posting. I would of course like to know more about this design/methodology. Please go through my research Problem as mentioned below to offer your valuable and explanatory notes/suggestions on that.

Dear All, I think the discussion has reached to a point where it becomes quite necessary for me to explain my research problem in more detail. My actual research objective is to analyse the Impact of various Financial Inclusion Initiatives on the beneficiaries. The Impact will constitute of the changes in socio-economic variables of the beneficiaries. So far as the Research design is concerned, I have 2 alternatives.

1) To collect data from the Program beneficiaries who have been with the program for quite a time, asking them about their Socio-economic conditions before joining the program and after they have been with the program. In other words, to understand the impact of Program on their Socio-economic condition using Pre/Post methodology. Since this design is not without limitations but due to time and financial constraints I am considering this Research Design.

2) To get the responses from program beneficiaries who have been with the program for quite some time and take them as Control Group. and to take responses from the new-comers to the program and take them as a Comparison group. Both the groups would be asked to respond to a same set of queries and after the collection of responses from both groups, any statistical techniques can be used to assess the impact like Annova, CAGR etc.

Both the alternatives are with their inbuilt limitations, but I believe when talking about Applied Economics, and where almost all variables are uncontrollable, there can be no perfection. There will be a set of limitations in both the cases but that will be mentioned while discussing results of the Research.

Now it is time for you all to kindly enlighten in detail as to which methodology can be used. I mean please help me out at the alternatives I am facing and also suggest techniques and tools that can be used in either case.

Thanks,

Khaki Audil

In terms of specific techniques it would depend entirely on the data collected. Many statistical techniques cater for pre-post testing analysis but can be divided into parametric and non-parametric. An alternative method might be to use a multiple wave data collection where you collect data at several points prior to intervention as a repeated baseline, pre- and post intervention and then at several points post intervention. This can be useful to determine potential intervention effects if you only have one sample.

It depends on your kind and number of variables, and also it depends on the sample size and other statistical assumptions.

If you have two conditions (program / no program, I mean, experimental and control groups), and several program sessions and you assess the same variables with quantitative measures you can compute a two-way mixed ANOVA. It's the best way to assess the impact.

But, as Erica Bowen says, it would depend entirely on the data collected. If you have problems with sample size and/or statistical assumptions, you have to proceed with non-parametric analysis.

It may be useful to start with a pilot study.

I hope this helps.

Good luck with your research!

Your question actually pertains to study design issues and not just statistical techniques.

I agree with Jan about the need for the control group. Although pre-intervention period is one kind of a baseline control, it alone is not enough to make a strong case (because temporal changes could easily be the reason for the observed outcomes). A stronger design would be to include, as Jan says, another group similar in all respects to your studied group but differing only with respect to access to program under study. The idea is that control group's outcomes would also be influenced by temporal changes but would not be influenced by the program.

I recently had a study published where pre-/post-rates alone did not show a difference and if I had stopped at that I would have concluded that there was no change in outcome under study. But because I compared with another city that presumably was similar to first city in several respects but differed from second city only with respect to program under study (legislation, in our case), I was able to show a modest change. The statistical approach we used was generalized estimating equations model.

Some program evaluations' literature suggests that you can further improve your design by studying secondary outcomes. In my case, I studied a secondary unrelated outcome that I did not expect to be influenced by the program at all. The underlying idea was that both cities would have similar outcomes because program was not expected to influence these secondary outcomes. My results helped me further modestly strengthen my conclusions about the program's effectiveness.

Finally, it is OK not to have pre-program measures collected earlier on--collecting them later via self-declared means will subject them to recall bias limitations, but this is why it is crucial you employ a control group as well as a secondary (related or unrelated) outcomes' approach.

Hope this helps.

This is my recent paper. I have a couple of good program evaluation references in there. http://injuryprevention.bmj.com/content/early/2012/06/29/injuryprev-2012-040389.full

Thanks Jan, Erica Bowen, Malathi Raghavan, Merche Ovejero for your valuable inputs regarding my problem. I am infact, working in the field of applied economics, wherein I am rather trying to study the impact of Financial Inclusion Program on its beneficiaries. Since the objective of the study is to assess the socio-economic impact of the program on beneficiaries, it is almost impossible rather perilous to use control group - comparison group design. The variables of the study are economic variables and are uncontrollable. Also due to time and Financial constraints, such kind of a design would not be effective to me. I was also of the opinion to use the same control group - comparison group design but due to other difficulties, my supervisor suggested a Pre-Post Methodology.

Erica, due to the limitations mentioned above it is also not possible to get the data at different intervals on a continuous basis.

I request all of you to kindly suggest any modifications that are necessary for this Pre-Post design to be more effective.

I think given the feasibility issues you face a pre-post test design may be the only option, but you will need to be very clear about the limitations of such a design - i.e. that it will not be possible to attribute any impact to the actual intervention due to lack of comparison/control group. This is particularly problematic if the focus is on impact only. Is there scope to look at other issues, such as the differential impact on subgroups of the population? Is the purpose for a qualification - you mention supervisor? There are several different perspectives on evaluation which may be worth looking into. As a psychologist I tend to be interested in individual differences and process as well as impact simply because impact is so hard to objectively determine with the populations I work with. I use Rossi et al's Comprehensive evaluation perspective for example. I would suggest having a read through back issues of Evaluation/American Journal of Evaluation/Evaluation review to see if there is a particular perspective which may be helpful to you.

Dear All: I think one can use the propensity score matching technique in this situation. I shall come back with a somewhat detailed exposition on the technique if you are interested.

This is the third (and last) time I enter my comment.

Previous comments disappeared for some reason.

My comment includes the use of propensity scores, mentioned by Milindo.

First, any conclusions drawn from the study you describe only apply for beneficiaries of the program.

Below, I assume that you have primary data (i.e. data from individual beneficiaries). If not, you should do a qualitative study which is outside my field.

If you have primary data you could proceed as follows:

1. Determine what you consider is a relevant pre-post difference

of the variabele(s) you use to assess impact.

2. Define two subgroups: one including persons who did benefit from the program and one with persons who did not benefit.

3. Calculate propensity scores (to correct for selection bias in the

assignment suggested in 2.) using pre-measurement variabels not related to SES or to the dependent variables used to measure impact.

4. Use logistic regression analysis in which the subgroup indicator is the dependent and the propensity score and the SES variables are the predictors (this step can be replaced by a more sophisticated technique like structural equation modelling or GEE, suggested earlier.

In the above procedure you have taken care of (some of the) selection bias that can occur in 2., which makes it plausible to assume that differences between the two groups can be ascribed to SES.

If you need more information, let me know.

Herman Adèr

Thanks Erica.. You are right.. The design will be with these limitations. Since the current research problem is for my Ph.D. Research and as my Research is not sponsored and since most of the variables are uncontrollable, and also it becomes very difficult rather impossible to find a comparison group with similar characteristics as that of control group. Please read my whole research problem in detail in this comment later so as to offer your valuable inputs.

Milindo: I am explaining my actual research problem in a bit more detail below. Please go through it and offer your valuable inputs regarding the Research design and the other Research Techniques.

Herman Ader: Thanks for your explanatory input. Thanks for making your comment available inspite of some technical problems with posting. I would of course like to know more about this design/methodology. Please go through my research Problem as mentioned below to offer your valuable and explanatory notes/suggestions on that.

Dear All, I think the discussion has reached to a point where it becomes quite necessary for me to explain my research problem in more detail. My actual research objective is to analyse the Impact of various Financial Inclusion Initiatives on the beneficiaries. The Impact will constitute of the changes in socio-economic variables of the beneficiaries. So far as the Research design is concerned, I have 2 alternatives.

1) To collect data from the Program beneficiaries who have been with the program for quite a time, asking them about their Socio-economic conditions before joining the program and after they have been with the program. In other words, to understand the impact of Program on their Socio-economic condition using Pre/Post methodology. Since this design is not without limitations but due to time and financial constraints I am considering this Research Design.

2) To get the responses from program beneficiaries who have been with the program for quite some time and take them as Control Group. and to take responses from the new-comers to the program and take them as a Comparison group. Both the groups would be asked to respond to a same set of queries and after the collection of responses from both groups, any statistical techniques can be used to assess the impact like Annova, CAGR etc.

Both the alternatives are with their inbuilt limitations, but I believe when talking about Applied Economics, and where almost all variables are uncontrollable, there can be no perfection. There will be a set of limitations in both the cases but that will be mentioned while discussing results of the Research.

Now it is time for you all to kindly enlighten in detail as to which methodology can be used. I mean please help me out at the alternatives I am facing and also suggest techniques and tools that can be used in either case.

Thanks,

Khaki Audil

I think ,it is better to used anova if you know the distribution ofb population,if you dont know,you dont have any chosie except non parametric anova

Dear Audil,

I.

You really have to consider how you define relevant impact of the program for a specific beneficiary. It seems that it has to be in terms of change of SES, but what aspects? Maybe put yourself in the place of a policy maker who has to initiate such a program. What should the program achieve? And how can the effects been measured?

Are SES variables the most appropriate? Which ones?

II.

As intrinsicly suggested in your explanation, you assume that beneficiaries benefit

more in some way or another when they are longer in the program, at least that impact depends on time in the program. This can easily be checked using (multilevel) logistic regression analysis. In this, the variable you come up with in I (a dichotomy), is the dependent. The indepent is `time in the program', possibly divided in relevant, equidistant classes (less than 3 month, between 3 and 6 month, 9 month etc), because this makes interpretation easier and takes care of the possibility that the `odds on benefit' are not linear in time (it would be an interesting result, when benefit is best after 9 month say, and is no longer significant after that. This would mean that the Program can be shortened).

In this way you get rid of any treatment selection problems. Your study becomes strictly descriptive.

Of course, the above can be extended to a study in which you include, apart from `time in the study', other background variables of the benificiaries. But maybe this is not too relevant and may even lead to unethical recommendations (for example: what if it turns out that girl profit more from the program than boys at lesser cost? Do you then decide that boys and girls have to be treated differently? Or do boys need extra stimulus?).

Let me know,

Herman

.