You might explore the literature on Bayesian nets. Since the work of Judea Pearl, statisticians have begun to step back from the classical assertion that "correlation is not causation." And external perturbations are key.

Alternatively, you could take your measurements or give your evaluation tests at multiple points over time during the intervention (rather than just pre- and post-) and find a mindful collaborator who enjoys building causal models framed as systems of ordinary differential equations.

Good luck! It sounds like a great project.

Thank you so much for your adivce! I will evaluate, if Bayesian nets are reasonalbe for my project by getting into the literature you recommended.

Best regards and have a great time!

Benedict

Hi Benedict,

It is possible to see how each of the independent variables (factors)  contributes to each of the dependent variables (output response). This can be done through calculation of percentage of contribution of each individual effect. Design of experiment such as factorial design helps to calculate percentage of contribution after determining the regression coefficients.

Regards,

Omar

Benedict, if you are new to the design of experiments and putative causal models, welcome to the "deep dive" of a fairly complex problem that you propose.  I think it would be helpful to consider Design, Modeling and Measurement as separate aspects of your problem.  First, it sounds like you have a theory of what should happen within the data space that you are exploring. Design your theory carefully and as competely as you can.   Ultimately you will need to construct some way to model and test your theory (against some other competing theory.)  There may be many ways to model your theory.  For example LISREL can be used to model a fairly complex multivariate structure of plausible causal paths.  Be ware of the assumptions that the tool is making however.  Within the limits of what you measure, the relationships should be linear and differences between levels of the same variable should represent linear changes in those variables.  Modeling your theory as a path diagram has many intuiitive advantages when trying to understand what fits and why some of your data does not fit your theory.  If competing models are hierarchical, differences in fit statistics have known distributions and you can make an assessment about which competing theory explains the data best.

Unless you have very large samples and an army of helpers for the manipulations you want to make, and given that your measured variables are mostly continuous, then LISREL may be a viable modeling tool.  If not, then you may want to consider fractional factorial designs that economize degrees of freedom at the expense of aliasing some variables with others. 

Measurement is a significant problem.  Assuring linearity and knowing what a score means is not simple.  For example, see the literature of Rasch models.  You will want to be sure that you can actually manipulate your independent variables, or you will be left with a hodge podge pile of survey data to contend with, and causation will be plausible, at best. I have kn idea of how you intend to manipulate "personal values."

Can you cut this into some smaller pieces to investigate?  Best wishes on a vary interesting area of inquiry. 

Thank you so much for all this Information. I will channel all of your Information the next days and start the journey in discovering the different statistical methods that I could use for analysing the causalities in my intevention. A short question concerning lisrel: Does it have other functions than the one you have with AMOS (part of SPSS). For example for structural equation models and cfas I used Amos and dont see the difference to lisrel now. I hope you can help me find out.

Best whishes

Benedict

I have not used AMOS since it was incorporated into SPSS many years ago so I am not competent to comment on exactly what it will and will not do easily in its current version.  Since these are both pretty general SEM engines, each with their own set of estimation methods, I suspect that with a little work they are probably pretty good substitutes for one another.  I believe that the current version of LISREL comes with a variety of add-on programs including PreLIS that may be helpful in calculating some of the input that may be useful for LISREL such as tetrachoric correlations.  It is also useful to clean the data. 

My suggestion would be to familiarize yourself with the operation of each program with some small problems and decide for yourself what is best for you.  You can download a very limited (20 variables) version of LISREL designed for teaching students what the program can do from the following.

http://www.ssicentral.com/lisrel/student.html

You apparently already have access to the SPSS version of AMOS. You could construct equivalent models in LISREL and AMOS and then decide which program meets your requirements and move in that direction based on the resources you have available.  Based on what you described, I believe that the choice of computer programs is not the most important issue to address.  I would probably make the choice based on comfort of human-computer interaction unless I learned of some compelling statistical or computational reason. to pick otherwise.  

I must let you know that I do have a bias toward LISREL mainly because I worked on that project for the publisher.  I have not been associated with them for more than 25 years, however.

If I may add a few more high-level thoughts on this:

It seems that no matter whether you talk about "correlation" or "causation", in the end, you are restricted to certain kinds of model assumptions (i.e., linearity, independence, certain kinds of covariance structures, etc.) that are needed to make model fitting possible within the multi-dimensional space you have here.

So, to me it seems that if you

• design the experiment as best as you can (making sure that the variables can be properly manipulated, controlling for order effects in the intervention, setting proper sample sizes given known reliabilities of the measured variables and possible effect sizes you are looking for) 
• get a model that lays open its assumptions clearly 
• and then get a good data fit (r^2, AIC/BIC, etc.)

you have what you want - a tentative model for how your variables affect the measurements.

It would be good as a next step to validate this using an independent manipulation of the intervention (or, as suggested above, by adding several time-points in between).  

To me that's as good as it gets (as you may see, I'm more of a experimental than a statistics person :) )...

Good luck with your experiment!! 

Thank you for these great advices of both of you. My work is now to see whish assumptions are necessary in designing a model that can lead to fitting data and which Software is the best translator of this Problem. I am sure during this process new Problems and questions ;-)

Thank you so much until now! Keep up the good work!

Best regards

Benedict

Here is a little fable.  One day I watched a grandfather, Leo, looking after his young granddaugther, Sarah.  Sarah had a pile of small cubes, some red and some blue.  There were two cups, one black and one white nearby.  She was picking up the cubes an placing them in either the white cup or the black one.  Every time she put a red in the black cup or a blue in the white cup a laughing noise came from near the cups, but not otherwise.  Sarah soon knew that red in a black cup or blue cube in a white cup causes a laugh.  Leo however knew this wasn't the case as he had a different world model to that of Sarah.

My message here is that experimental data together with a model may lead to an assertion of causality, but the validity of that assertion relies on the validity of the model not that of the data and the data analysis.

I now dived a bit into the world of structural equation models. This triggered some new questions regarding my approach.

To recapture: MY GOAL is to enhance the four psychological variables (independent variables) "mindfulness", "personal values awareness", "autonomy of goals" and "intrinsic values orientation" in workshops with students in order to influence variables like Engagement, goal attainment, psych. well-being, subjective well-being and certain health conditions positively.

My OLD APPROACH was a pre-/post test multifactor experiment. But taking into regard that I plan to manipulate at least four variables (independent variables) through the workshop this would end in a lot of different treatments to be able to show causlities.

This together with your advices and comprehensive reading led to my thoughts about a NEW APPROACH. The opportunity I realized is that I have already built a hypothesis system of causlities based on correlations from other studies and comprehensive theoretical considerations. Based on this closed system of causal hypothesis I now ask myself, if, like some of you recommended it to me, it would be the best way to do the following 2 steps: 

1. Measure all of my independent and dependent variables (through my already existing quantiative measurements) with a big sample to prove my hypothesis system. As far as I understand the method path analysis as a type of structural equation model right I could test my hypothesis in that way with e.g. Lisrel and AMOS.

2. Based on the causalities I have shown I could now aim for an experiment in which I just measure pre and post the change of the independent variables. By that I should also be able to show that my workshop shows measurable results concerning the independent variables and can use the path analysis of the first step to show how each of the independent variables will effect the dependent variables.

I hope you were able to follow my thoughts. Do you think this approach is reasonable?

I would be more than delighted, if you share your thoughts with me and make recommendations.

Best whishes

Benedict

Hi Benedict... in reference to your #2: "Based on the causalities I have shown I could now aim for an experiment in which I just measure pre and post the change of the independent variables." ... you may have already considered this, but your conclusions will be even more convincing if there is any way that you could have some of your participants participate in a 'sham'/'placebo' workshop (ie a workshop that takes a similar amount of time, interaction, structure etc) as the real one but does not 'train' the variable in question... ie no longer just a pre-post design, but pre-post x treatment-control design... Elia

Thank you!

That is right! To do a quasi-experiment with pre/post measurement with a control group makes even more sense. After diving deeper into the methods that I could use to test an intervention that is able to manipulate four independent variables at a time on the one side, but also show hints for causalities between all Independent and dependent variables on the other side, I have now come to the following conclusion:

Doing just an experiment with four different independent variables may show that I was able to manipulate the independent variables, but showing how they all interact with each other and with the dependent variables would need to many different treatment alternatives and is far from feasibility in my case.

From my point of view, the best alternative for me would be now to do my research in two steps:

1. Collect longitudinal data for all relevant variables from a big sample (without an intervention) in order to conduct path analysis with my hypothesis system of causlities.

2. Based on the path analysis I am able to identify which variables have the most impact on the relevant variables and aim to manipulate them in e.g. a controlled quasi exeriment with pre- and post- measurement.

If this is successfull, I should have what I wanted:

1. A supported hypothesis-system of causalities between the Independent and dependent variables

2. An intervention that has proven that it can manipulate the Independent variables and by that influences the dependent variables in the way the path analysis has shown.

Does this make sense?

It would be awesome to get feedback as I am figuring out all alternatives thoroughly. Thank you so much for all the answers so far.

Yours faithfully,

Benedict