Dear Bruno,

I would propose that you think about ANCOVA. Treat  your post test scores as your DV and pre test scores as IV. You may also have to bring in your covariates of interest.

Hi Bruno,

This depends a little on whether you want to understand and quantify what variables impact on the outcome.

In that case, I suggest you consider RandomForest to indicate which of the variables are important to use in a suitable model. 

If you are just interested in predicting, RandomForest is very capable of doing that in the circumstances you mentioned. Neural Nets may be helpful too.

I hope that's helpful,

Don

The analysis should actually be clear before one starts to collect the data. So who planned these studies? What analysis did they have in mind while designing these studies?

If the answer is: "Oh, nobody though about the analysis. We just thought these data might be interesting, and some of the data were simply easy to get, so we got them. We just wanted to see if there is something interesting." then the question for the analysis is as arbitrary. Describe the data and use it as information to better plan studies next time on interesting-looking features you may find in the present set of data - with a clear idea of the analysis in mind.

There are tools for multivariate (descriptive) analyses, you can use multideminsional scaling techniques, supervised and unsupervised clustering techniques, random forrests (as Don suggested) etc., and you can also summarize the findings for each variable seperately (what I think is the for your kinds of data). The problem you will always have with your data is that the sample size is too small for the number of variables you have to draw confident conclusions.

Thanks for all replies.

As I see, statistical tools like trees, forest, neural nets must have a big number of samples to produce valid results.

@Jochen, these were studies that were properly conducted and questions replied. But we thought that we could have some more analysis based on the data.

In fact, these were some examples of research that is very common in certain areas (like education, studying class of students), so my question here is about the strategies. I'm a junior researcher exploring what kind of methods I could use in future problems.

I think that qualitative approaches are more suited to situations like these. But I want to know more about different kinds of analysis, preferably with some experiences.

You have measurements, tests scores, questionaires etc. Thus you have lots of quantitative or easy to quantify data (because they are highly structured). This is why your first option is statistical analysis. You cannot apply qualitative analysis on this kind of data.

Maybe on your observation and interview data... You can try coding procedures, thematic analysis.

The size of your sample will not be a problem anymore in qualitative analysis but you must have appropiate kind of data to do it.

Hi Bruno,

Your small sample is not necessarily a hindrance for random forests. I had a small sample of about 40 participants for which approximately 170 variables were measured.

Random forests indicated the important variables to predict the binary outcome. There were three important variables -  which were entered into a logistic regression with good sensitivity and specificity. 

Bedroom air quality and vacuuming frequency are associated with repeat child asthma hospital admissions - Vicendese et al. 2015.

However, in regards to any future research, you will do well if you take on board Jochen Wilhelm's comments above.

Regards,

Don