I have developed a framework for design professionals, that main goal is to enhance the quality of space in certain neighbourhoods.
Then I asked 3 independent design teams to follow this framework in practice, and create the design of the same site. Also, they got back to me with the feedback about their experience on working with this framework.
Is this enough to validate the framework? Or I should use another word to what I did? The control here would be their previous practices, without the framework?
Thanks for answers and comments!
If you follow the paths that you draw in your framewok and test its sequence and chronological order toward the outcome, then your statistical analysis are supportive to your framework. To validate, I think you, or others, need to test it in differnt settings and samples
I would say you this is a fair way to test a framework like this. It is called a validation by some and others would oppose to aplying this term. They would feel you need stronger evidence to call something 'validated'.
I think it is more important to reflect on the type of evidence you have gathered by comparing it to the evidence you could have gathered if you would have set up the evaluation in a different way. For example...
What you did sounds like a formative evaluation. You applied your franework in practice withe main goal of teasing out problems and to improve the framework. The experience of your practitioners gives you evidence they can use the framework and whether they find it valuable. This is sometimes called face validity or practical validity. I am framing this as the 'lightest' form of evaluation, but there are many ways in which your evaluation could have been even less strict: such as only applying it yourself, only using one group, not using a real setting and so on...
An alternative would have been to do a summative evaluation. In this case you would have set well defined, measurable goals for the effectiveness of your framework and measured to what extend they where met in your evaluation. This would have given you somewhat stonger evidence to support the validity claim, but would not given you evidence it worked each time and it would be generalisable to different settings. It is well known, for example, that interventions, such as aplying a novel framework are biased to give positive results. So to 'prove' your framework works across different settings you need to do an even stricter test.
In a randomised controled trial, for example, you would take a statistically representatieve set of teams and randomly assign them to conditions such as working with your framework, working with a different (placebo) framework and not working with a framework at all. Next you would evaluate whether the teams using your framework score higher on some predefined indicators. You could certainly claim validity after doing this, but the enormous costs of such an approach would, in my opinion, not outweigh the benefits.
My personal preference is generally to do more formative evaluations, leading to more refined frameworks with lower validity claims attached to them.
Hope this helps.
Thank you for your answers. It is true to really validate a framework for design practice it would take enourmous budget, and might as well be not worth it at all, for many reasons.
First of all there is no one indicator for a "succesfull design" or "successfull place" , but instead, multiple various indicators, or rather attempts to specify the place as succesfull. If we consider some very rigorous indicators such as environmental indicators for example. We can then measure if the environment improved after application of this framework, and after building that particular space.
Here, again the limitation would be enourmous costs of building a set of these places, and in the end they wouldn't even be comparable because each space has a different physical site specificity....
Obviously I agree - although your last comment suggests limitations to the general applicablity of your framework, which you still need to make explicit. It is up to you specify the scope of your theory (in what settings you believe your to be helpful) and to provide a justification of this claim. Maybe the attached paper, although written for a different context could advance your thinking about this subject.
Conference Paper How to evaluate technologies for health behavior change in H...