Data, data everywhere – a pursuit to increase the degree of belief in proposed models for change that lead to improvement

We must start somewhere. How to practically accomplish this is an important challenge for understanding what works in which contexts in complex systems. In healthcare, there is an established interest in the use of improvement methods, largely borrowed from engineering to examine processes within systems. This involves proposing change concepts / ideas for development and testing in practice to improve processes, in anticipation of improving outcomes of interest (e.g. infection, mortality, number of avoidable falls). In combination, those concepts and ideas may or may not work in synergy, although continuous measurement (and statistical process control methods) that is mindful of relationships between process and outcome, as well as unintended impacts on other parts of system (good or bad), help to assess whether the desired improvements are being achieved. But, where to begin? How does one generate the 'logic models' (or Driver Diagrams in improvement terms) that capture those concepts and ideas we believe could leverage the desired improvements in the complex system of interest?

The Primary Care Patient Safety (PISA) Group at the School of Medicine, Cardiff University, is tackling this method challenge to improve patient safety. What do we do when lots of interventions do not exist for the system we are trying to improve, or when we do not know how those interventions work together? Our approach has been to empirically identify the important concepts and ideas for change, around themed areas of interest (e.g. discharge between secondary and primary care, vaccination administration incidents involving children, primary care paediatrics, and others), as proposed in the free text descriptions written by healthcare staff when things go wrong in NHS funded care. Through analysis of these national patient safety incident reports  from primary care in England and Wales, we are building an overview of the complexity of system failures resulting in iatrogenic patient outcomes. To do this, we developed the PISA Classification System to support the systematic application of codes to describe the sequence of events leading up to the incident, contributory factors at the level of the patient, staff, environment, as well as harm outcomes at multiple levels. Thus, we have found it possible to both model the common sequences of proposed human and system failures resulting in a patient safety incidents, as well as yield insights about contributory factors which form the basis of recommendations for intervention. 

The 'hot spots' (i.e. the most common combinations of events and contributory factors occurring together) for intervention inform the 'change concepts' and 'change ideas' that inform the content of our initial logic model. In addition, we undertake focussed literature reviews to identify existing interventions for inclusion too. Corroborating findings with other sources of routine or new prospective data collected is an essential next step, but also not forgetting the contributions from subject matter experts and those leading the project in practice – they will have many ideas about how to improve the system that will not emerge in written accounts because of all sorts of socio-cultural influences. Iterating this working theory of change must be informed by process evaluation that aligns with the maturity of the improvement model too (my colleagues and I have written a paper on matching evaluation methods – qualitative and quantitative – to the maturity of an improvement initiative to determine how and which contexts changes achieve improvement, see the link below. 

So, just start somewhere. Use insights from subject matter experts, routine data, new data, to inform an initial logic model. Develop and test the ideas within the model in practice and use continuous measurement to track the success of those changes to build 'degree of belief' in the components (and their synergy) within the model.

What other sources of routine data have others used to inform their initial programme-level logic models –  patient / consumer experience forums, culture surveys, co-production methods e.g. focus groups, interviews? 

Article Recommendations for Evaluation of Health Care Improvement Initiatives

Article Disparities in the quality of primary healthcare for sociall...

Conference Paper Improving safety of care transfer between secondary and prim...

Conference Paper How to generate improvement solutions from large volumes of ...

I do agree with Andrew: introduce as many and trustful data and just start. Jump into the water!

If you are seeking a 'practical' solution, I have tended to follow a similar approach to Ellis Cooper's answer above: define the system first and recognise it exists within a larger complex system, and is itself composed of further complex sub-systems.  It might be feeble but I attempt to describe this diagrammatically in the first instance so that the reader can at least understand how I have contextualised it and is then at liberty to disagree and suggest an alternative, which in turn can generate useful debate. This is generally consistent with an Action Learning methodology (Revans).  Mathematical tools and others' mental models can then be brought into play.  But even in writing this, I feel I have probably only admitted to struggling with the same question as the one you ask. 

We looked at this question from the point of view of "complexity engineering". We did not identify a measure of complexity per sé, but pointed at diverse ways to engineer complexity in different domains (mostly in ICT, but also nano-technologies, materials, biology, ..). Another way to look at measures of complexity, is to look at characteristics or performances related to properties such as scalability, resilience, adaptivity, evolution, learning, ...

Frei, R. and Di Marzo Serugendo, G. (2011). Concepts in Complexity Engineering. Int. J. of Bio-Inspired Computation, 3(2), pages 123–139.

Frei, R. and Di Marzo Serugendo, G. (2011). Advances in Complexity Engineering. Int. J. of Bio-Inspired Computation, 3(4), pages 199–212.

Frei, R. and Di Marzo Serugendo, G. (2012). The Future of Complexity Engineering. Central European Journal of Engineering, 2(2), pages 164–188.

I would like to suggest a distinction before trying to answer in a simple way  to the question.

In a scientific study,  you can try to manage as many data you can put in play and arrange them in a complex model of system description.

In the operational practice, if you have to decide on something important you may find it difficult to make proper reference to more than 3 main key factors at a time. This is the limit of the human brain and if the 3 key factors are not sufficient , you change model, then another one and so on, to deal with complexity.

Complexity is connected with  the life, and simplicity is a desirable objective in a model: I deem it necessary, also in the complex analysis, to utilize simple models arranged or combined in a complex way.

That is: complexity remains, in my opinion  and  in most cases of life,  at the end of the game, a complex arrrangement of individually simple elements and descriptive models. When you feel to have  fully dominated the true complexity then you might find, often,  that you are in the process of  falling in a complex descriptive mistake.That is the intriguing part of our scientific knowledge development : it never ends and complexity is to be considered as the inexhaustible feeding factor for new goals!

Kind regards

Alberto

Tarride, Mario Ivàn. “The Complexity of Measuring Complexity.” Kybernetes 42, no. 2 (2013): 174–84. doi:10.1108/03684921311310558.

I like this article by Tarride that provides a good frame to help determine what complexity you want to measure and offers some strategies for doing so:

Check Jacek Marczyk's work on Quantitative Complexity. He's developed methodology and products which can actually map AND quantify the complexity of a system. www.ontonix.com

We quantity the complexity by ht-index: the higher the ht-index, the more complex.

Jiang B. and Yin J. (2014), Ht-index for quantifying the fractal or scaling structure of geographic features, Annals of the Association of American Geographers, 104(3), 530–541, Preprint: http://arxiv.org/abs/1305.0883

In a way, measuring complexity is trying to characterize something by means of its opposite, since measures are inherently simple descriptions. 

My dissertation was on which measures (of a system development project) that are related to complexity might actually predict problems in the project or product. I found three that were correlated with cost, schedule, and system performance, and about 20 that were related to 1 or 2 of them.

See 2012 item at http://seir.sei.cmu.edu/sheard/.