System thinking is being recognized as an important and valid philosophy. Many researchers keep on putting question mark upon the validity of system approach techniques such as Interpretive Structural Modeling technique etc.; however, these are found very useful to solve/model complex problems/issues by utilizing experts' opinions.

We are trying to develop a test that can predict systems thinking ability, which cognitive styles psychologists call "spatial" cognition.  I would welcome any advice, comments, suggestions.

Perhaps the participatory systems dynamics modelling methodology would be interesting for you? I have attached one of our papers here that references several articles on the subject.

Conference Paper Use of participatory system dynamics modelling to assess the...

to assess models based on the system thinking such as simulation model of System Dynamics, there are several evaluation test such as: sensitivity analysis, extreme value test, behaviour reproduction test and etc... If i have not misunderstood your question.

You may wish to look at some of the MMODS (mental model of dynamics systems) literature, which tries to elicit the internal cognitive structures by which individuals make sense of the world they inhabit. In particular, the works of Martin Schaffernicht and Stefan Groesser might be useful. I suspect that mental models that link a wide range of variables might be indicative of more systemic thinking, or "systems thinking ability", e.g.:

Schaffernicht, M., & Groesser, S. N. (2011). A comprehensive method for comparing mental models of dynamic systems. European Journal of Operational Research, 210(1), 57-67.

For an applied example, the case study below by Fokkinga et al., uses the number of variables and the number of feedback loops in a mental model as indicative of systems thinking:

Fokkinga, B., Bleijenbergh, I., & Vennix, J. (2009). Group model building evaluation in single cases: a method to assess changes in mental models. In 27th International Conference of the System Dynamics Society, New Mexico, USA.

These methods may introduce a confounding effect - creating a diagram to describe a mental model may have an organising effect on the individual's thinking. However, this may be less important in your case, if you are looking to assess systemic thought between individuals rather than how mental models change over time. 

Hope this helps.

I’m working on a paper about systematic management (in my area in projects) The used models are the OSTO-model and the Sankt-Gallen Model. Both allow to steer but also to analyze the system you like to manage.  The first step is to create a model of the system. Second step you can manage the system by comparing output with targets or you can analyze the system by defining an input and compare the output with the “theoretical” output designed in the model. By changing the input you can either steer or analyze the systems changes.

This article will be published in the next months. Some information you can find in my publications.

Article Case-study of a system-cybernetic project management in a mo...

There are two practical ways to answer this question, if we look at organizations

1)  see if you can map different parts of the organisation to the viable systems model

2) look for governance mechanisms, the more indirect they are the more system thinking was involved

As Dieter Masak answered in his first point is similar to my answer some minutes ago. His additional point of governance processes is part of the OSTO-system model.

Hi Michael,

There is a game called Ecopolicy developed by Malik Management, which is quite famous in educating and testing systems thinking abilities of both individuals and groups. It can be applied for all, including school children, students, government officials and companies. People can buy a disk for personal use or a licence for company use. You just need to google the key words and you will see a lot of websites and videos about the game.

Cheers,

Tuan

The simple stock and flow problem of Cronin, González and Sterman could be a good starting point http://www.sciencedirect.com/science/article/pii/S0749597808000447

Mike:

I am uncertain as to what you mean by 'not a questionnaire.' For example, do you want something that would be based (at least, to some degree) on third-party observation and not ('completely') on self-assessment?

Assuming that an answer to that question (or a similar question) is 'yes or somewhat,' if you are open to developing something, let's talk. I developed some checklists (I use the term "Direct Outcomes") via which it might be easy to develop something useful.

 Dear Thomas:

Answer = Yes.

Jaime Andrés Castañeda recommended a  simple stock and flow problem of Cronin, González and Sterman, (see his post)

http://www.sciencedirect.com/science/article/pii/S0749597808000447

Which led me to Sweeney and bathtub dynamics

http://www.arndt-sowi.de/lehre/ws0708/forschung/sterman_bathtub_dynamics.pdf

We're looking at a 20 to 30 minute test  that can assess the systems thinking ability --- whatever that is --- of science, technology, engineering and math (STEM) majors between the ages of 16 (senior high) to 26 (undergraduates).

So far the test consists of 5 or 6 Raven Matrices visual non-verbal questions, that we think measure several systems elements.

Raven, J. C., & John Hugh Court. (1998). Raven's progressive matrices and vocabulary scales. Oxford Psychologists Press.

Mike-

Michael - The document you shared assumes a fixed definition of systems thinking, but what if someone is a systems thinker, just not in the exact way that the definition in that document categorizes systems thinking?  

Consider the problem of gathering intelligence.  You may want to deter a terrorist attack or the plan for a financial compromise, but what if the perpetrator is using a different lexicon in their communication from what you are expecting?  What you actually want is to be able to determine what they are thinking from the words that they are using.  More than that, you want to be able to categorize what they are thinking.  Someone may be planning terrorism - or they may be planning tourism.  It could be a very subtle difference in the communication while it's important to be able to draw the distinction.

In you problem, you want to be able to objectively assess (classify) levels of systems thinking quality.  The questionnaire you reference may assess that level in an individual based on their Likert scale responses.  What if you are looking at the work of a team? What if you are looking for improvement over time?  Does that team have documentation that describes their approach and the manifestation of their systems thinking?  If so, an assessment that classifies the quality of that documentation may be useful.

We took a supervised learning approach to natural language classification in our publication: Use of Natural Language Processing to Discover Evidence of Systems Thinking in the IEEE Systems Journal, published earlier this year.  I invite you to review it and the preceding material, and provide us with feedback.

Best of luck!  -N. Peter Whitehead-

The best, in fact only, way to assess success of a systems thinking approach is by results, by answering the three Baldrige questions.

Dr. Deane

deanewaldman.com

Ask what the parts are of the system, ask how they interact, then ask what role the system plays in its context, and vice versa.

In our control systems assessment training, we start by explaining the concept of control to those students who come from the IT side of the house.  We use several examples of control systems that are familiar, everyday systems that most people don't think of as control systems.  A typical drink cooler that you see on picnics and at sporting events is an example of an open loop control system (the user provides both sensor and actuator).  We also show a range of temperature control systems (thermostat and a microPLC with a thermistor and electrical breaker).  For the purposes of your question, however, I fall back on one of the simplest closed loop control systems, the toilet bowl.

If you want to engage the students, show them (because many will have not seen it) the innards of a toilet.  Then tell them about dual-flush toilets.  Finally, ask them what needs to change and what will be affected by dual-flush toilets.  Few of them will be plumbers, but the system is simple enough that they should understand it and, if they are systems thinkers, they should be able to imagine the changes necessary to make a toilet be dual-flush.

There is a lot of development work to be done, but the basic scenario will suffice.  One thing to decide upon is whether you make it an "essay" question or provide multiple choice questions (possibly guided by the student's choices).  For those who have never paid attention to the inside of a toilet tank, you will need to provide basic understanding.  This assumes that the student has basic physics understanding (water is a fluid, it flows downhill, et cetera).

I am probably stating something that is notorious (and thus redundant), but from a methodological point of view, all is in the way we operationalize: what indicators we propose for an indicatum -- i.e., hypothetical construct like system thinking in your case. And the more creative you are in that, and the more they are converging, the more certain you are in the existence of that construct etc. By saying that, I want to point out how much I admire your effort to have that convergence.

Next, while skimming the items, I got an impression that the answers could end quite "refracted" due to language use: from what you intended to state, to what participants understood, to what they guessed would be "nice to give as an answer". And this is totally not about bad characteristics of the instrument. It's just ubiquity of language use.

If you would like quite different angles/views on the matter, then, maybe you could start with something more qualitative; like focus group that would discuss what system thinking might be, and/or in what situation it emerge, and for what purposes, and so on. That way you would have better insights into what is the commonsense about system thinking is, and what the responses to your scale actually mean.

Another wild brainstorm would be to try applying TAP (the Thinking Aloud Protocol), which would be quite different animal than the scaling techniques, or the focus group. There are quite some options in the world, but to repeat myself: the more divergent the techniques and the more they converge in results, the happier you should be.

Best

PM

Dear Mike, 

Besides the ecopolicy game as I mentioned previously, please see our publication below, in which we combined both questionnaire surveys (before & after a systems thinking course) and individual, group reflections to measure their systems thinking abilities, and KASA (knowledge, attitude, skills and aspiration) change elements.

Cheers,

Tuan

Article Can Advancements in Economic and Managerial Practice be achi...

You can use Scenario Analysis/Assessment or building as a tool to analyze different systems

If you consider a system as a cognitive framework for understanding complexity (essentially based on metaphors) then the test of a systems approach relates to possible changes in how people frame complex situations, presumably with the aim of gaining new insights and building requisite variety. You may wish to read John Grove (2015) Aiming for Utility in 'Systems-based Evaluation', IDS Bulletin 46:1, January for some thoughts on how one might go about this.

Ultimately the way ahead is to view complex situations using multiple systemic frameworks- see work by David Kolb in relation to using Pepper's "World Hypotheses" which is summarised in my paper on Action Research, which is cited by John Grove.

why not use the Appreciative Inquiry Method [AIM] see

Stowell, F.A. [2012] The Appreciative Inquiry Method – A Suitable                             Framework for Action research?, Systems Research and Behavioural                        Science, Volume 30, Issue 1 pp15-30 [Jan-Feb 2013]  

Dear Tuan:

Looks interesting.  Am exploring your lead.

Mike

Dear Kathryn:

You posted: "For spatial thinking it depends on whether you mean the mathematical or the geographical concept ..."  I think I understand the geographical aspect of systems thinking, but who is math spatial thinking?

Mike

Dear Radianti:

Thanks for the reference.  We have your article and found it very useful.

Mike

John Sterman developed a number of exercises/experiments to assess the individual "systems thinking" ability, e.g. understanding notions of stock versus flow (or accumulation/depletion vs change). See:

Sterman, J.D. (2010). Does Formal System Dynamics Training Improve People's Understanding of Accumulation? System Dynamics Review. 26 (4), 313-334.

Cronin, M.A., C. Gonzalez and J. D. Sterman (2009) Why Don't Well-Educated Adults Understand Accumulation? A Challenge to Researchers, Educators, and Citizens. Organizarional Behavior and Human Division Processes 108(1): 116-130.

http://jsterman.scripts.mit.edu/docs/Sterman-2009-DoesFormalSystemDynamics.pdf

http://jsterman.scripts.mit.edu/docs/Cronin-2009-WhyDontWellEducated.pdf

Tom Wujec might suggest that you assess systems thinking by considering how your subjects draw making toast:

http://www.drawtoast.com/

Mike Tang, Thanks for helping me track down research on "how does one test for systems thinking skills".  The address to the systems thinking scale manual that Case Western school of nursing has recently changed, it is now at:

https://nursing.case.edu/research/labs-studies/systems-thinking/

I can offer you a practical "test" of systems thinking. I literally applied systems thinking to a sick system called U.S. healthcare, uncovered the root cause and thereby established a "fix that will actually fix." If you want to see how systems theory really works in the real world, read "The Cancer in the American Healthcare System" (http://amzn.to/1QN2wBB).

Dr. Deane, www.wecanfixhealthcare.info. Also, Director of Center for Health Care Policy at the Texas Public Policy Foundation

 Systems thinking is determined by the subject area of its application. Two cases are known: the first - systems thinking in solving complex problems and the second - system thinking in the study of complex systems.

In the first case, the attributes of systemic thinking are systematization, formalization and goal-orientation. Solving the problem, it is necessary to perform an analysis of previously developed principles, models, methods and tools, determine their features, advantages, disadvantages and formulate the corresponding task verbally. At the second stage, in general, the mathematical formulation of the problem is carried out and a method for solving it is developed. All these actions are carried out taking into account the purpose of solving the problem.

If the goal is to study a complex system, then it is necessary:

- to form and streamline the goals of its development and operation;

- determine the internal structure and the elemental basis of the system;

- define the functions of the system as a reaction to interaction with the environment;

- offer an assessment of the performance indicators of the system as the degree of its functions performance;

- identify the integral criterion of system effectiveness.

The implementation of such stages will allow to predict the functioning of the system and perform its optimization.

You may also want to look at John Sterman's experimental approach to assess the system's thinking ability of students, professionals and others. At his GoogleScholar page you'll find many studies in this area. I add a link to one particular overview article.

Let's assume you are not caught in the MIT subterfuge (Forrester, Senge, Meadows, Richmond, et al.) and you do not consider systems thinking (aka: The Learning Organization) to be a derivative of system dynamics modeling.  If systems thinking in your mental model is based on a process like systemigrams and not a cognitive paradigm, please read no further.

If, however, you possess some variation of the cognitive paradigm mental model, then consider the core definition of systems thinking: system + thinking.  Definitions of system are abundant, but there is exactly one definition of critical thinking: Critical thinking examines assumptions, discerns hidden values, evaluates evidence and assesses conclusions (Myers).  This is a characteristic of subjects that is accurately quantifiable just like IQ.  Alas, it requires a questionnaire, the California Critical Thinking Disposition Inventory (CCTDI).

I posit that measuring critical thinking via CCTDI is an accurate reflection of a subject's disposition towards systems thinking.  I hope to publish sometime soon my thesis on how the seven categories of the CCTDI map to systems thinking.

I believe everyone is missing the point.

The validation or refutation of systems thinking is PROOF––hard evidence–of the outcomes/results from a system. If the desired outcomes occur in the quantity and location intended, then the interaction of parts "worked" and the system is well thought out. If not, then not. "The proof of the pudding is in the eating."

Dr. Deane Waldman, MD MBA, who applied systems thinking in The Cancer in the American Healthcare System and thereby uncovered the root cause as well as the route to curing a sick system called healthcare.

http://amzn.to/1QN2wBB

What we must avoid is the Potter Stuart approach to systems thinking.  True ST is much more refined and well developed than that.  There are quantitative ways to measure it before it is implemented and, I believe, that is what Dr. Tang is looking for.

Exactly.  Potter Stuart says he doesn't know what systems thinking is, but like pornography, he knows it when he sees it.  Meanwhile, Whitehead has pointed out that Wujec on Ted Talk tells us that we can assess systems thinking by asking people to draw toast.  I thought Whitehead was joking until I saw the video: http://www.drawtoast.com/. Whitehead and company are working of a Baysian statistical semantic/linguistic approach, if I understand him correctly, which shows promise that hopefully will take beyond pronography analogies and shallow Ted Talks, where I'm beginning to think most people are getting their ideas. 

In answer to Viegrig's posting (pg 1), we're looking at how to design a general domain systems thinking test, which can be validated with a cognitive styles test and a non verbal cognitive ability test, specifically the Raven.  The reason for looking for a non-verbal test systems thinking text , is that if "the medium is the message," then a verbal test  may have a non-systems bias, although this is just a hypothesis.  

In addition, much thanks to the people who posted answers and recommendations in the context of  complexity analysis and systems dynamics, which is highly advanced in regard to systems theory and systems thinking (for example, see Scott's posting).  When I looked "systems dynamics" up in Wikipedia, Google also suggested to look into "stocks and flows," in my search for a better understanding of this topic of systems thinking.  As a sequitur, I would add, that those who suggested that we look at Sterman's tests in that regard, are much appreciated and well taken.  Sterman-like  tests should be a part of any systems thinking test, but in themselves they appear too fragmented and as actual tests have not been sufficiently validated.  

In regard to complexity theory, however, I'm not sure if complexity theory should be conflated with systems theory and thinking even though definitions of systems theories often involve directives such as "a non-linear way of dealing with complexity, " 

Dear Michael,

maybe the research paradigms or tests in the area of complex problem solving (and dynamic decision making) are interesting for your research. Some years ago I developed a computer-based test called the EcoSphere - the participants have to explore a complex system by interaction and have to draw their experiences afterward (and their previous knowledge before the task). More details can be found here:

Tobinski, D. and A. Fritz  (2017), "EcoSphere: A new paradigm for problem solving in complex systems", in The Nature of Problem Solving: Using Research to Inspire 21st Century Learning, OECD Publishing, Paris.

DOI: http://dx.doi.org/10.1787/9789264273955-15-en

Best

David

http://www.oecd-ilibrary.org/education/the-nature-of-problem-solving/ecosphere-a-new-paradigm-for-problem-solving-in-complex-systems_9789264273955-15-en

https://journals.ub.uni-heidelberg.de/index.php/jddm

Hello David:

I read your article, Ecosphere, and found it most interesting and enlightening.  I'm looking for available systems thinking tests to develop one of my own because such a test is needed for further research and to operationally define "systems thinking."  Your article briefly talks about the need for a systems thinking assessment instrument and describes computerized test you've developed.  Is that software application to assess problem problem solving in complex system available?"  I'm trying to develop a pen and pencil assessment instrument using concept maps to measure systems thinking, what ever that is.  Eventually the pen and pencil version will of course be automated and placed on the Cloud.

Mike

 Hello Peter:

I surfed the jCCTDI url and caught the jest of what your are research on to be:

The CCTDI measures seven  attributes that influence an individual’s capacity to learn and to effectively apply critical thinking skills: the disposition toward truth-seeking or bias, toward open-mindedness or intolerance, toward anticipating possible consequences or being heedless of them, toward proceeding in a systematic or unsystematic way, toward being confident in the powers of reasoning or mistrustful of thinking, toward being inquisitive or resistant to learning, and toward mature and nuanced judgment or toward rigid simplistic thinking.

. . . which fits into the taxonomy of critical thinking as to what I call the Paul/Elder school of critical thinking which has a strong philosophical if not a neo-liberal political  bias.  While I believe engineers need this kind of thinking to be more effective in this new age of increasing complexity. I'm not sure the a "disposition toward truth-seeking, etc. are really attributes of systems thinking, as I would define it.

Mike

Hello Peter:

You wrote a while back:

Critical thinking examines assumptions, discerns hidden values, evaluates evidence and assesses conclusions (Myers).  This is a characteristic of subjects that is accurately quantifiable just like IQ.  Alas, it requires a questionnaire, the California Critical Thinking Disposition Inventory (CCTDI).

Does anyone know of a test to assess "systems thinking" that is not a questionnaire?. Available from: https://www.researchgate.net/post/Does_anyone_know_of_a_test_to_assess_systems_thinking_that_is_not_a_questionnaire [accessed Jun 23, 2017].

The reason for asking about a non-verbal questionnaire is because, generalizing McLuhan's law, "the medium is the message," from a cognitive perspective, a questionnaire, because it is verbal, will measure whatever with a verbal bias.  Again, from a cognitive perspective are you familiar with cognitive style theory specifically that of Blazhenkova and Kozhevnikov?  I'm working on using their theory as basis of defining systems thinking as a matter of style rather than ability.  For example, according to the two neuro-psychologists, each of has a preference as to how best to solve problems in the external world.  Artists, multi-media, multi-sensory professionals prefer to deal with reality through an "object" perspective, most of us academicians especially in the social sciences, business, and law, operate from a largely "verbal" construct, while scientists and engineers prefer to see the world through a "spatial" style, which is highly abstract, deals with patterns and relationships, changing dynamics, etc. which I correlate with systems thinking.  A non-verbal IQ test that appears to measure these cognitive aspects, happens to Raven Matrices, which I'm looking at as an ability test to validate what I'm beginning to see as a systems thinking style.  

Mike

MIchael - Just saw your reply. You've cited some very interesting points that I will explore as my time allows. In my work, I considered Noam Chomsky's theories on language and thought coupled with the CCTDI work on quantifying critical thinking. The style aspect was broken down via elemental semantic processing, the bag of words approach in tf-idf for example. In my perspective, the object view is more systemic and the verbal view is more systematic, but my lines fall a bit differently. Successful lawyers and artists think more systemically, doctors, car mechanics, pilots and computer programmers think more systematically.

One of the problems with "systems' is that everyone has an unispected notion of what that name means. The above answers show the breadth of ideas from so called 'hard' systems , soft and into social science. It seems to me that a general answer is not possible nor undesirable. Each application will be based upon a particular notion of Systems and it is to that the researcher should turn for inspiration. {Checklands FMA model might be helpful here] For our part [soft systems] we advocate "authenticity' as a worthy 'tool' for thinking about the outcome [with the clients] See

Champion, D. and Stowell, F.A. (2001) “PEArL: A Systems Approach to Demonstrating Authenticity in Information System Design”, Journal of Information Technology, Vol. 16, pp. 3-12.