Andreas,

Having taught a problem solving module for the first time last academic year, I realised that simply asking them to solve a range of problems wasn't adequate. Our assessment strategy was based on a portfolio of problems that they were asked to solve each week. As part of the portfolio, I asked them to write up their understanding of the problem and their approach to solving the problem but these are meta level tasks and many seem to lack this meta-level reflective ability or didn't see it as relevant. The consequence was poorly formed solutions.

For next year, I am reducing the number of problems but placing more emphasis on the components where the reflect on their understanding of the problem and their problem solving process. The first exercise will be purely about how do you understand this problem. The second will look more at how do we work toward a solution. The later exercises will involve solving problems using domain tools and techniques.

We do teach a problem solving method (Vickers 2009). If you then ask students to describe their problem solving process, many will simply parrot back the Vicker's process believing that is what you want. That is they are not internalising the process and may not even be attempting to apply it. I see a similar problem when we provide sample solutions to exercises and then ask students to solve a similar exercise. The solutions often are little more than adaptation of a sample solution. The question then becomes can they actually solve these types of problems or can they only copy and adapt current solutions. Of course from the perspective of design patterns, this might be argued that this is precisely what experts do. However, experts manage to adapt the solutions far more than students do.

Coming back to your question, I don't believe that it is as clear cut as your question implies. What do you mean by validated assessment instruments? It seems that you want something that you can apply to your assessments and have them signalled as good or bad. All such instruments will do is confirm whether your problem solving test conforms to the biases of the developer of the assessment instrument. Surely, it is more appropriate to ask what is it that you hope to be able to see from these assessments and whether your test is accessing these objectives.

I am suspicious of the idea that using appropriate verbs in questions or assessments mean that students will actually perform at the cognitive level supposedly indicated by those verbs.

I totally agree. But even if no single assessment instrument can cover all the skills and abilities relevant for all kind of problems (but merely reflects the test developers field of focus) I think there may be many tests out there which are known to address certain cognitive and non-cognitive aspects relevant for solving a wide range of problems. Maybe the tests tell me at least which developers' biases my tests conform to.

Do you think your tasks may reliably assess a kind of "meta-level reflective ability"?

Hey Andreas,

I agree with Errol's response. There isn't an easy way to test "problem solving" since it's such a wide area. There are a couple of research papers out there on a battery of cognitive assessment while testing video games. These tests include various aspects of cognition, such as memory, visual acuity, math problems, logic puzzles, etc. Boot, Blakely, & Simons (2011) is a good place to start looking at these battery tests.

You ask "Do you think your tasks may reliably assess a kind of "meta-level reflective ability"?" Recognising the problems in assessing any skill that doesn't have a clear yes, you have it skill, I think my approach is reasonably reliable as far as academic judgement is concerned.

It is dependent on a student's ability to reflect at a "meta-level" and me being able to detect that their reflection is genuine. That isn't always easy.

Reflective writing requires development and like any skill the novice will start out trying to follow templates. When I read the writing, I have to determine whether the student is just modifying keywords in a template or actually reflecting. Let me try and explain my basis for doing this.

Dreyfus and Dreyfus (Hunt 2008) developed a model of the progression from novice to expert.

A novice tends to rely on recipes or follow rules. Learners at this level will tend to reflect a template back with minimal change in its structure or content. Many of our students fall into this category. They are not really interested in learning just determining the rules to get a degree of the level that they are aiming for. Sadly if they don't get out of this level then they end up with a degree far below their potential.

The advanced beginner can begin to break away from the fixed rules but they are not really interested in the big picture. That is they don't want the theory, they want enough to be able to adapt it for a solution to the problem at hand. The template here is adapted to better fit the problem at hand but the understanding of the bigger picture and relationships between ideas are not there. Again these students often don't achieve what they set out to achieve.

The competent learner / practitioner begins to look for the bigger picture and begins to develop a conceptual model of the problem domain and to work with these models. As a result, they can begin to troubleshoot problems and to solve problems that they haven't seen before. They tend to work to deliberate plans and past experience. Hunt says that these people are "typically described as "having initiative" and being "resourceful" (p 21). They also tend to be in leadership roles whether formal or not. People at this level tend to be able to describe what they have done and to adapt it but not show the "meta-level reflection" that we desire.

The proficient learner / practitioner need the big picture and "will seek out and want to understand the larger conceptual framework around this skill" (Hunt 2008, p 22). They can begin to self-correct and seek self-improvement that was not available at previous stages. This I would contend is when we begin to see "meta-level reflection" beginning to happen.

The expert tends to work from intuition based on conditionallised knowledge. They may not be able to explain how they came to their solution as it is ingrained into the way that they operate. Hunt says that they know "the difference between irrelevant details and the very important details, perhaps not on a conscious level" (p 25). They are "very good at targeted, focused pattern matching" (p 25). We can detect an expert by the way that they seem to intuitively solve problems but they may struggle with "meta-level reflection" which tends to be more of a logical operation.

Taking the Dreyfus model, I expect many of my students to use the templates that I provide both to solve problems and to reflect on their learning (novice level). Adaptation of the templates begins to show advanced learner operation but they still will struggle with reflecting in a way that shows the bigger picture. Competence begins to show through when I see them seeking big picture understanding but they struggle to reflect on their work in terms of this big picture. They are still keen to apply past experience and rules. It is only when they move to proficiency that we begin to see them being able to reflect on what they are doing against the big picture models. This is often what we are seeking from our students when they are not capable of doing it. If we strike students operating at the expert level, we find ourselves often forcing them back to behave as proficient learners since we want an explanation of why they came to that solution. However, we might see their expertise because they come up with solutions that we don;t really expect.

The Dreyfus model is useful for understanding where students might be in their development and understanding but it doesn't necessarily act as good assessment criteria or I haven't yet formalised it for assessment. Instead, I have used the SOLO taxonomy (Biggs 1999). The SOLO taxonomy is a bout the structure of what is presented. The five levels are:

1) Prestructural: At this level, what is presented bears little relevance to what is asked of the learner. The response tends to miss the point.

2) Unistructural: At this level, what is presented tends to focus on a specific aspect or feature of the problem ignoring all other issues that might have a bearing on a solution. I am inclined to say that they are blinkered by the rules that they know (i.e. novice).

3) Multistructural: At this level, what is presented will appear almost like a shopping list or coverage of a lot of areas with little linkage between them. A surface level learner would tend to simply reflect back what they are given while a deeper level learner attempts some connection between the themes or ideas but still doesn't connect them all together.

4) Relational: At this level, what is presented shows an understanding of the whole and the relationships between the themes. They are seeing the big picture.

5) Extended abstract: At this level, what is presented shows a relational understanding of the problem domain but they are also drawing on wider understandings of what could be applicable to the problem. Some of these connections may seem right outside the box but they have linked it back into the problem in meaningful ways.

In our studies into code reading ability, we have made extensive use of the SOLO taxonomy. Someone who ignores the code and takes a stab at an answer reflects a prestructural understanding. A solution that focuses on a single construct is showing a unistructural understanding. A line by line description of the code tends to be a multistructural understanding. This would also include descriptions that summarise small portions of the code but fail to see the whole. A relational understanding would tend to summarise what the code does as a whole. It may also include some discussion of the inner workings but this isn't essential for a relational understanding. An extended abstract understanding would provide the relational answer but may also see the code patterns that have been used in arriving at the solution.

I tend to oversimplify SOLO when explaining to to others but I find it relatively easy to write assessment criteria that possibly include multiple levels within the multistructural level for programming and problem solving assessments that I write. I have also found it relatively easy to apply in the marking process since I am looking at the structure of the solution and not the minute detail.

I tend to see conventional marking rubrics as focusing on multistructural answers since they tend to focus on whether the learner has included specific concepts or ideas. Unfortunately, I am forced at times to write conventional rubrics for external examiners or assessment reviewers. In such cases, there is always a rubric that relates to showing the relationship between ideas and/or drawing on ideas from a wider context. Some would say the extension activities.

References:

Hunt (2008) Pragmatic thinking and learning: Refactor your wetware. The pragmatic bookshelf.

Biggs (1999) Teaching for quality learning at university. Open University Press.

For assing "Social Problem Solving" the Social Problem Solving Inventory-Revised (SPSI-R; D'Zurilla, Nezu, Maydeu-Olivares, 2002) is often used.

Lars, The inventory looked interesting but I can't afford the cost especially when I would need to adapt it to a different problem solving context. I like the idea of dimensions of problem solving ability and wonder whether these can be adapted to a more general problem solving context. I think I can see in my students both the rational and impulsivity-carelessness styles but I suspect the reason for using them has less to do with soling the problem and more to do with how they believe they can obtain the marks.

I would be interested in a general discussion around these inventories and their applicability.