This may help https://www.promint.hu-berlin.de/veranstaltungen/doktorandenkongress/eilerts.pdf

I do not use tests for competence assessment  rather presentations, posters or in-situ exercises. These techniques are the most appropriate to show different skills of my students such as communication, confidence, reflective thinking etc.

We utilized a combination of standardized testing (multiple choice, fill in the blank, short essay, matching etc.) with a day of practicals (application based) testing.  Although this takes longer, this combination of testing addresses the needs of many different types of learners and informs instructors of where students are struggling.  I personally do not think that standardized tests tell an instructor much about what a student knows beyond their ability to regurgitate facts, however, I very much agree with the practical.  This form of evaluation allows students to practice the application of concepts in a context that is closer to real life than a multiple choice test.  

Unfortunately, examinations are widely seen as the best competence assessment. Sometimes, you notice that students write verbatim the details the lecturer has given them but few days after the exam, they cannot provide answers to some of the questions they have had 'A' in. I think a combination of strategies would better make for competence assessment including verbal communication, presentations, critical thinking or reflective thinking to test the student's ability to communicate and convince. In terms of examination, all strategies are good too including MCQ, short essays, structured questions etc and most importantly, asking questions that will force the student to think well before being able to attempt it. This totally discourages cram work rather it makes the student remember the details of the exam later.

Depending on discipline, different exam methods may be suitable, but where it is not about learning factual knowledge by heart and reproduce it accurately in a test, it is not so easy to assess the competence of your students. In tendency I would suggest not to rely on only one form of test or exam. I am a historian and my students have to give a short (15 minutes) presentation in class, they have to participate in discussions, hand in some homework and at the end of the semester they have to write a paper/essay. This means you have an awful lot of work reading them and giving feedback but I can't imagine a better way to really see how much your students might have learned. 

As competence means a lot of different things to different people and anyway, it appears impossible to come up with a clear-cut all-inclusive definition of it which will also imply (as opposed to allow) the type of test which would be appropriate, let's take a pragmatic and broadest possible interpretation of competence, which is: the more you move upwards in Bloom's hierarchy (or to the right in the usual table representation of this hierarchy), the larger the proportion of the factor competence in the learning that takes place and thus in any appropriatte assessment designed to capture "competence" in some way.

Furthermore, there is far from unanimity or clarity about what constitutes valid tests or assessments of "competences", because in Bloom's days and still to a large extent in our days theories of learning and theories of assessment are rather unsatisfactorily or not at all coupled: i.e., we have two worlds of scholarship, and it is entirely left to practicing teachers / lecturers to come to grips and integrate the two "in the classroom". Thus again let's take the broadest possible view on testing and allow all forms you can find in practice, from simple one-minute quizzes to large-scale high-stakes examinations over several days. This is conveniently labelled assessment in most handbooks, whereas testing and examinations are very special forms of it, mostly exclusively used for summative evaluation at the end of a course or study.

Having said this, my interest and experience is mainly with classroom-based assessment (where the teacher is fully responsible and in charge) and with a small number of other faculty-based forms of assessment (oral examinations with presentations and questions, student research projects spanning one year, batchelor projects). In all those cases, the expected form of assessment is by and large competence-oriented, I would say (see my pragmatic definition above).

In one of my classes, the teaching paradigm is PBL (problem-based learning), which is definitively competence-oriented, and thus I have arranged an elaborate assessment scheme consisting of (at its zenith) 18 distinct assessments of different type: classical tests (false/correct), working papers, oral presentations, design documents, review documents (self-review, peer review, groups reviews), software demo's, etc. IMHO, really adequate competence assessment should cover not just one type of assessment, but by its very purpose a selection of different assessment types, so that distinct components of the to-be-assessed-competence will be covered. If not, you are paying lip service to a hype instead of full service to the student.

Agreeing with Paul Hubert Vossen here upper, I’d say that you need to know how you define a competence before trying to measure it. Jacques Tardif (2003, p37) defines it as a « complex knowing to act supported by the effective mobilization and use of a variety of resources ». The author considers the competence as being « more about heuristic hands-on learning than it is about algorithms ». Following that definition, we try to design evaluations allowing that « knowing to act » to be observed and judged. It can occur either during internship, role play, focussed presentation/discussion, or, more often, through a portfolio. In his/her portfolio, the student is in charge of bringing together the best evidences of his/her development of the 5 to 8 key competences of his/her program. Evidences ask for the students’ analysis and reflective comments on a series of elements brought by him/herself like pictures taken during internship, videos, testimonies, pieces of texts or traces of work the student thinks demonstrate his/her competence development.

Such an integrated evaluation, through portfolios, is pretty much time consuming but it is multidisciplinary, so it’s taken in charge not by a single teacher but by a team of teachers, so sharing the time demand. The main advantage is that it allows to approach competence evaluation (in Tardif's sense) without being « on field ». Anyways, even on field, it would be impossible to test not only what the student knows and does, but also what he/she would have done in such and such situations, that don’t occur daily. The portfolio allows it... and of course, it's pretty far from MCQs.

Fully agree, Marianne, in particular also with your last paragraph: sound valid competence assessment can be very time-demanding.

P.S. Could you add a link to (Tardif 2003) on Amazon.com to your comment, to the benefit of all? Thanks in advance!

Here it is, as attached file. Tardif's recent work, and especially his 2006 book on competence assessment, is published in French and unfortunately I don't know any English translation of it :-(

My competence in french is probably not enough to read the book - although quite recently I embarrassed a friend of mine, who has studied french linguistics but is now a retired professor in quite another discipline, by asking what the figurative sense of "Je ne suis pas une valise!" is - he didn't know ... (probably forgotten after so many years)

Now to the paper of Tardif. In that, he brings the problems with the Scoring Rubric approach eloquently to the point: a Scoring Rubric integrates qualitative and quantitative aspects of competence assessment in one and the same framework, but it doesn't tell the assessor explicitly and unequivocally how he or she should end up with a final asessment score. Sometimes that may be OK, e.g. if a teacher is only interested in a rough and provisional sort of assessment (as in the context of formative assessment), but Scoring Rubrics fail badly in the case of assessment scoring for summative purposes.

Remarkably, Tardif - if I have understood him correctly - seems to be of the opinion that it would be better to discard any references to quantitative parameters as he calls them (i.e. criteria), and thus to ignore any mathematical basis for this type of assessment.

I can't agree at all with him on this point. As the underlying approach of a Scoring Rubric is very clearly borrowed from measurement theory, in that it defines what is called a measuring standard (i.e., levels in the terminology of SR), one can't escape taking all what is required to turn it completely in a form amenable to sound and consistent quantitative scoring, even if it is only in the weak form of ordinal measurement (mostly fully adequate for the weak and scattered theories we currently have in the area of competence assessment). If one aschews these necessary steps - for whatever reason - what one gets IMHO is a false sense of doing something meaningful. That is exactly what I feel, when I use some Scoring Rubric, which unfortunately I have to do in some assessment situations.

My motto is: there is no royal road to competence assessment if one wants to avoid the many pitfalls and mistakes while turning qualitative judgements into robust quantitative scores for reporting and grading.

On the other hand, if the stakes are really very low and you are the only one using the Scoring Rubric in your own classroom to inform yourself about learning progress in the class or for quick and dirty feedback to your students, please go ahead. However, if you feel uncomfortable with the technique of Scoring Rubrics, don't blame yourself: it's the technique which is confusing you, not the other way around!

Wright, Paul Hubert, your last paragraph will probably speak to many teachers. 

Even though I don't use rubrics to judge on competence development anymore, we still need rubrics in order to inform the students on their mastery of some elements that the students should use and combine in action. Those elements of the competence (Tardif calls them "resources" for the development of the competence) are knowledge, comprehension and other mental processes described in Bloom's taxonomy, but also skills and attitudes. For grading the mastery of those "resources", rubrics are still valuable tools and the students really need to be informed on their progresses on those isolated elements.

It's only when you intend to assess the level of the competence, i.e. the integration and combination of all those resources, that the rubric is often considered as no more valid. I see at least two reasons for that lack of validity : you will not be able to list all necessary resources the students should combine, and you have no basis for deciding on the comparative weight of those resources in the competence. At that high complexity level, there are too many different ways to end up with an efficient "knowing-to-act" in a given situation. 

It doesn't mean that you don't need any criteria for the evaluation of the portfolios and the evidences brought by the students. You still need criteria (we used 6 criteria this year). The difference is that  the final grade is not given by the addition of numbers from a rubric. You introduce a touch of subjectivity, so you also have to introduce a collegiality of judgement (we usually have 3 judges for the most complex performances). The difficult part, for the teachers, is still to agree on the criteria !

Some further questions I don't have an answer on:

• who has the competence and the resources (e.g. time) to develop high quality scoring rubrics (aka qualitative ordinal scales) in the first place?
• what are the consequences for our students if we use scoring rubrics which aren't robust enough?
• if scoring rubrics are used not for formative purposes but for summative purposes (entering the realm of high-stakes debates), shouldn't those rubrics be thoroughly validated and certified by established accreditation agencies?
• if for some reason you decide to ignore the implicit ordering of the levels of competences on a scoring rubric, what else is left as a simple checklist of different but related quality criteria which together define that competence?
• why is it difficult for the teachers to agree on the quality criteria in the first place: don't they have themselves the competences they are supposed to transfer in some way to their students? or aren't they able themselves to articulate what they know and what they do when they teach their competence?
• what about educational researchers: are they in a better position (than teachers in the classroom) to agree on, articulate and communicate the basic resources and quality criteria for the competences which  teachers are supposed to transfer in some way to their students?
• if defining and agreeing on the underlying quality criteria (parameters for Tardif) is already difficult and perhaps impossible, how can we hope to be ever able to construct valid and useful Scoring Rubrics which build upon those criteria and go one step further by spelling out degrees of competence? 

---

Looking at this long list of questions I haven't found an answer on in the literature it seems to me that we are unfortunately still far away from a educational science of professional competences.

I even wonder, just wonder, if it might be the case that in spite of a decade long scholarly discussion about the concept of competence we haven't really made any substantive progress and still aren't in the position to tell teachers:

• if you - teacher - want to do a decent job of competence-oriented teaching and assessment, here are the concepts, methods, techniques and tools to do it;
• Please go ahead ("Here is the starter kit") and if you happen to have any questions, feel free to contact us: our experts are trained to answers all your remaining questions. We are well-prepared.

***

While developing my performance assessment scoring system PASS (cf. my publications on RG) I always told and tell my audience:

• "I am not going to spell out any specific quality criteria for the contents of your respective courses and trainings, because I know that you can find all such information in your field of expertise much better than I can tell you here and now.
• But if you have such a catalogue of basic quality criteria for the competences you want to assess, then please here you have a comprehensive and flexible system to deal with them, no matter whether those criteria are qualitative or quantitative, and irrespective of the type of assessment(s) that you want to conduct."

Perhaps I have been too optimistic!?

there's quite a lot written about it in the medical education literature, searchable on pubmed.

For an introduction I would suggest:

Hodges, B. (2006). Medical education and the maintenance of incompetence. Med Teach, 28(8), 690-696. doi: 10.1080/01421590601102964

van der Vleuten, C. P. (1996). The assessment of professional competence: developments, research and practical implications. Advances in Health Sciences Education, 1(1), 41-67.

And then for a more modern view:

van der Vleuten, C. P., & Schuwirth, L. W. (2005). Assessing professional competence: from methods to programmes. Med Educ, 39(3), 309-317. doi: 10.1111/j.1365-2929.2005.02094.x

van der Vleuten, C. (2000). Validity of final examinations in undergraduate medical training. BMJ, 321(7270), 1217-1219.

and relevant to the discussion above:

White, C. B., Ross, P. T., & Haftel, H. M. (2008). Assessing the assessment: are senior summative OSCEs measuring advanced knowledge, skills, and attitudes? Acad Med, 83(12), 1191-1195.

Varkey, P., Natt, N., Lesnick, T., Downing, S., & Yudkowsky, R. (2008). Validity evidence for an OSCE to assess competency in systems-based practice and practice-based learning and improvement: a preliminary investigation. Acad Med, 83(8), 775-780

and

Pell, G., Fuller, R., Homer, M., & Roberts, T. (2010). How to measure the quality of the OSCE: A review of metrics - AMEE guide no. 49. Med Teach, 32(10), 802-811.

Thanks Dason, seems there is a lot of expertise already in the medical field (not surprisingly, I would say, because the medical profession has always been strongly performance- and competence-oriented). How about other professional fields, i.e. fields in which there is a clear bifurcation into research on the one hand, and application on the other hand? 

This may help https://www.promint.hu-berlin.de/veranstaltungen/doktorandenkongress/eilerts.pdf

Thanx Jesus Garcia Laborda

Without going into the other comments.  In competency based teaching & learning there are many ways of assessing.  One good one is observation.  Just watch what & how students address the materials (initially you get to see whether the level you have set is adequate or lacking or too high) Then you adjust & in my area I gather materials almost weekly whereby students are involved in the assessment process throughout the unit.  The end or mid reviews are very simple as they are effectively a) tallies of the materials submitted.  b) the initial & mid unit reviews signal T&L issues re. each student & also their capacity to meet deadlines, do the work & address the unit's criteria.  That is standard VET delivery.  N.B The above examples are from Digital Art (computer based units) & Introductory Painting (Visual Art) units. 

Thanks for this good example, Jeremy. Have you ever used electronic portfolios for the students to store, then select and comment the pieces of their art work they submit for assessment, explaining themselves how they meet the criteria ?

If anyone is still reading here, please see my project on Peer Assessment Scoring Systems. Here you will find all ingredients for a modern assessment toolkit. It is still "in the making" but the basics are there to stay for a long time... Thanks for your interest!