I think that you can approach learning at three levels. Firts. Individuals learning capability,. Secondly is the learning organization level. So there is the team learning job with the orgnization governanza. Finally is the third level that is the colaborative learning that occur within and betwen partners organizations doing collaborative arragements. Learning capability is the key issue. So individuals, organiization and parterns confront challenges, the way they deal with this problems is the learning capability. Therefore learning outcomes appear at three level. Finally, you can put all together at the four level that is policy recomendations.

Best regars

Are you sure that the strategy alone, independent of people, place, time, recent past experience, aims and resources?

For example, someone who has developed a relationship with students in which conjecturing is valued, as are attempts at justification, may find a particular strategy (eg getting students to present their work on a board in public, or getting students to discuss their approaches in twos and threes) 'works', whereas someone who is used to a formal relationship with students, lecturing (while other tutors do the marking) may find that those strategies do not 'work' for them.

JohnM

This depends on what you think has been changed by the learning strategy. First one can consider the development of new concepts and changes in the interelationships between existing concepts. Equally one should expect that an increased ability to engage in new but related material shows that more learning of the original material has occurred - and can now act as a foundation for further learning.

A strategy that simply alters cognitive orientation is probably not going to be productive in itself - for example, if you exchange breadth of material for depth of understanding you have no normally changed the total amount of material learned. A succcessful strategy needs to have consequences that cannot be explained by the simple motivating effect of the strategy intervention and measurement process.

Cheers,

Please consider the possibility of carrying out an experiment to assess knowledge + skills gained using that particular learning strategy.

As you may be aware, that includes the identification of specific variables and the related indicators, the selection of the experimental group and the control group, and the development of the experiment itself.

Among the indicators, I have found the competence-based approach to be particularly useful.

Have fun!

I am uncertain as to the question being asked. In order to measure learning you have to ASSESS the students. I think some clarity is needed on what you mean by "beyond student assessment."

If I interpret it to mean "Without Student Assessment," then I don't know what can be done. This is because observing he students solve problems with the new knowledge or, having the students give feedback on what they have learnt are all means of assessment.

If the question is interpreted to mean "assessing student learning to demonstrate that it is the leaning strategy that resulted in the different student performance," then the design of your experiment is extremely important.

You'll have to survey students and group them by common demographics, experience and pre-knowledge. The groups would have to be split into two (2) half for the controlled strategy and the other for the strategy which you are trying to assess. Please note that you'll then have to conduct the experiment under the same conditions as to eliminate any bias which can be caused by the environment.

It is only under such conditions of eliminating or reducing eternal variables to learning can you then determine that it was the learning strategy which contributed to the performance of the students.

Gain is always assessed in the comaprison of a pre-test and a post-test surrounding the learning strategy you are interested in. But recent work in neuroeducation (fMRI, EEG, etc.) can provide other kind of research results DURING the task itself ...

The first thing that crossed my mind when I looked at the question was what Ruel has already pointed out, that measuring and assessing o together. I then wondered what Carola may mean by making such a distinction. A measure is what we want to achieve and assessment is a means by which we do this. The word assessment is commonly associated with ‘formal’ and often, pencil and paper, tasks. Furthermore, it is also often associated with the measurement of content acquisition. So, I thought, why not open up the idea of ‘assessment’ by brain storming the different ways in which we can gain insights into students’ learning. So, here are some ideas:

1. Pencil-and-paper tests

2. Observation: behavior, interactions with peers and teachers, task engagement

3. Interview/ask

4. Attitude questionnaires (change of attitude => change in understanding)

5. Punctuality, attendance

6. Brain scans (ref. Patrick’s contribution)

7. …

Over to you!

Carola, with regards to, “I would appreciate some ideas in order to state and not suggest that a particular strategy is effective”, you can ‘state’ that a particular strategy has been found to be effective, but many of us would assume that, although you are using definitive language, you mean that the strategy has been applied to one or more particular groups under particular conditions and learning has been found to have been improved in comparison with a control group and that this differential has been attributed to the strategy.

In other words, when referring to the results of research to make conclusions, some authors prefer to say that the results ‘suggest’ that this method is effective and others may say that the results ‘show’ the method to be effective. To some extent, the qualitative word used to ‘interpret’ the quantitative result depends on level of confidence that the investigator is comfortable with. So, for example, with a p

I second Charland's suggestion of pre-/post- test approaches to quantify learning gains in students. If there is a Valid/reliable test for what you are looking ofr, then administering the test prior to and then after a certain course can help. concept inventories are diagnostic tools that can be used for that - see for example: http://www.physics.indiana.edu/~sdi/ajpv3i.pdf ). However, this is not teh same as validating a "learning strategy", since the latter is influenced by several other factors - institutional, cultural, personal, etc. The learning gains are one piece of evidence in the puzzle: perhaps you need to define better what you mean by "effective".

Thanks for your comments. Sorry, I meant beyond students' perception.

Bom dia Carola Bruna!

A aprendizagem está em contínua construção.Por um lado, depende do desejo, da receptividade e da capacidade de processamento e da incorporação do aprendiz; por outro depende do "processo ensinar". E aí entra família, vizinhanças, convívio, ambiente, escola; metodologia, etc. Este conjunto de fatores (que envolve as condições do aprendiz e do ensinar) determinam o diferencial no processo. A mensuração deste diferencial de aprendizagem deve ter um momento inicial (momento zero=to) e outro momento em que se supõem que já tenha ocorrido ganho de aprendizagem (momento t1). São necessários estabelecer indicadores adequados que sejam altamente sensíveis e manifestem as variações ocorridas com a aprendizagem. Estes indicadores podem ser cognitivos, comportamentais, etc. Depende do interesse e do objetivo maior envolvido.

atenciosamente

AR

Carola, this is such a huge question. A few things come to mind that may be useful and the Kirkpatrick model of evaluation is useful.

I agree with Patrick & Manuel that there is always the pre-test/post test and you may consider administering both of these at the end of the teaching intervention as this has been shown to strengthen results since the learner can more accurately reflect on what they did't know after the teaching has taken place.

In the student assessment, have you considered self and peer assessment and triangulating this with conventional assessment results?

Ruel describes a case/control type study design but these are always difficult in education due to the number of confounders/biases including learner philosophical underpinning, prior experiences and the fact that at times the unintended learning outcomes may be more valuable to the learner than the intended outcomes.

The ultimate assessment benchmark or indeed evaluation of any teaching intervention is a measured improvement in the learners performance and its sustainability.

The question to be answered by any assessment is "has it changed the learner in terms of how they think and/or do things and for how long?"

Student assessment is necessary in order to determine the impact of any focused instructional strategy. The question you raise is more about the what and how of assessment, which has been mentioned above. From my experience working mostly with adults in commercial and government R&D environments we focus a lot upon the quality of learner productive efforts and error rates before and after instructional interventions. We also ask about perceived knowledge gains and will administer content area assessments but what matters most, for us, is the learners' ability to apply their new knowledge and skills to effectively make things or solve problems.

Out of the box answer: when you give a student a task, rather it is written, verbal, or hands on, the next step would be to give another task that involves using the previously learned task. If the student is able to accomplish the task, then learning has taken place. The assessment is simply the next step in application. For example: if I teach a child the letter a, then ask them to say the letter, then write the letter, if this student can track what is being asked of them and accomplish all the tasks, then learning has taken place. There is not need for formal assessment. The next step would be showing them how the letter as becomes part of a word. Following the same steps above, if the student is able to comprehend and do the required tasks. learning has taken place.

Our nation is so stuck on pencil paper testing, that we have forgotten that there are multiple ways to see if learning is taking place. We need to quit making it so complicated and just watch, listen, and observe what is taking place. All the sudden the child that learned the letter a is now reading stop signs, and other written material all over the place.

How do we measure this? Why? Ensure that the first step is learned and then move on. It would be impossible for me to jump off of a bench and ski down a slope if I did not even know how to put the skis on.

Carola, Learning is the respoonsibility of the Student. Therefore, if we want to measure Learning, then yes, I agree, a pre/post test is required. That way, the amount of "Change" occuring during the course is easily identified. I also agree that the "Test" ( I prefer to use Measurement) should not simply be a paper/pencil exercise. There are so many ways to measure change occuring in students. Discussions, Debates, Performances, Report Developments, Show and Tell, etc. and etc. Measuring Student Change and tracking the results often opens new insights into how better to deliver the course data.

Have an individual and group observation based on their performance or hands on. Improved performance indicates improved learning. Researchers whose work has informed much of this assessment reform include Ken O’Connor, Grant Wiggins, Jay McTighe, Richard Stiggins, Paul Black, Dylan Wiliam, Gordon Stobart, Caroline Gipps, Joanna Goodman, Thomas Guskey, Damian Cooper and Ronán Howe.

@Dear Carola, very nice and thought provoking question you have posted. Very simple answer to measure learning gain is the use of Intended Learning Outcome Survey, Just design a survey and tailor it according to the particular strategy to be assessed and seek feedback from the students who are supposed to acquire that ILOs, Interns as well as fresh graduates. For more accuracy or validation, seek feedback from the employers about level of that specific learning domain/s.

As far as I know each subject has a specific way to be assessed. For example, if I want assess math skills in young children (4 to 7 aged), I use the Early Numeracy Test (r) by H. Van Luit. ENT-r assessment is not linked to any particular learning strategy and it is reliable and accuracy. ENTr has 3 versions and you can check gains using a longitudinal design. Attached you find an article (and link) were we used the ENT with Spanish children.

Navarro, J. I., Aguilar, M., Marchena, E., Ruiz, G., Menacho, I. & Van Luit, H. (2012). Longitudinal study of low and high achievers in early mathematics. British Journal of Educational Psychology, 82, 28-41. doi:10.1111/j.2044-8279.2011.02043.x

http://jose-navarro.wikispaces.com/

Pre and post measurement are important here. It is importnt to know the knowledge level of the students before the introduction of the new concept. After the introduction then the knowledge level can be measured again. If the new concept involve the use of skill, then observation method can be used. Adesoji, S.A.

Hi Carla and discussion group,

The evaluations suggested through this discussion seems to be focussed on the educational forum. My area of teaching and learning is in post exerience post registration healthcare practitioners. The question that we are battling with is how do you assess the transfer of learning and knowledge into the workplace. Traditionally we have identified with Kirkpatrick and other theories of learning and proficiency (Benner) but the powerful assessment is to measure the impact on their practice and how that can in turn affect the performance of a team, group or organisation. This usually requires a different set of measurements and parameters to identify.

I would be glad of anyones thoughts.

Carola, with regard to measuring learning gain you might find it interesting to look at the 'Dreyfus Model of Skill Acquisition.' Here are a few links to get you started:

http://www.slideshare.net/mike734/dreyfus-model-by-pete-goodliffe#btnNext

http://www-personal.umich.edu/~mrother/KATA_Files/Dreyfus.pdf

http://www-personal.umich.edu/~mrother/Materials_to_Download_files/1980%20Dreyfus%20Report.pdf

We can not get away with assessment to know what students can and are able to do. What we can do is use more than one way to assess students, such as how well can they take notes, can they talk about it and answer questions at the same time, can they teach it to somebody else.

Mike Frey, Assessing workplace training can be easily accomplished by a two step process. First, if there is formal classroom training prior to the work performance, then you may want to establish a Skill/Knowledge Questionaire that would be given to those who have already passed the course and are in the workplace performing job functions. These questionaires should be given a the 30 Day, 60 Day and maybe 90 Day post formal training date.

Using this, you can easily determine levels of knowledge/skill that has been retained by the class participant, what impact (positive/negative) the work place performance has on the course objectives. I think we will all agree that actual live work performance is often very different that the processes we present in our training classes. Also, as processes change in the work place, at times our objectives and our lesson plans become out of date.

The second step would be to establish a post formal training OJT program for each individual. Try NOT to make this a grand production, but a solid simple task outline with areas for the individual and the individuals Supervisor to sign off when BOTH of them agree that the individual has met the requirement. Of course, a monthly review by the Training Department of each record to ensure progress would be required.

Now, I have implemented just such a dual program in many companies and the biggest problem I have faced is within the Training Departments. They often feel that the OJT portion does not have sufficient depth (that means all kinds of charts, graphs, verbage and other non-sense). The goal is Individual performance. To get that, I do make my programs short and sweet, otherwise, no one would participant, and getting better performance is certainly worth giving up on some charts.

Good morning, everybody!

Students must enjoy being testing and they must participate with their souls.

The results must show them what is missing, how they can improve 'themselves'.

A student of mine (aged 11) was doing extremely well at school, but when he had to change to a new school (closer to his new house), her mother looked for me to have a talk.

She said that after a composition test, he had arrived at home so enthousiastic... he had never showed such happiness.

He immediately handed his test with a smile... She got confused. It said he had got 3/10! He had never had problems with tests!

He asked her to read the teacher's note... She congratuluted his about his great ideas. He had never been congratulated! (From this moment on, I 'always' looked for something to congratulate when marking my students' tests!) (This also happened to me! I have always been a good student, but I was first congratulated at college! And this compliment has totally changed my professional life!)

When I saw the test, another good thing of it (that's not new) was that the teacher hadn't correct the mistakes. She highlighted them and asked the student to rewrite it in a better way.

When I taught at PUC University in São Paulo City (for about 10 years), I got surprised with some of my students when they got the their tests back with my marks and comments. Because of my compliments, despite the result, most of them tried to strive for a better result. Some looked at me and thanked.

All previous answers to the initial question demonstrate the complexity of educational evaluation, depending on whether it evaluates knowledge, skills, attitudes and values ​​or complex skills performance or whether the evaluator is only the teacher, peers or the student.... or simultaneously several of those issues.

To add to the specific question of Carola "evaluate the gain" the most reliable answer is through a quasi-experimental model: evaluation at baseline (before the educational intervention) and after the educational intervention; for excellence, you can also use a parallel control group.

In fact, we are applying this model to evaluate educational interventions to teach small innovative issue contents about science, technology, society and nature of science with real teachers and classrooms and a powerful pre-designed instrument.

The best ANSWER is to read Bloom classifications and High Order Skills Learning

Teachers are the foundation of a strong education system. An effective teacher is the number one predictor of student success. Education is to encourage younger people to go into the profession of teaching.Developing educational products of new models based on flexibility and learner's choice:

 Preparing students for the knowledge society;

 Providing methods and styles of working for life-long learning;

 Ensuring total quality management in the higher education system; & Catering to the changing market demands, instead of providing narrow specialization.

Generally, a pre-study assessment before deploying the learning strategy helps in refining the strategy. After deploying the learning strategy, one may like to post assessment and get a feel of the gain. Assessments with questions tagged to one of the levels of Bloom's taxonomy could be a way.

Usage of the four quadrant matrix of competence vs consciousness (Kirkpatrick Model) could be one, way, but it will take time. Peer assessment is another technique, but has its own challenges.

I would like to suggest the examination of other research that has been done regarding that particular learning strategy, first. As a researcher I always find that there's research that has already attempted to answer the "assessment" component of your question. Second, assessments can be carried out in multiple ways and many of these are imbedded in the learning strategies themselves. Meaning that they can go beyond paper and pencil tasks. They can include project based assessments, discussions which examine the discourses that students engage in, or articulation of the students' thoughts or thought patterns.

Hi Carola:

I think the first thing that the teacher had to been in mind is the objetive and goals of the learning activity in orden to design their evaluation strategies. For example if your goal is training a student in a particular laboratory skill, you can perform a set of evaluation strategies. One of them could be a theorical test (this one can help to measure the theorical fundaments of the skill). Other could be the quality of student replication of the lab skill, coupled with the aplication of the technique on a resolution of a specific field problem. By the end, you can invite the student to explain his or her results in front of the class and doing so invite the others class students to share their point of view about the student performance and achievements. The teacher in this case may be aware of design and facilitate a specific evaluation templates in order to keep a fair play on the co-evaluation by peers.

Hello! I like the retrospective self assessment method. This involves a pretest, e.g., how much do you know about these topics? How competent are you at these skills? Then there is an immediate posttest, with the same questions, Include questions about "compared with your knowledge and skills at the beginning of class, show me where your knowledge and competence have changed".

I do not take the pretest answers into account because students seldom are aware of they know/do not know BEFORE the course. But this pretest serves to alert them to the content of the class. It wakes them up.

The immediate posttest data are the best measure, based on their awareness of what they have learned. Then check these scores against a regular test of knowledge and competence.

A powerful way to measure learning gain is to measure generalization in different domains/processes than those targeted by an intervention. In the psychology of intelligence there is a hot current discussion about how working memory or attention training (two markers of general intelligence) generalizes to reasoning and inference. Moreover, once you have careful pre- and post-tests in the generalization domains of interest, there are powerful statistical methods in the context of structural equation modeling that can be used to specify what and how much generalization occurred.

Andreas

One way is also when students are able to teach the topic to someone else or by designing a graphic organizer

I use group project-based learning for medical cybernetics (medical informatics) education. Each group of 5-year students develop software for medical personnel (doctor, nurse in different medical fields) workstation. These workstations work on common database (developed for the task of certain year), but each workstation's software has different user interface, supplemental information, decision support module, etc. Final evaluation of each workstation's software is carried out by final user. It means medical personnel participate in the evaluation of students projects, during the work and finally. It is model of market, but without money. Besides, we can ask students about role of other students in project, about corresponding to their functions and role in the project team.

The simplest and most dependable approach I've found is to measure retention and recall of knowledge. It's a "low level" cognitive measure, but it's replicable and easily done. I select and slightly change a number of exam/quiz questions given over the course of the semester that focus on retention and recall. I re-administer them months after the end of the course. This is a particularly powerful method if there are two groups of students with different pedagogical interventions that can be compared in the same semester.

Why do we assume that 'learning gains' can be attributed, in whole or in part, to a particular 'learning strategy'? The underlying metaphor or model appears to be something akin to physics, particularly mechanics, in which each action has an effect, hence 'cause-and-effect' as the core mechanism in learning. But human beings are not mechanical agents, at least not totally, though sometimes they do act like that.

I prefer a metaphor akin to chemistry, in which multiple reagents react in a chemical soup, approaching, perhaps never reaching equilibrium. So learning is influenced by multiple factors whose relative weighting or significance changes dynamically in relation to the others. I am not aware of an analogue to 'cause-and-effect' that sums this up, apart from 'maintaining complexity'. Every attempt to reduce complexity in teaching and learning misses out significant influences, in my view.

Furthermore, a 'learning strategy' is an observer's description of a flow of actions in response to felt (and sometimes unrecognised) influences made up of desires, dispositions, propensities and habits in a mixture of affective, cognitive and enactive contributions. There is rarely a consistent 'strategy' over time.

The analysis becomes even more complicated if the notion of 'learning' is inspected closely. When, for example, does learning take place? I suggest not in classrooms, nor when reading a book or watching a film or engaging in some task. It might be useful to think of learning takes place during sleep, when the brain chooses to forget some impressions, thereby strengthening the likelihood of fresh actions to come to the fore in future situations.

We look at learners and decide they have learned something when something changes ... but as others have pointed out, tests don't really do this, as they are artificial probes which predict little about what actions a person will have come to the fore in the future (I don't say 'come to mind' because they sometimes come directly to body, that is actions, or to emotions, that is affect).

to measure learning we can use observations and observation checklist and interviewing or asking critical questions

I believe that is difficult to measure learning gain using a mathematical measure. I my profile, I observe the feed-back from the students any the way they try to apply to understand if it is any progress. Also, the pre-tests and the post-tests can be usefull.

The learning is complex since there are so many variables to play for perceived knowledge. In content to the area of assessments & the learners' ability to apply the new technology based knowledge for self satisfaction & growth in his/her life. The change in the classroom learning to Technological learning (the generic skills -problem identification and troubleshooting) is needed for the new initiatives to be undertaken. On the bases of selecting teaching strategy - which can enhance the skills of students to meet the challenges of life & student - uses the acquired learning effectively to solve problems of coming future - and in turn growth of student can prove the real effectiveness of teaching of the teacher.

C.Girija Navaneedhan Madras University India

One of the simplest way by which learning gain could be measured is to make the students do the presentations on the topic already discussed in class using technology or conventional way, by doing this it is possible to understand to what extent learning gain has taken place in the individual as he/she is totally involved in the presentation.

From an evaluation perspective, ask the students. They know what works. They can accurately assess what helps them grasp a concept.

Tests are merely testing one's ability to learn by rote or guess, wherever required.

Educational psychologists define learning as "change of behaviour". This assessment takes some time, even months/years...to be really meaningful.

Present methods of instruction mostly are compartmental i.e., one subject for itself. Can we change this approach? For example, a maths teacher has just taught ,say, decimals. THere is an immediate opportunity for the Physics and/or Chemistry teacher, to use it in their lessons on "speed/acceleration/distance covered" and/or ""finding the volume of Hydrogen gas liberated when H2SO4 is added to Zinc".

THis is just one example. WE could extend this to continuously test as well as make the students realise the value of Maths in education.

Dr MV Ananthakrishnan,mUmbai 9820217151`

If the teacher can grasp "cognitive apprenticeship", then assessments become a matter of the continuing "give and take" between student and teacher, the measure being the gradual arrival of the student at milestones on the way towards mastery readiness. While all that sounds a bit fuzzy and abstract, it certainly can be one subjective component of the educator's assessment toolbox.

Good morning!

Dear colleague

What I think now the question:

(my contribution)

First: Indentify other possible interferences that are different from the strategy that we are trying to assess;

Second: Isolate or control these other interferences as well as the effect of these;

Third: Establish time zero (the time is right before you start to implement the strategy in assessment);

Fourth: describe / measure the initial situation learning, ie: a moment of zero (AEM0);

Fifth: Plan procedures and frequency (can be several times (Mi): M0, M1, M2, M3, ...., Mn) learning evaluation before inducing strategy;

Sixth: Begin implementation of strategy evaluation;

Seventh: Track progress, have control over other possible interference and make the assessment at any time before defined.

Eighth: Make the assessment of the final moment of the strategy in question;

Nono: If the planning has provided ex post evaluations the closure of the strategy under test, then continue the evaluation (Note: Consider that the manifestation of the effects of the strategy assessment may be manifested after his ministry and slowly).

Tenth: Closed assessments, perform analysis using appropriate tools, chosen and applicable to the case (in general need to advisory / consulting).

Eleventh: Interpret and, based on existing knowledge, discuss and conclude

Twelfth: Give value and applicability of the results;

Thirteenth: It is suggested publishing / socialize the results

My views are

1. By good attendence in the subsequent classes.

2. Popularity of a teacher among students .

3. By giving them assignment (incorporating all what u intended to make them learn) which they would bring in next class .

4. Asking the key question to b answered randomly by one of the students during the teaching at the end of every important content u delivered to them which u want them it to remember.

5. Divide them in groups and ask them to ask each other about the topic and u observe the participation and asking question and in replying the answers.if every one in group is either questioning or answering -Learning gains are 100%.

6. Give them exercise to prepare projects which involves the applied use of the skills or contents you have taught them and then you assess the quality of projects in terms of how much content they could incorporate to be used for preparing .

When a student becomes excited with the process of discovery in radiology clinics and says how much fun it was, as well as realising that the more they know, the more they will see. Learning without understanding will often get a pass on a test but the student can not apply that learning to different clinical situations

ًWe have applied "Active-Learning Strategies" in short-term English language summer courses. Continuous assessment and evaluation was based on a single comprehensive test presented to students as a pre- and post test (placement and final). The results gained in both tests were compared where remarkable progress and regress were clearly shown.

Currently growing popularity in the USA is the "flipped classroom",wherein, the student "teaches" the content, and the assessments are analogous to those assessing the methods of the teacher.

"Every attempt to reduce complexity in teaching and learning misses out significant influences". I agree. The solution is then to teach with the odds on our side, i.e., make time and opportunities for diverse learners to develop.

What i have assessed from your question is that you are asking about the learning acquired after a particular learning strategy beyond what that is measured .i think we could devise the method for the assessment of that particular domain on which your particular learning strategy is based,

A specific learning or teaching strategy should embraced what specific methods of implementation be used while implementing such strategy. Then find out the effects of the strategy used in relation to student's learning behavior such as academic performance, attitudes and other parameters provided you need to identify the research statistical tool to determine any significant relationships. With this, your assessment as to the teaching approach would demonstrate any gains toward student's learning.

measure learning gain can be applied in many ways; having elementary students show-case academic work, test results and art work during a parent teacher conference was great to see how much they have learned and modeled understanding of their own learning. The student does the talking and the teacher facilitates the confersation between student and parent.

Learning strategies are key to learning. And learning happens consciously, not unconsciously; despite what 'religious' leaders like Krashen might say. Learning strategies are used consciously in working memory, the only place they can be used.

They are procedural knowledge (linked to experience and encyclopaedic knowledge and LTM), and reflect that learning is a process - not a spontaneous event, like some have claimed, or strongly implied that there is no process involved.

Learning may happen quickly sometimes, but drawing unwarranted conclusions that it happened spontaneously are absurd.

To observe learning gains as an effect of employed learning strategies, one must do some complex research work. First, you must know which strategies are being employed in any particular moment in the learning task, and learning tasks are diverse, so, and second, be sure to carefully operationallise the task, otherwise validity will diminish.

John Mason's 'cause-and-effect' theory (above) must be explained carefully. The way John talks about it is as if there is nothing happening between the stimulus moment and the end result, effect. This is too vague to mean anything. What is happening in this moment is what we must become aware of as teachers.

The stimulus enters working memory and lingers there a moment, sometimes for an instant, other times a few second. It is in this instant that the learner can deal with it, process it, determining its future. And it is the moment a learner employs a learning strategy. For instance, it may be a new word (e.g. written or spoken). What the learner does with it will be essential in its becoming part of their knowledge base, or experience. If nothing is done with it, it will disappear from working memory.

If it is a new word, the learner may choose a 'look-up dictionary' strategy to know its meaning. The new word may not be kept in working memory long enough during this process and she may need to look at the word again,may several times. Becoming aware of its meaning, though, may create a new link between the new word and its L1 meaning. What comes next is even more essential to the new word becoming part of the learners lexicon. The learner may write the new word several times, doing so will strengthen the link, the association, between the new word and its L1 meaning. Several other strategies can be employed to strengthen the association, until it becomes part of the learner's memory.

What a learner is doing when they are trying to learn a new language item, is what you will need to know to create a valid measure of learning.

Joel, watching a student perform a task in accordance with the course stated objective and measuring sucess factors is the assessment of the course. Whether that task being performed is responding to questions, completing an essay, developing a report, performing a hands-on skill, etc. Even if the student has the abiity to look up data on the internet, compile a report and the course objective dictates such action, it is still the assessment.

Now, beyond the course assessment, I build performance evaluations and measure the student, post trainng and on the job. Compare output to job requirements and if the data supports changing the course material, so be it. If the data dictates further training, then so be it.

The above paragraph references Corporate Training, but academic education can achieve the same by building relationships in the local business community and assisting that business community by gaining access to perform post education training. In cases where I have been sucessful, the local Human Resources Dept. was gratefull for the assistance.

I suppose students' questions (by their number and level) can help.

Learning outcome survey as well as the questions about the PILOs asked from the employers, parents, Alumni etc.

Hello

Depends on the type of learning we are measuring. Getting students to apply the knowledge gained to an authentic real life task can be a good indication of the learning gained.

I agree with Mr Boginnidis. I would add that solving a new but related problem, even if it is not "an authentic real life task" (which would be of course be a better one) gives you some indication about learning. We have used this approach in evaluations of Biology cooperative games. It is of course limited, because it is unlikely that we can assess the different things that different students can learn from a same experience. In my view, however, we can address the learning of very specific problems/concepts/skills, as long as we do not assume that students who did not learn what we are measuring "did not lealearn anything from the experience".

That’s a simple question, Carola.

Everyone knows, that if a box contains a compressed spring, that box is heavier than an identical box with it’s lid open, containing an uncompressed spring. That’s because E=mcc and the energy stored in the compressed string adds extra mass to the box. We also know that information is related to mass and energy.

So, if you want to know how much information has learned your student, you have to measure her/him (very carefully). Than give the lesson. After the class measure them again: the amount of information stored by the student during the lesson can be read on the scale. Extra care has to be paid to the inventivity of the students, because some of them will try to gain mass instead of knowledge (by drinking, eating, etc).

But wait a minute. It’s possible after all, that information – although related to mass and energy – could not interact with the Higgs field. Consequently – it has no mass. What a pity – such an elegant strategy to let down because the Higgs filed!

We must try to find an other approach to determine what is information. Some attempts has been made in a parallel discussion on the quality of information (here on RG) and some progress was noted. Than we learned that scientists can have different approaches to the same question – and those differences can be greater than the conditions found in brain versus soul, or different sides of a black hole’s event horizon – but at least I’m convinced that I have a friend on the other side of the rainbow.

So, maybe your question is not so simple, Carola.

Let’s try again. Learning is much more than memorizing. Learning is that cognitive process which enables the brain to embed in it’s neural networks new procedures. (So, zou can see on which side of the rainbow I am). These procedures can be related to physical activities (walking, talking) or mental activities (understanding – that is constructing operational mental models of our environment). The main objective of learning is to build a mental model to the relevant connections between the concepts – and to be able to apply these mental models to new subjects. So the best strategy to learn is to challenge your students to apply what they just learnt in new situations.

This strategy is a very old one and it can be observed in animals also. It’s called *play*.

As freshmen, we have been taught at anatomy the pelvic organs. We learned about the uterus, the rectum and so on, but later we had to imagine how these organs are covered with the peritoneum. So our professor called in front of the classroom an eminent girl and said: - Bow a little ahead and stand still like an uterus in anteversion! Now, dear students, we have here a beautiful uterus – told us the prof (weird jokes has to withstand who wants to be a doctor). Now, let’s take a rectum. You, there – pointed to a lazy and not too brilliant minded boy (who later become an eminent doctor) – come here and stand behind your colleague. Now, dear students, we have also a beautiful rectum here – told us the prof. Now here comes the peritoneum – and the prof took a large bed-sheet and covered them. That was the moment when laughing and joking we learned the walls of the Douglas pouch and the in interrelations of pelvic organs.

An other important feature of learning is critical thinking – to be able to see through the appearances and observe what is really important.

We had forensic medicine classes before graduation. We were preparing for our final examinations and the prof gave us advices to be exact in our observations and expressions – in everyday life also, not only in medico-legal circumstances. Quite boring. Suddenly a guy broke in the classroom, with some weapon stabbed the prof in his neck – more than a liter of blood was on instantly on the floor. The guy, wearing prison-garments rushed out from the classroom. The prof died quietly and rapidly in front of the petrified students. We knew, that many convictions were made in court based on the statements of our prof – but never imagined that it could be so dangerous to be a forensic expert. But the show went on. In ten seconds the police occupied the class. No one moves – ordered an officer. Everyone takes a sheet of paper. Write exactly your name, your location in the classroom at the moment of the murder and describe what you have seen. You have ten minutes!

After ten minutes the prof raised from dead and collected the papers. Took a look at them and announced that we have very faint chances to pass the final exam at forensics. No one noticed that the guy in prison-garment was our assistant prof – without his white coat. A large arsenal of stabbing weapons was described in our papers: no one noticed that the assistant had a frozen mackerel in his hand. Let’s don’t talk about the red ink and so on – we were really taught a lesson. And nobody died – it was just a play.

Of course, it’s not easy to elaborate plays which help the player to learn. But in my opinion a good teacher has to choose this strategy. And to assess the learning gain, the teacher has to observe her/his students playing.

That’s an interesting question, Carola!

Salve,

András

If you are discussing about teaching-labs, my answer would be yes. Let's pretend that the student is a physician, an engineer, a physicist or biologist, whatever. The teacher should describe the concepts of the instruments and materials present in the lab - and then start a contest between the students to synthetize, mount or determine the presence of something what for the lab is designed for. Students make their best when they have to prove their fitness to play. As a collateral damage, they accept even to learn to reach better results. A good teacher can sail on these winds of competition. Of course, a close supervision of the playing students is mandatory

The situation is completely different in case of *real* laboratories.

There sometimes mother Nature is playing with the experimenters : )

We can not stop giving assessments, tests. we also need to allow students to demonstrate their understanding in any of several ways such as; project based learning, design videos, develop presentations, design posters etc

I use a very simple mechanism to check what the students have absorbed. I have been evolving a template for feed back form. I ask them, on a regular basis, what new things they have learned, what skills they acquired, what concepts were new to them, what applications they foresee and how they can connect it up with topics they knew earlier and so on and so forth.

To my astonishment many a times their responses were totally off the expected mark.

This approach I believe can be very revealing in making assessment of any teaching strategy.

The question was very interesting as are some of the replies.I think we have to consider what it is we want students to learn in the first place.If it is just memorisation and retention of factual information then many standard tests will test the recall of these.However if we are seeking to develop other aspects in students learning such as critical thinking, creativity etc, these are much more diffcult to measure. There is also the possibility that assessment can stifle such aspects of learning. For example a student might only seek to learn aspects relating to assessment. I found A Kapoor's findings interesting in that students often produce un expected results.

Great topic, research has demonstrated that both teachers and students can be encouraged to apply the skill of effective or critical questions. Asking effective questions used to assess require us to involve others and obtain and analyze relevant information

MARIO LEON

The students can not do or handle something (1)

You pretend to teach accordingly (2)

Now students can do or handle something (3)

They have learned something (4)

¿ O.K.?

I agree with Mario

Students should sit for a placement test to determine their level and amount of knowledge they have, the same process should measure the amount of knowledge gained at the end of the given course.

Any course of study has immediate observable and measurable learning outcomes for those who take that course. However, whether we know it or not, there is more learning beyond what can readily be observed or measured at the end of the course. The heavily inclination of education system towards behavioural objectives not only limits possibilities for teaching but also for student learning. Under such as system of education, though teachers do not have non-bahavioural objectives for learning, students do learn beyond what has been selected in the form of behavioural objectives for them. And how a teacher can possibly assess learning which he/she did not think of while planning a course or lesson. I hope more insights will come from colleagues on this important question asked by Bruna.

The standard old-timer answer to thisd questionn is "you can't measure student learning gain". This answer has long been invalidated and does not move education forward. In TQM language "if you can't measure it, you can't improve it." A rational anaswer depends on teh time frame over which youwant to measure learning gain. Over multiple years and very generally, the matter is indeed very complex and I know of any really persuasive research that can tie what happens in fifth grade to overall learning on high school graduation because there are so many factors involved, and many are not or at least not easily quanitifiable.The question suggests a much shorter time frame and am much narrower focus. For example, it may have in mind the question whether an inquiry-based approach produces better learning than a lecture approach. There is lots of research in that area, and there is now increasing effort to find out whether college students learn some specific concept better woith different approaches. Tyhat ios called Siscipline Based Education Research (DBER). Last year the Natrional Researtch Council did an extensive and careful, lengthy report on this subject that you can get on-line.

To get back to your question though, how can you measure STUDENT learning gain unless you find out what a STUDENT has learned? There are numerous studies of studnet learning gains but assessing student learning gains in relatyion to a specific mode of insturction has to recognize that it takes fairly large numbers of students and control groups, and statistical analysis of results. What you will finds is a relatively small shift in the bell curve that describes student learning gains no matter how you measure it.

Medical sciecne did not really make much progress until it adopted the controlled experiemnt approach comparing different tratments. Statins are great but they too only shift the probability of adverse events. The effects of different student "treatments" are similarly best understood by looking at student learning distribution shifts.

You can't define any form of 'learning gain' in abstract. Unless you hypothesize some possible effects (learning gains) that might be produced by a particular strategy and test them/measure them you have no way to know that. Student assessment is one common way to measure 'learning gain' because students' ability to do or know something is a possible (common) effect that might occur as result of a particular learning strategy.

Francisco: "You can't define any form of 'learning gain' in abstract." Yes, that's why we need to operationalise 'learning gain'; e.g. use of relative clauses.

I know some people will say, and this a research project of mine but in relation to vocabulary learning, that such things as relative clause are an emergent aspect of a developing additional language. Think about Pinneman's developmental stages for a moment - in other words, we could predict, allegedly, when the thing we want to measure as a learning gain will appear and prepare for it.

Francisco: "Unless you hypothesize some possible effects (learning gains) that might be produced by a particular strategy and test them/measure them you have no way to know that." That's right; that's as I suggested - and that's why I said that it was complicated research. Complicated because often it won't be a single strategy that is involved in producing a single learning gain.

For example, if a learner meets a new word and immediately tries to guess its meaning - guessing its meaning will involve a lot of activity, and that's why research needs to dig deeper into strategy use - but fails at the attempt. Would we conclude that there was no learning gain? perhaps.

But then we see the student go for a dictionary, and looks up the new word in the dictionary. Even this simple act of using a dictionary is not at all simple. But soon the student exclaims understanding, would we conclude that this 'understanding' is a learning gain? Probably.

Francisco: "Student assessment is one common way to measure 'learning gain' because students' ability to do or know something is a possible (common) effect that might occur as result of a particular learning strategy." Yes, and assessment is normally summative rather formative. But I do not believe assessment as it occurs today is concerned with how a student acquired a learning gain, whatever it is. However, the assessment will presume that learning has occurred, evident by the answer to a question in the test - unfortunately this result will not assess whether the student really knows it or not if the student gives the wrong answer.

What Carola's question is asking is how to MEASURE learning gain from the use of a particular strategy. Of course, no one has as yet defined what Carola means by measure. Carola hasn't even defined it. But we assume she means qualitatively measure learning gain more than quantitatively, because she surely isn't concerned with counting learning gain, but more concerned with observing it. Why would she wanted to count learning gain from the use of a strategy? learning gain is singular not plural, surely?

Unless something is happening that I am not aware of in learning from strategy use that suggests the learner learns more than one thing from the use of the same strategy? Possibly, but highly unlikely. If I use a dictionary to look up a new word, what else do I learn from this activity other than gain insight into the meaning of the new word? Besides improved skill at looking up new words in a dictionary e.g. increased speed at finding new words in the dictionary and numerous other things associated with this activity. But surely not more insight into the meaning of the new word which was the point of using the 'look up the new word in a dictionary' strategy?

We are almost there, aren't we? If the learning gain from the look up strategy is insight into the meaning of the new word, then that is the learning gain. It surely is not the actual word because to learn the new word, the learner will need to engage or employ a bunch of the other strategies to actually learn the new word. So the learning gain 'measured' from the use of this particular strategy is 'the meaning of the new word'. One caveat: it is the dictionary meaning of the new word. So the definition of 'learning gain' will either be this one or some other one you want to use as its definition.

So, as far as I am aware, there are at least 60 pus vocabulary learning strategies in use in any given English as a foreign language learning context - usually formal learning contexts. If there are others, they must be unearthed from within the learning contexts they are employed in. And whichever ones are being employed to learn a particular aspect of language needs to be clarified and highlighted - e.g. are the same strategies being used to learn verbs as are being used to learn nouns? and so on and so forth.

It would help the discussion greatly if Carola could provide us with her definition of learning gain, if she has one.

As pointed out by a number of respondents, measuring gain especially those involved with strategies creates a problem for analysis. To determine which strategy or group of strategies a student used to learn a particular point requires some form of student self-report. Before any self-report can be generated, the researcher must provide a list of categories and criteria for the strategies he/she is testing or has taught. In other words, there must be clarification from the start about exactly what we plan to teach before we can arrange some form of assessment of those elements.

Finding a quantifiable means to measure qualitative learning continues to present difficulties for anyone attempting to measure student gains in the higher-order thinking skills--evaluation, application, incorporation of new information. The lower-level taxonomy skills (Bloom's Taxonomy) are easy to check through matching, cloze test items, straight recall. As soon as we begin to work with skills that require analysis and then synthesis, we are moving into areas hard to test. We can evaluate those skills in research papers or other forms of academic composition,but again there is a strong subjective element in evaluating any form of written language.

Another complication results from the use of multiple strategies interwoven with one another. Most learners combine a number of mental strategies one right after another or in combination with one another. Decoding a new vocabulary word may involve phonetics, context clues, drawing upon personal knowledge or experience of the context, not to mention the levels of acculturation in the reader. These strategies are then dependent upon the reader's language competence--is she reading in her first or second or even third language? I'd love to hear about strategies for these higher-order cognitive skills because our composition courses are growing more complex in requirements for strong academic writing, and that writing is dependent upon the higher-order skills that are just as hard to teach as to evaluate.

Aysha Bey

Learning form is mainly qualitative with some quantitative ( measurable)aspects

Some quantitative learning gain can be "measured"observing that after teaching students can do something that they cannot do before the teaching activity,Desing

carefully the " items" for both moments of the teaching learning process.

Mario

Aysha, you said: "To determine which strategy or group of strategies a student used to learn a particular point requires some form of student self-report."

Yes, and if you're going to measure some kind of gain from using them you must know which strategies are being used to learn different aspects of the language being learned.

You also said: "Before any self-report can be generated, the researcher must provide a list of categories and criteria for the strategies he/she is testing or has taught. In other words, there must be clarification from the start about exactly what we plan to teach before we can arrange some form of assessment of those elements."

Yes, and this is where there is a gap in the research into strategy use. There is some research into individual strategy use e.g. guessing from context, keyword, but it is very general, in that it doesn't specify which apsects of language are being learned when used. Someone has to do this research. I hope to do some when I finish what I'm doing now.

I think strategy researchers have gone quiet since Dornyei's (2005) criticism of it a few years back. In a nutshell, Dornyei said strategies don't exist. And this made the magical realist very excited, but just because Dornyei says they don't exist doesn't make it true. I think it was just strong criticism of strategy research because the existence of a language learning strategy can only be observed indirectly through self-reports.

Macaro (2006) was brave trying to situate them in working memory alone and nowhere else. But that is problematic because learners possess a memory of them, thus making them part of LTM. They are not one-off phenomena created at the instant they are being used. A language learning strategy is procedural knowledge e.g. look up a dictionary.

All that the researcher (and teachers) need to do is keep track of what is being taught and just ask students what they did to learn what the teacher taught. What needs to be clarified, is what learners are doing to learn what is being taught. The researcher then matchs the elements, if you will, the taught material with the strategies used to learn it. It will be hard work because each learning context will be different because of other factors, which must be highlighted as the research is being undertaken e.g. age, level, setting, teaching style, material, etc, etc.

The learning capability in your studets is given by the transformation of the information in knowledge requires the realization of a series of activities. The review of the theoretical work of Kolb (1984), Huber (1991), Kim (1993) and Nonaka and Takeuchi (1996) Argyris (1999) to identify such activities as information acquisition, evaluation, interpretation and reflection, the re - evaluation and integration in mental models and their dissemination and implementation.

The learning process starts with the collection of data for the individual who carries out this activity, it processes them, organizes, analyzes and interprets giving them meaning becomes information, on the interpretation, processes and synthesizes information that internalized through reflection, in the looks of your knowledge, skills, values ​​and principles-stored mental models. (Senge, 1990, Huber, 1991; Argyris, 1999)

Thus, once internalized, runs the activity of re-evaluation, where values ​​your contribution and interest for the present situation, you can use it directly on actions to be undertaken (single loop learning), but when it relates to the prior knowledge existing in their mental models and contextualized, turning information into knowledge that values ​​not only for the shares present, but retains it for future use, and to apply, processes, organizes, analyzes and interprets the results, whereupon, whatever the positive evaluation and negative; retained in the memory for the evaluation of new situations, we develop a double-loop learning.

You need to stablish the strategy in accordance what you are trying to they learn.... Know Why or Know How or Both!!!!!!!

Dea Miriam

Thanks for your comment, but ...

You said: "The learning process starts with the collection of data for the individual who carries out this activity, it processes them, organizes, analyzes and interprets giving them meaning becomes information, on the interpretation, processes and synthesizes information that internalized through reflection, in the looks of your knowledge, skills, values ​​and principles-stored mental models. (Senge, 1990, Huber, 1991; Argyris, 1999)."

The thing we teachers need to know is HOW do learners do what you have described here. How do students 'collect the data'? What data do they collect? How do they carry out this activity? How do they process this data? How do students organize the data? How do they analyze it? How do they interpret it? How do the student do they do reflection? And how do they internalize the new data?

This is what we need to know in order to measure what the student gained from all this activity. I mean, a summative test run at the end of semester certainly tells that there was some gain, but it doesn't tell anything else. This is good for people who just want such information, e.g. administrators. But it isn't useful for the type of research Carola and I are involved in. What data do students work with? It won't be a very large amount of data, otherwise they would experience information and cognitive overload. So what you told us, which is all very interesting, isn't very useful because it doesn't tell us what the actual data was that students worked on and the actual process involved in that activity, and doesn't bring us very close to being able to measure the precise gain from that activity. And that is precisely what Carola wants to do.

The information you have provided us is too general and doesn't help us at. It's all gloss and no substance; nothing to do with explaining these very general activities.

Carola, I dont know if it will help you, but you might like to take a look at what we are doing. Our research tries, first of all, to just describe what is happening in learning, from both the teachers point of view (the 'design footprint') and from the learner's point of view (a series of footprints of emergent learning), and see where and how they differ.

In our research so far, we have found that there is a much bigger, and much more interesting, gap between what the teacher imagines is going to happen (the planned learning strategy) and what actually happens (the learner's experience and actual strategy).

We dont for a minute assume that the two 'should' be the same - it depends on many things, and besides, learners differ from each other anyway. The interesting thing is to try to describe all these 'views' and then engage in a conversation about them, to see if and how we can make sense of them.

See: http://footprints-of-emergence.wikispaces.com/

Hi Roy

Emergence is the end product, the result of learning. But Carola wants to go deeper than that; she wants to measure the gain from using particular strategies, and so do I.

My question would be: how do you describe learning? How do you conceptualise learning?

It seems to be that you're labelling the end product learning. I could be wrong, so please clarify.

When you combine emergent and learning in the same phrase, what you are really doing is saying that 'emergent' is learning. But whatever emerges is the end product of learning, not the learning itself. Learning is not a noun in this case, it is a process; a verb; the thing done. And a process that involves interaction, as you say, which is good.

You said: "In our research so far, we have found that there is a much bigger, and much more interesting, gap between what the teacher imagines is going to happen (the planned learning strategy) and what actually happens (the learner's experience and actual strategy)." And that is often true. But that is the purpose of strategy research, to see what students actually do when they learn language. It isn't focused on the teacher. But its end result will inform teachers so teachers can design better learning activities to meet learners' distinctly individual approach to learning.

You said: "We dont for a minute assume that the two 'should' be the same." Neither do I. But they must eventually become strongly linked, if not, language teaching is going to go the way of the Dodo. If teachers are going to help their learners make great gains from learning activities in the classroom, teachers MUST be aware of precisely what their students are doing when they do learning, or interact with the learning materials/activies/teachers/peers in the classroom.

You also said: "The interesting thing is to try to describe all these 'views' and then engage in a conversation about them, to see if and how we can make sense of them." Actually, no; no it isn't. The kind of research Carola and I are doing doesn't need input from areas that aren't doing strategy research, or are using theoretical perspectives like 'universal' grammar or pure social theory. Neither of these perspectives have anything, or very little, to contribute to the discussion of gains from using specific language learning strategies.

You may well observe behaviour in the classroom, interaction with a textbook, with peers, with a game, with a teacher, which is all very social, but very little can be gained from observing such behaviour regarding the precise language learning strategies being used to learn language in these situations. Which precise language learning strategies were being used by the student when reading specific parts of the textbook? To observe that learning was gained from reading is not the same as highlighting which strategies were involved, and determining the precise gain from that particular set of strategies. Which is what Carola wants to do.

If y...

Hi Robert, I think we are talking at cross-purposes. I'm not sure I understand what you are saying. We are certainly not trying to use "theoretical perspectives like 'universal' grammar'. God forbid. We are engaged in rather more prosaic descriptions of how the learner experiences the course's implicit strategy, and how that might or might not relate to their own learning strategy.

But perhaps our interests and the way we think about strategy are just very different from yours.

I used to use a very simple way: metacognition: How would you use this information in real life? Give examples of this information applied to a different context? What do you need to apply it? What does the environment or patient of specific situation need to apply your ideas?.

OP - I cannot envision a way in which the organism in which the learning took place was not in some way the unit assessed. They are the units that "learned" there is no external unit to examine. You can collect he data via self-report, "test" or other work product, live demonstration of "learning", observation or other ways but each of these is an assessment of learning done with the student.

Roy Williams, here's a question for you, because it isn't obvious in anything you've said so far. Do you know what the 'learning strategy' is of each of your students? Being individuals, they are likely to have different ways of learning; similar in some ways, different in others. This is a general question I've asked you, not a very specific one, so I expect your answer to be very general.

Also, it is highly likely that the 'implicit' strategy of the course may well be very different from students' learning strategy; in fact, I believe they will be different because a single 'implicit' course strategy is unlikely to harmonise with more than one student's learning strategy.

Further more, a student does not have a single learning strategy, they use more than 62 general and specific strategies to learn language, with each general strategy containing multiple sub-strategies. And this is where more research needs to be done, finding out what all the sub-strategies are.

Yes, I would say that your view and understanding of langauge learning strategies is different from the field of language learning strategy research which is my field of study and research. Because you speak in quite generalised terms and hardly provide a defintion, even a clear definition, of the terms you use, it makes it hard for me to know exactly what you're talking about. This would help. :)

Alma Goodin, you, on the other hand, are closer to what we are talking about here.

Yes, metacognition is one of at least 9 categories of language learning strategies that i am aware of. Given that language larning is part of cognition, it makes sense that students will think about and observe their own learning behaviour.

You provided some intersting examples, some of which closer to strategies, some are not. The first was: "How would you use this information in real life?" This is an excellent question because it provides us some very important information on language, as well as insights into the student's language use strategies.

The next two questions, "Give examples of this information applied to a different context?What do you need to apply it?", are interesting because they provide use with insights into metacognitive strategies as well as cultural strategies. If the student is from a non-English speaking country then their social interaction and their culture are going to influence English language use, including learning strategies.

The last question is not so clear. "What does the environment or patient of specific situation need to apply your ideas?" You need to explain what 'environment' means, and what 'patient' means. What 'situation' are you asking about? There are so many, it is impossible to cover them all in language teaching. For instance, is the learner asking for directions to a hospital while standing on a busy street she's never been on before in a crowded city she's been in for a day and doesn't know anyone? Knowing exactly what these are will help us figure out the language learning strategies, if any, or language use strategies that are used and go from there.

Dale Pietrzak, by using the term 'organism' you have distanced yourself from learners by at least 200 hundred years. If you are talking people doing learning then you should use 'person' rather than the other. Yes, the learning did take place inside the learner, and that is our starting point but not the end point of our investigation.

You said: "They are the units that "learned" there is no external unit to examine." Well, yes; but we don't call people 'units'; we call them people. In strategy research, we are not concerned with examing 'external' phenomena, we are investigating internal phenomena. Investigating 'external' phenomena in relation to strategy research concerns what a learn is interacting with at the moment they are learning something (e.g. knowledge or a new skill), a person or a boo, for instance.

You also said: "You can collect he data via self-report, "test" or other work product, live demonstration of "learning", observation or other ways but each of these is an assessment of learning done with the student." Yes, these are very common ways to collect the data on learning; some are much better than others. Some researchers have access to MRI technology to observe the electro-chemical activity in the brain, but even this will not help language strategies researchers.

What we are concerned with is not so much with data collection, though it is central to what we are investigating. Self-report is still the best way to collect data on language learning strategies. Dörnyei (2005, p. 189) said, learning strategies offer us "unique insight into the mechanisms of the learning process", and are a significant factor promoting academic achievement. What Carola wants to know, and so do I, is exactly which set of strategies are involved in learning which bits of language (i.e. from a single word to the whole sentence it appears in). The 'look up a dictionary' strategy provides us with some clues, but we need to dig deeper. Try to think about everything that could possibily be involved in this simple activity, and then proceduralize them all. Firstly, the strategy must have been used many times before and therefore have a connection with long-term memory, otherwise the learner is going to be creating this strategy from scratch every time they meet a new word and want to turn a dictionary for information on it.

What Carola wants to know, and so do I, is exactly what is learned from using this language learning strategy (e.g. look up dictionary), and the many other strategies used in language learning. If a learner grabs a dictionary or uses the dictionary in her smart phone, to get information on a new word, what exactly is involved in this simple activity, and more importantly, what exactly is she going learn from this activity?

Learning is a controlled activity. Any "subject" will learn what they choose to learn. Given any experimental issue; the character, life style and integrity are what chooses what actual process is learned. It will also control what they choose to display what has been learned. With an unpleasant experiment they will "experiment" themselves to find out what will get them away from the unpleasantry; but once in their own comfortable arena will display what they choose to display; deciding factor being what will give them control over any given circumstance. An experiment may control the variables but once the response is ingrained; given another thought or variable; the response could change. How do you actually know what has been learned? Interaction on a comfortable level--find their comfort zone, talk to them, ask questions, then shut up and listen.

I have found that some of the best ways of measuring learning gain produced by a particular learning strategy is to allow the student to utilize what has been learned in

producing term projects. It is also useful to set up a hands-on laboratory and observe how the student manipulates equipment and integrates it in problem solving situations.