Learning outcomes are often used as a way of making clear what a course expects from its learners. MAEL has just produced a position paper of this topic, outlining the background to the educational theory behind them and the way it intends to use them.
to read further click
MAEL Position Paper Two
The Use of Learning Outcome Specification to Facilitate the Assessment of Competence.
Competencies are taken as broad aims. However they are too broad to be assessable with any degree of consistency. Learning outcomes are more specific and have been defined (Ohio, n.d.) as “a statement in specific and measurable terms that describes what the learner will know or be able to do as a result of engaging in a learning activity.“ The use of learning outcomes (LOs) to act as a means of specifying measurable processes which may be taken as indicators of competence has become a part of educational orthodoxy. Some reservations about the behaviouralist aspect of concentrating on measurable features, and on the relative absence of taking student-centred learning-direction into account, have been expressed (Adams, 2002) but nevertheless the developing use of LOs does represent a paradigm shift towards:
- student-centred learning,
- outcome-focused approaches to teaching, learning and assessment and
- supporting the role of the teacher as a facilitator.
The first and third of these developments are particularly congruent to the Montessori approach.
Gagne (1984) proposed that LOs fell into five categories:
- intellectual skills which may be termed procedural knowledge. This is the understanding and use of rules and procedures, the distinction being that procedures consist of a greater number of sequential steps.
- verbal information or declarative knowledge. This is knowledge which can be demonstrated by a person’s ability to articulate it.
- cognitive strategies or strategic knowledge. This is the ability to bring to new tasks previously learned declarative and procedural knowledge and also the skill of knowing which to use.
- motor skills which are characterised by their gradual improvement through repetition.
- attitudes which are inferred internal states of mind.
Gagne did not at that time discuss what types of assessment processes would be appropriate for each category.
Kraiger et al (1993) refined Gagne’s scheme by recognizing three main categories:
- cognitive outcomes, which included verbal knowledge, knowledge organization and cognitive strategies. These are Gagne’s first three categories, though they put declarative knowledge s the first stage.
- skill-based outcomes. This is Gagne’s fourth category but they followed Anderson et al in distinguishing three stages. Skill acquisition involves firstly the evolution of declarative into procedural knowledge, enabling the reproduction of the desired behaviour. Compilation is the continuing practice and automaticisation is the mastery when the skill becomes fluid, accomplished, individualized and no longer requiring conscious effort.
- affective outcomes, which includes Gagne’s attitudinal outcome but also motivational outcomes.
They did discuss the types of assessment processes which would be appropriate to each category under the heading of ‘measurement implications’ as shown in table 1.
| Category | Measurement process(es) |
| declarative knowledge | multiple-choice, true-false or free-recall examinations(divided into speed and power testing). Speed tests assess the number of items answered in a given time and this tests reaction times. Power tests assess the degree of accuracy in unlimited time and this is appropriate when the consequences of error are high. Kraiger at all note that these are frequently conflated in practice (i.e. counting correct answers in a set time) but they argue this is mistaken. They also argue that such tests should be used fornatively early in a course as they test the lowest of the cognitive skills. |
| procedural knowledge | modal techniques involve assessing the learner’s cognitive structures by requiring judgements on the relationships between a previously defined set of core concepts. |
| strategic knowledge | probed protocol analysis involves a verbal analysis, conduted by an expert, of the steps involved in a particular task – this is akin to sustained shared thinking using questions like “Why would you run this test, and what would it mean if it failed?”
learner’s self-assessment of learning outcomes is also important. Questionning learner’s awareness of the level of proceduralization, the degree of additional learning required and the awareness of mistakes are appropriate. |
| initial skill acquisition | n/a |
| skill compilation | targeted behavioural observations or hands-on performance tests |
| skill automaticity | performing the trained task whilst simultaneously performing another task.
using a single task but having both normal and interference problesm. The latter is when one or more parameters are altered to make the task impossible or more difficult.
asking for conscious attention to the task to see if performance slows or is impaired. |
| attitudinal outcomes | self-reporting on an attitude scale showing both direction and strength of feelings (e.g. 5 strongly agree to 1 strongly disagree.)
Other indicators include measuring how important the feelings are to the learner’s self image, how often they think about it, how often they express their feelings to contacts, how likely they are to change their mind, how certain they are about the issue, how it makes them feel and how central the issue is to them (i.e. how many other issues come up when discussing it.) |
| motivational outcomes | self-assessment of confidence/commitment in achieving mastery |
Table 1: Kraiger et al’s suggested learning outcome measurement processes
Lim et al (2013) reviewed the field using the ERIC data base. 113 cases covered by 59 empirical studies were identified from between 1992 and 2006. They acknowledge the limitations of their survey population but found that only eight cases featured LOs at the performance level. They attribute this to a lack of guidance on the topic generally. They also drew attention to a critical dimension of evaluating learning outcomes, the differentiation of learning outcomes from transfer outcomes, i.e. do learners effectively transfer their learning to the workplace?
On that basis they used Bloom’s taxonomy, as updated by Bloom and Foreland, to propose a new framework for LOs in the cognitive domain:
- the knowing (or perceptual) level involves two sub-levels, remembering and understanding.
- the competency (or semantic) level involves four sub-levels, applying, analysing, evaluating and creating.
- the performance (or pragmatic) level involves the same four sub-levels.
On this basis they showed that only 7% of the surveyed studies involved performance level LOs. They argued that this framework would enable the proper identification of learning targets, whether foundational comprehension, internalization of competence or demonstrative work performance. However, they did not consider the question of what constitutes an appropriate measurement.
Learning Outcomes should (Mager, 1990) all have the following characteristics:
- The performance that is expected of the adult learner on the completion of their learning. This should be expressed in terms of action verbs – terms such as ‘know’ or ‘understand’ should be avoided as they do not communicate to the adult learner what is to be expected;
- The conditions under which the adult learner is expected to perform;
- The level of performance (‘criterion’ in Mager’s terms) which is required for the learning outcome to be deemed to have been fulfilled.
- Learning outcomes should never contain any reference to the learning process or the role of the facilitator.
Learning outcomes should also reflect the various levels of Bloom’s (Anderson & Sousniak, 1994) taxonomy of educational objectives through the use of appropriate action verbs, as shown below.
| Level | Appropriate verbs |
|
describe, define, list; report, relate… |
|
re-state, discuss, identify, explain, review… |
|
interpret, apply, practice, demonstrate, use, schedule… |
|
distinguish, differentiate, analyse, compare, contrast, examine… |
|
compose, plan, design, create, organise, manage, construct, prepare… |
|
judge, appraise, measure, value, assess… |
Table two: Appropriate descriptors for Bloom’s taxonomy
Drawing on these frameworks, with the practicalities of producing reliable and manageable assessment tasks, it is proposed that MAEL should standardise the wording of LOs, and the modes of assessment which may be used, in the following formats:
- Declarative knowledge (remembering)
“An ability to concisely/comprehensively describe/define/list/relate…
through a written theory examination/self-assessed quiz.”
- Declarative knowledge (understanding)
“An ability to concisely/comprehensively discuss/identify/explain/review…
through a written theory examination/a written coursework essay or report/portfolio compilation.”
- Procedural knowledge (applying)
“An ability to concisely/comprehensively interpret/apply/demonstrate…
through a written theory examination/written coursework essay or report or case study.”
- Strategic knowledge (analysing & evaluating)
“An ability to concisely/comprehensively distinguish/analyse/compare/examine…
through a written theory examination/written coursework essay or report or case study.”
- Strategic knowledge (creating)
“An ability to compose/plan/design/organise/construct (to a standard which would be usable in the workplace)…
through a written coursework essay/construction of teaching materials/construction of lesson and longer-term (project) plans.”
- Initial skill acquisition
“An ability to accurately and comprehensively demonstrate the steps involved in the acquisition of the skill of…
through a viva practical examination.”
- Skill compilation
“An ability to clearly demonstrate a developing facility with the skill of…
through workplace observation by a tutor or mentor.”
- Attitudinal changes
“An ability to reflect on own progress in their attitude to…
through an interview or discussion with a tutor mentor/self-assessed quiz/reflective diary
Skill automaticity outcomes, as proposed by Kraiger et al (1993), have been excluded as not being appropriate for initial training courses. Motivational outcomes, also proposed by Kraiger et al, have been excluded on the grounds of the difficulty of devising reliable measures.
References:
Adams, S. (2002). Using Learning Outcomes. United Kingdom Bologna seminar background report accessed 17.11.14 from http://aic.lv/bolona/Bologna/Bol_semin/Edinburgh/
G agne, R.M. (1984). Learning outcomes and their effects: Useful categories of human performance. American Psychologist 39(4): 377-385.
Kraiger, K. et al (1993). Application of cognitive, skill-based and affective theories of learning outcomes to new methods of training evaluation. J. Applied Psychology 78(2): 311-328.
Lim, D.H. et al (2013). integrating learning outcome typologies for HRD: review and current status. New horizons in Adult Education & Human Resource Developmen 25(2): 33-48.
Mager, R.F. (1990). Preparing Instructional Objectives. London: Kogan Page.
Ohio. (n.d.) Writing Learning Objectives: Beginning with the end in mind. Ohio University, accessed 7.1.14 from http://www.oucom.ohiou.edu/fd/writingobjectives.pdf
Category: Uncategorized