Student Assessment – Page 4 – Centre for Teaching Excellence Blog

Passion – Veronica Brown

This is Georgian Bay. North of Parry Sound.

As a long distance swimmer, it is my favourite place in the world to swim. Not only because it is fresh water, has fewer scary creatures than the ocean (no sharks or jelly fish here), is warm (usually a balmy 75F in the summer), and is relatively calm (unlike the English Channel). But it is also where I learned to swim.

But many changes have occurred in the Great Lakes since my Great Uncle and his father bought the island 100 years ago. The ’30s and ’60s were marked by extremely low water levels while the ’80s had some of the highest water on record. The challenge today, among others, is low water. You see, this is where I learned to swim.

In the background, you can see a small green bucket. That’s where our dock used to start. It was moored to the rock in the foreground. The one with the chain attached. There was enough water here to park a 14′ aluminum with a 35HP, our canoes, and, depending on the wind that day, a small sailboat.

Where I used to paddle, now there are trees.

And if you look closely at the island below, you can see the high water mark. The line where the rock colour changes from grey to beige.

Consider the size of the Great Lakes and then look at that image. The water is four or five feet lower. What has happened to all the water?

During the past several weeks, I have shared my exploration of the affective domain. Appreciation. Uncertainty. Honesty. Integrity. Ethics. Awareness of Limits. Open-mindedness. Commitment. Compassion. Cooperation. When I work with departments across campus, these themes arise regardless of the discipline or degree-level. These affective elements give our students a shared experience.

Now, why did I share the water story above? Not because I want you to know about dropping Great Lake water levels but because it is an example of an activity (and assessment) that you could try in your own class to encourage expression of ideas in the affective domain. Here are some suggestions.

End a class with a picture that relates to key themes in the class. Ask students to find connections between the image and the theme as part of a short assignment that functions as a review of the past few weeks and helps you assess their readiness for the next unit.
Create a 3-Minute Thesis contest in your class around a theme that requires a sense of more than just the knowledge and skills components of the course. If I had presented the above water story in class, it could be done in 3 minutes.
Encourage creative responses to assignments through flexible formats for submission. If writing is not a specific objective of the assignment, why not encourage video, poster, or presentations. A well-designed rubric could be used to assess all these formats.

And now, the title. Passion. It is yet another element of the affective domain. In all this need for measurement – grades, program evaluation, accountability – I worry that we are squeezing out the affective elements that are, I believe, critical to success, in school, the workplace, and in life. Several weeks ago I shared that the affective domain is a mystery to me. I think that mystery was tied to a fear of not “measuring it properly”, as if there was a single answer. Ironically, it is not unlike how my students must sometimes feel when faced with a complex problem, one in which there is no one single answer, one that cannot be measured to two significant digits.

Thank you for sharing this journey with me. I do not have a single answer because it does not exist. But I better understand the tools that can be used, it has reaffirmed my idea that we need to provide multiple opportunities to our students to explore these ideas, and that while they might not all fully embrace these affective elements, we can provide the activities, opportunities, and experiences, that can help them move in that direction.

Reducing student anxiety in the classroom — Karly Neath

Many educators are unaware of what anxiety is, how it affects their students, and what they can do to reduce it.

To cope with anxiety students:

Do not participate
Skip class
Avoid enrolling in classes with participation

These students may be missing out on learning opportunities.

From research literature in neuroscience, it is clear that stress and anxiety inhibit learning through powerful brain mechanisms. The stress response has evolved to avoid threatening situations, however it impairs new learning. By caring about students, and doing our best to reduce anxiety in the classroom, we can help utilize brain processes that contribute to learning.

What can we do to reduce anxiety in our classrooms and help our students learn and succeed?

Below are a few ideas from research conducted by Birkett and Shelton (2011) in neuroscience and practices in higher education:

Be predictable. Numerous studies have demonstrated the anxiety-provoking nature of unpredictable stressors. This does not mean that you have to give up flexibility or spontaneity in your classroom, but it means that you need to make your expectations explicit. For example, you specify the requirements for a research project but you do not need to specify the topic. This entails providing a clear, detailed and explicit syllabus at the beginning of a course, with the assignments described, due dates listed, and policies for late submissions. This can go a long way towards reducing stressful unpredictability. This is especially important at the beginning of a course when the students’ anxieties about the course are high.
Provide opportunities for student control. In neuroscience and stress research, lack of control is the second ingredient in creating anxiety. Control or even perceived control of a situation is capable of reducing the physical and psychological reactions to stress. Giving students opportunities to control some aspects of their experiences in our classes is an effective way to reduce anxiety. This might range from flexible due dates to late assignment policies to allowing students to select their own topic for a research project, or using a class poll to determine the next topic in class, to fully student-led projects for classes.
Trust students. Ken Bain claims that the most successful teachers trust their students. Bain writes “trust and openness produce an interactive environment in which students can ask questions without reproach or embarrassment” (p.142). Bain suggests that we can demonstrate trust by sharing a sense of humility with students, occasionally sharing paths in our own learning, expressing our own curiosity about learning, and setting an intention to share a classroom with students as fellow learners.

Each of these elements can help convey student caring. Each can be considered a characteristic of a classroom environment designed to reduce student anxiety, but a thoughtful and intentional combination of these aspects is required to be successful.

What strategies have you used to promote student caring and reduce anxiety in your classrooms?

References

Bain, K. (2004). What the best college teachers do. Harvard University Press.

Birkett, M.A., Shelton K. (2011). Participating in an introductory neuroscience course decreases neuroscience anxiety. Journal of Undergraduate Neuroscience Education, 10(1), A37-A43.

The One Hit Wonder – Veronica Brown

For several years now, all Ontario degree programs have been expected to demonstrate their students’ fulfillment of degree-level expectations as part of their program review process. There are different requirements at the undergraduate (UDLEs) and graduate (GDLEs) levels (more info is available in the Program Review area of the CTE Curriculum site). There are six UDLEs, which institutions could choose to use, adapt, or create their own and demonstrate how their own fulfilled the six required. At Waterloo, we adopted the six required UDLEs.

Depth and breadth of knowledge
Knowledge of methodologies
Application of knowledge
Communication skills
Awareness of limits of knowledge
Autonomy and professional capacity

But wanted to capture other elements that uniquely define Waterloo and added two more.

Experiential learning
Diversity

Consider the UDLE, “Awareness of limits of knowledge”, which is defined as

… an understanding of the limits to their own knowledge and ability, and an appreciation of the uncertainty, ambiguity and limits to knowledge and how this might influence analyses and interpretations. (Ontario Council of Academic Vice-Presidents in University of Waterloo, n.d.)

How do you measure “an appreciation of uncertainty, ambiguity and limits of knowledge”? I don’t think you can, not as it is stated here. You need to better define this UDLE as it relates to the experience your students have had and what you might expect them to experience. I find Eisner’s suggestions for expressive activities that will lead to expressive outcomes (see my February 18 blog) more and more appealing. Every student might have a different limit of their own knowledge but we can provide experiences that help them explore what those limits are. We can expose them to examples where a lack of knowledge has led to serious analysis and interpretation issues. We can give them labs or problem sets or case studies, etc., that have no single, right answer to help them gain comfort or an appreciation of uncertainty. The final outcome for each student might not be the same but we can control and define the activities that lead to that outcome.

The challenge, however, is to ensure that these experiences are scaffolded throughout the degree rather than being a one-hit-wonder. How can a student gain an appreciation of uncertainty if every question they are asked to answer has a single answer? How is that comfort or appreciation demonstrated by questions like “Will that be on the test”? How do we gauge student’s limits of their knowledge if we gather no evidence of the reflective process they use to review (or not) their performance in our class?

Next week, as I conclude this blog, I’ll explain why I posted the images throughout these blogs. Have a look at the images and try to guess where they are from, why I posted them, and what they have to do with the affective domain.

Veronica

References

University of Waterloo. (n.d.). The degree level expectations. Retrieved from https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/curriculum-development-and-renewal/program-review-accreditation/8-degree-expectations on March 4, 2014.

Taking one for the team – Jane Holbrook

Last week CTE had our annual professional development day and Mary Power, Samar Mohamed and I facilitated a short exercise with our CTE staff on team-based learning that introduced our group to the use of IF-AT (Immediate Feedback Assessment Technique) cards. We simulated a typical team-based learning activity (where group work is a critical part of the learning process) by having our participants take a multiple-choice test individually and then as a group. Known as RATs (Readiness Assurance Tests), the individual test encourages students to prepare for class, through a reading or other homework, and assesses their level of understanding of the concepts addressed in that pre-class work. A group of 5-7 students then take the same test again together, and in the process they discuss the questions and their answers and then come to a collective decision on the best answer to each question. If they select a wrong answer the IF-AT cards will indicate that and they can discuss and answer the question again … and again until they are correct. By using the IF-AT cards the group gets immediate feedback on their answers, see IF-AT First You Don’t Succeed…. Mary Power’s blog post about an activity using IF-AT cards and students’ reactions to this activity in a Pharmacy class.

Our CTE teams were carefully handpicked for diversity (longstanding CTE staff and new people mixed together) and we assigned them tricky questions on a range of topics (copyright, Canadian history, a math-based brain teaser) hoping that we could prove a key point about this type of team-based activity – that the score of the group is always better than the same test taken by an individual, rather than being dependent on the knowledge of the most competent group member. And even with our tricky questions this was true, people were learning from each other and not just being led by one strong group member. Larry Michaelsen, who has championed team-based learning for many years has tested group decision making in post-secondary courses and has found that when teams are engaged in “contextually relevant and consequential problem-solving” that the group will outperform the most competent individual (Michaelsen, Watson & Black, 1989).

Team-based learning activities using IF-AT cards are most effective when students are applying concepts to solve problems, analyze situations or data, or make diagnoses; when there may be many different approaches to answering the question but where there is one best, defensible answer. Teams should be thoughtfully assembled to include students with diverse backgrounds or skill levels and ideally they work together for a whole term on a series of these activities. The activities can also be used as a springboard to deeper class discussions and/or a preamble to more in depth group projects. The groups in a class should be working on the same problems so that after the RAT is completed, the groups can share their reasoning and conclusions with each other. There’s an aspect of competitiveness in the activities too, with groups vying to do better than each other on the tests. Another important aspect of the RAT process is that groups can challenge the instructor on the correctness of an answer – and that was certainly what happened during our PD day (we’re a fairly argumentative bunch).

See http://www.teambasedlearning.org/ for more information on team-based learning and some convincing testimonials about its effectiveness in large classes. Instructors report higher attendance and participation levels in these classes and importantly students are engaged and motivated.

If you would like to try an activity like this in your class, we have a good supply of IF-AT cards at the Centre to get you started, so please be in touch! Designing some team-based learning opportunities in your course can be a great way to flip some classes in your course too. See Course Design: Planning a Flipped Class.

Michaelsen, L. K., Watson, W. E. & Black, R. H. (1989). A realistic test of individual versus group consensus decision making. Journal of Applied Psychology. 74(5), 834-839.

Standards, Judgments and the Finnstep – Veronica Brown

Until today, I had never heard of the Finnstep. Now, it is streaming across social media as yet another debate emerges on the subjectivity (and potential corruption) of judging figure skating. It’s rather a timely debate as I work my way through Eisner’s (1985) The Educational Imagination. Having skimmed the first few chapters (which deal with social factors that influence the curriculum, an assessment of the state of education at the time, and some curriculum basics), I have landed at Chapter 6, “Educational Aims Objectives, and Other Aspirations”. He begins with a nice overview of behavioural objectives, how best to define and use them, their merits, and their shortcomings.

I am see great value in behavioural objectives because they provide specificity, measurability and, one hopes, objectivity in assessment. Those behaviours are often tied to standards, which Eisner describes as “crisp, unambigous, and precise” (p. 116). They are best when you know what the end product will look like. Consider swimming lessons. A swimmer must be able to fulfill all criteria for a given level (e.g., complete a front dive off a 1m board, swim front crawl 50m, swim back crawl 25m, tread water 1 minute, etc.) before moving to the next level. Swimming lessons exemplify a competency-based system based on well-defined standards. A swimmer does not move to the next level until all elements of the current level are completed. It is not a system where you can pass 60% of the elements and move on, you must pass everything or you re-do that level. The outcomes are very specific and the same standard is applied across all swimmers in that level. There is a little bit of room for judgment (e.g., the quality of the front crawl might vary) but most elements are fairly objective (e.g., you can tread water for one minute or you can’t).

The reality, however, is that not everything is as clear as your success swimming across a pool. Eisner acknowledges there is a place for such behavioural objectives, but exposes the limitations of this approach. He asks,

But what about the rhetorical force of a students’ essay ? What about the aesthetic quality of her painting? What about the cogency of his verbal argumentation? What about her intellectual style, the ways she interprets the evidence in a science experiment, the way in which historical material is analysed? Are these subject to standards? I think not.

But to say that such qualities cannot be measured by standards is not to say that judgments cannot be made about them. It is not to say that one can have no criteria through which to appraise them. Judgments can say much about such qualities, not by the mechanical application of prespecified standards, but by comparison of the qualities in question to a whole range of criteria that teachers or others making the judgment already possess. (p. 116)

This is the part that makes me nervous about assessing the affective domain. Before joining CTE, I was an instructional coordinator, managing very large classes. Leaving all this to “judgment” makes me nervous. Some of our courses had more than 40 markers. How could I be sure their judgment was the same? How could I minimize variance in that judgment? Can you really trust “judgment”? How many judging scandals have we heard about? Just how “fair” is judging? OK, I admit maybe I’m just in a slightly cynical mood having watched the Olympics all day only to hear tonight that people are talking about figure skating judges, again. Before worrying about these larger issues, let’s get back to the question of the day, how do we define outcomes related to the affective domain? Not to say behavioural objectives cannot be used in the affective domain, but I do think they are limited.

Eisner gives two alternatives to behavioural objectives, not to replace behaviour objectives but to complement them. First, he describes problem-solving objectives. One of the limitations of behavioural objectives he identifies is the need to know what the end product will be before students begin. But many of the questions we pose to our students do not have a single, clearly-defined answer. Instead, we ask them to solve a problem, with varying constraints. Even in the introductory programming course I taught, students solved the problems in different ways. My solution to the problem was not the only one and so I could not judge them based on whether they solved the problem my way, I had to develop an evaluation scheme that provided for that flexibility. An example Eisner shares is that of an architect, who must meet the constraints provided by the client, such as budget, site, and architectural style, but the product cannot be fully assessed until it is completed. He explains that, “what is known is the problem; what constitute appropriate solutions remains to be seen after the work has been done” (p. 119). Those constraints can help to form the criteria against which the solution is evaluated but there is no single solution to the problem.

The second alternative is expressive outcomes. Something I missed as I read the earlier part of the chapter was that he used the terms “behavioural objectives” and “problem-solving objectives” but calls these “expressive outcomes“. I tend to use the two interchangeably but he’s sees a clear distinction. Objectives represent the goals we have for our students, which lead to activities. For example, we might have a goal of evaluating students’ ability to analyse a budget, which leads to an activity in which we present a case study and ask them to analyse someone’s budget. But for expressive outcomes, we begin with the activity and the outcome is

essentially what one ends up with, intended or not, after some form of engagement. Expressive outcomes are the consequences of curriculum activities that are intentionally planned to provide a fertile field for personal purposing and experience. (p. 120)

I have read this chapter several times but it is only now that I realized my error. I didn’t pick up this subtle difference because I use objectives and outcomes with the same meaning. But what is truly different about expressive outcomes is that it is the activity that we plan, not the outcome. As an educator, I cannot foresee all the outcomes that activity might yield but I might have the sense that it has value. We trust that while each student might have a different experience, participation in the activity will have value. To achieve such an outcome, Eisner recommends we “have students engage in activities that are sufficiently rich to allow for a wide, productive range of educationally valuable outcomes” (p. 121).

I like this idea of planning the activity and allowing the outcome to come forth. But I am still uneasy. At the end of the day, I have to give my students a grade. How can I assess students who have a different outcome based on that experience. Is one outcome better than another? What is the criteria against which I judge this experience? How can I be assured that a panel of judges (or the 40 people marking the assignment) will yield the same result? Thankfully, we are not starting from scratch. There are many valuable tools that can help us to evaluate these experiences, experiences that I think are critical to the development of elements in the affective domain.

Veronica

Eisner, E. (1985). The Educational imagination : On the design and evaluation of school programs (2nd ed.). New York: Macmillan.

Begin with the end in mind – Veronica Brown

After writing the title, I could not for the life of me remember where that was from. So I Googled it. Turns out it’s Habit 2 from Stephen Covey’s (2004) 7 Habits of Highly Effective People. Don’t worry, I’m not going to write a blog on Covey. But his premise of having a clear direction to take makes sense with our ultimate goal of measuring the affective domain. It’s why I keep coming back to outcomes. If we want to develop certain characteristics in our students or at least encourage a different viewpoint, it is really important to be able to define those characteristics in measurable ways.

Continuing with Krathwohl, Bloom and Masia (1964) from last week, one approach to writing affective outcomes is to consider the outcome as it passes through the various levels from receiving to internalizing. Consider an outcome related to conflict and power. Depending on the level, the outcome might take one of these forms.

Be aware of the different power relationships that exist (Receiving) (NB – This is different than a cognitive outcome that might expect the student to be able to list various sources of power, such as positional or identity)
Demonstrate the use of a specific source of power in a given contextual setting (Valuing)
Defend the use use of a specific source of power in a given contextual setting (Organization – unlike Valuing, this higher level expects students to compare or synthesize the value, similar to analysis in the cognitive domain)

These examples demonstrate how the same construct can be defined at different learning levels. In lots of ways, the hierarchy makes sense as each level forms a building block for the next step. But this structured approach to defining (and then evaluating) affective outcomes is not without criticism. A concern that I sometimes hear related to defining outcomes is the loss of opportunity for students to find their own way. If we are too prescriptive, do we squash learning opportunities? There are many examples where these behavioural outcomes (i.e., specific, observable, measurable behaviours) makes sense, particularly in some of the lower levels (i.e, receiving, responding, and valuing). But what happens as you move into areas when these “behaviours” cannot be measured or, at least, cannot be measured in a consistent, valid way. For example, how would you evaluate a how a student values critical thinking or demonstrates behaviours related to ethics, diversity, humility, etc.?

While measuring these higher-level elements of the affective domain can be difficult, there might be associated behaviours that are observable. The design of the outcome, known as an expressive outcome, helps us describe or define the learning that might occur through experience, situations, and reflection. My blog next week will explore the work of Elliot Eisner (1985), who describes expressive outcomes as “the consequences of curriculum activities that are intentionally planned to provide a fertile field for personal purposing and experience” (p. 120). He continues by expressing that

it is perfectly appropriate for teachers and others involved in curriculum development to plan activities that have no explicit or precise objectives. In an age of accountability, this sounds like heresy. Yet surely there must be room in school for activities that promise to be fruitful, even though the teacher might not be able to say what specifically the students will learn or experience. (p. 121)

Food for thought.

Tree growing in rock

References

Covey, S. R. (2004). The 7 Habits of Highly Effective People: Powerful Lessons in Personal Change (Nov. 2013 edition). New York: Simon & Schuster.

Eisner, E. (1985). The Educational imagination : On the design and evaluation of school programs (2nd ed.). New York: Macmillan.

Krathwol, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of Educational Objectives: The Classification of Educational Goals. New York: David McKay.

The mysteries of the affective domain – Veronica Brown, CTE

I spent a lot of time during the past year thinking about assessment at all levels. In the Instructional Skills Workshop, we talk about pre- and post-assessments during a lesson to evaluate where our learners are before and after the lesson. During the Teaching Excellence Academy, we discuss assessment as it relates to the overall design of the course. Julie Timmermans and I presented on assessment for learning and asked participants to explore their own assessment philosophy at a recent workshop. Most of the curriculum work I support looks at assessment at a macro level as programs evaluate themselves.

Despite all this focus, the affective domain still remains a mystery. Back in the 50’s, Bloom and his colleagues created their Taxonomy of Learning and split learning into three domains: cognitive; psychomotor; and affective. Today, in course design and curriculum work, we simplify these to knowledge, skills, and values. The affective domain is meant to capture our values, beliefs, attitudes, and behaviours. It encompasses areas such as ethics, impact on society, diversity, creativity, humility, openness to failure, questioning, appreciating complexity, valuing teamwork, professionalism, exploration, critical thinking, etc. It ranges from simply knowing about a concept (i.e., being aware of a given phenomena) to fully internalizing that particular value.

If the affective relates to changing perceptions, attitudes and behaviours, how can we teach it? Moreover, how are we supposed to assess it? Can we actually change behaviour? Should we? We’re expected to. Just look at accreditation requirements from the past 10 years. It’s no longer enough to teach specific subjects for a certain number of hours. Now, we must demonstrate how specific outcomes have been fulfilled, such as life-long learning, professional ethics, awareness of limits of knowledge, etc. But more importantly, we want to. When working with departments on their curriculum, we often begin with an “Ideal Graduate Brainstorm” during which department members list the knowledge, skills, and values they expect an ideal graduate to embody by the time they graduate from their program. And the list of values is often just as long as the knowledge and skills.

Over the next few weeks, I will be exploring ideas surrounding assessment of the affective domain, specifically the use of media in teaching and assessing affective elements, Eisner’s Expressive Outcomes (Eisner, 1985), and what I’m going to call, the one-hit wonder phenomena. Throughout the blog posts, you’ll see one picture in each post. At the end of these posts on affective assessment, I’ll share the theme of the images with you (sorry – no prize if you guess the theme before it’s revealed).

See you next Tuesday,

Veronica

Bloom, B. S. (1956). Taxonomy of educational objectives : The classification of educational goals. New York: D. MacKay.

Eisner, E. (1985). The Educational imagination : On the design and evaluation of school programs (2nd ed.). New York: Macmillan.