Veronica Brown – Page 2 – Centre for Teaching Excellence Blog

Standards, Judgments and the Finnstep – Veronica Brown

Until today, I had never heard of the Finnstep. Now, it is streaming across social media as yet another debate emerges on the subjectivity (and potential corruption) of judging figure skating. It’s rather a timely debate as I work my way through Eisner’s (1985) The Educational Imagination. Having skimmed the first few chapters (which deal with social factors that influence the curriculum, an assessment of the state of education at the time, and some curriculum basics), I have landed at Chapter 6, “Educational Aims Objectives, and Other Aspirations”. He begins with a nice overview of behavioural objectives, how best to define and use them, their merits, and their shortcomings.

I am see great value in behavioural objectives because they provide specificity, measurability and, one hopes, objectivity in assessment. Those behaviours are often tied to standards, which Eisner describes as “crisp, unambigous, and precise” (p. 116). They are best when you know what the end product will look like. Consider swimming lessons. A swimmer must be able to fulfill all criteria for a given level (e.g., complete a front dive off a 1m board, swim front crawl 50m, swim back crawl 25m, tread water 1 minute, etc.) before moving to the next level. Swimming lessons exemplify a competency-based system based on well-defined standards. A swimmer does not move to the next level until all elements of the current level are completed. It is not a system where you can pass 60% of the elements and move on, you must pass everything or you re-do that level. The outcomes are very specific and the same standard is applied across all swimmers in that level. There is a little bit of room for judgment (e.g., the quality of the front crawl might vary) but most elements are fairly objective (e.g., you can tread water for one minute or you can’t).

The reality, however, is that not everything is as clear as your success swimming across a pool. Eisner acknowledges there is a place for such behavioural objectives, but exposes the limitations of this approach. He asks,

But what about the rhetorical force of a students’ essay ? What about the aesthetic quality of her painting? What about the cogency of his verbal argumentation? What about her intellectual style, the ways she interprets the evidence in a science experiment, the way in which historical material is analysed? Are these subject to standards? I think not.

But to say that such qualities cannot be measured by standards is not to say that judgments cannot be made about them. It is not to say that one can have no criteria through which to appraise them. Judgments can say much about such qualities, not by the mechanical application of prespecified standards, but by comparison of the qualities in question to a whole range of criteria that teachers or others making the judgment already possess. (p. 116)

This is the part that makes me nervous about assessing the affective domain. Before joining CTE, I was an instructional coordinator, managing very large classes. Leaving all this to “judgment” makes me nervous. Some of our courses had more than 40 markers. How could I be sure their judgment was the same? How could I minimize variance in that judgment? Can you really trust “judgment”? How many judging scandals have we heard about? Just how “fair” is judging? OK, I admit maybe I’m just in a slightly cynical mood having watched the Olympics all day only to hear tonight that people are talking about figure skating judges, again. Before worrying about these larger issues, let’s get back to the question of the day, how do we define outcomes related to the affective domain? Not to say behavioural objectives cannot be used in the affective domain, but I do think they are limited.

Eisner gives two alternatives to behavioural objectives, not to replace behaviour objectives but to complement them. First, he describes problem-solving objectives. One of the limitations of behavioural objectives he identifies is the need to know what the end product will be before students begin. But many of the questions we pose to our students do not have a single, clearly-defined answer. Instead, we ask them to solve a problem, with varying constraints. Even in the introductory programming course I taught, students solved the problems in different ways. My solution to the problem was not the only one and so I could not judge them based on whether they solved the problem my way, I had to develop an evaluation scheme that provided for that flexibility. An example Eisner shares is that of an architect, who must meet the constraints provided by the client, such as budget, site, and architectural style, but the product cannot be fully assessed until it is completed. He explains that, “what is known is the problem; what constitute appropriate solutions remains to be seen after the work has been done” (p. 119). Those constraints can help to form the criteria against which the solution is evaluated but there is no single solution to the problem.

The second alternative is expressive outcomes. Something I missed as I read the earlier part of the chapter was that he used the terms “behavioural objectives” and “problem-solving objectives” but calls these “expressive outcomes“. I tend to use the two interchangeably but he’s sees a clear distinction. Objectives represent the goals we have for our students, which lead to activities. For example, we might have a goal of evaluating students’ ability to analyse a budget, which leads to an activity in which we present a case study and ask them to analyse someone’s budget. But for expressive outcomes, we begin with the activity and the outcome is

essentially what one ends up with, intended or not, after some form of engagement. Expressive outcomes are the consequences of curriculum activities that are intentionally planned to provide a fertile field for personal purposing and experience. (p. 120)

I have read this chapter several times but it is only now that I realized my error. I didn’t pick up this subtle difference because I use objectives and outcomes with the same meaning. But what is truly different about expressive outcomes is that it is the activity that we plan, not the outcome. As an educator, I cannot foresee all the outcomes that activity might yield but I might have the sense that it has value. We trust that while each student might have a different experience, participation in the activity will have value. To achieve such an outcome, Eisner recommends we “have students engage in activities that are sufficiently rich to allow for a wide, productive range of educationally valuable outcomes” (p. 121).

I like this idea of planning the activity and allowing the outcome to come forth. But I am still uneasy. At the end of the day, I have to give my students a grade. How can I assess students who have a different outcome based on that experience. Is one outcome better than another? What is the criteria against which I judge this experience? How can I be assured that a panel of judges (or the 40 people marking the assignment) will yield the same result? Thankfully, we are not starting from scratch. There are many valuable tools that can help us to evaluate these experiences, experiences that I think are critical to the development of elements in the affective domain.

Veronica

Eisner, E. (1985). The Educational imagination : On the design and evaluation of school programs (2nd ed.). New York: Macmillan.

Begin with the end in mind – Veronica Brown

After writing the title, I could not for the life of me remember where that was from. So I Googled it. Turns out it’s Habit 2 from Stephen Covey’s (2004) 7 Habits of Highly Effective People. Don’t worry, I’m not going to write a blog on Covey. But his premise of having a clear direction to take makes sense with our ultimate goal of measuring the affective domain. It’s why I keep coming back to outcomes. If we want to develop certain characteristics in our students or at least encourage a different viewpoint, it is really important to be able to define those characteristics in measurable ways.

Continuing with Krathwohl, Bloom and Masia (1964) from last week, one approach to writing affective outcomes is to consider the outcome as it passes through the various levels from receiving to internalizing. Consider an outcome related to conflict and power. Depending on the level, the outcome might take one of these forms.

Be aware of the different power relationships that exist (Receiving) (NB – This is different than a cognitive outcome that might expect the student to be able to list various sources of power, such as positional or identity)
Demonstrate the use of a specific source of power in a given contextual setting (Valuing)
Defend the use use of a specific source of power in a given contextual setting (Organization – unlike Valuing, this higher level expects students to compare or synthesize the value, similar to analysis in the cognitive domain)

These examples demonstrate how the same construct can be defined at different learning levels. In lots of ways, the hierarchy makes sense as each level forms a building block for the next step. But this structured approach to defining (and then evaluating) affective outcomes is not without criticism. A concern that I sometimes hear related to defining outcomes is the loss of opportunity for students to find their own way. If we are too prescriptive, do we squash learning opportunities? There are many examples where these behavioural outcomes (i.e., specific, observable, measurable behaviours) makes sense, particularly in some of the lower levels (i.e, receiving, responding, and valuing). But what happens as you move into areas when these “behaviours” cannot be measured or, at least, cannot be measured in a consistent, valid way. For example, how would you evaluate a how a student values critical thinking or demonstrates behaviours related to ethics, diversity, humility, etc.?

While measuring these higher-level elements of the affective domain can be difficult, there might be associated behaviours that are observable. The design of the outcome, known as an expressive outcome, helps us describe or define the learning that might occur through experience, situations, and reflection. My blog next week will explore the work of Elliot Eisner (1985), who describes expressive outcomes as “the consequences of curriculum activities that are intentionally planned to provide a fertile field for personal purposing and experience” (p. 120). He continues by expressing that

it is perfectly appropriate for teachers and others involved in curriculum development to plan activities that have no explicit or precise objectives. In an age of accountability, this sounds like heresy. Yet surely there must be room in school for activities that promise to be fruitful, even though the teacher might not be able to say what specifically the students will learn or experience. (p. 121)

Food for thought.

Tree growing in rock

References

Covey, S. R. (2004). The 7 Habits of Highly Effective People: Powerful Lessons in Personal Change (Nov. 2013 edition). New York: Simon & Schuster.

Eisner, E. (1985). The Educational imagination : On the design and evaluation of school programs (2nd ed.). New York: Macmillan.

Krathwol, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of Educational Objectives: The Classification of Educational Goals. New York: David McKay.

Why Krathwohl helps make the affective a bit less mysterious – Veronica Brown

Let’s face it. I’m a mathie. Not in the Big Bang Theory way. It’s just that I tend to think in numbers. Numbers help me understand things and explain things. My teaching career reflects that. I taught general math at an agricultural college. Later, I taught introductory courses in computer science (computer usage and introductory programming). As a result, until recently, I haven’t given the affective domain a lot of thought. Not that it wasn’t on my mind. In my research, which has focused on different kinds of support systems (at the workplace and for students), it was a critical element – I just didn’t call it the affective domain.

As I discussed last week, the affective relates to our values, beliefs, behaviours, and attitudes. I am trying to figure out how to effectively assess these elements. Again, my early teaching career was spent teaching first year courses that focused primarily on knowledge and skill development. Topics that progressed in a fairly linear way. It didn’t make sense to ask students to create arrays or debug complex code before they had a decent understanding of the basics, such as variables, repetition, and selection. There’s no need to consider the affective if I’m just trying to teach them these basics, right?

Well, actually, it’s not that simple. As I think back to those programming courses, one of the basics we introduce is documenting your code (adding short comments throughout that explains the code to other users). These were small programs. Most of the code was self-explanatory, documentation wasn’t really necessary. But I wasn’t just teaching them about basic documentation. I wanted them to understand why it was important. I didn’t want them to write good documentation because they were expected to, I wanted them to value documentation so that it became an ingrained habit – where code just wouldn’t look right without a few lines of documentation.

But how could I instill that value with such short programs. Imagine writing a program that is only 20 lines long and having to write 5 lines of comments to explain it. Hardly seems worth it. And so I shared a story with them about why documentation was so important to me, hoping it might inspire them to find value in those comments. My first co-op job involved designing software for small robots. We used a language called Behavior Language (Brooks, 1990), which I had never seen before. I had four months of programming experience (one term of C++) and there was little available to help me learn the language. My best option was to read the code. The problem, I soon discovered, was that the last co-op student decided to use Alice in Wonderland characters for all of the variable and function names. So, instead of writing something like RobotTurnLeft, it said MadHatter.

Did sharing this anecdote mean I saw beautiful documentation in every project at the end of the term. No. But, it exposed my students to a reason to make an effort on documentation. At the very least, now it was on their radar. In 1964, Krathwohl, Bloom, and Masia expanded on the taxonomy of learning and explored the affective more deeply. They described five levels: receiving (awareness); responding (complies with set of values); valuing (expresses values through behaviour); organization (considers more critically – compares, synthesizes values); and internalizing (value system that is pervasive and consistent in behaviour). My students probably reached the awareness phase – they recognized that there was value in writing good documentation but might not have internalized it to the point of changing their behaviour.

As I consider how to assess elements of this domain, I come back to the idea of alignment among outcomes, teaching and learning activities, and assessment. So next week, I’ll spend some time talking about outcomes in the affective domain. By defining where I want my students to be by the end of the course, I can plan the activities and assessments appropriately. But the first step is to define the outcome. What would an outcome be in an area such as lifelong learning or diversity? What does it mean to value humility or critical thinking or creativity? How do I define an outcome related to ethics?

These are the questions I’ll be pondering for the next week. I’ll let you know what I find out.

Veronica

References

Brooks, R. A., (1990, April). A.I. Memo 1227. The Behavior language; User’s guide. Retreived Feb 4, 2014 from http://people.csail.mit.edu/brooks/papers/AIM-1227.pdf.

Krathwol, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of Educational Objectives: The Classification of Educational Goals. New York: David McKay.

The mysteries of the affective domain – Veronica Brown, CTE

I spent a lot of time during the past year thinking about assessment at all levels. In the Instructional Skills Workshop, we talk about pre- and post-assessments during a lesson to evaluate where our learners are before and after the lesson. During the Teaching Excellence Academy, we discuss assessment as it relates to the overall design of the course. Julie Timmermans and I presented on assessment for learning and asked participants to explore their own assessment philosophy at a recent workshop. Most of the curriculum work I support looks at assessment at a macro level as programs evaluate themselves.

Despite all this focus, the affective domain still remains a mystery. Back in the 50’s, Bloom and his colleagues created their Taxonomy of Learning and split learning into three domains: cognitive; psychomotor; and affective. Today, in course design and curriculum work, we simplify these to knowledge, skills, and values. The affective domain is meant to capture our values, beliefs, attitudes, and behaviours. It encompasses areas such as ethics, impact on society, diversity, creativity, humility, openness to failure, questioning, appreciating complexity, valuing teamwork, professionalism, exploration, critical thinking, etc. It ranges from simply knowing about a concept (i.e., being aware of a given phenomena) to fully internalizing that particular value.

If the affective relates to changing perceptions, attitudes and behaviours, how can we teach it? Moreover, how are we supposed to assess it? Can we actually change behaviour? Should we? We’re expected to. Just look at accreditation requirements from the past 10 years. It’s no longer enough to teach specific subjects for a certain number of hours. Now, we must demonstrate how specific outcomes have been fulfilled, such as life-long learning, professional ethics, awareness of limits of knowledge, etc. But more importantly, we want to. When working with departments on their curriculum, we often begin with an “Ideal Graduate Brainstorm” during which department members list the knowledge, skills, and values they expect an ideal graduate to embody by the time they graduate from their program. And the list of values is often just as long as the knowledge and skills.

Over the next few weeks, I will be exploring ideas surrounding assessment of the affective domain, specifically the use of media in teaching and assessing affective elements, Eisner’s Expressive Outcomes (Eisner, 1985), and what I’m going to call, the one-hit wonder phenomena. Throughout the blog posts, you’ll see one picture in each post. At the end of these posts on affective assessment, I’ll share the theme of the images with you (sorry – no prize if you guess the theme before it’s revealed).

See you next Tuesday,

Veronica

Bloom, B. S. (1956). Taxonomy of educational objectives : The classification of educational goals. New York: D. MacKay.

Eisner, E. (1985). The Educational imagination : On the design and evaluation of school programs (2nd ed.). New York: Macmillan.

The disposition to think critically – Veronica Brown

As I write this post, several Waterloo colleagues are attending the Society for Teaching and Learning in Higher Education’s (STLHE) annual conference. Seemed like a good opportunity to reflect on my experience at last year’s conference. STLHE was the first conference I attended when I joined CTE three years ago. It was held in Toronto that year, wrapping up on the same weekend as the G-20 summit. Last year, it was in Montreal, where I watched people march a block or two from our hotel as part of their day of protest.

Interestingly, the session that continues to haunt me was related to critical thinking. In her session, Beyond skills to dispositions: Transforming the critical thinking classroom, Shelagh Crooks, a professor at Saint Mary’s University, explored elements of the instruction of critical thinking, her goal to “raise questions in the participants’ minds about the purpose of critical thinking education, rather than propose clear solutions”(Abstract, para.3). She certainly fulfilled that goal in my case.

This idea of the disposition to think critically is what is really stuck in my head. Not just for critical thinking, but other areas of the curriculum in which we must move beyond the knowledge and skills of a topic and encourage thought in the affective domain. Consider themes such as health and safety, societal or environmental impact, ethical behaviour, integrity, teamwork, management, etc. As educators, what is our role in the development of our students? Take health and safety for instance. Is it enough for our students to know about hazardous materials, for example, and to have the skill to work with them appropriately? Or is there a third element, to actually value health and safety? To look critically at a situation, to question a current practice when appropriate, to have the disposition to continuously look at the lab through a health and safety filter.

And so here I am, a year later. I find myself with more questions about this disposition idea than answers. It is something I am exploring as part of the curriculum work I support. Many of us are wondering not only about teaching and learning in the affective domain but, as a next step, how to assess it. If developing this disposition is our instructional goal, how will we know our students have achieved it? If this is a question you are pondering, too, let me know, I’d love to chat with you about it.

WaterlooWinter — I realize this image isn’t related to critical thinking but I thought I would share it for anyone missing the snow…

Finding opportunities and taking new directions – Veronica Brown

Today is the Opportunities and New Directions (OND) conference at the University of Waterloo. I just learned that we’re expecting almost 160 participants, most from Waterloo but lots from other institutions, too. When I realized my blog date was the same as the conference, I thought it would be exciting to blog from the conference. Instead, as I began to think about what to post, I realized there are a few stories I’d like to share as a participant for the past five years. If you’d like to check out a play by play of the conference, there’s a Twitter feed that I understand will appear on the OND web page (the hashtag is #OND2013 – oh, I just noticed the Twitter feed is already there!). I’d like to share two opportunities and a new direction with you today.

Opportunity – Presenting my research

For a variety of reasons, disseminating my research was not on my radar as a grad student. I’m sure that sounds strange but grad school was a part-time affair juxtaposed with family (two toddlers), a full-time job and all the other things life throws at you. I was in a professional program that many would consider a “terminal” degree. As such, sharing my results beyond finishing my thesis wasn’t a priority. The OND conference gave me the chance to share my findings in, as a new researcher, a fairly safe place. It was just a short research presentation (I think it was 20 minutes with 5 minutes for discussion) but it was an important first step, which led to opportunities at larger conferences, both within my discipline and beyond.

Opportunity – Finding a community of practice

Having completed my degree, I felt a bit lost. My work did not necessarily afford opportunities for research and yet I was left with many unanswered questions thanks to the “future considerations” area of my thesis. How would I find time to answer these questions? What I discovered at OND that many of my colleagues shared the same tension. Research on teaching and learning was an add-on to a plate already full of scholarship, teaching, and service. Yet, there they were, sharing new findings or best practices. Their efforts encouraged me to continue to pursue research, particularly action research, despite a perceived lack of opportunity.

New direction – Becoming an educational developer

Finally, the most significant impact OND has had on my career was my decision to join the Centre for Teaching Excellence. Strange how we spend so much time “planning” our lives yet never realizing what hidden opportunity might arrive at our door suddenly. I think it’s fair to say that I stumbled into educational development, having every intention of staying with the Professional Development (WatPD) program for a very long time. Then a role as instructional developer came along and, having met some lovely CTE folks through the conference, I took a closer look and I’m so glad I did. I love my job, which shares many attributes with OND. I spend time with colleagues from across campus and beyond, who genuinely care about the success of their students and strive to make things better for them. I’m encouraged to use a scholarly approach in all aspects of mywork. And, most importantly, I get to spend my day focused on every possible aspect of teaching and learning.

Many, many, many thanks to everyone who has supported and contributed to the conference. Best wishes to my colleagues today who are presenting, collaborating, sharing, investigating, and just plain enjoying!

The hidden classroom

Public School, 1892. (Boston Public Library)

As you stand at the front of your class, about to start the session, what do you see? What do you notice about your students? Who sits at the front? Who sits at the back? Who is chronically late? Who rarely makes it to class? Who always sits alone?

While we see our students at class, what do we really know about them? There was once a student in my class who consistently fell asleep. As a very new teacher, I was really surprised and, at times, wondered if there was something I was doing wrong. Was my class really that bad? I also wondered about the student and the choice to come to class just to sleep through it. The reality is that I don’t know why the student was falling asleep. Perhaps it was from the fatigue of working two or three part-time jobs in order to pay for school. Possibly, the student’s role outside class was the main caregiver for a sick relative. Maybe 3:30pm on Monday in a darkened 250-seat lecture hall just didn’t work.

In course design workshops, we ask participants to explore the context of their classroom. As part of that reflection, we discuss factors such as the class size, required/elective course, the distribution of majors and non-majors, TA support, etc. But what about the hidden context, the part we rarely hear about.

The reality is that we don’t know what is impacting students’ participation or engagement in our classes. I remember speaking with a colleague several years ago who spoke about how she never attended class. Having never pondered the idea of missing class when I was a student, and knowing how successful my colleague was at school, I was surprised. In essence, the lecture hall didn’t work for her. It was much better for her to learn from the textbook and the assignments. She wasn’t “skipping” class, she was making a choice that made sense for her as a learner.

Given that there is no way for us to know about all the contextual factors that are impacting our students’ success in our class (and, of course, online), what can we do? One option is to consider universal design, which focuses on design for all. Although its roots are in designing accessible places and spaces for persons with disabilities, anyone can benefit from this design modality. Consider curb cuts. Although originally designed to support people with mobility limitations, as a mom, I can tell you they are a blessing when you are pushing a stroller, your toddler is on their tricycle, or your pre-teen is pulling a wagon full of flyers to deliver. There are examples all around campus of design choices that originally served one purpose but benefited many others.

The same holds true for our courses. Thoughtful design that focuses on engaging a broader spectrum of learners, or one that was made consciously to support a specific group, can have far-reaching benefits. Consider transcriptions of video content. While this text helps students with hearing impairments, it can also be really helpful for students whose first language is not English. I would have loved podcasts and video content as a student simply because I am not the best note-taker in the world and am sure I lost a lot by scrambling to take notes. With a podcast, I would have the chance to go back to catch what I’d missed.

There are countless examples of how we can make small changes in our course’s design, our teaching methods, learning activities, and assessments, that can provide deep learning experiences for a broader spectrum of students. If you would like to explore this idea, why not join Jay Dolmage and I next Friday, March 8, as we explore universal design. For information about the workshop, please go to CTE’s events listing.