Category: Student Assessment

Peer Grading in Higher Education – Hadi Hosseini

phd020608s

Most educators regard peer assessment and peer grading as a powerful pedagogical tool to engage students in the process of evaluating and grading their peers while saving instructors’ time. This process helps improve students’ understanding of the subject matter and provides an opportunity for deeper reflection on the subject matter, accessing higher levels of Bloom’s thinking taxonomy.

Designing and distributing tasks and assignments for peer assessment should be as easy as assigning a few papers to each student and wait for the magic to happen, right? …. Not really!

As instructors, we care about the fairness of our evaluation methods and providing effective feedback. Yet, throwing this crucial responsibility to the shoulders of novice students who (hopefully) have just learned the new topic seems like an awfully risky behavior. There are two major concerns when it comes to peer grading; inflated (or deflated) grades and poor quality feedback. Both of these issues seem to be originating from the same sources of insincerity of the graders and the lack of effort each student invests on grading the peers [2]. To address these issues, recently researchers have raised the question of whether we can design a peer-grading mechanism that incentivizes sincere grading and discourages any type of secret student collusion.

The simplest possible design is to evaluate the quality of the peer reviews or simply put by “reviewing the reviews” [1, 3, 4, 5]. This procedure is bullet proof since no student can get away from a poor-quality feedback or deliberately assigning insincere grades. However, even though this technique may help us achieve the goal of involving the students in the higher levels of learning, in most situations this mechanism is either costly in terms of TA-Instructor time or simply impossible in large classes. In fact, this fully supervised approach defeats one of the main purposes of using peer grading by doubling or tripling the required grading effort: each marked assignment has to be reviewed by one or two TAs for quality.

As a partially automated solution, the system may randomly send a subset of graded papers to the Teaching Assistants (TAs) to perform a sanity check (instead of doing this for every single paper). In contrast, fully automated systems provide a meta-review procedure in which students evaluate the reviews by rating the feedback they have received [1, 3, 5] or by computing a consensus grade for assignments that are initially graded by at least two or three peer graders [5, 6].

In a different approach, students are treated as potential graders throughout the term and only those who pass certain criteria will be take the role of independent graders [6]. The premise is that once an individual reaches a level of understanding, he or she can now act as a pseudo-expert and participate in the assessment procedure. Of course, to ensure fair grading the system randomly chooses a subset of graded papers to be reviewed by the instructor.

Peer assessment is still in its infancy; nevertheless a number of researchers in various disciplines are developing new techniques to address the critical issues of efficiency, fairness, and incentives. Each of the above methods (and many others that exist in the peer-grading literature) could potentially be adopted depending on course characteristics and intended outcomes. I do believe that such characteristic, to very least, must include the following:

  • Skill/knowledge transferability: Do marking skills and the knowledge of a previous topic  automatically transfer to the next topic? If so, are they sufficient?
    For example, an essay-based course may contain similar marking guidelines in all its assignments and training students once could be sufficient in transitioning students to effective peer graders.
  • Course material and structure: How are the topics that are covered in the course dependent on one another? Is the course introducing various semi-independent topics, or are the topics all contribute to building a single overarching subject.

What do you think? Have you ever used peer-assessment in your classes?

 

References

  1. Cho, K., & Schunn, C. D. (2007). Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system.Computers & Education, 48(3), 409-426.
  2. Carbonara, A., Datta, A., Sinha, A., & Zick, Y. (2015) Incentivizing Peer Grading in MOOCS: an Audit Game Approach, IJCAI.
  3. Gehringer, E. F. (2001). Electronic peer review and peer grading in computer-science courses.ACM SIGCSE Bulletin, 33(1), 139-143.
  4. Paré, D. E., & Joordens, S. (2008). Peering into large lectures: examining peer and expert mark agreement using peerScholar, an online peer assessment tool.Journal of Computer Assisted Learning,24(6), 526-540.
  5. Robinson, R. (2001). Calibrated Peer Review™: an application to increase student reading & writing skills.The American Biology Teacher, 63(7), 474-480.
  6. Wright, J. R., Thornton, C., & Leyton-Brown, K. (2015). Mechanical ta: Partially automated high-stakes peer grading. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (pp. 96-101). ACM.

Ipsative Assessment, an Engineering Experience

How will students demonstrate learning? What types of Assessments will you use? https://www.flickr.com/photos/gforsythe/

Last month I attended and presented at the Canadian Engineering Education Association Conference that was held in McMaster University.  It was a wonderful learning experience that allowed all participants to connect with engineering educators not only from Canada, Continue reading Ipsative Assessment, an Engineering Experience

The Debate Over Accommodations: Making Space for Mental Health in the Classroom — Sarah Forbes

Equality doesnt mean EquityMost professors are aware of their responsibility to accommodate students with disabilities in their classroom.  Many of them may not be as aware that this responsibility extends to students with a documented mental health condition as well. While mental health issues are often invisible, they create many difficulties for students in academia. By allowing reasonable accommodations, instructors can encourage these students to reach their full potential.

What do these accommodations look like?

Accommodations can take many forms. For students with difficulty focusing in crowded environments due to issues like ADHD, alternative exam locations allow them to write their exams in smaller rooms. Often other resources are used alongside alternative exams such as peer note-takers, where a student will take lecture notes on behalf of someone who may not be able to multi-task or focus as well. For students with depression or anxiety who may have difficulties with motivation, short negotiated extensions on assignments may help them to manage their time. Other changes in assignment structure can be negotiated with specific students as well, such as changing a public speaking presentation to a prerecorded lecture for a student with social anxiety. In any of these cases, accommodations require the student to document their issue with AccessAbility Services. For extensions and other personalized changes in exam or assignment structure, the student and instructor can collaborate to find a solution that fits both the assessment needs of the instructor and the issues faced by the student.

cartoon accommodationsThere is some controversy over the idea of accommodations that change assignment structure or allow extra time. However, as illustrated by the cartoon accompanying this article, expecting all students to achieve the same results based on their different abilities and starting points in life is unrealistic. Accommodations given to students who need them simply gives them the chance to truly show the work they have put into the class and the knowledge they have gained.

 The debate over content warnings

The most controversial accommodation by far appears to be the “trigger warning” or “content warning.”  The idea is exactly the same under either name. For controversial or difficult topics that must be discussed in class, the instructor will present a short warning prior to the introduction of the topic. This allows students for whom this topic may be upsetting or trigger flashbacks/anxiety attacks to choose how they interact with the subject matter. This is especially important in the arts, where controversial discussions are the backbone of many classes. While discussions about rape culture and sexual assault on campus are important and help to eliminate stigma as well as introduce students to new viewpoints, they can send a student who has survived sexual assault into a debilitating panic attack, forcing them out of the conversation. Many professors view these warnings as an escape route from difficult conversations and assignments. Anyone can claim to be “triggered,” they argue, and then skip out on important lecture material and assignments with no penalty. However, the content warning does not mean that the material is not mandatory – it just allows students to be prepared for the discussion. If a student knows that they will not be able to handle the material, they can then approach the professor privately and negotiate any other accommodations necessary.

These warnings are easy to add to a syllabus. They can be placed in the class schedule, next to lectures in which topics such as sexual assault, eating disorders, violence, and any other potentially graphic or disturbing topic are discussed. The discussion culture of university is incredibly important for allowing students to experience many different ideas and viewpoints – but by including upsetting subjects without any warning it can alienate many students with mental illnesses, leaving them out of a discussion that often focuses on them. The voices we most need to hear when talking about some of these issues are from students who have personally experienced them. To encourage them to speak up, we need to keep our classrooms welcoming.

‘Seeing beyond the self’: Using reflective writing as an assessment tool – Dan McRoberts

82648702_800bccf11eFor many years, post-secondary educators have been encouraged to move outside the classroom and create transformative learning experiences for university students. Field courses, service learning, and cooperative education are all examples of the kinds of programming that have become increasingly common and popular amongst undergraduates looking to incorporate some unique and useful experiences in their university careers.

Despite the popularity and growth of transformational learning, questions persist about the most effective ways of assessing student learning that results from these experiences. Experiential learning is hard-to-measure so traditional assessment measures often fall short of the mark. Reflective writing is often at the heart of assessment measures employed to qualitatively measure transformative learning, with self-evaluation, and journaling common assignment formats. There are significant challenges with using reflection to assess students, related to the highly personal nature of the transformations being recorded. Pagano and Roselle (2009) find that there is usually little clarity or systematization involved in using reflective practice. What is involved can vary substantially between courses or instructors. Also, reflection tends to rely on students’ own accounts of events and responses and as such it is very hard to discern if learning has indeed taken place. Woolf (2008) also identifies concerns with the confessional ‘dear diary’ approach to reflective writing, as he aligns this with highly personal change or transformation. Given that much of the possible value in transformative learning comes from the opportunity to ‘see beyond the self,’ the question becomes how to design assignments and assessments that will help students develop this awareness and critical reflexivity.

Sometimes it helps to divide the task into two parts, one which focuses on personal development and the other that relates to key academic objectives or themes. Peterson (2008) profiles a service-learning course assessment that combined personal narrative with more academic analysis. Students were asked to prepare two journals with these respective foci, rather than being asked to write whatever came to mind. Doubling the student (and instructor) workload may not be the ideal solution, but fortunately there are models for designing reflective writing that can assess several components in the same assignment.

One is the DEAL model developed by Patti Clayton, which involves students Describing their experience, Examining the experience in light of specific learning objectives, and Articulating their Learning. The assignment is guided by specific prompting questions that encourage students to complete these various tasks in their reflective writing, from the who, what when and where of an experience (describing learning) to more detailed prompts about what was learned and how (examining and articulating learning).

Another, perhaps less well-known, approach is the ‘refraction model’ proposed by Pagano and Roselle (2009). Refraction tries to incorporate critical thinking into the process of reflection to encourage students to move beyond their own perceptions and consider how to address problems or scenarios they may have experienced in their course. This process begins with reflection and activities that are common to the assessment of transformational learning outcomes. From here, however, the authors propose using critical analytic and thinking skills to refract this knowledge and generate learning outcomes. The first stage – reflection – involves asking students to log events and journal reactions. The critical thinking phase asks students specific questions about these experiences, and the refraction stage invites them to suggest solutions and interact with others and their ideas about the same events or issues.

Whether or not the DEAL approach or refraction model are applied, it is useful to remember what Nancy Johnston from Simon Fraser University says about reflection as a means of assessment. “We are looking for evidence of reflection, which means that students are challenging their assumptions, appreciating different points of view, acknowledging the role of power and discourse, the limitations of their conclusions and in short moving from black and white understandings towards recognizing varied shades of gray.”

(image credit: Paul Worthington)

Teaching and All The Feels — Aimée Morrison

This post has been reprinted (with permission) from the Hook & Eye blog. 

feelingsI have that nervous feeling in my stomach again–those butterflies, or that flip-flopping feeling, a vague nausea and discomfort. It’s final paper time, and while I’m not writing any myself, I assign them. And it makes me incredibly nervous, shepherding my grad students through their projects’ various stages. I want so badly for them to succeed; I worry so much about how tired they look, or frustrated, or, worse, how silent they get.

Teaching. It’s very emotional.

Over the past ten years, as I have wrestled with my teaching persona, teaching practices, teaching goals, one thread runs constant–trying to manage my own emotions. I started out perhaps over-attached to results: if a student did poorly on an exam, say, I would take all that on as a personal failure of mine. There was a lot of crying. It was not helpful. I tried to learn to not take it as a personal affront when students were often absent. I had to learn that sometimes it’s not about me when students look bored and tired every single semester once week 7 rolls around. I was very emotional but about the wrong things and it was gruelling and ineffective.

Then, for a while, I tried too hard to swing the other way. Teaching became more contractual and transactional. I would lay out some rules and try to enframe the teaching situation as mutually beneficial but largely impersonal: trying to protect my own feelings and recover from my over investment in outcomes that were beyond my control, I tried to take my feelings out of the classroom. But even as I tried to pull away from my misguided mother hen tendencies, my students still sometimes cried, or got angry, and I was doing them a new disservice by trying to deny them that reality.

Real learning is transformative–and all transformations are fraught with fear and excitement and loss and gain. The crucible of the new self is necessarily hot; it burns. Teaching, I find, is as emotionally and personally wrenching as learning is, and I need to find new ways to incorporate this reality into my work, even as I create some boundaries for myself and my students.

For me this starts with acknowledging that I care a lot about the material I teach, and I am, actually, really invested in having students learn it. This might be an ethical and respectful methodology for research on the internet, or it might be the history of the www, or it might be the difference between technological determinism and social construction, or it might be the design theory of affordance, or it might be feminist pragmatics, or it might be how to make a daguerrotype. It really matters to me a lot that students understand these things and, crucially, see the value in them.

When I teach, I necessarily make myself incredibly vulnerable to my students, by reaching out to them with ideas and sources and methods and assignments and illustrations, and asking them to hold on. It requires, I find, an incredible outlay of empathy for me to try to find where the students are at already, intellectually and ideologically or whatever, and go to them there to ask them to come with me to where the class is designed to take us. It is rarely the case now that I teach just from what I want to say; I’m always doing this sort of dance where I try to figure out the emotional temperature of the room, poll the interests, prod the knowledge base, and figure out a context-specific approach.

The best way I can find to describe it is this: It feels like being on a first date with 40 people at the same time. Every single time I teach.

To be clear, I’m not in it to be “loved” or even liked. I’m trying to put myself–Aimée Morrison, the situated human being–behind the ideas but of course teaching and learning are human acts so I’m still there. Reaching out, trying to get in 40 heads and hearts at the same time, trying to shift something in someone’s understanding: “even though this was a required course, it was surprisingly useful.”

I begin finally to understand that this is why teaching days are so gruelling. Why if I teach in the morning, I’m not going to be writing in the afternoon. It’s the interpersonal work, the mutual vulnerability, the work of empathy, the work of caring. In my worst moments I want to withdraw–I say things like, “If they won’t do the readings, to hell with them.” But really, I am usually overwhelmed with the sheer importance of the work I’m trying to do, and how much I care and how much I care about having students come to care about what I teach as well. I’m not naturally empathetic and I’m much more inclined to try to structure the world into rule-based interactions we can process cognitively and rationally, so the empathy required of teaching is not something I come to naturally. It’s something over time I’ve come to learn is crucial: learning is transformative, and thus scary and personal. Teaching must be these things too. All the feels.

— Aimée Morrison

Image courtesy of Nic Walker.

Integrated Testlets- What are they?–Samar Mohamed

IF-AT
Sample IF-AT card

Last month I attended the annual Society for Teaching and Learning in Higher Education (STLHE) 2014 Conference that was held in Queens University, Kingston. It was an excellent opportunity for me to learn from colleagues across Canada and exchange ideas with them. One of the workshops that attracted my attention was facilitated by Aaron Slepkov and Ralph Shiell from the Dept. of Physics at Trent University. In their workshop, Ralph and Aaron focused on their newly developed testing technique: “Integrated Testlet (IT)”. The presenters started by talking about the benefits of Constructed Response (CR) questions, a common term for questions where students must compose the answer themselves, and how these types of questions enable instructors to gauge their students’ learning. CR questions also allow instructors to give part marks to partially correct answers. The presenters also commented on the trend to switch from CR questions to Multiple Choice (MC) questions in the field of Physics due to increasing class size and the resulting contraints on time and personel resources. However, traditional MC questions don’t allow for part marks or, more importantly from a pedagogical standpoint, enable the instructors (and students) to know where the students went wrong. The integrated testlet method is different in that not only does it allow the students to keep trying each MC question until they get the correct answer, “answer-until-correct question format” enabling the granting of partial marks, but student do not leave the question without knowing the correct answer enabling them to move on to the next integrated question. The method presented changed complex CR physics questions into IT questions. The IT method is based on a traditional testlet, which is a group of MC questions that are based on a common stem (or scenario). In an IT, an answer to a question (task) leads to the next task procedurally, and in this way the students’ knowledge of how various concepts are connected can be assessed. Therefore, items in an integrated testlet are presented in a particular sequence in which the answer for part (a) is used to solve for part (b) and so on. The IT rely on the use of an “answer-until-correct response format”, where the students can keep on making selections on a MC question until the correct response is identified and can be used in subsequent related items. The presenters used the Immediate Feedback Assessment Technique (IF-AT) to allow the students to do several attempts and to get part marks for their response. For more information about IF-AT cards, see Epstein Educational Enterprises website. Moreover, for a sample application of the IF-AT cards at the University of Waterloo see the CTE blog by my colleague Mary Power, The faculty of Science Liaison. In their published paper, the presenters explain the method they have used to transform CR questions to IT questions and analyzed the students’ responses for both question types; it is a very useful and interesting reading that I recommend for instructors thinking about this method.

Very Varied Backgrounds – Maxwell Hartt

student facesAs students get farther along in their program, they are often streamed by discipline or sub-discipline. This creates an environment where the students know each other well and are entering the class with very similar pre-requisite academic experiences. They’ve taken many of the same classes, from the same instructors and will even have many shared experiences from outside of the classroom. However, when students first begin their post-secondary education, they come from all different types of schools and varied academic backgrounds. Furthermore, the larger introductory classes pool different programs, departments, and even faculties together. This makes for a potentially difficult teaching and learning experience.

How can an instructor best prepare themselves to deliver course content to very varied students? How can one ensure that the material is at the appropriate level when some students are far more advanced then others? How does one even tailor examples, projects, tests and other materials to be relevant and captivating to students with a wide range of academic interests?

I have had the good fortune to teach a class of this sort for the past two years. An introductory, (mainly) first-year statistics class for the entire Environment Faculty. This means that there are students from Geography and Environmental Management, Planning, Knowledge Integration, Environment and Resource Studies, Environment and Business, Geography and Aviation, Geomatics, and International Development. Among the slew of differences these programs boast, entrance pre-requisites is one that looms large. Will a student in Geography and Aviation really want to concentrate on the same examples as one from International Development? Maybe. But in my experience, it has not been the case.

The first year I taught this course, I concentrated on the conceptual material and then simply used easy-to-grasp but not overly provocative or discipline-specific examples. Basically, I shot for the middle ground. Something that everyone could understand but that, perhaps, no one would be writing home about. This strategy worked, but only in the sense that the students comprehended the material (more or less) and could relate somewhat to the examples used to get across the concepts. I wanted more than that. Being a first year student is an exciting time to be discovering new knowledge, sharing with others and beginning to forge your way into academia. I wanted to be able to harness this more.

To do so, I put the material into the hands of the students. The overarching concepts and methods remained the same, however, they controlled the specifics of their work. The course was restructured around a group project (rather than midterms and exams) and the students worked with TAs to develop their own research areas, research questions, hypothesis, and methods. Instantly, there was an overwhelmingly positive response. Attendance at optional tutorials went up, office hours had lines down the hall, I could overhear students excitedly talking about their topics and debating the merits of different approaches. In short, they were excited. And so was I. By giving them the ability to steer their education, it demonstrated my confidence in them, even in first year, to get motivated and engaged in their studies and in their field. It was at times difficult for myself and the TAs to provide topic-specific feedback on their reports, but even this barrier had its benefits as it encouraged them to become ‘experts’ in their chosen topic. They developed the research question, they performed the literature review and then found the sources.

We even took the creative element one step further. We allowed them to create stories for where their data (which was distributed without labels or metadata at random) came from, what the variables were, how it was collected, when it was collected, etc. This proved to be a fun exercise as students debated the potential bias or accuracy of different fictional data collection techniques.

In short, rather than trying to find a safe middle ground, the students streamed themselves and then taught each other. We provided the conceptual and course-specific content and techniques and they filled in the rest. And boy was it exciting.