Peer Grading in Higher Education – Hadi Hosseini

phd020608s

Most educators regard peer assessment and peer grading as a powerful pedagogical tool to engage students in the process of evaluating and grading their peers while saving instructors’ time. This process helps improve students’ understanding of the subject matter and provides an opportunity for deeper reflection on the subject matter, accessing higher levels of Bloom’s thinking taxonomy.

Designing and distributing tasks and assignments for peer assessment should be as easy as assigning a few papers to each student and wait for the magic to happen, right? …. Not really!

As instructors, we care about the fairness of our evaluation methods and providing effective feedback. Yet, throwing this crucial responsibility to the shoulders of novice students who (hopefully) have just learned the new topic seems like an awfully risky behavior. There are two major concerns when it comes to peer grading; inflated (or deflated) grades and poor quality feedback. Both of these issues seem to be originating from the same sources of insincerity of the graders and the lack of effort each student invests on grading the peers [2]. To address these issues, recently researchers have raised the question of whether we can design a peer-grading mechanism that incentivizes sincere grading and discourages any type of secret student collusion.

The simplest possible design is to evaluate the quality of the peer reviews or simply put by “reviewing the reviews” [1, 3, 4, 5]. This procedure is bullet proof since no student can get away from a poor-quality feedback or deliberately assigning insincere grades. However, even though this technique may help us achieve the goal of involving the students in the higher levels of learning, in most situations this mechanism is either costly in terms of TA-Instructor time or simply impossible in large classes. In fact, this fully supervised approach defeats one of the main purposes of using peer grading by doubling or tripling the required grading effort: each marked assignment has to be reviewed by one or two TAs for quality.

As a partially automated solution, the system may randomly send a subset of graded papers to the Teaching Assistants (TAs) to perform a sanity check (instead of doing this for every single paper). In contrast, fully automated systems provide a meta-review procedure in which students evaluate the reviews by rating the feedback they have received [1, 3, 5] or by computing a consensus grade for assignments that are initially graded by at least two or three peer graders [5, 6].

In a different approach, students are treated as potential graders throughout the term and only those who pass certain criteria will be take the role of independent graders [6]. The premise is that once an individual reaches a level of understanding, he or she can now act as a pseudo-expert and participate in the assessment procedure. Of course, to ensure fair grading the system randomly chooses a subset of graded papers to be reviewed by the instructor.

Peer assessment is still in its infancy; nevertheless a number of researchers in various disciplines are developing new techniques to address the critical issues of efficiency, fairness, and incentives. Each of the above methods (and many others that exist in the peer-grading literature) could potentially be adopted depending on course characteristics and intended outcomes. I do believe that such characteristic, to very least, must include the following:

  • Skill/knowledge transferability: Do marking skills and the knowledge of a previous topic  automatically transfer to the next topic? If so, are they sufficient?
    For example, an essay-based course may contain similar marking guidelines in all its assignments and training students once could be sufficient in transitioning students to effective peer graders.
  • Course material and structure: How are the topics that are covered in the course dependent on one another? Is the course introducing various semi-independent topics, or are the topics all contribute to building a single overarching subject.

What do you think? Have you ever used peer-assessment in your classes?

 

References

  1. Cho, K., & Schunn, C. D. (2007). Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system.Computers & Education48(3), 409-426.
  2. Carbonara, A., Datta, A., Sinha, A., & Zick, Y. (2015) Incentivizing Peer Grading in MOOCS: an Audit Game Approach, IJCAI.
  3. Gehringer, E. F. (2001). Electronic peer review and peer grading in computer-science courses.ACM SIGCSE Bulletin33(1), 139-143.
  4. Paré, D. E., & Joordens, S. (2008). Peering into large lectures: examining peer and expert mark agreement using peerScholar, an online peer assessment tool.Journal of Computer Assisted Learning,24(6), 526-540.
  5. Robinson, R. (2001). Calibrated Peer Review™: an application to increase student reading & writing skills.The American Biology Teacher63(7), 474-480.
  6. Wright, J. R., Thornton, C., & Leyton-Brown, K. (2015). Mechanical ta: Partially automated high-stakes peer grading. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (pp. 96-101). ACM.

Navigating the Pitfalls of Peer Evaluations – Kyra Jones

4208086358_a65947e1c3

Having students work as a team for summative and formative assessment can be challenging, but implemented thoughtfully, it can be highly beneficial to students. Teamwork allows an instructor to pose more difficult problems that encourage deep learning. Group work can also be an effective way to engage students in a large class as well as prepare students for the workplace. Despite these benefits, group work can be challenging to implement.

One tool that can help group work succeed in a classroom is to incorporate peer evaluations. Peer evaluations help to provide a key benefit of group work in the classroom: teaching students how to give and receive constructive feedback. Peer evaluations provide a method to keep students accountable for their contributions to the team’s task, can help reduce group conflict, and lead to a more evenly distributed workload amongst group members. Finally, this tool can help alert instructors to conflicts in between group member. Peer evaluations have many benefits, but like group work, using this tool effectively takes careful planning.

First, the expectations of the students in the team setting must be communicated clearly and directly. Students need to be aware of the criteria by which they will be assessed and use to assess their peers. It is also essential that instructors formulate expectations that are realistic and align with the course objectives. It can be helpful to involve the students in creating the peer evaluation criteria and designing procedures surrounding peer assessment. This can motivate students to take the process seriously and address student anxiety surrounding group work. Students take more control of the peer evaluation criteria and process, promoting validity and reliability of the peer assessments.

It is also important to have multiple peer evaluations during group assessments. This allows students to develop clear expectations of their responsibility as a group member and gives students who under-perform, especially those who do not realize the are not meeting their peer’s expectations, a chance to improve.

Additionally, we need to provide students with the tools and skills to give and receive constructive feedback. Giving constructive feedback is not a natural skill and many students have not had the opportunity to participate in peer evaluation.  One method to introduce students to this process is through demonstration, looking at a journal article or other work as a class and providing constructive criticism. Further, the class can work together to restructure examples of inappropriate feedback to create constructive comments. This skill that takes practice, which further exemplifies the need for multiple formative peer assessments throughout the project.

In my view, one of the most important aspects of implementing a peer evaluation system is to take into consideration your own teaching style and goals for the class. One model of peer evaluation may work for one instructor, but this model may be ineffective when implemented in your classroom. As with all teaching tools, it is important to tailor peer evaluation models to your own personality, teaching style, course, and institution. Being a good teacher is something we all strive for, but it is important to be a good teacher in a way that reflects who you are.

Peer evaluations can be a tricky component of group work, but with diligent planning and consideration, this tool can make group work a more realistic and successful exercise in the classroom.

Aggarwal, P. & O’Brien, C.L. (2008). Social Loafing on Group Projects: Structural  Antecedents and Effect on Student Satisfaction. Journal of Marketing Education, 30(3), 255-64.

Cestone, C.M., Levine, R.E. & Lane, D.R. Peer Assessment and Evaluation in Team-Based Learning. In Michaelson, L.K., Sweet, M., Parmelee, D.X. (Eds.), Team-Based Learning: Small-Group Learnings Next Big Step (69-78). San Francisco: Wiley Periodicals, Inc.