Crowdmark – Online grading for large courses

This Fall 2016, the University of Waterloo will have 25 courses with stockvault-pile-of-paper117595a class size of between 500 and 1000 students and 10 courses of  between 1000 and 2000 students.

The amount of paper handling to administer the potential 33,000 final exam papers from these large courses will be monumental. (For fun, guestimate the volume of paper this amounts to.)

The Mathematics Faculty has been  successfully experimenting for a year with a online grading system called Crowdmark, a company founded by Professor James Colliander of the Mathematics Department of the University of Toronto.

Professor Colliander was faced with a similar problem: grading 5000  Canadian Open Mathematics Competition (COMC) papers each year with  100 volunteers.  As with final exams, each paper is typically graded by a number of markers so keeping track of which questions are graded on which papers and when the papers are free to be passed to another marker is a time consuming and error prone business.

Crowdmark (CM) attempts to eliminate some of the time and trouble spent managing the grading process.   We are not talking about a quiz system with automatic grading. Crowdmark is hand-marking done online.  Skilled people still grade, and tests and assignments are still created for printing on paper so there is nothing new in this part of an instructor’s routine.

So, what is it that makes the marking process more efficient when done online?

  • Markers are able to grade the same paper at the  same time.  No more locating and waiting for a paper that someone else is grading.  Or waiting for a batch of papers to arrive at your location to begin your stage of grading.  Grading can be done concurrently at multiple locations and times.
  • Grades can be automatically summed, collected, summarized, distributed and recorded in a Learning Management System without needing to check for arithmetic or transcription errors.
  • No time needs to be spent returning piles of exam papers.

There is a time and money cost to using online grading.  The physical papers have to be scanned into digital format (PDF file) before grading can start. High speed scanners (500 pages per minute) can process 1000 10-page exams  in 20-30 minutes once delivered to the scanning machine.

Here I’ll briefly discuss how instructors and students use CM.

Steps for an instructor:

  • upload one test or exam pdf file into CM (leave 2 inches blank on the top of each page for CM ID info and set 1 question per page)
    • CM duplicates the test pdf for each student and adds a paper and page ID to each page
  • download from CM the pdf file of student tests and print it
  • after the test scan all written test papers into a pdf file and upload the file into CM
    • CM arranges the pdf file pages into a grid pattern: each row holds a student’s test pages
  • each marker clicks on a page in the grid to read, comment, and grade it
    • when grading is complete page grades are summed for each test paper by CM
  • match each test paper cover page student ID with a student name in your CM course (assigned seating at UW can eliminate this step)
  • you choose whether CM sends each student their grade and a CM link to their graded test paper or to keep the grades and graded papers private and just download the grades for inclusion into a course grade

Steps for a student:

  • write the test paper by hand as usual
  • may receive an email from CM with a link to a CM page showing their test results

The links at the end of this post provide further details about Crowdmark.  In addition, 2 live sessions demonstrating Crowdmark are coming up at the end of August and the beginning of September.   The first is an introduction to Crowdmark on Wednesday August 31 and  the second follows up a week later on Wednesday September 7 (1:30-3 PM) with details about a University of Waterloo system named Odyssey that works with Crowdmark.  Odyssey organizes test papers, students and exam room seating providing relief from some time-consuming management overhead.

Crowdmark is not a free service, but the University of Waterloo has a licence so there is no charge to individuals (instructors or students) at the university.

If you are interested in learning more about online grading for your course please get in touch with me.

Paul Kates
Mathematics Faculty CTE Liaison, x37047, MC 6473

Intro to Online Marking using Crowdmark: Wednesday, August 31, 2016 – 10:30 AM to 11:30 AM EDT
Crowdmark home page,   help pages and  youtube channel.
UW Odyssey Examination Management

Program Outcomes – Join our new learning community – Veronica Brown

Goals. Aims. Objectives. Outcomes. Metrics. Performance Indicators. Ideal Graduate Attributes.

Last week, I spent some time with colleagues debating the meaning of these various terms. They are often used interchangeably but, depending who you ask, they don’t mean the same thing. I tend to lump goals, aims and objectives together because they represent our intentions – what we will work towards during a given learning experience. I see outcomes and attributes as what students are actually able to do by the end of that experience (specific behaviours, knowledge, skills, attitudes they have developed). Finally, I place metrics and performance indicators into a category of measurements of those outcomes. Our discussion last week verified that while we use these terms in the similar ways, it’s worth taking the time to clarify our shared understanding of these key terms.

Now, it’s time to expand that conversation across campus. I’m excited to announce a new learning community at CTE – program outcomes assessment. Many departments across campus are engaged in program assessment through academic program review, accreditation, and curriculum design and renewal. Bob Sproule (a member of the School of Accounting and Finance’s Learning Outcomes Committee) and I will be leading this group as we explore various aspects of program outcomes assessment.

The first session, on May 12, 2016 12:00-1:15pm in EV 241, is a brainstorming session to explore topic ideas for the coming year. Our goal is to meet twice per term, starting in Fall 2016, and we want to ensure the sessions reflect areas of interest for you, our community members. If you are unable to attend the session but are interested in joining the community, please email me, Veronica Brown (, Sr. Instructional Developer, Curriculum and Quality Enhancement, Centre for Teaching Excellence.

Small portion of a curriculum map
A slice of a curriculum map – a great tool in assessing program outcomes

A band-aid for mental health – Maggie Bradley

imagesBefore you read this blog post, let me assure you that I believe every student should have access to a positive learning environment. I am in no way advocating that we should start making students fight to survive in a distressing domain. However, there is a movement spreading wildly through post-secondary campuses to offer students increasing protection from ideas they do not like and words that make them uncomfortable. To date, there is no scientific evidence that coddling students is having a positive impact on society or on their future; the widespread use of trigger warnings could be dangerous for mental health. In an article written for The Atlantic, Greg Lukianoff and Jonathan Haidt pose the question: “What exactly are students learning when they spend four or more years in a community that polices unintentional slights, places warning labels on works of classic literature, and in many other ways conveys the sense that words can be forms of violence…?” (1). Using warnings to protect students from potentially harmful ideas is setting them up for larger issues once they leave the “safe space” campuses that have been created for them. Instead, let’s help teach students how to cope with these potentially triggering situations.

Assisting people’s efforts to avoid their fears is misguided. Cognitive behavioural therapy (CBT) is a non-pharmaceutical treatment of mental illness that would be more beneficial to students than using trigger warnings. CBT involves the patient working with a mental cc7e34fe0b549e5eb409d27207689bdehealth counselor to help him or her “become aware of inaccurate or negative thinking so [he/she] can view challenging situations more clearly and respond to them in a more effective way” (2). The basis of the process is simple: notice that you’re being affected by a stressor, name it, describe the facts of the situation, and consider alternative interpretations. After treatment, people are less likely to suffer from anxiety, depression, and anger. Over time, this process becomes more automatic and also helps enhance critical thinking skills. This method can further be combined with exposure therapy, where a patient’s cognitive distortions are diluted with a gradual increase of exposure to the offending scenario. Over time, using either approach, the triggered reaction would deflated. You can’t always control the situation, but you can control how your thoughts, actions and feelings affect the situation.

Keep in mind that I am not an expert. I am not prescribing treatment, nor am I making false promises for results I cannot guarantee (which in this case – I can’t). I’m also not naïve enough to think that cognitive behavioural therapy or exposure therapy are one-stop solutions that will work for everyone. Every person and situation is different. Furthermore, not every reaction to trauma will require the use of one of these coping mechanisms.

Where is all this coming from? Straight out of high school, I attended college. I earned my diploma, worked in my field for a couple cbt-diagram-1years, and then decided it wasn’t the right fit for me. As someone who has recently made the transition back to academia from the “real world,” I’m rather shocked at how hard it is to say the right thing anymore. It can be very tricky to live in a world where “I’m offended” could be used at any given time as an unbeatable trump card. Constantly having to worry about whether an answer you give in class will elicit an upsetting response while discussing the assigned material is exhausting. As a student, I would prefer to utilize the nonthreatening environment of the classroom to discuss controversial topics instead of feeling limited by uncertain restrictions.

The above methods do not entirely invalidate the need for trigger warnings on certain material. They are an excellent temporary solution. It would be more practical long term to negotiate coping mechanisms in a classroom environment before students are released into the “real world.” Throughout elementary and into high school, students are sheltered to protect them. A safe space is provided to nurture students as they mature. In the infamous life after school people keep talking about, it is likely you will have to engage with people and ideas that make you uncomfortable. Outside of the classroom, there will be significantly less protection from any potential stressors. The long-term benefits of developing proper coping mechanisms should not be diminished.


This post was inspired by “The Coddling of the American Mind” written by Jonathan Haidt and Greg Lukianoff for The Atlantic’s September 2015 issue. You can view their full article here:
To read more on trigger warnings, check out the Centre for Teaching Excellence’s tip sheet here:

(1) The Atlantic –
(2) The Mayo Clinic –

Pathways to feminist pedagogies — Charis Enns

femnist pedagogiesMy introduction to feminist pedagogy took place as an undergraduate student in an upper-level geography class at the University of British Columbia. The final assignment for this class involved conducting a community-based research project. Students were assigned small groups, in which they partnered with community organizations to design and carry out research based on the organization’s needs. My small group partnered with a local food bank, which asked us to investigate how the food bank could partner with community gardens in order to contribute to food security in Vancouver.

This learning experience was transformative. Ultimately, it shifted my thinking about social life in Vancouver and motivated me to pursue my interests through research. Importantly, through this class, I was introduced to spaces within the university that were orientated towards building community and social action. This is one explicit intention of feminist pedagogy. In classrooms grounded in feminist values, “there is a need and desire to move learning beyond the walls of the classroom” (Shrewsbury 1993, p. 171). Students are encouraged to extend theory to action, and then action is brought back into the classroom in order to inform theory. 

What is feminist pedagogy?

My initial experience with feminist pedagogy not only shaped my research, it laid the foundations for my approach to teaching and learning. Broadly, feminist pedagogy is a way of thinking about teaching and learning that is grounded in feminist values. It is therefore more appropriate to speak about feminist pedagogies, and then to highlight what these different approaches to teaching and learning have in common. Common principles of feminist pedagogies include:

  • Resisting hierarchy and empowerment: In a traditional university classroom, the instructor holds power over the class and knowledge is passed from instructor to students.  In contrast, feminist pedagogy involves the de-centering of power. Instructor and students exist in a “symbiotic” relationship and knowledge is constructed through discussion, dialogue, and critical inquiry. Importantly, students are also invited to play a role in influencing the design of the class and to participate in the delivery of content.
  • Building community and using personal experience as a resource: Like a traditional university classroom, many classrooms grounded in feminist values continue to rely on traditional sources of information, such as textbooks and academic journals. However, feminist pedagogy also involves drawing on students’ and teachers’ own experiences as learning materials. This promotes critical thinking, as students are challenged to bridge scholarship with “real-life.” Drawing on personal experience in order to deepen and widen understanding of course content also encourages students to value difference and diversity.
  • Transformative learning: This is the principle that was put into action in my first experience with feminist pedagogy as an undergraduate student.  Traditional university classrooms may provide limited space for critical thinking and problem solving. However, in classrooms grounded in feminist values, teaching and learning aim to shift thinking in new directions. Students are asked to examine either their own experiences or social phenomena in new and critical ways. This often involves creating learning experiences that draw attention to real world problems or power differences that contribute to inequality.

Importantly, feminist pedagogy is a way of thinking about teaching and learning that shapes both what we teach and how we teach it. But how can this translate into classrooms that might, at first glance, not appear conducive to such approaches? In other words, are there practical applications of the principles of feminist pedagogy in all university classes? Or, is feminist pedagogy best reserved for certain levels of learning and certain disciplines?

Feminist pedagogies in practice

Research suggests that there is space for feminist pedagogy in all university classrooms and at all levels of learning; however, what this looks like in practice is likely to vary significantly. Here are some examples of feminist pedagogy in practice, along with links to research in support of this practice:

  1. Feminist pedagogy can be used to change teaching strategies and deepen learning outcomes in engineering. For example, Cashman and Eschenbach (2004) use labs to teach students how to work in small-groups to design their own approach to solving real-world problems. They ask students to solve problems that are community-based or locally relevant. Cashman and Eschenbach (2004) have found that this empowers students to extend their classroom learning into the community – some students even become involved in community projects or local politics. This approach also encourages students to approach exams and homework assignments using real-life scenarios.
  1. Feminist pedagogy can be used in psychology education to de-centre power in the classroom, contribute to more diverse curriculum, and to guide students in developing their feminist consciousness. For example, Robinson-Keilig et al. (2014) adopted the photovoice research methodology for a classroom project on violence against women. The project encouraged self-disclosure of students’ own lived experiences, as a means of integrating student knowledge into classroom content. The authors of this study found that this particular project facilitated critical consciousness, as students became more aware of the multiple systems of oppression that exist in society and became empowered via new insights and self-reflection to make change.
  1. Feminist pedagogy can be integrated into economic classrooms to include students in the learning process and urge students to grow as critical thinkers. For example, Nelson and Goodwin (2005) argue that economics learning materials often do an inadequate job at integrating feminist concerns into introductory economics courses. Accordingly, they have published alternative learning materials that introduce students to neoclassical economic principles but also make room for ecological and feminist concerns. They argue that this deepens students’ understanding of economics, as they are exposed to less mainstream debates about the interconnection between economics, gender, and the environment.

Ultimately, if we understand feminist pedagogy as a way of teaching and learning that empowers students to be reflective and critical learners and to apply their learning through social action, it is possible to imagine how this approach is relevant across the university and beneficial to students, regardless of discipline. 


Crawley, S. L., Lewis, J. E., & Mayberry, M. (2008). Introduction—feminist pedagogies in action: Teaching beyond disciplines. Feminist Teacher19(1), 1-12.

GEA – Gender and Education Association (n.d.). Feminist Pedagogy.

Shrewsbury, C. M. (1997). What is Feminist Pedagogy? Women’s Studies Quarterly, 25(1/2), 166-173.

Charis Enns is a Graduate Instructional Developer in CTE and a PhD Candidate at the University of Waterloo and the Balsillie School of International Affairs. 

What is the “Case Method”?

Teaching using case studies has typically been used in Business Schools, Law Schools, and Medical Schools but it is a technique being used by other disciplines to provide exposure to complex real world problems for which there is no “right” or “wrong” answer. At Waterloo, cases have been used in disciplines including Engineering, Biology, Accounting, Social Work, Environment and Business, English and others.

The traditional “Case Method” used in Business Schools involves a three stage process where:

  1. students are given the case and asked to work on it individually to come up with a recommendation or course of action (done outside of class time). The key here is for students to be able to justify and support their choices or decisions.
  2. students meet in small groups of 4 or 5 to discuss the case and their recommendations (done outside of class time) – the objective here is to share perspectives, not come to a consensus as a group
  3. the case is discussed in class with the entire class with the Professor acting as a facilitator to guide discussion.

The amount of learning increases over each stage with exposure to different perspectives.

Learning using the Case Method

While this is the typical method used in MBA programs where cases are used in most courses, it can be modified and adapted. For instance, students can read the case and prepare before class and class time can be used for small group discussion and then discussing the case as a large group (i.e. the entire class). It is important to communicate expectations to students about coming to class prepared as the quality of discussion depends on proper preparation. One technique to encourage students to prepare is to give them questions about the case to answer and submit before class begins.

These techniques of using small group work for peer teaching (i.e. small group work to share perspectives) and facilitating a discussion with the entire class can be adapted and used for other contexts than just cases.

If you’re interested in learning more about the Case Method or using cases in your course, contact Scott Anderson in the Centre for Teaching Excellence.

Waterloo Cases in Design Engineering also writes and supports the use of cases in Engineering courses.


Erskine, J., Leenders, M., and Mauffette-Leenders, L. (2012). Learning with Cases, 4th Edition, Ivey Publication Services, Richard Ivey School of Business, Western University, London, Ontario, Canada.

Erskine, J., Leenders, M., and Mauffette-Leenders, L. (2003). Teaching with Cases, 3rd Edition, Ivey Publication Services, Richard Ivey School of Business, Western University, London, Ontario, Canada.

Mauffette-Leenders, L., Erskine, J. and Leenders, M. (2001) Writing Cases, 4th Edition, Ivey Publication Services, Richard Ivey School of Business, Western University, London, Ontario, Canada.

Learning from Challenge and Failure – Julie Timmermans


This year’s University of Waterloo Teaching and Learning Conference theme, Leaning from Challenge and Failure, is an opportunity to open up discussions with our colleagues, our students, and ourselves around the beliefs we hold about challenges, setbacks, and failure in the context of teaching and learning at the University.

How do these beliefs shape the ways in which we teach, learn, and lead? How do we work to cultivate a culture that encourages risk-taking, growth through experimentation, and learning from our earnest attempts that lead to failure? What measures can we put in place to ensure that the members of our community have the opportunity to flounder, perhaps fail, and flourish?

During the Conference, we will explore not only challenges and failures, but the work of learning from these challenges and failures. The difficult cognitive and emotional work of learning from these experiences does not happen automatically or autonomously. It takes time and must be guided by people who care deeply about our development.

Airing our experiences of challenge and failure publically may certainly feel vulnerable and risky. But what might be the risks of not sharing these stories? Engineers Without Borders Canada (EWB) publishes annual ‘Failure Reports’ in which they highlight a dozen or so stories of failure – and learning from failure – in their international development efforts. This is risky in many ways – financially, for an organization that depends on contributions from donors; emotionally, for the people in the field sharing their stories. But EWB has determined that the benefits of disclosing these failures outweigh the costs of hiding them. Because hiding them does not help them, or other organizations, solve the problems which they are hoping to solve – poverty, access to clean water, food security, etc. This approach recognizes that we are involved in a collective endeavour to improve our communities.

As we began to introduce and discuss the Conference theme with others on campus, we discovered that conversations about failure, challenge, and resilience are already going on in residence rooms and in meetings rooms. Often, however, these rooms are behind closed doors. Through the Conference, we hope to bring these conversations out into the public spaces of our University – a learning organization – so that when we share our stories of innovation, experimentation, and publication, they integrate the stories of uncertainty, failed attempts, and rejections. Because the whole story of our successes often include failure. We hope that the Conference will be one space of many in which we can collectively explore our potential to learn and grow from challenge and failure.

For links to resources on learning from challenge and failure, including an excellent blog series from the Faculty of Arts Teaching Fellows and CTE’s Kyle Scholz, please visit the “Resources“section of the Conference website.

Register for the Conference

Keys for a TA to Succeed in the Classroom — Aser Gebreselassie

TutorialAs an undergraduate student currently in my third year of ERS at the University of Waterloo, I have had the chance to interact with various types of Teaching Assistants (TAs) over the course of my studies, whether it be in labs, tutorials, in class, via email, or having assignments marked by them. There are plenty of great stories about TAs whom I have had in the past, and unfortunately, a few stories of some questionable TAs as well. Being a successful TA consists of many different aspects, but the three characteristics that I appreciate in a TA is their ability to relate to students, knowledge of the course content, and an ability to communicate effectively and efficiently.

Relating to your students helps build trust between the TA and the student which helps to manage the classroom effectively, as the students will have respect for the TA. Quick but effective activities which I have personally seen in my classes include icebreakers during the first day of meeting your students, as well as having a sense of humour and giving out a positive vibe. A few new things I learnt during the Building Rapport with Students workshop earlier this week was that maintaining positive body language throughout the session gives the students a positive impression about yourself, and learning the student’s names as soon as possible to help develop trust and understanding between the TA and the student.

Knowledge of course content is also key. Most people think that their TAs are those who have taken the course before and have done fairly well in it. This isn’t always the case. Sometimes the TA may never have taken the course, or sometimes they didn’t complete their undergraduate degree in the same faculty as the course they are TAs for. If this is the case, doing the readings and making detailed notes would help a lot. The students understand that a TA is a student as well, and if you as a TA can’t answer a question but are willing to do some research to find the right answer, students find that extremely helpful and are willing to wait to get a right answer, instead of getting a wrong or incomplete answer immediately.

Being able to communicate successfully can make or break the trust and respect that students have for a TA. Setting basic rules on the first day can help a TA significantly. I have had TAs in the past tell us a couple of ground rules: for example, they only will respond to emails during business hours (9 am to 5pm), and that students should not email questions about a majoTutorial 2r assignment the night before it is due as it will be too late to get a response of any value. Prompt responses and setting ground rules can help alleviate pressure from students, and can significantly help boost a TA and student’s relationship. Sometimes TAs respond weeks, even months after receiving an email and it destroys any rapport that they have built with the student.

Lastly, my pet peeve: it is frustrating when students compare grades after an evaluation, and have written two very similar things on their paper, but get two completely different marks. The TA has now lost a lot of the positive feelings that they may have gained over the semester by being inconsistent. Both students are now alienated and concerned, and will go through every little detail of their evaluation to make sure nothing else was missed. Both students will come to the TA with many concerns about their marks. Being consistent, whether it be giving both those students an 85% or a 55%, will save a TA a massive headache.