Crowdmark – Online grading for large courses

This Fall 2016, the University of Waterloo will have 25 courses with stockvault-pile-of-paper117595a class size of between 500 and 1000 students and 10 courses of  between 1000 and 2000 students.

The amount of paper handling to administer the potential 33,000 final exam papers from these large courses will be monumental. (For fun, guestimate the volume of paper this amounts to.)

The Mathematics Faculty has been  successfully experimenting for a year with a online grading system called Crowdmark, a company founded by Professor James Colliander of the Mathematics Department of the University of Toronto.

Professor Colliander was faced with a similar problem: grading 5000  Canadian Open Mathematics Competition (COMC) papers each year with  100 volunteers.  As with final exams, each paper is typically graded by a number of markers so keeping track of which questions are graded on which papers and when the papers are free to be passed to another marker is a time consuming and error prone business.

Crowdmark (CM) attempts to eliminate some of the time and trouble spent managing the grading process.   We are not talking about a quiz system with automatic grading. Crowdmark is hand-marking done online.  Skilled people still grade, and tests and assignments are still created for printing on paper so there is nothing new in this part of an instructor’s routine.

So, what is it that makes the marking process more efficient when done online?

  • Markers are able to grade the same paper at the  same time.  No more locating and waiting for a paper that someone else is grading.  Or waiting for a batch of papers to arrive at your location to begin your stage of grading.  Grading can be done concurrently at multiple locations and times.
  • Grades can be automatically summed, collected, summarized, distributed and recorded in a Learning Management System without needing to check for arithmetic or transcription errors.
  • No time needs to be spent returning piles of exam papers.

There is a time and money cost to using online grading.  The physical papers have to be scanned into digital format (PDF file) before grading can start. High speed scanners (500 pages per minute) can process 1000 10-page exams  in 20-30 minutes once delivered to the scanning machine.

Here I’ll briefly discuss how instructors and students use CM.

Steps for an instructor:

  • upload one test or exam pdf file into CM (leave 2 inches blank on the top of each page for CM ID info and set 1 question per page)
    • CM duplicates the test pdf for each student and adds a paper and page ID to each page
  • download from CM the pdf file of student tests and print it
  • after the test scan all written test papers into a pdf file and upload the file into CM
    • CM arranges the pdf file pages into a grid pattern: each row holds a student’s test pages
  • each marker clicks on a page in the grid to read, comment, and grade it
    • when grading is complete page grades are summed for each test paper by CM
  • match each test paper cover page student ID with a student name in your CM course (assigned seating at UW can eliminate this step)
  • you choose whether CM sends each student their grade and a CM link to their graded test paper or to keep the grades and graded papers private and just download the grades for inclusion into a course grade

Steps for a student:

  • write the test paper by hand as usual
  • may receive an email from CM with a link to a CM page showing their test results

The links at the end of this post provide further details about Crowdmark.  In addition, 2 live sessions demonstrating Crowdmark are coming up at the end of August and the beginning of September.   The first is an introduction to Crowdmark on Wednesday August 31 and  the second follows up a week later on Wednesday September 7 (1:30-3 PM) with details about a University of Waterloo system named Odyssey that works with Crowdmark.  Odyssey organizes test papers, students and exam room seating providing relief from some time-consuming management overhead.

Crowdmark is not a free service, but the University of Waterloo has a licence so there is no charge to individuals (instructors or students) at the university.

If you are interested in learning more about online grading for your course please get in touch with me.

Paul Kates
Mathematics Faculty CTE Liaison
pkates@uwaterloo.ca, x37047, MC 6473

Intro to Online Marking using Crowdmark: Wednesday, August 31, 2016 – 10:30 AM to 11:30 AM EDT
Crowdmark home page,   help pages and  youtube channel.
UW Odyssey Examination Management

Paul Kates

Paul Kates

Centre for Teaching Excellence (CTE) Liaison to the Faculty of Mathematics (pkates@uwaterloo.ca)

More Posts

Presenting and Recording Notes Using a Tablet – Carsen Banister

onenoteTablet technology has been available to consumers since the 1990s, but was not originally heavily adopted. New generation tablets, most of which are designed without dedicated keyboards, are becoming more common.

I am currently taking a Graduate course with an instructor that uses a tablet to present his lecture content. The environment created is very similar to the typical blackboard or whiteboard classroom, but instead the instructor’s laptop screen is projected for the students to follow. I am recording notes for this class in the same way, using my tablet to reproduce what is being created on the projector. For both of these applications, tablets provide some advantages over traditional methods, but also introduce some drawbacks.

As a note-giver
From a lecture or presentation perspective, a tablet simplifies many of the activities required, such as eliminating the need to erase the board, allowing the marking up of images, and offering many drawing tools, such as coloured inks of varying thicknesses. With the addition of a desktop recording software, the lecture can be recorded and saved to the class website. Students are able to review these screen-castings and catch up on content they missed. This can be particularly helpful for graduate students, where travel to conferences can interfere with a class schedule.

Many of these advantages also have direct disadvantages, such as reducing the movement of the instructor in the classroom. Since the tablet is typically used at a podium, the instructor will remain rooted here during the lesson. It is much harder to interact and gesture with a projection screen than a blackboard. This reduced movement can impact the interactivity of the class, confining it to a style of note-copying rather than getting input and involvement from the students. Although no erasing is required, the screen is much smaller than a collection of blackboards, so less content can be shown at one time. This can hinder students’ ability to visualize the progression of course material and make connections.

As instructors, we should be aware of these advantages and disadvantages because our choice of presentation method affects all members of the class. Switching perspective now to the role of student, a tablet and note-taking software can also be used to record notes, regardless of the method of presentation.

As a note-taker
I have used this method for a couple equation-intensive engineering courses and, as expected, also found some advantages to be balanced by drawbacks. A strong benefit I have found is the high speed of typing compared to writing. In courses with a lot of text note-taking, a lot of time can be saved. An added benefit is being able to look up while recording these notes, something that is not easy when writing on paper. This allows for more focus during the lesson and less time bogged down blindly copying notes.

I don’t find drawing on a tablet faster than on paper, but the added tools do provide benefit. Being able to draw shapes or axes with a couple clicks saves time and improves quality. Drawings can also be modified after they are created, rather than having to pull out an eraser and start over. For mathematical equations, there is a choice to write it out in full or use a built-in equation editor to type the equation. The latter option provides a nice appearance that is less likely to be misread in the future, but it requires some getting used to and can be slower than the former method.

Recording all the content digitally means that it can be easily searched using keywords, even handwriting in many software packages. This can save time if you need to find something quickly and don’t want to look through all your notes. But being able to leaf through your notes is exactly what you lose when going digital. No longer can you quickly flip through them to look for a key image or figure.

A major concern with taking notes electronically is the reliance on electronics and electricity. What if your computer crashes during a lecture? What if you don’t have an electricity source for the 3 hours of class coming up? A backup is always required in case your tablet is unavailable. A pencil and paper don’t have these technical limitations and are much more reliable.

Although it may not be reasonable to expect tablets to replace pencil and paper or chalk and blackboard, they have established a place in the classroom. In both cases, there is a significant learning curve when beginning to deliver or take notes using a tablet. If you want to start using this technology in the classroom, take the time to become comfortable with it before taking the leap!

cbanister

Carsen Banister, B.A.Sc. Ph.D. Candidate, Mechanical Engineering Graduate Instructional Developer, Centre for Teaching Excellence University of Waterloo

More Posts

Presenting lectures with iPad and 2Screens — Paul Kates

Please visit the Presenting lectures with iPad and 2Screens page to learn about combining the ability of the iPad app 2Screens to make presentations to students in the classroom with the iPad’s ability to capture handwriting with a stylus.

Display, annotate, browse, or write-freehand with 2Screens on an iPad tablet. Using files from Microsoft Office or Apple iWorks; or files (pdf), images and video from the web you can present to your students or audience through the lightweight iPad with a VGA adaptor. With an additional pen stylus, pages can be annotated or new pages written, live in class. Files in 2Screen are displayed on tabs, so it is easy to switch among different content pages, jumping from a slide tab to a whiteboard tab for example, to work on a problem or exercise.

Paul Kates
Mathematics Faculty CTE Liaison
pkates@uwaterloo.ca, x37047

Paul Kates

Paul Kates

Centre for Teaching Excellence (CTE) Liaison to the Faculty of Mathematics (pkates@uwaterloo.ca)

More Posts

Wikispaces Goes Free for Higher Ed — Trevor Holmes

I have been using Wikispaces for many years now. It’s been a space to collaborate with peers on research, a space to house an organizational website that needs to be very flexible and easy to use while we seek permanent solutions, and a space for my students to go when other tools go down. By no means have I used Wikispaces to its full potential, but I do administer several wikis there and I’m thrilled that they are extending their ad-free version to higher education, after serving nearly a million K-12 users this way. A couple of examples of how I’ve used it:

  • A backup site for my Cultural Studies 101 course over at WLU
  • A working site for the Council of Ontario Educational Developers

If you’re interested in wiki use, we do have advice for you. And for ease of use plus the newly free adless version, I’d recommend Wikispaces as a strong contender for your time and energy.

trevorholmes

As Senior Instructional Developer, Curriculum and Programming, Trevor Holmes plans and delivers workshops and events in support of faculty across the career span. Prior to joining the Centre for Teaching Excellence, Trevor worked at a variety of universities teaching courses, supporting faculty and teaching assistants through educational development offices, and advising undergraduates. Trevor’s PhD is from York University in English Literature, with a focus on gothic literature, queer theory, and goth identities. A popular workshop facilitator at the national and international levels, Trevor is also interested in questions of identity in teaching and teaching development.

More Posts - Website

…and so it goes – Trevor Holmes

Bit of a dry spell on the blog this term! We’ll try to be more regular.

So I’m sure readers have been holding their collective breath, awaiting eagerly my update from the first day of class a couple of Fridays ago. That’s right: in my first blog post of 2011, I imagined a perfect pedagogical storm of a first day. I did do what I intended to. Many of the students in lecture contributed good thoughts to the definition of culture we were coming up with, collectively. They didn’t seem to tire of the pairs of images so much as previous years’ cohorts have. And in tutorials, when confronted again with some of the same images, they deepened their analysis still more, becoming comfortable with each other in the smaller setting. I even had them fill out tutorial logs at the end of each tutorial, so those who didn’t get a chance to contribute could let us know what they were thinking. Continue reading …and so it goes – Trevor Holmes

trevorholmes

As Senior Instructional Developer, Curriculum and Programming, Trevor Holmes plans and delivers workshops and events in support of faculty across the career span. Prior to joining the Centre for Teaching Excellence, Trevor worked at a variety of universities teaching courses, supporting faculty and teaching assistants through educational development offices, and advising undergraduates. Trevor’s PhD is from York University in English Literature, with a focus on gothic literature, queer theory, and goth identities. A popular workshop facilitator at the national and international levels, Trevor is also interested in questions of identity in teaching and teaching development.

More Posts - Website

Restoring attention and memory by disconnecting?

Montserrat Hermitage
Catalonian Hermitage

For some time now, I’ve been (along with certain friends and colleagues) advocating for at least occasional Slow experiences in higher education teaching and learning. Somewhat akin to Slow food (which of course has its detractors, Continue reading Restoring attention and memory by disconnecting?

trevorholmes

As Senior Instructional Developer, Curriculum and Programming, Trevor Holmes plans and delivers workshops and events in support of faculty across the career span. Prior to joining the Centre for Teaching Excellence, Trevor worked at a variety of universities teaching courses, supporting faculty and teaching assistants through educational development offices, and advising undergraduates. Trevor’s PhD is from York University in English Literature, with a focus on gothic literature, queer theory, and goth identities. A popular workshop facilitator at the national and international levels, Trevor is also interested in questions of identity in teaching and teaching development.

More Posts - Website

Wikis + Interactive Teaching + Class notes = A new take on an old assignment – Seanna Davidson

I’ve often had trouble with the assumption made by students, that professors will post their PowerPoint slides or lecture notes online, and that they will contain all of the content covered in class. There are certainly some instances in which sharing lecture notes may be appropriate, such as for students with learning disabilities who need additional support.  But for the large majority of the class, if we are seeking to encourage active and engaged learning, posting notes online seems like a bit of a contradiction to me.   Many will skip class knowing the notes will be online, or attend class but play games or check their Facebook page instead of tuning in. Continue reading Wikis + Interactive Teaching + Class notes = A new take on an old assignment – Seanna Davidson

seanna davidson

As a Teaching Assistant Developer for CTE's, Certificate in University Teaching (CUT) program, Seanna is responsible for observing teaching events, facilitating workshops and providing feedback for participants in the program . She began her Geography Ph.D. program at the University of Guelph and while there, completed the University Teaching: Theory and Practice course. She is now enrolled with Department of Geography and Environmental Management at the University of Waterloo and is in the third year of her studies. Seanna has been a teaching assistant at York University, Guelph University and the University of Waterloo.

More Posts