Category: Teaching Resources

Crowdmark – Online grading for large courses

This Fall 2016, the University of Waterloo will have 25 courses with stockvault-pile-of-paper117595a class size of between 500 and 1000 students and 10 courses of  between 1000 and 2000 students.

The amount of paper handling to administer the potential 33,000 final exam papers from these large courses will be monumental. (For fun, guestimate the volume of paper this amounts to.)

The Mathematics Faculty has been  successfully experimenting for a year with a online grading system called Crowdmark, a company founded by Professor James Colliander of the Mathematics Department of the University of Toronto.

Professor Colliander was faced with a similar problem: grading 5000  Canadian Open Mathematics Competition (COMC) papers each year with  100 volunteers.  As with final exams, each paper is typically graded by a number of markers so keeping track of which questions are graded on which papers and when the papers are free to be passed to another marker is a time consuming and error prone business.

Crowdmark (CM) attempts to eliminate some of the time and trouble spent managing the grading process.   We are not talking about a quiz system with automatic grading. Crowdmark is hand-marking done online.  Skilled people still grade, and tests and assignments are still created for printing on paper so there is nothing new in this part of an instructor’s routine.

So, what is it that makes the marking process more efficient when done online?

  • Markers are able to grade the same paper at the  same time.  No more locating and waiting for a paper that someone else is grading.  Or waiting for a batch of papers to arrive at your location to begin your stage of grading.  Grading can be done concurrently at multiple locations and times.
  • Grades can be automatically summed, collected, summarized, distributed and recorded in a Learning Management System without needing to check for arithmetic or transcription errors.
  • No time needs to be spent returning piles of exam papers.

There is a time and money cost to using online grading.  The physical papers have to be scanned into digital format (PDF file) before grading can start. High speed scanners (500 pages per minute) can process 1000 10-page exams  in 20-30 minutes once delivered to the scanning machine.

Here I’ll briefly discuss how instructors and students use CM.

Steps for an instructor:

  • upload one test or exam pdf file into CM (leave 2 inches blank on the top of each page for CM ID info and set 1 question per page)
    • CM duplicates the test pdf for each student and adds a paper and page ID to each page
  • download from CM the pdf file of student tests and print it
  • after the test scan all written test papers into a pdf file and upload the file into CM
    • CM arranges the pdf file pages into a grid pattern: each row holds a student’s test pages
  • each marker clicks on a page in the grid to read, comment, and grade it
    • when grading is complete page grades are summed for each test paper by CM
  • match each test paper cover page student ID with a student name in your CM course (assigned seating at UW can eliminate this step)
  • you choose whether CM sends each student their grade and a CM link to their graded test paper or to keep the grades and graded papers private and just download the grades for inclusion into a course grade

Steps for a student:

  • write the test paper by hand as usual
  • may receive an email from CM with a link to a CM page showing their test results

The links at the end of this post provide further details about Crowdmark.  In addition, 2 live sessions demonstrating Crowdmark are coming up at the end of August and the beginning of September.   The first is an introduction to Crowdmark on Wednesday August 31 and  the second follows up a week later on Wednesday September 7 (1:30-3 PM) with details about a University of Waterloo system named Odyssey that works with Crowdmark.  Odyssey organizes test papers, students and exam room seating providing relief from some time-consuming management overhead.

Crowdmark is not a free service, but the University of Waterloo has a licence so there is no charge to individuals (instructors or students) at the university.

If you are interested in learning more about online grading for your course please get in touch with me.

Paul Kates
Mathematics Faculty CTE Liaison
pkates@uwaterloo.ca, x37047, MC 6473

Intro to Online Marking using Crowdmark: Wednesday, August 31, 2016 – 10:30 AM to 11:30 AM EDT
Crowdmark home page,   help pages and  youtube channel.
UW Odyssey Examination Management

Online Math Numbers at Waterloo, and Comparative Judgments as a Teaching Strategy — Tonya Elliott, CEL

math equationOnline Math Numbers

If you weren’t already aware, here are a few numbers about online math at the University of Waterloo:

  • The Math Faculty has been offering fully online courses since Fall 2003 and, since that time, has offered 55 unique online courses to more than 21,000 students
  • The Centre for Education in Mathematics and Computing (CEMC), with support from the Centre for Extended Learning (CEL) and local software company Maplesoft, was the first group on campus to release a large set of open educational resources (OERs). Called CEMC courseware, the OERs include lessons, interactive worksheets, and unlimited opportunities for students to practice skills and receive feedback. At the time of this post, the resources have received over 1.8 million hits from 130,000 unique users in 181 different countries.
  • In 2015, the Canadian Network for Innovation and Excellence (CNIE) recognized CEMC, CEL, and Maplesoft for their OERs through an Award of Excellence and Innovation.
  • The Master for Mathematics for Teachers (MMT) program has the highest enrolment of all the fully online Masters programs offered at the University of Waterloo. MMT and CEL staff who work on the program were one of three teams from Waterloo who won a 2016 Canadian Association for University Continuing Education program award.
  • Maplesoft is using a focus group from Math, CEL, and CTE to develop a new authoring environment that will specifically target the needs of online STEM course authors. It is anticipated that this tool will be released in early 2017 and, over time, should save development costs by 50%.
  • The Math Faculty, together with the Provost’s office, has dedicated $1.2 Million over the next three years for additional work on online projects; over 90 course development slots allocated by CEL have already been filled.

These numbers are some of the reasons Waterloo is considered a leader in the area of online math education.

Comparative Judgments

From June 19 – 22, a small group from Waterloo and I joined an international team of mathematics educators to discuss digital open mathematics education (DOME) at the Field’s Institute in Toronto. Lots of great discussions happened including opportunities and limitations of automated STEM assessment tools, integrity-related concerns, and practical challenges like lowering the bar so that implementing fully online initiatives isn’t the “heroic efforts” for Faculty it’s often viewed as being today. Of all the discussion topics, however, the one that got me most excited – and that my brain has returned to a few times in the month since the conference – is using Comparative Judgement (CJ) in online math courses.

colour shadesThe notion behind CJ is that we are better at making comparisons than we are at making holistic judgments, and this includes judgments using a pre-determined marking scheme.  It doesn’t apply to all types of assessments, but take this test on colour shades to see an example of how using comparisons instead of holistic rankings makes a lot of sense. Proof writing and problem solving may also lend themselves well to CJ and three journal articles are listed at the end of this blog for those who would like to read more.

Here are some of the questions I’ve been pondering:

  • Are there questions we aren’t asking students because we can’t easily “measure” the quality of their responses using traditional grading techniques? How much/when could CJ improve the design of our assessments?
    • Example: Could CJ, combined with an online CJ tool similar to No More Marking, be used by students in algebra courses as a low-stakes peer assessment activity so students could see how different proofs compare to one another? Perhaps awarding bonus credit to students whose proofs were rated in the top X%.
  • Which Waterloo courses would see increases in reliability and validity if graders used CJ instead of traditional marking practices?
  • How much efficiency could Waterloo departments save if high-enrolment courses used CJ techniques instead of marking schemes to grade exam questions or entire exams? Could CEMC save resources while using CJ to do their yearly contest marking?

I don’t have answers to any of these questions yet, but my brain is definitely “on” and thinking about them. I encourage you to read the articles referenced below and send me an email (tonya.elliott@uwaterloo.ca)  If you like the idea of CJ, too, or have questions about anything I’ve written.  If you have questions about Waterloo’s online math initiatives, you’re welcome to email me or Steve Furino.

References

Jones, I., & Inglis, M. (2015). The problem of assessing problem solving: can comparative judgement help? Educational Studies in Mathematics, 89, 3, pp. 337 – 355.

Jones, I., Swan, M., & Pollitt, A. (2014). Assessing mathematical problem solving using comparative judgement. International Journal of Science and Mathematics Education, 13, pp. 151–177.

Pollitt, A. (2012). The method of Adaptive Comparative Judgement. Assessment in Education: Principles, Policy, & Practice. 19, 3, pp. 281 – 300.

__________

Blackboard image courtesy of AJC1.

 

 

 

Meaningful Conversations in Minutes – Mylynh Nguyen

ConversationWith constant media stimulation, increase in competitiveness, and stress overload, “Is it possible to slow down” (1)?  Our culture can be self-driven and individualistic so it is no surprise that for many, time is a finite resource that is draining away. As a result, we try to do as much as we can in a very short time period. Our minds are filled with constant distraction, thus limiting opportunities for self-reflection to ask oneself “Am I well or am I happy?” (1).

We’d like to believe that we have been a good friend, partner, or child at various points in our life. However, upon remembering that significant person in your life, do you know or have you ever asked what were the moments when they were the happiest? The times when they were crying from tears of joys to the time when they felt the most accomplished? Surprisingly for many, we are unaware of these stories that ultimately define whom that individual has become today. We mindlessly pass every day without pondering about the conversations that we had or the connections that were made.  By simply being mindful of the questions that we pose, more specifically “questions that people have been waiting for their wholes lives to asked … because everybody in their lives is waiting for people to ask them questions, so they can be truthful about who they are and how they become what they are,” as beautifully said by Marc Pacher (2).

So what is the action plan?

1.Invite people to tell stories rather than giving answers. Instead of “How are you” substitute

  • What’s the most interesting thing that happened today?
  • What was the best part of your weekend?
  • What are you looking forward to this week? (3).

2. Enter a conversation with the willingness to learn something new

  • Celeste Headlee in her TED Talk 10 Ways to Have a Better Conversation describes how she frequently talks to people whom she doesn’t like, and with people whom she deeply disagrees yet is still able to have engaging and great conversations. She is able to do this as she is always prepared to be amazed and she seeks more to understand rather than to listen and state her own opinion and thoughts.

3. Lastly “being cognizant of [your] impact is already the first step toward change. It really does start at the individual level” my friend once said (5).

  • Brene Brown in her Power of Vulnerability talk said, “Many pretend like what we’re doing doesn’t have a huge impact on other people”. But we’d be surprise of what we are capable of when you allow yourself to be vulnerable as this “can be the birthplace of joy, of creativity, of belonging of love… the willingness to say, “I love you” the willingness to do something where there are no guarantees” (6).

That being said, you don’t have to be the most intellectual or outspoken person in the room, but what is key is the willingness to be open and the questions that are posed. There are many simple things that can be easily integrate into our daily lives, by being more mindful of the question that we ask to ultimately have a more memorable and enriching conversation. In the end it is to have better connections, new understanding and awareness to savor the moment.

At CTE, Microteaching Sessions are offered where you can choose from various topics to conduct an interactive teaching lesson. For my first topic I will be talking about the importance of communication. All participants will not only be giving feedback but will receive constructive feedback and ways to improve from knowledgeable facilitators. It’s a safe environment where you have the chance to present to fellow graduate students from various departments. Many have found these sessions beneficial as you are working on skills relevant to work, field of study or for your own personal growth. I am excited and nervous for this opportunity to talk about something I am passionate about and I hope I can successfully engage others and deliver the content well. In order to help participants formulate an effective teaching plan, the Centre for Teaching Excellence website has provided many resources such as well written guidelines, lesson plans outlines, and facilitators review the lesson before you present.

As a follow-up post, I had the chance to facilitate an hour session for an AIESEC conference for participants from various universities such as Toronto, Waterloo, Laurier, and York, that recently returned from their international exchanges. There were lots of discussion so thank you to the Graduate Instructor Developers, Charis Enns and Dave Guyadeen, and Instructional developer, Stephanie White for their great feedback and helping me make this session more successful!

Sources:

Early Student Feedback — Paul Kates

Feedback from students doesn’t have to wait until the time of end-of-term course clipboard and arrowevaluations. Getting feedback early and often in a course allows you to build on what’s working and make changes towards what can work better, all in time to have an impact on your students.

Asking students at the start of the term about their expectations for the course, the lectures, the textbook even their own work habits can give you an insight into why your students are in your course and let you address expectations immediately should they be out-of-line with the way the course is going to be run.

Eric Mazur in his book Peer Instruction gives a start-of-term “Introductory Questionnaire” to his Physics class where he asks

  • What do you hope to learn from this course?
  • What do you hope to do with this new knowledge?
  • What do you expect the lectures to do for you?
  • What do you expect the book to do for you?
  • How many hours do you think it will take to learn all you need to know from this course? Include everything: lectures, homework, etc.

(This and the following set of questions are attributed to Prof James Sethian, Department of Mathematics, University of California at Berkeley.)

With the answers in hand he addresses each of the questions in class – supporting and encouraging his students and expanding on student answers with his own goals for the class:

I want the material we cover to be useful to you beyond the exam. I want you to become good critical and analytical thinkers, able to tackle not just familiar problems but also unknown new problems or questions. Not only to plug numbers into equations but able to develop new models and theories, to make qualified assumptions, and then use those models and assumptions to break new ground in science and technology.

He also has the opportunity to address student expectations, realigning and influencing those expectations about the lectures, text and workload.

He gives a sample reply to all the questions above (ask me for a copy), but here I’ll only quote the answer to the question
What do you expect the lectures to do for you?

There were many very thoughtful responses to this question, but I did encounter a number of misunderstandings about the lectures that I should address to avoid falling short of your expectations. The most serious misconception I encountered is that the lectures will present and explain the fundamental concepts, while the book will clarify the ideas presented in the lecture. This is not what is going to happen. You will be reading the material before coming to class. The book will introduce the basic terminology and definitions, hopefully raise some questions, perhaps even confuse you a little (“to wonder is to begin to understand”). The lectures are intended to challenge your thinking and thereby help you assess your understanding of the concepts you read about, to further and deepen your understanding of these concepts, to stimulate and inspire you, and to show you how things “fit together.” The book will then provide further reference. In addition it will be a source for questions and problems.

Some of you expect to practice problem-solving in lecture, but problem-solving is not the main focus of this class. I want you to understand things, not just be able to “plug and chug.” This is clearly reflected in the way you will be tested – take a good look at the exams in the back of the syllabus. Close to half of the questions on each exam are not the traditional, quantitative problems you may have seen before. The solutions to many of these don’t involve even a single equation. Rest assured, the sections and homework assignments will offer ample opportunity to sharpen your traditional problem-solving skills. The lectures are meant to stimulate your thinking, to further your basic understanding. I guarantee that a better understanding of the concepts will improve your problem-solving abilities, whereas the reverse is not necessarily true. Here is what I think of some other answers given: …

After a month Prof. Mazur uses the following questionnaire to gauge how students are settling in to the course. This is another early opportunity to address concerns, misunderstandings and expectations.

  • What do you love about this class?
  • What do you hate about this class?
  • If you were teaching this class, what would you do?
  • If you could change one thing about this class, what would it be?

If you have your own in-course questionnaire and want to share it I’d be happy to use it along with any comments you care to include in a follow-up post.

Further readings:

What is the “Case Method”?

Teaching using case studies has typically been used in Business Schools, Law Schools, and Medical Schools but it is a technique being used by other disciplines to provide exposure to complex real world problems for which there is no “right” or “wrong” answer. At Waterloo, cases have been used in disciplines including Engineering, Biology, Accounting, Social Work, Environment and Business, English and others.

The traditional “Case Method” used in Business Schools involves a three stage process where:

  1. students are given the case and asked to work on it individually to come up with a recommendation or course of action (done outside of class time). The key here is for students to be able to justify and support their choices or decisions.
  2. students meet in small groups of 4 or 5 to discuss the case and their recommendations (done outside of class time) – the objective here is to share perspectives, not come to a consensus as a group
  3. the case is discussed in class with the entire class with the Professor acting as a facilitator to guide discussion.

The amount of learning increases over each stage with exposure to different perspectives.

Learning using the Case Method

While this is the typical method used in MBA programs where cases are used in most courses, it can be modified and adapted. For instance, students can read the case and prepare before class and class time can be used for small group discussion and then discussing the case as a large group (i.e. the entire class). It is important to communicate expectations to students about coming to class prepared as the quality of discussion depends on proper preparation. One technique to encourage students to prepare is to give them questions about the case to answer and submit before class begins.

These techniques of using small group work for peer teaching (i.e. small group work to share perspectives) and facilitating a discussion with the entire class can be adapted and used for other contexts than just cases.

If you’re interested in learning more about the Case Method or using cases in your course, contact Scott Anderson in the Centre for Teaching Excellence.

Waterloo Cases in Design Engineering also writes and supports the use of cases in Engineering courses.

References

Erskine, J., Leenders, M., and Mauffette-Leenders, L. (2012). Learning with Cases, 4th Edition, Ivey Publication Services, Richard Ivey School of Business, Western University, London, Ontario, Canada.

Erskine, J., Leenders, M., and Mauffette-Leenders, L. (2003). Teaching with Cases, 3rd Edition, Ivey Publication Services, Richard Ivey School of Business, Western University, London, Ontario, Canada.

Mauffette-Leenders, L., Erskine, J. and Leenders, M. (2001) Writing Cases, 4th Edition, Ivey Publication Services, Richard Ivey School of Business, Western University, London, Ontario, Canada.

A primer on case-based teaching – Stephanie Verkoeyen

6765941543_7e250390fb_m

Last semester I was allocated a half RA-ship to develop case studies for a second year course in the Faculty of Environment. Case studies have traditionally been applied within the fields of Engineering, Business, and Health Sciences, though their application is slowly starting to trickle into other disciplines. In the social science context that I was working in, students would be applying some of the concepts and ideas discussed in lecture to a real world situation in an attempt to offer insight into some of the complexities surrounding environmental decision-making, and to get students comfortable with there being no ‘right’ or ‘wrong’ answer.

Having never developed a case study previously, I struggled with where to start. Much of the material I read dealt directly with writing cases or teaching with cases, but did not go into detail on getting started with case-based teaching. As a result, I’ve put together a few steps to help anyone else who should wish to use case-based teaching in their own course.

  1. Integrating the case into your course

The first step is to decide how your case study fits into the existing course. This means thinking about the content it relates to (i.e. what concepts/ideas will students apply?), the amount of time required (some case studies are designed to be completed within a lecture, others span the duration of a course), as well as grade allocation, if there is to be an evaluation method attached. It is worth spending time mapping out these details because adopting a case-based teaching method can result in significant changes to how a course is taught.

  1. Defining the learning objectives

This step is closely tied to decisions relating to content. The goal here is to define why the case study is being presented – what are the goals of the discussion to take place? It is useful to refer back to lesson outcomes and/or course objectives to understand how a case study can be used to reinforce these aims. Or you may find that using a case study allows you to introduce new aims.

  1. Selecting the case

This may be the most challenging step of the whole process. Having clearly defined teaching objectives helps to simplify this stage by providing a set of parameters your case has to fit within. Once this has been established, there are a number of different suggestions about how to pick a good case example – it should be current, interesting or provocative, and/or relatable. When making my own selections, I tried to think about the range of views and opinions that might be expressed, the idea being that a broader range is more likely to stimulate discussion.

  1. Writing the case

Before writing the case, first define the questions students are expected to answer. This will determine what information needs to be included to allow students to answer these questions. Your teaching objectives will influence both the questions being asked and the presentation of the case. For example, if identifying what information is most relevant is an important learning outcome, you can also include irrelevant information in the case description. You need not restrict yourself to written materials when presenting a case study. Different multimedia, like videos and podcasts, can also be interesting alternative sources of information.

Once the case is developed, it is highly recommended that you create a teaching note. This document serves as a personal reference guide for how you will actually teach the case study, including information related to the planned agenda and your analysis of the case. The case analysis should consider the type of discussion that might result from each of the assignment questions, as well as follow-up questions that can be used to prompt this discussion.

For those interested in further information about case-based teaching, Teaching with Cases by Erskine, Leenders, and Mauffette-Leenders (2003) is an excellent resource. Developed for the Ivey School of Business, much of the information presented within can be adapted to a social science setting.

Graduate Student Teaching on Campus

As a Graduate Instructional Developer who works mainly with CTE’s Certificate in University Teaching (CUT) program, I have the privilege of observing graduate students teach in classrooms across campus. Over the past year, I’ve had the opportunity to observe over 35 classes taught by graduate students in all six faculties. I have been incredibly impressed by the quality of teaching by graduate students. They have taken concepts from CTE workshops (e.g., active learning, group work, formative assessment) and applied them directly in their teaching. They are using innovative teaching strategies, technologies, and engaging students in their lessons. The University of Waterloo community should be proud of graduate students’ dedication to, and passion for, teaching.

So how can we support graduate students in continuing to develop their teaching skills?

  • I think many of us would agree the best way to improve our teaching is to practice. In some departments, it’s difficult for graduate students to access teaching opportunities, but guest lectures are a great way to gain experience. If you’re teaching, consider asking the graduate students you supervise and/or your Teaching Assistants whether they’re interested in giving a guest lecture in the course.
  • If you know a talented Teaching Assistant or graduate student instructor, please nominate them for an award! Information regarding Graduate Student Teaching Awards can be difficult to find, so I’ve compiled a list here. If you know of any that are missing from this list, please post a comment and we will add them.

 

Graduate Student Teaching Awards

A) University-wide teaching awards

Amit & Meena Chakma Award for Exceptional Teaching by a Student (deadline: February)

B) Faculty-wide teaching awards

C) Department teaching awards

  • Biology – “Outstanding Graduate/Undergraduate Teaching Assistantship Award” (no link available)

 

Teaching Resources for Graduate Students: