Integrated Testlets- What are they?–Samar Mohamed

Sample IF-AT card

Last month I attended the annual Society for Teaching and Learning in Higher Education (STLHE) 2014 Conference that was held in Queens University, Kingston. It was an excellent opportunity for me to learn from colleagues across Canada and exchange ideas with them. One of the workshops that attracted my attention was facilitated by Aaron Slepkov and Ralph Shiell from the Dept. of Physics at Trent University. In their workshop, Ralph and Aaron focused on their newly developed testing technique: “Integrated Testlet (IT)”. The presenters started by talking about the benefits of Constructed Response (CR) questions, a common term for questions where students must compose the answer themselves, and how these types of questions enable instructors to gauge their students’ learning. CR questions also allow instructors to give part marks to partially correct answers. The presenters also commented on the trend to switch from CR questions to Multiple Choice (MC) questions in the field of Physics due to increasing class size and the resulting contraints on time and personel resources. However, traditional MC questions don’t allow for part marks or, more importantly from a pedagogical standpoint, enable the instructors (and students) to know where the students went wrong. The integrated testlet method is different in that not only does it allow the students to keep trying each MC question until they get the correct answer, “answer-until-correct question format” enabling the granting of partial marks, but student do not leave the question without knowing the correct answer enabling them to move on to the next integrated question. The method presented changed complex CR physics questions into IT questions. The IT method is based on a traditional testlet, which is a group of MC questions that are based on a common stem (or scenario). In an IT, an answer to a question (task) leads to the next task procedurally, and in this way the students’ knowledge of how various concepts are connected can be assessed. Therefore, items in an integrated testlet are presented in a particular sequence in which the answer for part (a) is used to solve for part (b) and so on. The IT rely on the use of an “answer-until-correct response format”, where the students can keep on making selections on a MC question until the correct response is identified and can be used in subsequent related items. The presenters used the Immediate Feedback Assessment Technique (IF-AT) to allow the students to do several attempts and to get part marks for their response. For more information about IF-AT cards, see Epstein Educational Enterprises website. Moreover, for a sample application of the IF-AT cards at the University of Waterloo see the CTE blog by my colleague Mary Power, The faculty of Science Liaison. In their published paper, the presenters explain the method they have used to transform CR questions to IT questions and analyzed the students’ responses for both question types; it is a very useful and interesting reading that I recommend for instructors thinking about this method.

IF-AT First You Don’t Succeed….

students working in group

For the first fifteen minutes the class was quiet, heads studiously bent over their papers as individual quizzes were taken. The only sound in the room, our footsteps as we moved about placing one card in front of each group of 4 students. When Professor Kelly Grindrod announced “Okay, 15 more minutes to do the cards in your groups”, the class erupted into: noisy chatter; intense discussion; whoops of joy and high fives; an occasional groan of dismay. Moving about the classroom this time it was invigorating listening to the students discuss their chosen answers, argue their opinions, and reasoning together as they worked to reach consensus on which answer was correct and thus which box to scratch. The energy was palpable.

So what were they doing? What caused this classroom transformation? The students were completing the same quiz that they had just done individually but this time as a group, checking their answers by scratching boxes on cards called IF-AT. The IF-AT (Immediate Feedback Assessment Technique) produced by Epstein Educational Enterprises  is a multiple-choice assessment and learning tool in which students scratch the box corresponding to their chosen answer and know immediately if that answer was correct. At first glance it looks a little bit like a scratch-and-win lottery ticket admittedly; if used thoughtfully by students – not at all. If an incorrect choice is made immediate feedback (no star in the box) is received. The students then have the opportunity to re-think, and in the case of Kelly’s class re-discuss, re-convince, re-argue, and then attempt again. Full marks are achieved for the first correct answer and progressively lower partial marks for subsequent attempts. In the class that I observed students actively and interactively worked  to discuss and understand the question principle before attempting again.

While collecting the individual quizzes and IF-AT cards at the end of the second 15 minutes I had a chance to chat with a number of the students. Some found it stressful, others found it less so than a “regular” quiz, but all agreed that the IF-AT format of quizzing was a great deal of fun (it was a low stakes 2.5% quiz). Being from a Teaching Centre, I just had to ask how they felt these quizzes affected their learning. Every student I spoke to said that, yes, they thought it helped them learn; noting the discussion, taking the time to really think about the problems and the immediate feedback. Incidentally, that is exactly what Dr. David DiBattista (the multiple choice exam guru from Brock U) and colleagues have reported in their studies.

What most struck me though was that these students were having fun! Fun, engaged, learning – seems good to me!


DiBattista, D. (2005). The Immediate Feedback Assessment Technique: A learner-centered multiple-choice response form. Canadian Journal of Higher Education, 35, 111-131.