Part 3 - "How to" Questions on Implementation

Here are the specific Questions. Click on these to go directly to the Question and the Answer.

1. How should I form the teams?
2. How big should the teams be?
3. What should I do if students have difficulty with the required pre-class reading?
4. How do I set up an effective grading system?
5. How can I provide grade-based incentives for group work and still avoid giving higher grades than the weaker students deserve?
6. Should I give a RAT every class period (or every week)?
7. How much should the RATs count?
8. What should I do when students are absent?
9. Should I use peer evaluations on all projects?
10. How should I handle the scoring of appeals?
11. How should I give feedback on multiple-choice questions?
12. How can I provide immediate feedback on the RATs if I don't have a scoring machine?
13. What are the benefits of using a "split answers" format on the RATs?
14. Are there alternatives to using multiple-choice questions on the RATs?
15. What kinds of questions are good questions for the RATs?
16. Can I use a writing assignment for the application phase of team-based learning?
17. Is it OK to use group presentations?
18. If I can not use group papers or presentations, what can I use?

Part 3 - "How to" Questions on Implementation

Question #1 How should I form the teams?

In forming groups, we recommend trying to do three things. One is spreading assets and liabilities (i.e., background factors that are likely to make a difference in students' performance in this course) across the teams. Assets are often such things as attitudes toward and/or performance in previous course work, course-related life experience; liabilities often include such factors as no (or poor) preparation in related courses, language barriers, etc. Thus, the team formation process should be criterion-based. A second objective of the team formation process is to avoid pre-existing, cohesive sub-groups (e.g., a group of three students from the same fraternity and three students who did not previously know each other would probably struggle). For these two reasons, teams should not be self-selected. Third, the process you use for team formation should foster the perception that none of the teams was given a special advantage. Thus, we recommend using a very public team formation process.

As a practical matter, I literally have students form a single line, in order of a set of categories with the "most rare but important" category first. For example, in undergraduate Management Principles courses the rare and important category is usually full-time work experience, so that is the first category. (By contrast, I rarely use work experience as a category in graduate courses because most students have worked for at least a year or two.) The next category for undergraduates is typically "Those who were born and raised outside of Oklahoma and the immediately adjacent states." Students array themselves from most to least distant; this makes it possible to spread out English-as-a-second-language students without specifically singling them out. Next, I have accounting and finance students join the line (they tend to see things in "black or white") followed by management and marketing students (who are more comfortable with various "shades of gray").

In graduate classes my most important but rare category is typically Ph. D. students and/or students from programs outside of business -- e.g., I often have several law students and want them in separate groups. As a result, I start off by asking students to raise a hand if they are from outside the college of business. Then I have them line up alphabetically by discipline. Then I invite the "born and raised outside of Oklahoma and the immediately adjacent states" category to come next and the final set of categories has to do with undergraduate majors.

When the entire class is standing in line, I then have them count off by the number of groups I want. When the counting-off process is finished, I have the groups assemble themselves in designated locations in the classroom.

Question #2 How big should the teams be?

The size of the teams always represents a compromise between being large enough to have sufficient intellectual resources to complete the assignments and small enough to develop into true teams. Historically, we have found that if teams have at least five members, they usually have the intellectual resources to complete the team assignments. On the other end of the spectrum, we have found that groups larger than 7 tend to have difficulty in the team development process. Hence the optimum size for team-based learning is 5 to 7 students.

Question #3 What should I do if students have difficulty with the required pre-class reading?

In some cases, students' reading limitations are a very legitimate cause for concern. These include: difficult subject matter, poorly written reading materials, insufficient reading skills, limited [English] language skills, and physical and/or mental handicaps. Although the answer as to what to do about it depends, to some extent, on the reason(s) that students are having difficulty, the severity of the problem can be greatly reduced by one or a combination of the following:

1. Prepare a reading guide containing questions that students should be able to answer after having read the text.
2. Allow a limited question and answer session (e.g., let each group pick 1 question immediately before each of the RATs).
3. Create short text supplements (often a single page or less) that clarify specific issues that create problems for students.
4. Create and/or locate web-based tutorials (note: publishers are frequently making these available as part of a text-purchase package).
5. Create and make available an audio "walk-through" of the text. For example, a physics colleague largely resolved his students' reading concerns in less than two hours by talking into a tape recorder as he thumbed through the pages of his text. He then allowed students to make their own copy of his tape to guide their individual study.
6. Provide tutorial help to individual students and/or teams.
7. Develop teams to the point that members voluntarily help each other prepare for the RATs.
8. If all else fails, use class time (usually a small fraction of that required in a lecture course) to remediate misunderstandings that surface during the Readiness Assurance Process.

Question #4 How do I set up an effective grading system?

An effective grading system for team-based learning contains three essential components: individual performance, team performance, and peer evaluation. The individual performance component provides a basis for student accountability to the instructor and to each other. The group performance component provides incentives for the development of group cohesiveness and justifies putting effort into group work. The peer evaluation solves two important motivational problems. One is providing an incentive for individuals to participate in group discussions. The other is removing the students' fear that they will have to choose between getting a low grade on the group assignments and having to "carry" the group work (if other group members fail to do their fair share).

The final weight of each of these three components should be a balance of three factors. First, each of the components should be given enough weight so it is clear to students that the instructor thinks it is important. Second, the instructor must be personally comfortable with the relative weights in the chosen grading system. Third, the grading system must be responsive to student concerns for fairness and equity.

In our classes, we involve students in the development of the grading system through an exercise called "Setting Grade Weights". This is an exercise in which we set maximum/minimum limits for each component of the course grade, and then representatives of the groups negotiate a mutually acceptable set of weights for each of the grade components. Over the years, it has proven to be a highly effective way to create a grading system that is both acceptable to all the parties involved and supports the individual and team behaviors that are necessary for learning and team development.

Question #5 How can I provide grade-based incentives for group work and still avoid giving higher grades than the weaker students deserve?

A relatively common problem with new team learning users is finding a way to count group work in their grading system without giving higher-than-deserved grades to some of their students. The apparent dilemma is created by the fact that most groups will score over 90 on most tests and other assignments. Further, even though team-based learning increases both attendance and pre-class preparation, users sometimes find that when a large percentage of the grade is based on group work, locking themselves into a 90 = A, etc. grade distribution forces them into "giving away" grades to students who really do not deserve them.

One way to at least partially avoid the problem is to not use percentages as the foundation for determining students' final course grades. Even though I carefully explain my rationale to the students, I still get some initial resistance. But it dissolves as students come to realize that you will be fair with them. (Note: Students are almost universally convinced that there is something magic about a 90. In my experience, however, it is really an arbitrary cut-off because most instructors try to peg the difficulty level of their tests in order to give the "appropriate" number of A's. If the test turns out to be too difficult, they curve it up, give an easier test the next time, or both. If the test is too easy, they give a harder one next time.)

As a practical matter, I generally work with an overall 1,000 point base and also "1,000-point within category" base. Thus, if group analysis of a movie or novel is worth 30% of the group part of the grade, I treat it as though a "perfect" score would be 300. I also have just over 100 questions on my total of 6 RATS and the Group RATs are 30% of the group grade. At three points/question, that adds up to be about 300 points for the groups. As a result, I simply have my spread sheet add up the total of the raw scores on the six RATs.

At the end of the semester, I multiply the subset totals by the weights determined by the class (e.g., if students have set the weight of the group part of the grade at 65%, I simply multiply the total of the raw group scores by .65). Then I sum the weighted points from the three major grading components (individual scores, group scores and, peer evaluations). This gives me an overall score for each student. I then have my computer sort the students into a distribution based on this weighted total score and look for "break points" for A's, B's, etc. at approximately the level that I think represents a "fair" overall grade distribution. I typically take three factors into account. These are: 1) grade norms for similar courses [I use this as a base.]; 2) my personal impressions about how well the students have performed relative to other classes that I have taught, and; 3) extenuating circumstances (e.g., textbook problems, weather problems, etc.).

Question #6 Should I give a RAT every class period (or every week)?

This answer is an unequivocal NO! In fact, when someone says they tried team-based learning and they got a negative student reaction my first question is: How many RATs did you give?" At least 80% of the time they will answer that they have given at least 12-15. If you give too many RATs, you are likely falling into a trap of pushing students into memorizing details that are not really significant to either you or the students.

The primary purpose of the RATs is to assess student readiness to engage in related application-focused activities, all of which are open-book. The RATs should focus on ensuring that students have an understanding of the basic concepts that is thorough enough to allow them to: 1) tackle the application-focused assignments, and 2) use their assigned readings as reference material to digest additional detail as they work. Thus, the test questions should focus on the concepts that would be found in a typical Table of Contents, not just in the Index. In my judgement, focusing on details is counterproductive for a variety of reasons. First, it limits the amount of material that students are willing to be responsible for, because they feel like they have to memorize everything. Second, if your goal is long-term recall of the material, then you are better off really reinforcing the fundamental concepts (the details are not all that important - partly because they are always changing - plus, students will never retain them anyway). Third, they can pick up - and are more likely to retain - the details when they are encountered in a meaningful context as they are working on the applications.

Question #7 How much should the RATs count?

A primary benefit of the individual RATs is that they make members accountable for their individual preparation. Students realize that, because of the give-and-take that occurs during the group test, they will not be able to hide a lack of individual preparation for the group RATs. Thus, regardless of how much the individual scores actually count toward the course grade, they make members accountable to their group. As a result, as long as the group RATs and the peer evaluation both have a significant impact on the course grade, it is not essential that individual RATs count very much at all. On the other hand, even though we typically develop a grading system using an exercise that gives students the option of not counting the individual RATs at all, they rarely choose that option. Instead, they typically set the weight for the entire set of individual RATs at about 10% and the group RATs at about 15% of the overall course grade.

Question #8 What should I do when students are absent?

In general, I try to create a situation that is similar to what students will experience in the workplace. Thus, I say something like:

In the workplace, when someone is gone, the group has to pick up the slack but the absent member still benefits from the group work. If the absent person has a good reason for being gone, explains the reason to the group, and does their best to make amends, most groups will gladly extend the benefit. If, however, members have doubts about the reason for the absence, feel like the member is trying to "freeload" or both, then the absence is likely to be a black mark that may not be forgotten when the peer evaluations come around. So, if you have to be absent, let your peers know in advance and make sure that you do your best to make up for it. Otherwise, you are at risk.

Then, if they do need to miss a RAT or some other graded activity, I give them the team score and also allow them the opportunity to take the individual test using one of two options. One is to take the test early so that I can let their team have access to their individual answers during the team test. The other option is to take the test later. Some professors are uncomfortable with the idea of giving make-up tests. They worry that allowing some students to take a late exam will give them an unfair advantage over other students but see the task of creating comparable but different exams as extremely time-consuming. It turns out that neither of these problems are as big as they seem.

In my case, I do not create an alternative exam. If a students misses a RAT when it is given in class (and few do), I leave a copy of the regular exam with the departmental secretary. Then, any student who needs to take a make-up RAT contacts her to make arrangements to take the exam in a nearby room. In the many years I have done this, the vast majority of students taking such make-up exams get scores that are lower than their average on the RATs they take at the normal time. This means I can simply use the same exam for the make-up exam, and doing so does not give students any significant, unfair advantage.

If allowing make-up RATs turns out to be too much of a problem, another option is to set up the grading system that allows students to drop one individual RAT score. When students calculate their overall score for the individual RATs, for example, they would only count the best 5 out of 6. If they have no absences, they get to drop their lowest score. If they have one absence, they simply have a "0" for that score and drop it. If they have more than one absence, they get to drop one of those but they have to calculate the other "0's" into the total. The advantage of this approach is that it reduces the hassles associated with giving make-up exams. The disadvantage, and one that I personally feel is important, is that it tends to eliminate the opportunity to reward students who are willing to put in the effort to be prepared on every test.

Question #9 Should I use peer evaluations on all projects?

Although it is essential to have a peer evaluation that counts at the end of the course, we see arguments both ways with respect to every project or even mid-course peer evaluations. The strongest argument in favor of frequent peer evaluations is that they give members earlier feedback on their behavior and hence an earlier opportunity to change it. Some students are honestly unaware that some of their behavior is seen as counter productive by their peers and are willing to try very hard to change, when they learn that change is needed.

There are, however, two arguments against frequent peer evaluations as well. One is that having members evaluate each other tends to disrupt the development of group cohesiveness. The other is that we have seen a few instances in which an assertive member received high evaluations for taking charge on early assignments. This encouraged the person to dominate the group even more on later assignments but led to the person receiving the lowest peer evaluations in the group at the end of the course.

Thus, if you choose to do peer evaluations along the way, we recommend: 1) providing a mechanism for giving anonymous comments; 2) making sure that members give each other both positive and negative (i.e., "room for improvement") feedback; and 3) counting the later feedback the most.

Question #10 How should I handle the scoring of appeals?

There are several issues in relation to handling score changes that result from appeals. However, in every case we think it is important to make sure that we are not reinforcing a behavior that we really do not want. For example, we make it clear that any team has the opportunity to submit appeals; but if they choose not to appeal, we hold them responsible for their choice (i.e., an appeal granted for one team will not improve the scores of other teams). In doing so, we also make it clear that it would be unethical to teach in a way that implied that it is OK for students to sit back and let others do their work for them. Similarly, we only accept appeals on the questions the teams have missed; allowing individual appeals would eliminate the incentive for members to stand up for their point of view during the team test. However, if we grant a team appeal, we also change the scores of each member who had the same answer as the team. Finally, we do not take away points from a member who had correctly answered the question in the first place.

Question #11 How should I give feedback on multiple-choice questions?

There are two aspects to the feedback process. One is the actual scoring of the tests. The other, which is by far the most important, is providing students with information on how their overall performance compares to other teams and whether or not each of their answers was correct. The simplest way to score the tests is using a portable test-scoring machine. We use a Scantron 888P+ scoring machine that we actually take into the classroom. This allows us to score the individual tests while students are taking the team tests.

We provide immediate feedback on the team tests in one of two ways. The most effective method is using IF AT answer sheets (Epstein, 2000, http://www.epsteineducation.com) to provide both immediate feedback on a question-by-question basis and partial credit if students have partial, but imcomplete, understanding of the concepts on which the questions are based. With IF AT answer sheets, students scratch off the covering of one of four or five boxes in search of a mark that indicated that they have found the correct answer. If they find the mark on the first try, they receive full credit. If not, they scratch until they find the mark, but their score is reduced with each unsuccessful scratch. The other method is having a member score their answer sheet on the Scantron scoring machine as soon as the teams complete their test. We then have the team member post the score on the board or on an overhead transparency (to provide the inter-team comparisons) and pick up the team folder containing the individual (already-scored) answer sheets.

Having immediate access to both the individual and team answer sheets provides immediate feedback on two key aspects of their work. One is their level of mastery of the assigned readings. The other is whether or not they effectively incorporated the input that was potentially available from each team member. Further, because the feedback is literally instantaneous, the IF AT answer sheets have a particularly powerful and positive effect on the teams for two quite different reasons. One is that they very rapidly learn the value of voicing and listening to each others' ideas. As a result, they are able to develop and benefit from effective interaction patterns in a fraction of the time it would take in a less feedback-rich environment. The other is that, even if they don't find the correct answer until the second or third scratch, members experience, and are motivated by, a series of immediate successes from working together.

We have also discovered a way to promote team development by limiting the feedback we provide in machine scoring the answer sheets. If we give information on which answer was correct (which is one of the automatic scoring options) teams often fail to discover situations in which the team missed a question that was answered correctly by a team member. On the other hand, when we limit our feedback to simply identifying the incorrect answers, their first question to each other is, "Did anyone get this one right"? Thus, with no coaching at all, teams obtain the feedback that enables them to develop a decision-making process that is effective in obtaining input from even their most timid members.

Question #12 How can I provide immediate feedback on the RATs if I don't have a scoring machine?

In reality, actually scoring the individual tests isn't essential for either learning or team development. What is important is providing immediate feedback. Thus, in large classes, if we do not have a scoring machine, we do not actually score the tests until later. Instead, we have team representatives turn in their team folder containing the individual answer sheets before they start the team test. When the teams finish their tests, we provide an answer key that they use to "score" their team test as soon as they turn in their answer sheet. Further, as long as members record their answers on the test itself (which they have to retain to take the team test), providing feedback on the team tests also enables members to "score" their own individual tests

You can also use class discussion to let students "discover" the correct answers. This involves having teams alternate in giving their answer to the individual questions and, when teams disagree, moderating the discussion between teams. Further, because the teams are challenging each other (and not the teacher), it is also possible to use the discussion to grant credit for an alternative answer in the same way as would normally be done with written appeals (i.e., declaring more than one "winner").

Another alternative is quickly hand-scoring the individual tests using an answer sheet with companion answer key in the form of a transparency overlay with clear windows (for correct answers) and shaded boxes (for incorrect answers). Then you can let the teams use the overlay to quickly score their own tests.

Question #13 What are the benefits of using a "split answers" format on the RATs?

Early on, most team members are somewhat timid about challenging each other out of fear of being perceived as overly aggressive. As a result, newly formed groups always look for way to incorporate everyone's input without having to risk a confrontation between members with differing views. With normal true/false and multiple choice questions, their natural inclination is selecting answers by a majority vote after a very brief discussion. However, voting with minimal discussion tends to hamper both learning and team development because members tend to withhold information that would stimulate give-and-take discussion. Further, the fact that voting often produces the correct answer tends to obscure the importance of obtaining input from everyone in the group.

Once we understood the dynamics of the process, we were able to develop a procedure that largely solves the problem. Our solution is providing a way for teams to incorporate everyone's input without having to risk a confrontation between members with differing views. This procedure highlights, instead of obscures, the importance of getting input from every team member. With this system, we have each question count for three points. If we are using machine-scored forms, we have students answer each RAT question three times (i.e., as though it was three separate questions and number the questions accordingly). Thus, the first question would be numbered 1-3, the second would be 4-6, etc. and we enable them to "split" their answers by filling-in up to three different answers to each question. For example, if they want to "hedge their bets" on the first question (which we would have numbered 1-3), they might want to answer A on answer spaces 1 and 2, and B on answer space 3. In this case, if the correct answer was A, they would get 2 of the three possible points. If the correct answer was B, they would get 1 point and they would get no points at all for a C, D or E. If we are hand-scoring the questions, we have students distribute three points in the boxes on each row of the answer sheet.

Early on, most groups choose politeness over rightness by splitting answers to smooth over member differences. By allowing split answers, however, we focus their attention on the fact that they incur a cost (i.e., the loss of a point) every time they reach a decision without exploring the reasons behind members' choices. This, in turn, provides a powerful incentive for members to start talking out differences instead of sweeping them under the rug. As a result, teams quickly develop to the point that give-and-take discussions are the norm and they are rewarded by increasing their ability to incorporate members' input into their decisions.

Question #14 Are there alternatives to using multiple-choice questions on the RATs?

The critical attribute of any RAT question is whether or not it can be used to provide immediate feedback on both individual and team performance. Although multiple choice questions are particularly well-suited for providing immediate feedback, other types of questions will work as well. For example, one faculty member hands out a sheet containing "fill-in the blank", short answer essay, and other questions that are designed to test concept understanding by completing simple computations. When students complete their individual RAT, they slip it into a photo album page so that it is visible on both sides but they won't be tempted to write on it during the team test (same questions but on colored paper). When they finish their team test, the teacher gives them a copy of the answer key that his graduate assistant will use to correct the test. After comparing their test to the answer key (which provides immediate feedback on both the individual and team tests), students have the opportunity to fill out an additional form on which they submit any "appeals" or comments that they would like to have considered in completing the grading process.

Question #15 What kinds of questions are good questions for the RATs

In addition to providing the basis for immediate feedback on both individual and team performance, the test questions should also emphasize key concepts (i.e., avoid asking questions about inconsequential details) and enhance learning. With respect to enhancing learning, one characteristic is that at least some of the questions must be difficult enough to stimulate discussion. Otherwise, teams will simply defer to their best member. In addition, using related questions that require increasingly complex levels of understanding are particularly helpful for two reasons. First, if the questions are correctly chosen and sequenced, students can learn from the questions themselves while they are taking the RAT. For example, by asking one or two recognition-type questions followed by a question that requires synthesizing the concepts from the two earlier questions provides students with the opportunity to develop a deeper understanding of the concepts themselves. Second, questions that require higher level thinking skills are far more likely to stimulate the kind of discussion that promotes peer teaching.

Question #16 Can I use a writing assignment for the application phase of team-based learning?

In many ways using "good" in relation to "writing assignments for groups" is an oxymoron. Writing is inherently an individual activity. As a result, group writing assignments typically set in motion a series of events that: 1) tend to inhibit learning, and 2) often foster very negative attitudes toward group work. Learning suffers because when groups are assigned to create a lengthy document, the only rational way to accomplish this overall task is to divide up the work so that each member independently completes part of the assignment (usually the part that he or she already knows the most about). The negative attitudes are caused by the fact that if any member does not do a good job on his or her share of the writing, his or her peers will be forced to accept a lower grade or engage in a last-minute attempt to salvage a disaster. In fact, high-achieving students often express the feeling that getting an acceptable grade on a group term paper feels like having crossed a freeway during rush hour without being run over.

Question #17 Is it OK to use group presentations?

Group presentations have both a positive and a negative side. The work leading up to a group presentation is usually more productive than writing a group paper. Because they realize they will all be in the public spotlight at the same time, members tend to spend more time working together to make sure they are as least somewhat familiar with the major content issues. As a result, the preparation phase of group presentations often enhances both learning and interpersonal relationships. Unfortunately, the actual presentations are often close to a waste of time. For listeners, it is just another lecture except that, they usually have serious doubts about whether or not their peers are expert enough to be taken seriously. As a result, the energy level is so low that, in far too many cases, the only ones who benefit from the experience are the presenters.

Question #18 If I can not use group papers or presentations, what can I use?

In general, the value of group assignments is largely a function of the extent to which they cause members to engage in give-and-take, content-related discussions. As a result, the best group assignments require groups to make decisions that involve complex applications of course concepts. The thinking required is simply too complex to be handed out to individual members working on their own. Further, in addition to having the teams make a specific decision, having them all: 1) work with the same problem or case, and 2) simultaneously report their decisions dramatically increases the value of the group assignments for two quite different reasons. One is the resulting give-and-take discussions between teams produces timely, content-related feedback on students' concept understanding. The other is that it builds commitment to and enhances the problem-solving capabilities within the teams.