## Scoring: for fun and extra credit!

January 3, 2013 in Uncategorized

PeerWise includes several *“game-like” elements* (such as badges, points and leaderboards) which are designed primarily for fun and to inject a bit of friendly competition between students. As an example, students accumulate points as they make their contributions and their score is displayed near the top right corner of the main menu.

In fact, if you have participated in your own course, perhaps you have noticed your score increasing over time?

Of course, not all students are motivated by such things, but a quick search of recent Twitter posts reveals that some students really seem to enjoy earning the various virtual rewards that are on offer:

Some instructors have even considered using these elements to award “bonus marks” or “extra credit” as a way of motivating their students. Obtaining the data to verify that students have met certain goals is trivial – for example, instructors can view the scores of all of their students in real-time by selecting “View scores of all students” from the Administration menu:

However, one difficulty with using the score for awarding such credit is that coming up with *realistic target scores* is complicated by the way the scoring algorithm works. The algorithm has previously been discussed in detail, but basically it rewards students for making contributions that are valued by their peers. In order to achieve the highest possible score, a student must make regular contributions and:

- author questions that their peers rate highly
- answer questions correctly before their peers
- rate questions as they are subsequently rated by their peers

What this means is the total number of points that a student can earn depends on how often their classmates endorse their contributions, and this is dependent not only on the size of the class but also on the “requirements” placed on the activity by the course instructor. This makes it tricky to set reasonable score targets for students to reach. Recently on the PeerWise-Community forum, member Brad Wyble raised exactly this point:

I’m interesting in linking the peerwise score to extra credit points but I’m a little stuck on how to proceed without an idea of what the range of possible values might be. I don’t need to know an exact number, but is it possible to provide a rough estimate given a course of size 60? And how would this estimate change for a size of 150?

So, what is the typical range of PeerWise scores for a class of a given size? Let’s start with a class of 50 students. It turns out the range can be quite wide, as exemplified by the two extreme cases in the figure below (each line represents the set of scores for a single class of 50 students – the average number of questions authored and answers submitted by students in each class are shown in the legend):

Not only were students in the “blue” class all highly active, but almost all of them made contributions in **each of the three areas** required for maximising the score: *question authoring*, *answering questions*, and *rating questions*. On the other hand, students in the “red” class were all quite active in answering questions, but **only a few students** in this class were active in **all three** areas. In fact, only the first 12 students had non-zero component scores for each of the three components. The remaining students scored 0 for the *question authoring* component (most likely because they chose not author any questions). Students 13-24 in the figure had component scores only for *question authoring* and *rating questions*, whereas the remaining students (all below student 25) only had a single component score (for *answering questions*). These students chose not to evaluate any of the questions they answered, and ended up with very low total scores (even though in some cases they may have answered many questions).

To calculate a “typical” range of scores for classes of varying sizes, we can average the class scores over a number of classes. For example, to calculate the typical range for classes of approximately 200 students, a set of 20 classes were selected (where class sizes ranged from 185 to 215) and the student scores for each class were listed in descending order. To calculate the average “top score”, the top score in each of the 20 classes was averaged. Likewise for the second top score, and so on, averages were calculated in decreasing order for all remaining scores. The figure below plots the average set of scores for classes of varying sizes (approximately 50, 100, 150 and 200) by averaging the class scores across a series of sample courses (in each calculation, between 15 and 20 classes were examined).

Brad also makes the following point in his forum post:

I suppose that another option would be to compute the grading scheme at the end of the semester once we see what the distribution of point values are.

This is an excellent idea – in many cases, the range of scores for a given course appear to be fairly consistent from one semester to the next (assuming the class size and participation requirements do not vary greatly). The figure below plots the set of PeerWise scores for one particular course over 6 different semesters. The class size was fairly consistent (around 350-400 students) and although the scores do vary, there is probably enough consistency to give instructors in future semesters some idea of what to expect (which may help them define targets for awarding bonus marks or extra credit).

In this class, only the very top few students achieve scores above 6000. It is interesting to note towards the right hand edge of the chart, the very sharp drops in the curves correspond to the students who have not made contributions in each of the three areas. Earning points in each of the *question authoring*, *answering questions*, and *rating questions* components is critical to achieving a good score – and it is probably important for instructors to emphasise this to students (although this information is shown when students hover their mouse over the score on the main menu).

Has anyone tried using the points (or the badges) as a way of rewarding students with extra credit or bonus marks? It would be interesting to hear of your experience – please share!

Mark Bender said on January 3, 2013

I use PW with third year undergrad business students in my Marketing Law elective subject. Typically 60 – 80 enrolled each semester. In order to ensure participation, PW activity forms part of the assessment. It is worth 10% of the final grade. I basically curve the total score, so the top score gets 10/10 and I use my Faculty presumptive curve to award marks below that, based on the score.

I do set a minimum ‘hurdle’, in that students are required to contribute at least 20 questions and answer at least 40 questions in order to pass this assessment. That hurdle may be a bit low?

Hayo Thielecke said on January 4, 2013

In my view, the gamification aspects of Peerwise are its best features. If you have a few good students in the module who enjoy the challenge of the leaderboards, it makes all the difference in the Peerwise experience for both students and teachers. So in order to reward good contributions, I have given bonus marks for particularly good questions. The bonus marks were only given when the teaching assistant and I thought the question and explanation were genuinely insightful, as opposed to automatically giving marks on score alone. I was a bit concerned that cliques of students could find a way to game the score algorithm by rating each other’s questions and comments. Nonetheless, the quality rating and score are a useful way to identify students who may deserve bonus points, so that we only needed to inspect a few questions and not hundreds when giving bonus marks.

Brad P Wyble said on January 4, 2013

Thanks Paul, this is extremely helpful.

The thought that occurred to me when reading this is to reward both extreme effort (e.g. the top X positions on the leader board earn Y points) but also allow everyone to earn something (e.g. extra points for hitting thresholds like 500, 1000 and 5000 points). Having seen these stats allows me to put those thresholds into the syllabus.