You are browsing the archive for Publications.

Students learn by generating study questions – but the content matters!

November 3, 2014 in Publications

About a year ago, in a large first-year science course at the University of Auckland, students were asked what they felt was most useful about creating, sharing and practicing with their own study questions (the class was using PeerWise, and this was one of several questions students were asked about their experience).

I remember reading through some of the students’ responses, and this one always stuck out to me:

“You don’t really understand how much or how little you know about a concept until you try to devise a good, original question about it”

It seemed to almost perfectly reflect the old adage that in order to teach something (which is, in essence, what students do when they explain the answers to the study questions they create), you must understand it well yourself. And this seemed to be a common perception in this class – overall, students felt that authoring their own questions was more helpful than answering the questions created by their peers. This is illustrated in the chart below, which compares the responses of the class when asked to what degree they felt authoring and answering was helpful to their learning in the course (rated on a typical 5-point scale from “strongly agree” to “strongly disagree”):

So, at least in this class, students felt that question authoring was most helpful to their learning. The instructors felt this was a positive sign, particularly because PeerWise meant they were able to run this activity in their large class with little moderation. But of course student perceptions of learning, and actual learning, are not the same thing! The central question remains – does this activity actually help students learn?

A good starting point is to look at the relationship between student engagement with the activity and their performance in the course – both of which can be measured in various ways. One measure of student engagement is the PeerWise reputation score (an approximate measure of the value of a student’s contributions), and the most obvious measure of course performance is the final mark or grade in the course. The chart below shows this relationship for the surveyed course described above:

To make the relationship clearer in the chart, students have been binned according to the final grade they achieved in the course. At the University of Auckland, there are 12 possible grades: 9 passing (from C- to A+) and 3 failing (from D- to D+). The chart plots the average final course mark, and the average PeerWise reputation score for all students who earned a particular final grade in the course. In this case, students who engaged most actively with PeerWise, tended to perform better in the course.

This relationship between student engagement with PeerWise and exam scores or overall course performance appears to be quite robust. A number of excellent studies have emerged this year, across various academic disciplines, that highlight this link. Hardy et al. report a significant positive correlation between students’ use of PeerWise and their exam performance in 5 large science courses, taught across 3 research-intensive universities in the UK, spanning the subjects of physics, chemistry and biology. Galloway and Burns examined the use of PeerWise by first year chemistry students and report a significant correlation between student activity with PeerWise (as measured by the reputation score) and their exam performance. Similar positive correlations have been reported recently by McQueen et al. (in biology), Singh (in computer science), and by Kadir et al. (in medicine):

So the evidence is clear – students who engage more with PeerWise also tend to perform better in their respective courses. While establishing this relationship is a useful first step, it does not answer our central question regarding student learning. The correlations do not imply that the use of PeerWise by the more successful students has caused their superior performance. In fact, it would be quite a surprise if we didn’t see this positive link. I think most instructors would probably agree that for any kind of course activity, the better students (who tend to earn the better grades) are the ones who are more likely to participate to a greater extent.

To further explore the impact on learning, we recently conducted a randomised, controlled experiment in a first-year programming course (in which engineering students were learning MATLAB programming) at the University of Auckland. Students were randomly assigned to one of two groups (called “Authoring” and “Non-authoring”) to control for their ability. Students in the “Authoring” group were asked to publish 3 study questions on PeerWise prior to a summative mid-semester exam. Students in the “Non-authoring” group could access all of the created questions on PeerWise, but did not author any of their own (NB: for fairness, at a later point in the course we switched these conditions for all students):

This was an “out of class” activity – students participated in their own time, and typically the hours between 8pm and 10pm were when most questions and answers were submitted. This took place over an 11 day period prior to a mid-semester exam that consisted of 10 questions. The activity was almost entirely student-driven – other than setting up the practice repository on PeerWise for their students and setting the exam questions, the instructors were not involved in the activity.

A total of 1,133 questions were authored by students in the “Authoring” group, and a total of 34,602 answers were submitted to these questions by students in both groups as they practiced prior to the exam. So, what was the impact on exam performance?

As a group, the “Authoring” students performed better than the “Non-authoring” students on 9 of the 10 exam questions – a binomial test reveals that this is statistically unlikely to have happened by chance (p = 0.0107). In terms of the average exam scores achieved by each group, there was a difference – but it wasn’t particularly large. As shown in the chart below, the “Authoring” students performed about 5% better than the “Non-authoring” students, again a statistically significant result (Wilcoxon test, p = 0.0197):

While the superior performance of the “Authoring” students on this mid-semester exam can be attributed to their use of PeerWise (more specifically, to their authoring of the study questions), this doesn’t necessarily mean that there aren’t more effective ways for students to study. For one thing, we don’t know how the “Non-authoring” students spent their time while the “Authoring” students were creating questions – we certainly can’t assume that they spent the same amount of time preparing for the exam.

What happens if we take a closer look at the content of the questions? This is where things get more interesting.

In an article published in 1994, entitled “Student Study Techniques and the Generation Effect“, Paul Foos suggests a reason for why we may have seen such a small difference between the average exam scores of each group. He argues that students who prepare for an exam by generating study questions may benefit only if they create questions on topics that are targetted by the exam questions. This certainly makes intuitive sense – some of the students in our “Authoring” group probably created perfectly “good” questions, but these questions did not target the concepts that were examined by any of the 10 exam questions, and thus they didn’t benefit as a result.

To explore this, we classified all 1,139 student authored questions according to the main topics that they targetted, and we did the same for the 10 exam questions. For simplicity, when we focussed on questions that targetted a single topic, we discovered that there were 3 core topics that were each targetted by 2 exam questions. For each of these three topics, the students can be classified into three groups:

  • the “Authoring” students who created at least one question on the topic
  • the “Authoring” students who did not create any questions on the topic
  • and the “Non-authoring” students

The chart below plots, for each of the three topics, the proportion of students in each group that correctly answered both exam questions on the topic:

We see virtually no difference between the performance of the “Non-authoring” students and the “Authoring” students who did not create questions on a topic, when answering exam questions on that topic – precisely as described by Foos’ earlier work. Students who did author questions on a particular topic performed far better on the corresponding exam questions. The impact of question authoring on learning also becomes clearer – with effect sizes of between 10% and 20% across the question pairs.

Of course, the story doesn’t end here. Although the question authoring activity did have a significant positive impact overall, some of the differences observed between the on-topic and off-topic “Authoring” students may be a result of students choosing to author questions on topics they already knew well, rather than learning much new from the process.

It is hard to say a lot more about this without more data – but like the correlation studies mentioned earlier, this helps to paint a picture of an activity which, with very little instructor involvement, can have a measurable positive effect on student learning!

Do badges work?

November 19, 2013 in Publications, Talking point

Badges everywhere

Have you ever wondered whether some of the “game-like” rewards that are becoming more and more common online actually have a measurable impact on user participation?  Does the promise of earning a “Hotel specialist” badge on Trip Advisor motivate travellers to write more reviews?  On Stack Overflow, a popular question and answer forum for programmers, do people answer more questions than they otherwise would so that they can increase their reputation score and earn a higher spot on the global leaderboard?

Of course, if you play games these kinds of rewards are nothing new – performance in many games is measured by points, leaderboards have been around since the earliest arcade games, and the Xbox Live platform has been rewarding players with achievements for nearly a decade.  Now, in an attempt to motivate users across a broad range of applications, we see these game-like elements appearing more frequently.  But do they work?

Badges in PeerWise

PeerWise includes several game-like elements (points have been discussed on this blog before), including badges (or “virtual achievements”).  For example, regular practice is rewarded with the “Obsessed” badge, which is earned for returning to PeerWise on 10 consecutive days and correctly answering a handful of questions each time.

Other badges include the “Insight” badge, for writing at least 2 comments that receive an agreement, the “Helper” badge for improving the explanation of an existing question, and the “Good question author” badge, awarded for authoring a question that receives at least 5 “excellent” ratings from other students.  A complete list of the available badges can be seen by clicking the “View my badges” link on the Main menu.

As you would expect, some badges are much harder to earn than others.  Almost every student earns the “Question answerer” badge – awarded when they answer their very first question.  The following chart shows the percentage of students with the “Question answerer” badge that earn each of the other available badges.  Only about 1 in 200 students earn the “Obsessed” badge.

The badges in PeerWise can be classified according to the roles that they play (there is a nice article by Antin and Churchill that explores this further):

  • “Goal setting”: helping the student set personal targets to achieve
  • “Instruction”: helping the student discover features of the application
  • “Reputation”: awarded when the quality of the student’s contributions are endorsed by others

It is interesting to note that most of the badges awarded for answering questions are of the “Goal setting” variety, whereas those awarded for authoring questions are mainly in the “Reputation” category.

And now back to our original question – do these badges have any influence over the way that students use PeerWise?  When considering this question, we must keep in mind that observed effects may not necessarily be positive ones.  One of the criticisms levelled at extrinsic rewards, such as game-like elements, is that they have the potential to undermine intrinsic motivation in a task, which is clearly of concern in an educational context.  However, this is a somewhat contentious claim, and very recent work by Mekler et al. showed no negative impact on intrinsic motivation in an experiment measuring the effect of using game elements to reward user participation in an online image-tagging activity (although it must be noted that this was a short-term study and motivation was self-reported).

Anecdotal support

There is certainly some anecdotal evidence that the PeerWise badges are being noticed by students in a positive way.  Examples of this include public tweets:

as well as responses to a survey conducted in 2012 at the University of Auckland:

“I didn’t think I was “badge” type of person, but I did enjoy getting badges (I was the first one to get the obsessed badge – yay!). It did help motivate me to do extra and in doing so, I believe I have learnt more effectively.”

“The badges did make me feel as if I was achieving something pretty important, and helped keep Peerwise interesting.”

Another example was nicely illustrated in a talk given by James Gaynor and Gita Sedhi from the University of Liverpool in June this year, in which they presented their experiences using PeerWise at a local teaching and learning conference.  On one of their slides, they displayed a summary of student responses to the question: “Was there any particular aspect of PeerWise you liked?

Across the two courses examined, “badges” and “rewards” emerged quite strongly (points, rewards, achievements and rankings were coded as “Other rewards”).

However, it should be noted that not all students are so positive about the badges.  Other responses to the previously mentioned survey indicate that the effect on some students is fleeting:

“well, it kinda increase my motivation a bit at the beginning. but then i get bored already”

“They don’t really affect my motivation now, but they did when I first started.”

and others pay no attention to the badges at all:

“I never cared about the badges -> simply because they dont mean anything -> i.e. does not contribute to our grade”

“They did nothing for my motivation.”

Controlled experiment

To understand the impact of the badges more clearly, we conducted a randomised, controlled experiment in a very large class (n > 1000).  All students in the class had identical participation requirements (author 1 question and answer 20 questions), however only half of the students were able to see the badges in the interface and earn them for their participation.  This group was referred to as the “badges on” group, whereas the control group who were not able to see the badges were referred to as the “badges off” group.  The experiment ran over a period of 4 weeks in March 2012, and the class generated approximately 2600 questions and submitted almost 100,000 answers.

Students in the “badges on” group, who were able to earn the badges, submitted 22% more answers than students in the control group.  The chart below plots the day to day differences over the course of the study – on all but one day, the “badges on” students submitted more answers than the “badges off” students.

The table below summarises the number of questions authored, answers submitted, and distinct days of activity for students in each group.


The presence of the badges in the interface had a significant positive effect on the number of questions answered and the number of distinct days that students were active with PeerWise.  Interestingly, although there was no effect on the number of questions authored by students, no negative effects were observed – for example, the increase in the number of answers submitted did not lead to a reduction in the accuracy of those answers.

If you you would like to see additional data from this experiment, as well as a more complete discussion and acknowledgment of the threats to the validity of the results, the full paper is available online (and on the PeerWise Community Resources page).  Of course, no experiment is perfect, and this work probably raises more questions than it answers, but it does provide some empirical evidence that the badges in the PeerWise environment do cause a change in the way that students engage with the activity.  Perhaps we could see similar effects in other, similar, educational tools?

TLDR: And for those who prefer movies to reading, the conference in which this work was published required a brief accompanying video.  If you are really keen, see if you can last the full 40 seconds!