Learners Teaching Novices: An Uplifting Alternative Assessment

Assessment often harbours negative, adversarial feelings from students, who can feel like they are being judged and measured.

Compared to this, we believe AVT is more motivating for learners. This is because it combines assessment and public service, drawing on the learning benefits of former with the fulfilment and motivation of the latter. Since the end goal is to help a real novice student, learners feel a strong sense of responsibility to study the material in preparation. This is deepened by a stake of personal identity and embarrassment of letting down their novice students. Moreover, a successful AVT session has an observable impact on novices—a feeling that is gratifying for the learner hosting the session.

Having to explain a concept is a powerful method to deepen understanding (Bargh and Schul, 1980). In AVT, explaining concepts effectively to a novice requires a thorough understanding of the course material. Novice students often have fundamental misunderstandings that require teachers to have a strong mental model of the concepts. As learners engage with novice questions, they refine their own mental model and, in the process, uncover new misunderstandings of their own. In addition, when mistakes in understanding are unearthed and resolved in teaching, these moments are extremely memorable and leave a lasting impression in the teacher’s mind.

Allowing students to repeatedly retake assessments is extremely desirable (Grover, 2021). However, the practicality of generating novel questions or activities for every instance of retesting is a huge challenge. AVT resolves this predicament by providing a standardised yet dynamic evaluation platform that accommodates repeated assessments without necessitating the continuous creation of new tests. In theory, learners should be able to redo AVT as much as they want, resulting in a constant improvement of their understanding while helping more novices.

High-touch, personalised 1:1 help for students is one of the most impactful educational interventions known to researchers (Bloom, 1984). Unfortunately, 1:1 tutoring is prohibitively expensive for most students. Multiple investigations have shown that early learners can be incredibly effective tutors if given the right support (Maas and Pressler, 1973; Keller, 1974; Reges, 2003). With AVT, we channel the efforts of bright students, who would otherwise waste hours sitting an exam, into instead providing a meaningful service of tutoring novices. Such a system, if implemented broadly, could support 1:1 help for learners at a global scale, especially those with limited access to resources.

Assessment often suffers from the famous Godhart’s law: “when a measure becomes a target, it ceases to be a good measure”. For many assessments,

students begin to optimise by studying for the test over improving their understanding. One benefit of AVT is that there is no clear way to study for the assessment itself. If a student just wishes to perform well at AVT, their best bet is to actually improve their understanding of the material.

Cheating is another major concern in assessment, especially in the new era of large-language models (LLMs) (Milano et al., 2023). Several assessment platforms have started adding biometric checks like video recording to ensure an individual is completing an assessment by themself. There is also an increasing distrust of written exams in general, with assessors instead opting for some sort of oral interview where a participant’s understanding can be adaptively probed. This is effective but impossible to scalable. AVT naturally integrates both these dynamics into its design. AVT is a live help session, which automatically makes it a biometric verification, and helping novices with their issues is similar to an adaptive oral interview where the learner must draw on their understanding dynamically in response to issues as they arise.

To understand the benefits of AVT, we collected at a few outcome variables. Firstly, for each learner, we looked at their eventual score on the midterm/final exam on the question relevant to the topics of their AVT sessions. For the midterm, this was Q1, and for the final this was Q3. We also sent the learners an exit survey to collect information about how they prepared for their AVT sessions and how enjoyable they were. Lastly, expert instructors manually analysed the AVT session recordings to quantify how much understanding learners demonstrated, how effective they were at teaching, and whether the sessions were helpful for the novices.

To compare the AVT sessions to a strong baseline of studying methods for exams, we had a random subset of participants sit a 1 hour practice exam session instead of doing AVT. In this practice exam, they were given the same question discussed in the AVT sessions. This group will be referred to as PracticeExam. We also wanted to control for the selection bias of people willing to join an experiment, so we chose a random subset of participants to not do any kind of session (AVT or practice exam). This group will be called the $\mathrm{Control}_{S}$ group. Lastly we had the $\mathrm{Control}_{C}$ group, consisting of students who did not interact with our experiment at all. The summary of these groups can be seen in Table 1.

To investigate the helpfulness of AVT in improving learner understanding, we looked at the performance of learners in the midterm/final exam following their session. Table 2 shows the average score (in percentage points) obtained by the AVT learners on each question in the midterm and final exam, as well as the class averages ($\mathrm{Control}_{C}$).

We compute a one-sided $t$-test for the significance between the AVT score and the class average, with $p$-values provided in parentheses.²²2We compare to this group due to larger sample size but our results hold even when compared to $\mathrm{Control}_{S}$ as well.

Our results suggest a strong effect of AVT teaching on learner understanding. In 3 out of 5 questions on the midterm, the AVT group outperformed the class average ($p<0.05$), with improvements of 25 to 30 percentage points (p.p.). A similar trend is seen in the final, although the improvements are in the 10-15 p.p. range³³3Note: The final exam was much easier than the midterm, with higher average score and lower variance in scores. Reassuringly, we always see a significant increase in questions directly related to the AVT topics (Midterm Q1 and Final Q3).

Next we looked at whether these effects were due to a selection bias in our experiment population. In Figure 2, we compare performance on Midterm Q1 and Final Q3 across all our experimental groups. We see that there is no significant difference between the class average and our experimental participants ($\mathrm{Control}_{C}$ vs $\mathrm{Control}_{S}$). We can see also see that AVT was at least as effective as doing a practice exam—an extremely compelling outcome, since practice exams are known to be good ways to prepare (Balch, 1998).⁴⁴4We note that in the midterm, the PracticeExam group seemed to preform badly, but we believe this is a statistical anomaly due to small sample size ($\approx$ 4 students).

We also explored how motivated the learners were to study for their AVT sessions. To do this, our exit survey asked participants how long they spent preparing for their session. Figure 3 (left) shows the fraction of participants in the AVT group who spent more than 1 hour preparing for the session compared to those in the PracticeExam group. We can see this fraction is almost twice as high as for the AVT group, suggesting that AVT participants took their preparation more seriously. We also tried to measure how enjoyable AVT was for learners. Figure 3 (right) shows that leaners found the AVT sessions more enjoyable compared to those who did a practice exam. In addition to this, the AVT group also gave an average recommender rating of 4.2 out of 5 for their experience and provided some positive qualitative feedback:

“I had a lovely experience! I was grateful that it forced me to prep for graphics/animation for the final, and my student was so nice :). I felt encouraged in my teaching skills and programming knowledge.”
— Learner 1

“Having to explain graphics/animations concepts to another person helped me realize my own weaknesses.”
— Learner 2

Expert instructors manually assessed each AVT session to measure the impact on novices that the AVT learner made. The experts checked for two things: (1) Did the novice make significant progress on their problem? (2) Was it because the learner just gave away the answer?

In all but one session (98%), the experts agreed that learners helped the novices make progress and did so in an educationally helpful way. For the one session where this was not the case, the novice student was particularly unprepared and struggled with basic syntax. That being said, the experts did notice signs of some learners being more comfortable “telling” rather than “guiding”. This is a major concern if AVT is to be used more broadly, and we discuss it further in §6.3.

The first indication of this possibility is a small investigation we conducted in which two expert teachers watched the midterm AVT interactions and tried to predict what kinds of mistakes the learner would or would not have made in the exam. A validation showed that the expert predictions were correct in practically all cases. However, there was a one-sidedness to their correctness: the experts could easily predict when a learner understood something well and likely wouldn’t make a mistake, but it was much harder to make strong claims about what the learner didn’t know.

This asymmetry is a notable difference from traditional exams, where the set of questions help instructors see what students both do and don’t understand. This suggests that AVT could be used to “grade-up” rather than “grade-down”, which has positive psychological benefits for learners. However, to get a comprehensive picture of the learner understanding would need the support of additional assessment or a different scheme for AVT.

It is also a challenge to find a reliable, affordable, and unbiased way to measure learner understanding from AVT sessions. In our experiment, we had expert instructors review the sessions, which is unsustainable at scale. One idea is to try grading learners based on how much their novice students improved after the AVT session. This would require assessing the novices in a more traditional way, through an autograded question before and after the AVT session. Another solution is to leverage the power of large-language models (LLMs) to go through the AVT session transcripts and identify moments of demonstrated learner understanding. This would require in-depth research to ensure the outputs were meaningful and fair for learners.

If we are to assess learners summatively using AVT, we need to account for the stochastic variance of the novices they are paired with; it might be easier to tutor someone who already has a good grasp of the material compared to a novice who is extremely confused. To this end, one could assess a learner over several AVT sessions with multiple novices to reduce variance. Alternatively, rather that randomly pairing novices with learners, we can use clever matching strategies to ensure the learner-novice pairing is “good” in some rigorous sense.