From {Solution} Synthesis to {Student Attempt} Synthesis for Block-Based Visual Programming Tasks^††thanks: This article is a longer version of the paper from the EDM 2022 conference. Authors are listed alphabetically.

Figures 1 and 2 illustrate this synthesis question for two scenarios in the context of the Hour of Code: Maze Challenge [9] by Code.org [8]. This question is akin to that of program synthesis [20]; however, instead of synthesizing a {solution} (i.e., program an expert would write), the goal here is to synthesize a {student attempt} (i.e., program that a given student would write). This goal of synthesizing student attempts, and not just solutions, requires going beyond state-of-the-art program synthesis techniques [3, 4, 25]; crucially, we also need to define appropriate metrics to quantitatively measure the performance of different techniques. Our approach and contributions are summarized below:

1. (1)

   We formalize the problem of synthesizing a student’s attempt on target tasks after observing the student’s behavior on a fixed reference task. We introduce a novel benchmark, StudentSyn, centered around the above synthesis question, along with generative/discriminative performance measures for evaluation. (Sections 2, 3.1, 3.2)

2. (2)

   We showcase that human experts (TutorSS) can achieve high performance on StudentSyn, whereas simple baselines perform poorly. (Section 3.3)

3. (3)

   We develop two techniques inspired by neural (NeurSS) and symbolic (SymSS) methods, in a quest to close the gap with human experts (TutorSS). (Sections 4, 5, 6)

4. (4)

   We publicly release the benchmark and implementations to facilitate future research.¹¹1The StudentSyn benchmark and implementation of the techniques are available at https://github.com/machine-teaching-group/edm2022_studentsyn.

Student modeling. Inferring the knowledge state of a student is an integral part of AI tutoring systems and relevant to our goal of predicting a student’s behavior. For close-ended domains like vocabulary learning ([42, 36, 22]) and Algebra problems ([12, 40, 43]), the skills or knowledge components for mastery are typically well-defined and we can use Knowledge Tracing techniques to model a student’s knowledge state over time [11, 33]. These modeling techniques, in turn, allow us to provide feedback, predict solution strategies, or infer/quiz a student’s knowledge state [40, 21, 43]. Open-ended domains pose unique challenges to directly apply these techniques (see [23]); however, there has been some progress in this direction. In recent works [28, 29], models have been proposed to predict human behavior in chess for specific skill levels and to recognize the behavior of individual players. Along these lines, [7] introduced methods to perform early prediction of struggling students in open-ended interactive simulations. There has also been work on student modeling for block-based programming, e.g., clustering-based methods for misconception discovery [18, 44], and deep learning methods to represent knowledge and predict future performance [49].

AI-driven systems for programming education. There has been a surge of interest in developing AI-driven systems for programming education, and in particular, for block-based programming domains [37, 38, 50]. Existing works have studied various aspects of intelligent feedback, for instance, providing next-step hints when a student is stuck [35, 52, 31, 15], giving data-driven feedback about a student’s misconceptions [45, 34, 39, 51], or generating/recommending new tasks [2, 1, 19]. Depending on the availability of datasets and resources, different techniques are employed: using historical datasets to learn code embeddings [34, 31], using reinforcement learning in zero-shot setting [15, 46], bootstrapping from a small set of expert annotations [34], or using expert grammars to generate synthetic training data [51].

Neuro-symbolic program synthesis. Our approach is related to program synthesis, i.e., automatically constructing programs that satisfy a given specification [20]. In recent years, the usage of deep learning models for program synthesis has resulted in significant progress in a variety of domains including string transformations [16, 14, 32], block-based visual programming [3, 4, 13, 47], and competitive programming [25]. Program synthesis has also been used to learn compositional symbolic rules and mimic abstract human learning [30, 17]. Our goal is akin to program synthesis and we leverage the work of [3] in our technique NeurSS, however, with a crucial difference: instead of synthesizing a solution program, we seek to synthesize a student’s attempt.

The space of tasks. We define the space of tasks as $\mathbb{T}$; in this paper, $\mathbb{T}$ is inspired by the popular Hour of Code: Maze Challenge [9] from Code.org [8]; see Figures 1(a) and 2(a). We define a task $\text{{T}}\in\mathbb{T}$ as a tuple $(\text{{T}}_{\textnormal{vis}},\text{{T}}_{\textnormal{store}},\text{{T}}_{\textnormal{size}})$, where $\text{{T}}_{\textnormal{vis}}$ denotes a visual puzzle, $\text{{T}}_{\textnormal{store}}$ the available block types, and $\text{{T}}_{\textnormal{size}}$ the maximum number of blocks allowed in the solution code. For instance, considering the task T in Figure 2(a), we have the following specification: the visual puzzle $\text{{T}}_{\textnormal{vis}}$ comprises of a maze where the objective is to navigate the “avatar” (blue-colored triangle) to the “goal” (red-colored star) by executing a code; the set of available types of blocks $\text{{T}}_{\textnormal{store}}$ is {move, turnLeft, turnRight, RepeatUntil(goal), IfElse(pathAhead), IfElse(pathLeft), IfElse(pathRight)}, and the size threshold $\text{{T}}_{\textnormal{size}}$ is $5$ blocks; this particular task in Figure 2(a) corresponds to Maze#$18$ in the Hour of Code: Maze Challenge [9], and has been studied in a number of prior works [35, 15, 1].

The space of codes.²²2Codes are also interchangeably referred to as programs. We define the space of all possible codes as $\mathbb{C}$ and represent them using a Domain Specific Language (DSL) [20]. In particular, for codes relevant to tasks considered in this paper, we use a DSL from [1]. A code $\text{{C}}\in\mathbb{C}$ has the following attributes: $\text{{C}}_{\text{blocks}}$ is the set of types of code blocks used in C, $\text{{C}}_{\text{size}}$ is the number of code blocks used, and $\text{{C}}_{\text{depth}}$ is the depth of the Abstract Syntax Tree of C. Details of this DSL and code attributes are not crucial for the readability of subsequent sections; however, they provide useful formalism when implementing different techniques introduced in this paper.

Solution code and student attempt. For a given task T, a solution code $\text{{C}}^{\star}_{\text{{T}}}\in\mathbb{C}$ should solve the visual puzzle; additionally, it can only use the allowed types of code blocks (i.e., $\text{{C}}_{\text{blocks}}\subseteq\text{{T}}_{\textnormal{store}}$) and should be within the specified size threshold (i.e., $\text{{C}}_{\text{size}}\leq\text{{T}}_{\textnormal{size}}$). We note that a task $\text{{T}}\in\mathbb{T}$ in general may have multiple solution codes; in this paper, we typically refer to a single solution code that is provided as input. A student attempt for a task T refers to a code that is written by a student (including incorrect or partial codes). A student attempt could be any code $\text{{C}}\in\mathbb{C}$ as long as it uses the set of available types of code blocks (i.e., $\text{{C}}_{\text{blocks}}\subseteq\text{{T}}_{\textnormal{store}}$); importantly, it is not restricted by the size threshold $\text{{T}}_{\textnormal{size}}$—same setting as in the programming environment of Hour of Code: Maze Challenge [9].

Distinct phases. To formalize our objective, we introduce three distinct phases in our problem setup that provide a conceptual separation in terms of information and computation available to a system. More concretely, we have:

1. (1)

   Reference task $\text{{T}}^{\text{ref}}$: We are given a reference task $\text{{T}}^{\text{ref}}$ for which we have real-world datasets of different students’ attempts as well as access to other data resources. Reference tasks are fixed and the system can use any computation a priori (e.g., compute code embeddings).

2. (2)

   Student stu attempts $\text{{T}}^{\text{ref}}$: The system interacts with a student, namely stu, who attempts the reference task $\text{{T}}^{\text{ref}}$ and submits a code, denoted as $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}}$. At the end of this phase, the system has observed stu’s behavior on $\text{{T}}^{\text{ref}}$ and we denote this observation by the tuple $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}})$.³³3In practice, the system might have more information, e.g., the whole trajectory of edits leading to $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}}$ or access to some prior information about the student stu.

3. (3)

   Target task $\text{{T}}^{\text{tar}}$: The system seeks to synthesize the student stu’s behavior on a target task $\text{{T}}^{\text{tar}}$. Importantly, the target task $\text{{T}}^{\text{tar}}$ is not available a priori and this synthesis process would be done in real-time, possibly with constrained computational resources. Furthermore, the system may have to synthesize stu’s behavior on a large number of different target tasks from the space $\mathbb{T}$ (e.g., to personalize the next task in a curriculum).⁴⁴4Even though the Hour of Code: Maze Challenge [9] has only $20$ tasks, the space $\mathbb{T}$ is intractably large and new tasks can be generated automatically, e.g., when providing feedback or for additional practice [1].

Granularity level of our objective. There are several different granularity levels at which we can predict the student stu’s behavior for $\text{{T}}^{\text{tar}}$, including: (a) a coarse-level binary prediction of whether stu will successfully solve $\text{{T}}^{\text{tar}}$, (b) a medium-level prediction about stu’s behavior w.r.t. a predefined feature set (e.g., labelled misconceptions); (c) a fine-level prediction in terms of synthesizing $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{tar}}}$, i.e., a program that stu would write if the system would assign $\text{{T}}^{\text{tar}}$ to the student. In this work, we focus on this fine-level, arguably also the most challenging, synthesis objective.

Performance evaluation. So far, we have concretized the synthesis objective; however, there is still a question of how to quantitatively measure the performance of a technique set out to achieve this objective. The key challenge stems from the open-ended and conceptual nature of programming tasks. Even for seemingly simple tasks such as in Figures 1(a) and 2(a), the students’ attempts can be highly diverse, thereby making it difficult to detect a student’s misconceptions from observed behaviors; moreover, the space of misconceptions itself is not clearly understood. To this end, we begin by designing a benchmark to quantitatively measure the performance of different techniques w.r.t. our objective.

We begin by curating a synthetic dataset for the benchmark, designed to capture different scenarios of the three distinct phases mentioned in Section 2.2. In particular, each scenario corresponds to a 4-tuple $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{tar}}})$, where $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}}$ (observed by the system) and $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{tar}}}$ (to be synthesized by the system) correspond to a student stu’s attempts.

Reference and target tasks. We select two reference tasks for this benchmark, namely $\text{{T}}^{4}$ and $\text{{T}}^{18}$, as illustrated in Figures 1(a) and 2(a). These tasks correspond to Maze#$4$ and Maze#$18$ in the Hour of Code: Maze Challenge [9], and have been studied in a number of prior works [35, 15, 1], because of the availability of large-scale datasets of students’ attempts for these two tasks. For each reference task, we manually create three target tasks as shown in Figures 3(b) and 4(b); as discussed in the figure captions, these target tasks are similar to the corresponding reference task in a sense that the set of available block types is same and the nesting structure of programming constructs in solution codes is same.

Types of students’ behaviors and students’ attempts. For a given reference-target task pair $(\text{{T}}^{\text{ref}},\text{{T}}^{\text{tar}})$, next we seek to simulate a student stu to create stu’s attempts $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}}$ and $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{tar}}}$. We begin by identifying a set of salient students’ behaviors and misconceptions for reference tasks $\text{{T}}^{\text{4}}$ and $\text{{T}}^{\text{18}}$ based on students’ attempts observed in the real-world dataset of [35]. In this benchmark, we select $6$ types of students’ behaviors for each reference task—these types are highlighted in Figures 3(c) and 4(c) for $\text{{T}}^{\text{4}}$ and $\text{{T}}^{\text{18}}$, respectively.⁵⁵5In real-world settings, the types of students’ behaviors and their attempts have a much larger variability and complexities with a long-tail distribution; in future work, we plan to extend our benchmark to cover more scenarios, see Section 7. For a given pair $(\text{{T}}^{\text{ref}},\text{{T}}^{\text{tar}})$, we first simulate a student stu by associating this student to one of the $6$ types, and then manually create stu’s attempts $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}}$ and $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{tar}}}$. For a given scenario $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{tar}}})$, the attempt $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{tar}}}$ is not observed and serves as a ground truth in our benchmark for evaluation purposes; in the following, we interchangeably write a scenario as $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{?}})$.

Total scenarios. We create $72$ scenarios $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{tar}}})$ in the benchmark corresponding to (i) $2$ reference tasks, (ii) $3$ target tasks per reference task, (iii) $6$ types of students’ behaviors per reference task, and (iv) $2$ students per type. This, in turn, leads to a total of $72$ ($=2\times 3\times 6\times 2$) unique scenarios.

We introduce two performance measures to capture our synthesis objective. Our first measure, namely generative performance, is to directly capture the quality of fine-level synthesis of the student stu’s attempt—this measure requires a human-in-the-loop evaluation. To further automate the evaluation process, we then introduce a second performance measure, namely discriminative performance.

Generative performance. As a generative performance measure, we introduce a $4$-point Likert scale to evaluate the quality of synthesizing stu’s attempt $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{tar}}}$ for a scenario $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{?}})$. The scale is designed to assign scores based on two factors: (a) whether the elements of the student’s behavior observed in $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\textnormal{ref}}}$ are present, (b) whether the elements of the target task $\text{{T}}^{\textnormal{tar}}$ (e.g., parts of its solution) are present. More concretely, the scores are assigned as follows (with higher scores being better): (i) Score $1$ means the technique does not have synthesis capability; (ii) Score $2$ means the synthesis fails to capture the elements of $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\textnormal{ref}}}$ and $\text{{T}}^{\textnormal{tar}}$; (iii) Score $3$ means the synthesis captures the elements only of $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\textnormal{ref}}}$ or of $\text{{T}}^{\textnormal{tar}}$, but not both; (iv) Score $4$ means the synthesis captures the elements of both $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\textnormal{ref}}}$ and $\text{{T}}^{\textnormal{tar}}$.

Discriminative performance. As the generative performance requires human-in-the-loop evaluation, we also introduce a disciminative performance measure based on the prediction accuracy of choosing the student attempt from a set. More concretely, given a scenario $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{?}})$, the discriminative objective is to choose $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\textnormal{tar}}}$ from ten candidate codes; see Figure 5. These ten options are created automatically in a systematic way and include the following: (a) the ground-truth $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\textnormal{tar}}}$ from the benchmark, (b) the solution code $\text{{C}}^{\star}_{\text{{T}}^{\textnormal{tar}}}$, (c) five codes $\text{{C}}^{\text{{stu}}^{\prime}}_{\text{{T}}^{\textnormal{tar}}}$ from the benchmark associated with other students $\text{{stu}}^{\prime}$ whose behavior type is different from stu, and (iv) three randomly constructed codes obtained by editing the solution code $\text{{C}}^{*}_{\text{{T}}^{\textnormal{tar}}}$.

As a starting point, we design a few simple baselines and compare their performance with that of human experts.

Simple baselines. The simple baselines that we develop here are meant for the discriminative-only objective; they do not have synthesis capability. Our first baseline RandD simply chooses a code from the $10$ options at random. Our next two baselines, EditD and EditEmbD, are defined through a distance function $\text{D}_{\text{{T}}^{\textnormal{ref}}}(\text{{C}},\text{{C}}^{\prime})$ that quantifies a notion of distance between any two codes $\text{{C}},\text{{C}}^{\prime}$ for a fixed reference task. For a scenario $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{?}})$ and ten option codes, these baselines select the code C that minimizes $\text{D}_{\text{{T}}^{\textnormal{ref}}}(\text{{C}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}})$. EditD uses a tree-edit distance between Abstract Syntax Trees as the distance function, denoted as $\text{D}^{\textnormal{edit}}_{\text{{T}}^{\textnormal{ref}}}$. EditEmbD extends EditD by considering a distance function that combines $\text{D}^{\textnormal{edit}}_{\text{{T}}^{\textnormal{ref}}}$ and a code-embedding based distance function $\text{D}^{\textnormal{emb}}_{\text{{T}}^{\textnormal{ref}}}$; in this paper, we trained code embeddings with the methodology of [15] using a real-world dataset of student attempts on $\text{{T}}^{\textnormal{ref}}$. EditEmbD then uses a distance function as a convex combination $\big{(}\alpha\cdot\text{D}^{\textnormal{edit}}_{\text{{T}}^{\textnormal{ref}}}(\text{{C}},\text{{C}}^{\prime})+(1-\alpha)\cdot\text{D}^{\textnormal{emb}}_{\text{{T}}^{\textnormal{ref}}}(\text{{C}},\text{{C}}^{\prime})\big{)}$ where $\alpha$ is optimized for each reference task separately. For measuring the discriminative performance, we randomly sample a scenario, create ten options, and measure the predictive accuracy of the technique—the details of this experimental evaluation are provided in Section 6.2.

Human experts. Next, we evaluate the performance of human experts on the benchmark StudentSyn, and refer to this evaluation technique as TutorSS. These evaluations are done through a web platform where an expert would provide a generative or discriminative response to a given scenario $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{?}})$. In our work, TutorSS involved participation of three independent experts for the evaluation; these experts have had experience in block-based programming and tutoring. We first carried out generative performance evaluations where an expert had to write the student attempt code; afterwards, we carried out discriminative performance evaluations where an expert would choose one of the options. In total, each expert participated in $36$ generative evaluations ($18$ per reference task) and $72$ discriminative evaluations ($36$ per reference task). Results in Table 1 highlight the huge performance gap between the human experts and simple baselines; further details are provided in Section 6.

Evaluation procedure. As discussed in Section 3.2, we evaluate the generative performance of a technique in the following steps: (a) a scenario $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{?}})$ is picked; (b) the technique synthesizes stu’s attempt, i.e., a program that stu would write if the system would assign $\text{{T}}^{\text{tar}}$ to the student; (c) the generated code is scored on the $4$-point Likert scale. The scoring step requires human-in-the-loop evaluation and involved an expert (different from the three experts that are part of TutorSS). Overall, each technique is evaluated for $36$ unique scenarios in StudentSyn—we selected $18$ scenarios per reference task by first picking one of the $3$ target tasks and then picking a student from one of the $6$ different types of behavior. The final performance results in Table 2 are reported as an average across these scenarios; for TutorSS, each of the three experts independently responded to these $36$ scenarios and the final performance is computed as a macro-average across experts.

Quantitative results. Table 2 expands on Table 1 and reports results on the generative performance per reference task for different techniques. As noted in Section 3.3, the simple baselines (RandD, EditD, EditEmbD) do not have a synthesis capability and hence have a score $1.00$. TutorSS, i.e., human experts, achieves the highest performance with aggregated scores of $3.85$ and $3.91$ for two reference tasks respectively; as mentioned in Table 1, these scores are reported as an average over scores achieved by three different experts. SymSS also achieves high performance with aggregated scores of $3.72$ and $3.83$—only slightly lower than that of TutorSS and these gaps are not statistically significant w.r.t. $\chi^{2}$ tests [6]. The high performance of SymSS is expected given its knowledge about types of students in StudentSyn and the expert domain knowledge inherent in its design. NeurSS improves upon simple baselines and achieves aggregated scores of $3.28$ and $2.94$; however, this performance is significantly worse ($p\leq 0.001$) compared to that of SymSS and TutorSS w.r.t. $\chi^{2}$ tests.⁸⁸8$\chi^{2}$ tests reported here are conducted based on aggregated data across both the reference tasks.

Qualitative results. Figures 9 and 10 illustrate the qualitative results in terms of the generative objective for the scenarios in Figures 1 and 2, respectively. As can be seen in Figures 9(d) and 10(d), the codes generated by human experts in TutorSS are high-scoring w.r.t. our $4$-point Likert scale, and are slight variations of the ground-truth codes in StudentSyn shown in Figures 9(c) and 10(c). Figures 9(f) and 10(f) show the Top-$3$ codes synthesized by SymSS for these two scenarios – these codes are also high-scoring w.r.t. our $4$-point Likert scale. In contrast, for the scenario in Figure 2, the Top-$3$ codes synthesized by NeurSS in Figure 10(e) only capture the elements of the student’s behavior in $\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\textnormal{ref}}}$ and miss the elements of the target task $\text{{T}}^{\textnormal{tar}}$.

Evaluation procedure: Creating instances. As discussed in Section 3.2, we evaluate the discriminative performance of a technique in the following steps: (a) a discriminative instance is created with a scenario $(\text{{T}}^{\text{ref}},\text{{C}}^{\text{{stu}}}_{\text{{T}}^{\text{ref}}},\text{{T}}^{\text{tar}},\text{{?}})$ picked from the benchmark and $10$ code options created automatically; (b) the technique chooses one of the options as stu’s attempt; (c) the chosen option is scored either $100.0$ when correct, or $0.0$ otherwise. We create a number of discriminative instances for evaluation, and then compute an average predictive accuracy in the range $[0.0,100.0]$. We note that the number of discriminative instances can be much larger than the number of scenarios because of the variability in creating $10$ code options. When sampling large number of instances in our experiments, we ensure that all target tasks and behavior types are represented equally.

Evaluation procedure: Details about final performance. For TutorSS, we perform evaluation on a small set of $72$ instances ($36$ instances per reference task), to reduce the effort for human experts. The final performance results for TutorSS in Table 2 are reported as an average predictive accuracy across the evaluated instances—each of the three experts independently responded to the instances and the final performance is computed as a macro-average across experts. Next, we provide details on how the final performance results are computed for the techniques RandD, EditD, EditEmbD, NeurSS, and SymSS. For these techniques, we perform $\text{numEval}=5$ independent evaluation rounds, and report results as a macro-average across these rounds; these rounds are also used for statistical significance tests. Within one round, we create a set of $720$ instances ($360$ instances per reference task). To allow hyperparameter tuning by techniques, we apply a cross-validation procedure on the $360$ instances per reference task by creating $10$-folds whereby $1$ fold is used to tune hyperparameters and $9$ folds are used to measure performance. Within a round, the performance results are computed as an average predictive accuracy across the evaluated instances.

Quantitative results. Table 2 reports results on the discriminative performance per reference task for different techniques. As noted in Section 3.3, the initial results showed a huge gap between the human experts (TutorSS) and simple baselines (RandD, EditD, EditEmbD). As can be seen in Table 2, our proposed techniques (NeurSS and SymSS) have reduced this performance gap w.r.t. TutorSS. SymSS achieves high performance compared to simple baselines and NeurSS; moreover, on the reference task $\text{{T}}^{\text{4}}$, its performance ($87.17$) is close to that of TutorSS ($89.81$). The high performance of SymSS is partly due to its access to types of students in StudentSyn; in fact, this information is used only by SymSS and is not even available to human experts in TutorSS—see column “Student types” in Table 2. NeurSS outperformed simple baselines on the reference task $\text{{T}}^{\text{18}}$; however, its performance is below SymSS and TutorSS for both the reference tasks. For the three techniques NeurSS, SymSS, and EditEmbD, we did statistical significance tests based on results from $\text{numEval}=5$ independent rounds w.r.t. Tukey’s HSD test [48], and obtained the following: (a) the performance of NeurSS is significantly better than EditEmbD on the reference task $\text{{T}}^{\text{18}}$ ($p\leq 0.001$); (b) the performance of SymSS is significantly better than NeurSS and EditEmbD on both the reference tasks ($p\leq 0.001$).