Research and Practice of Delivering Tabletop Exercises

Our literature review examines five research questions:

1. (1)

   What formats of tabletop exercises are used?
   Namely how the exercises are prepared, delivered and what tools are applied.

2. (2)

   Who are the participants of the exercises?
   What are the target groups of the exercises (trainees) and by whom are they organized (instructors, designers)?

3. (3)

   How are the exercises developed, assessed, and evaluated?
   We will examine methods for exercise preparation, assessment of trainees, and evaluation of the exercise itself.

4. (4)

   How are the research results applied in practice?
   Do the publications provide any supplementary materials or artifacts for other educators?

5. (5)

   What are the future research directions and challenges?
   What is the ongoing and long-term goal?

We used the Scopus citation database of peer-reviewed literature (Elsevier, 2024). Scopus indexes a representative portion of the databases of individual publishers, including ACM Digital Library, IEEExplore, SpringerLink, or Elsevier ScienceDirect. We did not use Google Scholar since it additionally indexes many non-peer-reviewed papers.

When constructing the search query, we focused on the subject area of computer science and engineering and used “tabletop exercise” as the primary keyword. We also sought not to miss potential alternative names for tabletop exercises, such as “simulation” or “communication exercises”. After several pilot searches, we concluded with the query presented in Figure 2. The asterisk is a wildcard to match both adjectives “table top” and “table-top”. However, the wildcard is not used for the stem of the word “communication” because the pilot search yielded numerous false positives. Similarly, we omitted “simulation” matching a huge number of papers unrelated to education and training. The search was case-insensitive.

We started the search on May 11, 2023, and then subscribed to Scopus e-mail notifications that informed us about newly indexed papers, which we gradually added to the candidate set. We stopped adding new candidates on July 19, 2023. In total, we found \papers candidate papers.

After multiple pilot tests, we set ten criteria.

1. (1)

   Papers must deal with TTXs in IT or operational technology (OT) operations or security.

2. (2)

   Papers describing exercises and tools in other domains, such as transportation or utilities, without considering cybersecurity or IT/OT operations aspects, are excluded.

3. (3)

   We include papers written in English with full text available¹¹1We consider a paper to be available if we can access it using tens of licensed electronic resources our institution has access to, or if the full text is freely available online.. We do not set any page limit to include short papers, which report ongoing and future work at the time of their publication.

4. (4)

   Other types of documents yielded by the automated search, such as conference reviews or records, are excluded.

5. (5)

   Papers must describe exercising or supporting the exercise of a complex process or standard operation procedures in a safe, simulated environment. For instance, this includes experience reports with lessons learned or descriptions of tools supporting the preparation, execution, and analysis of TTXs.

6. (6)

   Papers that describe only exercising essential communication and writing skills (e.g., technical writing, proposal paper, presentation and its delivery) are excluded.

7. (7)

   Papers must report on an exercise involving teams or groups, not only individuals.

8. (8)

   The paper must support an educational goal, for example, to train or assess participants, or help instructors conduct the exercise, or understand the learning processes.

9. (9)

   We exclude papers that use TTXs solely as an evaluation tool in non-educational settings, such as for testing of software or IT systems.

10. (10)

    Generic methodologies applied or applicable to TTXs in IT and OT operations and security are included.

Two authors of this paper each screened all \papers candidate papers and applied these criteria independently. The authors followed a simple algorithm for screening the paper content (Švábenský et al., 2022):

After both authors finished their reading, they compared candidate papers they identified after applying the inclusion and exclusion criteria. If both authors decided to include the paper, it was selected for review. If both decided not to include the paper, it was excluded. In case of a disagreement, the authors discussed their views and agreed on the final decision.

In the first round, the authors agreed that out of 140 candidate papers, nine (6.4%) were fitting the selection criteria. The authors had opposite opinions on seven (5%) papers. Out of the 16 papers, which passed the first round, the authors selected \papersselected papers and rejected two papers in the second selection round. Their inter-rater agreement (Krippendorff, 2004) measured by Krippendorff’s $\alpha$ for nominal data was 0.69, which is substantial. The coefficient was calculated using the Python NLTK module (Natural Language Toolkit Project(2023), NLTK).

From seven papers (namely P1, P2, P4, P7, P12, P13, P14) out of eight that contain enough information about the exercise format, we see that the exercises are designed as a series of injects (events, problems, or situations, see Section 2). These injects form a scenario that is unknown to the exercise participants beforehand. The injects are provided by exercise facilitators to participants who discuss appropriate actions, processes, or best practices within their team or with all participants.

The eighth paper (P11) reports two kinds of exercises that extend this common format. One TTX uses cards that drive the exercise itself (attack cards) or stimulate participants to think about several options (action cards, situation awareness cards, and information-sharing cards). Another TTX in P11 tasked participants to create a scheme depicting information and workflows within an organization affected by an incident.

Only four papers mention the use of any software tool during the exercise. P7 presents a web application for delivering a TTX involving various roles according to a scenario defined by the facilitator. Participants use a graphical interface to respond to presented injects. They can submit a short description of how they would solve the inject, inform other roles about it, or delegate the resolution to other roles. P11 mentions an exercise where participants use software to interact with virtual participants. Participants choose a response to presented injects from a limited set of predefined actions. P12 reports a remote exercise facilitated through Microsoft Teams web conferencing application. P1, P6, and P14 use a software tool before or after the actual exercise; see Section 5.3.

The exercises last from several hours (P2, P13, P14) through one day (P4, P7, P8, P9, P11, P12) to a few days (P6).

Trainees come from diverse sectors. The most frequent were critical infrastructure organizations, such as energy distribution operators (P4), a water management centre (P6), industrial control systems stakeholders (P8, P11), or oil and gas suppliers (P9, P13). Two exercises were carried out for university students (P2, P7). One exercise was conducted for a large law enforcement organization (P12). The number of trainees ranged from 20 to 108.

Exercises were designed by representatives of national or transnational authorities (P4, P9) or academic staff (P2, P8, P11, P12). The type of the organizing entity determines the target group and its diversity (employees of one organization, multiple organizations in a single country or multiple organizations from multiple countries).

No reviewed paper refers to supplementary materials (such as software tool implementation or exercise scenario) that other educators can directly use. In particular, P1, P6, P7, and P14 describe software tools, but the papers do not refer to any materials, such as software repositories or websites presenting the tools. P13 only outlines eight scenarios as an inspiration for creating a new exercise.

A few papers distill recommendations and lessons learned from TTX preparation or delivery applicable to other exercises. P2 lists steps describing the preparation of a lightweight TTX and the most common problems encountered. P3 provides three design ideas for more realistic and efficient exercises (see Section 5.3). P4 states recommendations for conducting exercises.

Finally, P5 shows that the methodology by ENISA (ENISA, 2009) is the most compliant with the ISO 22398 standard out of the three reviewed methodologies, yet not entirely.

The reviewed papers reported diverse future work. P2 suggests developing more detailed instructions, creating mock scenarios, and formalizing and publishing student feedback. P3 advises the development of an online tool for running a TTX, evaluating the proposed design ideas, and studying how TTXs are used in incident recovery. P4 recommends studying best practices and challenges of organizations conducting TTXs regularly and studying real-life incident responses to design more useful future exercises. P6, which deals with assessment, plans to replace binary scoring with a four-point scale in the presented method. Since P7 piloted the exercise with university students, future work includes conducting the exercise with professionals working in the target industry. In addition, it propounds the development of an assessment of trainee stress levels, which can then be used to adapt the exercise progression dynamically. P8 mentions sharing the instances of errors among facilitators. While P9 simply recommends conducting more exercises, P11 suggests developing a method to demonstrate the risks due to failure to act. P12 plans to improve the presented exercise using lessons learned from its first run and develop a training program for staff at all organizational levels. P13 would like to test the proposed criteria for realistic and expedient scenarios in practice when deploying the proposed scenarios. Finally, P14 lists several concrete directions to improve the tool for generating exercise content.

The topics in the identified papers differ widely. Most papers are reports from runs of a particular exercise or describe an exercise format. The typical TTX format is based on pieces of information gradually provided to the trainees by exercise facilitators. The trainees discuss and respond to the information in teams. The exercises are usually held for tens of trainees on site for several hours or days, with little support from dedicated software tools.

The explicit learning phase usually happens at the end or after the exercise, when trainees reflect on their decisions with other participants or when the previously unknown parts of the scenario are disclosed to them. Other types of assessment or feedback are rare because the complexity and labor of the manual preparation of the exercises hinder instructors from focusing on trainee assessment and exercise evaluation. Out of only three papers (P4, P6, and P8) that addressed assessment, only one (P6) suggested a method that goes further than unstructured assessment by the observers and facilitators. Similarly, the evaluation of the exercise itself is also underdeveloped. No paper uses advanced feedback methods but trainees’ reflections, discussions, or surveys.

The reviewed papers did not provide many actionable artifacts or supplementary materials (such as software supporting exercise preparation and delivery, exercise playbooks, scenarios, or checklists) for others considering conducting their own exercise. The absence of publicly available materials prevents educators from adopting the concept of tabletop exercises as a teaching method.

The current practice of conducting TTXs relies on manual preparation of exercise content (namely injects that form exercise scenarios), with no automation and limited future reusability. We believe this practice can be improved by leveraging large language models for semi-automated content generation (P14) or employing dedicated software for TTX preparation and delivery (see Section 5.1).

Finally, future directions in the papers often mention evaluating proposed methods by field tests or running the exercise for more trainees or different target groups. These plans indicate that the described methods and exercises should be developed further and provide new research opportunities in computing education.

This review is limited by narrowing its scope by only using the Scopus citation database of the peer-reviewed literature. This decision was explained in Section 4.2. Also, when using Scopus, we did not have to deal with duplicate records, which may occur when using multiple other databases.

The second limitation is keyword choice. Using other keywords may have led us to find different papers fitting the selection criteria. However, pilot searches with related terms, such as “crisis exercise” returned an excessive number of unrelated papers (see Section 4.2).

Finally, the data extraction was done manually, which may lead to misinterpretation of the presented information. To mitigate this risk, we performed cross-author discussion and validation.