Design and Results of ICCMA 2021

Jean-Marie Lagniez
CRIL, University of Artois and CNRS
[email protected]
&Emmanuel Lonca
CRIL, University of Artois and CNRS
[email protected]
\ANDJean-Guy Mailly
LIPADE, University of Paris
[email protected]
&Julien Rossit
LIPADE, University of Paris
[email protected]

Abstract

Since 2015, the International Competition on Computational Models of Argumentation (ICCMA) provides a systematic comparison of the different algorithms for solving some classical reasoning problems in the domain of abstract argumentation. This paper discusses the design of the Fourth International Competition on Computational Models of Argumentation. We describe the rules of the competition and the benchmark selection method that we used. After a brief presentation of the competitors, we give an overview of the results.

Formal argumentation [1, 2] is a major topic in the domain of knowledge representation and reasoning. This formalism allows to reason with conflicting information, and has applications e.g. in automated negotiation [3] or decision making [4]. The most classical reasoning tasks in this kind of formalism are intractable in the general case (see e.g. [5] for an overview of computational complexity in formal argumentation). This has conducted to the organization of the International Competition on Computational Models of Argumentation (ICCMA), that allows to compare the efficiency of the different algorithms that have been proposed for these reasoning problems. The previous competitions [6, 7, 8] have shown that, in spite of the theoretical hardness of argumentative reasoning, some powerful techniques allow to handle them efficiently.

After a short introduction to abstract argumentation in Section 1, we describe the rules of the competition in Section 2. In particular, we describe the various (sub)tracks as well as the scoring system and the benchmark selection. Note that, contrary to what was initially announced [9], there has been no track dedicated to dynamic argumentation or structured argumentation at ICCMA 2021. However, there has been some interest in the community that conducted us to add a new track dedicated to approximation algorithms for reasoning with abstract argumentation frameworks. Then, we present the competitors and the results in Section 3.

1 Background: Abstract Argumentation

An abstract argumentation framework (AF) [10] is a directed graph $F=\langle A,R\rangle$ , where $A$ is the set of arguments, and $R\subseteq A\times A$ is the attack relation. For $a,b,c\in A$ , we say that $a$ attacks $b$ if $(a,b)\in R$ . If in turn $b$ attacks $c$ , then $a$ defends $c$ against $b$ . Similarly, a set $S\subseteq A$ attacks (respectively defends) an argument $b$ if there is some $a\in S$ that attacks (respectively defends) $b$ . For $S\subseteq A$ a set of arguments, $S^{+}$ is the set of arguments that are attacked by $S$ , formally $S^{+}=\{b\in A\mid\exists a\in S$ s.t. $(a,b)\in R\}$ . The range of $S$ is $S^{\oplus}=S\cup S^{+}$ .

Different semantics have been defined for evaluating the acceptability of (sets of) arguments.

Definition 1

Given an AF $F=\langle A,R\rangle$ , a set of arguments $S\subseteq A$ is conflict-free iff $\forall a,b\in S$ , $(a,b)\not\in R$ . A conflict-free set $S$ is admissible iff $\forall a\in S$ , $S$ defends $a$ against all its attackers. Conflict-free and admissible sets are respectively denoted by ${\sc\bf CF}(F)$ and ${\sc\bf ADM}(F)$ .

Now, we formally introduce the extension-based semantics. For $S\subseteq A$ ,

•

$S\in{\sc\bf CO}(F)$ iff $S\in{\sc\bf ADM}(F)$ and $\forall a\in A$ that is defended by $S$ , $a\in S$ ;
•

$S\in{\sc\bf PR}(F)$ iff $S$ is a $\subseteq$ -maximal admissible set;
•

$S\in{\sc\bf ST}(F)$ iff $S$ is a conflict-free set that attacks each $a\in A\setminus S$ ;
•

$S\in{\sc\bf SST}(F)$ iff $S\in{\sc\bf CO}(F)$ and there is no $S_{2}\in{\sc\bf CO}(F)$ s.t. $S^{\oplus}\subset S_{2}^{\oplus}$ ;
•

$S\in{\sc\bf STG}(F)$ iff $S\in{\sc\bf CF}(F)$ and there is no $S_{2}\in{\sc\bf CF}(F)$ s.t. $S^{\oplus}\subset S_{2}^{\oplus}$ ;
•

$S\in{\sc\bf ID}(F)$ iff $S\in{\sc\bf ADM}(F)$ , $S\subseteq\cap{\sc\bf PR}(F)$ , and there is no $S_{2}\subseteq\cap{\sc\bf PR}(F)$ such that $S_{2}\in{\sc\bf ADM}(F)$ and $S\subset S_{2}$ .

CO, PR, ST, SST, STG and ID stand (respectively) for the complete, preferred, stable [10], semi-stable [11], stage [12] and ideal [13] semantics. We refer the interested reader to [14] for more details about these semantics.

For $\sigma\in\{{\sc\bf CO},{\sc\bf PR},{\sc\bf ST},{\sc\bf SST},{\sc\bf STG},{\sc\bf ID}\}$ a semantics, an argument $a\in A$ is credulously (respectively skeptically) accepted in $F=\langle A,R\rangle$ with respect to $\sigma$ iff $a\in S$ for some (respectively each) $S\in\sigma(F)$ .

We are interested in various reasoning problems:

•

CE- $\sigma$ : Given an AF $F=\langle A,R\rangle$ , give the number of $\sigma$ -extensions of $F$ .
•

SE- $\sigma$ : Given an AF $F=\langle A,R\rangle$ , give one $\sigma$ -extension of $F$ .
•

DC- $\sigma$ : Given an AF $F=\langle A,R\rangle$ and $a\in A$ an argument, is $a$ credulously accepted in $F$ ?
•

DS- $\sigma$ : Given an AF $F=\langle A,R\rangle$ and $a\in A$ an argument, is $a$ skeptically accepted in $F$ ?

Most of these problems are computationally hard in general, under the semantics from Definition 1. For instance, the decision problems DC and DS might be complete for the first or second level of the polynomial hierarchy (see [5] for more details).

2 Rules of the Competition

2.1 Tracks and Subtracks

The competition is made of two main tracks. The first one, the most “classical" one, is dedicated to exact algorithms for reasoning with abstract AFs. The second track has been introduced for the first time at ICCMA 2021, and is dedicated to approximation algorithms for abstract argumentation.

Exact Algorithms

The first track is made of six subtracks, each of them corresponding to one of the six semantics under consideration. Each subtrack is made of several reasoning tasks. More precisely, for $\sigma\in\{{\sc\bf CO},{\sc\bf PR},{\sc\bf ST},{\sc\bf SST},{\sc\bf STG}\}$ , all the problems CE- $\sigma$ , SE- $\sigma$ , DC- $\sigma$ and DS- $\sigma$ must be solved. For $\sigma={\sc\bf ID}$ , since any AF possesses exactly one ideal extension, CE- $\sigma$ is trivial, and DC- $\sigma$ is equivalent to DS- $\sigma$ . So, only SE- $\sigma$ and DS- $\sigma$ are considered.

Approximate Algorithms

The second track is also made of six subtracks corresponding to the six semantics. However, the problems CE- $\sigma$ and SE- $\sigma$ are not included in the subtracks this time, i.e. only the decision problems DC- $\sigma$ and DS- $\sigma$ appear. This means that every subtrack is made of two problems, except $\sigma={\sc\bf ID}{}$ which is made of only one task.

2.2 Input/Output and Environment

The input/output formats are the same as the formats from ICCMA 2019 for all the reasoning tasks that were considered then; only CE- $\sigma$ requires the definition of a new format. Each benchmark is provided in two different file formats: trivial graph format (tgf) and ASPARTIX format (apx). For the following examples, we use the AF $F=\langle A,R\rangle$ with $A=\{a_{1},a_{2},a_{3}\}$ and $R=\{(a_{1},a_{2}),(a_{2},a_{3}),(a_{2},a_{1})\}$ , depicted at Figure 1.

Figure 1: The AF

F

The Trivial Graph Format describes a graph by giving a list of identifiers for nodes, then a list of edges, separated by a $\#$ symbol. See https://en.wikipedia.org/wiki/Trivial_Graph_Format for more details. We give below the content of myFile.tgf that corresponds to the AF depicted at Figure 1.

The ASPARTIX format (named after the ASP-based argumentation solver ASPARTIX [15]) describes the argument names as rules arg(name)., and attacks as rules att(name1,name2). We give below, as an example, the content of myFile.apx that corresponds to the AF depicted at Figure 1.

arg(a1).
arg(a2).
arg(a3).
att(a1,a2).
att(a2,a3).
att(a2,a1).

For both tracks, solvers must write the result to standard output exactly in the format described below.

•
DC- $\sigma$ and DS- $\sigma$ , for $\sigma\in\{{\sc\bf CO},{\sc\bf PR},{\sc\bf ST},{\sc\bf SST},{\sc\bf STG},{\sc\bf ID}\}$ . The output must be either
```
ΨYES
Ψ
```
if the queried argument is (respectively) credulously or skeptically accepted in the given AF under $\sigma$ , or
```
ΨNO
Ψ
```
otherwise.
•
SE- $\sigma$ , for $\sigma\in\{{\sc\bf CO},{\sc\bf PR},{\sc\bf ST},{\sc\bf SST},{\sc\bf STG},{\sc\bf ID}\}$ . The output must be of the form
```
Ψ[a1,a2,a3]
Ψ
```
meaning that $\{a_{1},a_{2},a_{3}\}$ is a $\sigma$ -extension of the given AF, where $a_{1},a_{2}$ and $a_{3}$ are arguments. If $\sigma={\sc\bf ST}$ , there may be benchmarks that do not possess any extension. In that case, the output must be
```
ΨNO
Ψ
```
•
CE- $\sigma$ , for $\sigma\in\{{\sc\bf CO},{\sc\bf PR},{\sc\bf ST},{\sc\bf SST},{\sc\bf STG}\}$ . The output must be of the form
```
Ψk
Ψ
```
where $k\in\mathbb{N}$ is the number of $\sigma$ -extensions of the given AF.

The solver interface is also inspired from ICCMA 2019. The new problem CE has a similar command line to the previous EE problem.¹¹1See http://argumentationcompetition.org/2021/SolverRequirements.pdf for more details.

The competition has been run on a computer cluster where each machine has an Intel Xeon E5-2637 v4 CPU and $128$ GB of RAM. The runtime limit for each instance is $600$ seconds for the “exact" track, and $60$ seconds for the “approximate" track. The memory limit is the machine’s memory, i.e. $128$ GB.

2.3 Scoring Rules

There is one ranking for each sub-track, i.e. six rankings for the “exact" track and six rankings for the “approximate" track. To be ranked, a solver must participate to the full subtrack (but without any obligation to participate to all the (sub)tracks). The scoring system is slightly different between both tracks.

For the “exact" track, any wrong result on an instance $i$ in a subtrack conducts to the exclusion of the solver from the said subtrack. It does not prevent the solver from being ranked for other subtracks if there is no wrong results for these other ones. Then, the score of a solver $\mathcal{S}$ on the instance $i$ is

Score(\mathcal{S},i)=\left\{\begin{array}[]{lr}1&\text{the correct answer is given in the runtime limit}\\ 0&\text{timeout or non-parsable output}\end{array}\right.

On the contrary, wrong results do not lead to an exclusion in the “approximate" track:

Score(\mathcal{S},i)=\left\{\begin{array}[]{lr}1&\text{the correct answer is given in the runtime limit}\\ 0&\text{wrong result, timeout or non-parsable output}\end{array}\right.

Then, the score of the solver $\mathcal{S}$ for the task $\mathcal{T}$ is

Score(\mathcal{S},\mathcal{T})=\sum_{i\in\mathcal{T}}Score(\mathcal{S},i)

and the score for the subtrack $\mathcal{ST}$ is

Score(\mathcal{S},\mathcal{ST})=\sum_{T\in\mathcal{ST}}Score(\mathcal{S},\mathcal{T}).

In the case where two solvers have the same score for a given subtrack, the cumulated runtime over the instances correctly solved is used as a tie-break rule (the fastest is the best).

2.4 Benchmark Selection

ICCMA 2019 Instances

We have selected $165$ instances from the previous competition. The goal is to observe the evolution of the algorithmic techniques for abstract argumentation during the last two years. These instances are the hardest ones from ICCMA 2019, with respect to two criteria:

•

the number of times some solvers have reached the timeout on these instances;
•

the average runtime for solving these instances.

New instances

We have defined a new method for generating challenging benchmarks. This method has been used for generating $422$ new instances. For creating an instance, we use this procedure:

1.

Generate a (meta-)graph $G$ following a classical generation pattern (e.g. Erdos-Renyi, Barabasi-Albert, $\dots$ ).
2.

For each node $n_{i}$ in this graph, generate a new graph $F_{i}$ .
3.

For each edge $(n_{1},n_{2})$ in $G$ , pick some arguments $a_{1}$ in $F_{1}$ and $a_{2}$ in $F_{2}$ , and add an edge $(a_{1},a_{2})$ .

Intuitively, this method is used to create graphs with “communities of arguments".

Query argument selection

The last part of the benchmark selection is the choice of an argument to be queried for skeptical or credulous acceptance (DS, DC). Simply, for each AF, one argument is randomly chosen, and this argument is used for all the DS and DC queries on the same AF.

3 Participants and Results

3.1 Competitors

There were $9$ solvers participating to ICCMA 2021, $7$ in the exact track, and $2$ in the new approximate track.

Exact solvers

•

A-Folio DPDB (Fichte, Hecher, Gorczyca and Dewoprabowo) [16]
•

ASPARTIX-V21 (Dvorák, König, Wallner and Woltran) [17]
•

ConArg (Bistarelli, Rossi, Santini and Taticchi) [18]
•

FUDGE (Thimm, Cerutti, Vallati) [19]
•

MatrixX (Heinrich) [20]
•

$\mu$ -toksia (Niskanen and Järvisalo) [21]
•

PYGLAF (Alviano) [22]

Observe that $3$ solvers are new submissions (A-Folio DPDB, FUDGE and MatrixX) , while $4$ of them (ASPARTIX-V21, ConArg, $\mu$ -toksia and PYGLAF) are updated versions of solvers submitted to previous editions of ICCMA. Various techniques have been used, like translations into SAT, Constraint Programming or ASP, and dedicated algorithms.

Approximate solvers

•

AFGCN (Malmqvist) [23]
•

HARPER $++$ (Thimm) [24]

AFGCN is mainly based on neural networks, while HARPER $++$ uses the grounded semantics as a tool for approximating the results of the various decision problems.

Participation to Subtracks

The solvers are registered to ICMMA 2021 (sub)tracks as described by Table 1.

	Exact Track						Approximate Track
Solver	CO	PR	ST	SST	STG	ID	CO	PR	ST	SST	STG	ID
AFGCN							$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
A-Folio DPDB	$\checkmark$		$\checkmark$
ASPARTIX-V21	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
ConArg	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
FUDGE	$\checkmark$	$\checkmark$	$\checkmark$			$\checkmark$
HARPER $++$							$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
MatrixX	$\checkmark$		$\checkmark$
$\mu$ -toksia	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
PYGLAF	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Participation of the solvers to the various subtracks

3.2 Results of the Exact Track

Results for the track dedicated to exact algorithms are described in Table 2. Half of the subtracks are won by new solvers (A-Folio-DPDB for the complete and stable semantics, and Fudge for the ideal semantics), while updated versions of existing solvers perform well in the other subtracks (PYGLAF for the preferred and semi-stable semantics, and ASPARTIX for the stage semantics). An interesting point is the global correctness of the solvers. Only two solvers presented minor bugs during the running of the competition (Fudge on the stable semantics, and PYGLAF on the stage semantics). We gave some feedback to the competitors, which allowed to correct the mistakes, and only PYGLAF had to be excluded from one subtrack (the one dedicated to the stage semantics). Detailed results (ranking for each subtrack, with the cumulated runtime over the successfully solved instances) are given in Appendix A.

Rank	Solver	Score
1	A-Folio DPDB	1838
2	PYGLAF	1835
3	$\mu$ -toksia	1803
4	ASPARTIX-V21	1787
5	FUDGE	1695
6	MatrixX	759
7	ConArg	428

(a) Complete Semantics

Rank	Solver	Score
1	A-Folio-DPDB	1862
2	PYGLAF	1743
3	FUDGE	1585
4	$\mu$ -toksia	1441
5	ASPARTIX-V21	1429
6	ConArg	429
7	MatrixX	259

(b) Stable Semantics

Rank	Solver	Score
1	PYGLAF	1299
2	$\mu$ -toksia	1210
3	FUDGE	1190
4	ASPARTIX-V21	1052
5	ConArg	429

Rank	Solver	Score
1	FUDGE	492
2	ASPARTIX-V21	306
3	PYGLAF	238
4	$\mu$ -toksia	216
5	ConArg	214

(d) Ideal Semantics

Rank	Solver	Score
1	PYGLAF	1515
2	$\mu$ -toksia	1103
3	ASPARTIX-V21	744
4	ConArg	428

(e) Semi-Stable Semantics

Rank	Solver	Score
1	ASPARTIX-V21	879
2	$\mu$ -toksia	788
3	ConArg	425

(f) Stage Semantics

Table 2: Rankings for the Exact track

3.3 Results of the Approximate Track

The results for the various subtracks of the Approximate Track are described in Table 3. AFGCN wins most of the subtracks, but it is dominated by HARPER $++$ for the complete and ideal semantics. Detailed results (ranking for each subtrack, with the cumulated runtime over the successfully solved instances) are given in Appendix B.

Rank	Solver	Score
1	HARPER++	747
2	AFGCN	668

(a) Complete Semantics

Rank	Solver	Score
1	AFGCN	637
2	HARPER++	457

(b) Stable Semantics

Rank	Solver	Score
1	AFGCN	567
2	HARPER++	438

Rank	Solver	Score	Cumulated Runtime
1	HARPER++	108	9.848397
2	AFGCN	108	470.655630

(d) Ideal Semantics

Rank	Solver	Score
1	AFGCN	522
2	HARPER++	351

(e) Semi-Stable Semantics

Rank	Solver	Score
1	AFGCN	392
2	HARPER++	349

(f) Stage Semantics

Table 3: Rankings for the Approximate track

4 Conclusion

This paper presents a short overview of the organization of ICCMA 2021 as well as the results of the competition. Detailed results and benchmark descriptions will be available soon at http://argumentationcompetition.org/2021/index.html.

Several ideas could be interesting for the organizers of the future editions of the competition. The first one consists in reviving the dynamic track that was introduced at ICCMA 2019. Unfortunately, the lack of participants for this track conducted us to remove it from ICCMA 2021, but it seems to us that real world application of argumentation requires techniques dedicated to dynamic scenarios, which makes it an important feature of next ICCMA editions. The same comment applies to structured argumentation, which was also removed from ICCMA 2021. On the contrary, two interesting topics were introduced to ICCMA 2021 thanks to suggestions from the community. The first one is the track dedicated to approximate algorithms. For this first introduction of approximate algorithms at ICCMA, we chose to focus on the decision problems which are quite easy to evaluate. For the other reasoning tasks, it is not so easy to determine a relevant metric for evaluating the answer of an approximation algorithm. For instance, if the task is to determine a preferred extension of an AF, should we rank equally a solver that returns a (possibly non-maximal) admissible set and a solver that returns a conflicting set? This question, and similar ones, conducted us to exclude CE- $\sigma$ and SE- $\sigma$ for the approximate track. Metrics for evaluating approximate algorithms for these problems are thus an interesting question for the future of ICCMA. The other question that arose during the organization of ICCMA 2021 is the issue of parallel computing. One solver ( $\mu$ -toksia) was submitted in two versions, one single-threaded and one multi-threaded. While we decided to run the multi-threaded versions without ranking it, a special track dedicated to parallel computing could be interesting in the future.

For concluding this short description of ICCMA 2021, we thank all the participants, as well as the ICCMA steering committee for providing some valuable feedback during the preliminary phases of the organization. The competition was run on the CRIL computer cluster, that was funded by the French Ministry of Research and the Région Hauts de France through CPER DATA.

References

[1] Pietro Baroni, Dov M. Gabbay, Massimiliano Giacomin, and Leendert van der Torre. Handbook of Formal Argumentation. College Publications, 2018.
[2] Dov Gabbay, Massimiliano Giacomin, Guillermo R. Simari, and Matthias Thimm. Handbook of Formal Argumentation, volume 2. College Publications, 2021.
[3] Yannis Dimopoulos, Jean-Guy Mailly, and Pavlos Moraitis. Arguing and negotiating using incomplete negotiators profiles. Auton. Agents Multi Agent Syst., 35(2):18, 2021.
[4] Leila Amgoud and Srdjan Vesic. On the use of argumentation for multiple criteria decision making. In Proc. of IPMU’12, pages 480–489, 2012.
[5] Wolfgang Dvorák and Paul E. Dunne. Computational problems in formal argumentation and their complexity. In Pietro Baroni, Dov Gabbay, Massimiliano Giacomin, and Leendert van der Torre, editors, Handbook of Formal Argumentation, pages 631–688. College Publications, 2018.
[6] Matthias Thimm and Serena Villata. The first international competition on computational models of argumentation: Results and analysis. Artif. Intell., 252:267–294, 2017.
[7] Sarah Alice Gaggl, Thomas Linsbichler, Marco Maratea, and Stefan Woltran. Design and results of the second international competition on computational models of argumentation. Artif. Intell., 279, 2020.
[8] Stefano Bistarelli, Lars Kotthoff, Francesco Santini, and Carlo Taticchi. A first overview of iccma’19. In Proceedings of the Workshop on Advances In Argumentation In Artificial Intelligence 2020 co-located with the 19th International Conference of the Italian Association for Artificial Intelligence (AIxIA 2020), Online, November 25-26, 2020, pages 90–102, 2020.
[9] Jean-Marie Lagniez, Emmanuel Lonca, Jean-Guy Mailly, and Julien Rossit. Introducing the fourth international competition on computational models of argumentation. In Proceedings of the Third International Workshop on Systems and Algorithms for Formal Argumentation co-located with the 8th International Conference on Computational Models of Argument (COMMA 2020), September 8, 2020, pages 80–85, 2020.
[10] Phan Minh Dung. On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artif. Intell., 77(2):321–358, 1995.
[11] Martin W. A. Caminada, Walter Alexandre Carnielli, and Paul E. Dunne. Semi-stable semantics. J. Log. Comput., 22(5):1207–1254, 2012.
[12] Bart Verheij. Two approaches to dialectical argumentation: admissible sets and argumentation stages. In Proc.of NAIC’96, pages 357–368, 1996.
[13] Phan Minh Dung, Paolo Mancarella, and Francesca Toni. Computing ideal sceptical argumentation. Artif. Intell., 171(10-15):642–674, 2007.
[14] Pietro Baroni, Martin Caminada, and Massimiliano Giacomin. Abstract argumentation frameworks and their semantics. In Pietro Baroni, Dov Gabbay, Massimiliano Giacomin, and Leendert van der Torre, editors, Handbook of Formal Argumentation, pages 159–236. College Publications, 2018.
[15] Uwe Egly, Sarah Alice Gaggl, and Stefan Woltran. Answer-set programming encodings for argumentation frameworks. Argument Comput., 1(2):147–177, 2010.
[16] Johannes K. Fichte, Markus Hecher, Piotr Gorczyca, and Ridhwan Dewoprabowo. A-Folio DPDB – system description for ICCMA 2021. http://argumentationcompetition.org/2021/downloads/a-folio-dpdb.pdf, 2021.
[17] Wolfgang Dvorák, Matthias König, Johannes Peter Wallner, and Stefan Woltran. ASPARTIX-v21. CoRR, abs/2109.03166, 2021.
[18] Stefano Bistarelli, Fabio Rossi, Francesco Santini, and Carlo Taticchi. ConArg: A constraint-programming solver for abstract argumentation problems. http://argumentationcompetition.org/2021/downloads/conarg.pdf, 2021.
[19] Matthias Thimm, Federico Cerutti, and Mauro Vallati. Fudge: A light-weight solver for abstract argumentation based on SAT reductions. CoRR, abs/2109.03106, 2021.
[20] Maximilian Heinrich. The MatrixX solver for argumentation frameworks. CoRR, abs/2109.14732, 2021.
[21] Andreas Niskanen and Matti Järvisalo. $\mathrm{\mu}$ -toksia: An efficient abstract argumentation reasoner. In Diego Calvanese, Esra Erdem, and Michael Thielscher, editors, Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning, KR 2020, Rhodes, Greece, September 12-18, 2020, pages 800–804, 2020.
[22] Mario Alviano. The PYGLAF argumentation reasoner (ICCMA2021). CoRR, abs/2109.03162, 2021.
[23] Lars Malmqvist. AFGCN: An approximate abstract argumentation solver. http://argumentationcompetition.org/2021/downloads/afgcn.pdf, 2021.
[24] Matthias Thimm. Harper $++$ : Using grounded semantics for approximate reasoning in abstract argumentation. http://www.mthimm.de/pub/2021/Thimm_2021.pdf, 2021.

Appendix A Results of the Exact Track

A multithreaded version of $\mu$ -toksia has been submitted, and was run out of competition. Its results are presented here with those of single threaded solvers, although it was not eligible for an ICCMA award.

A.1 Complete Semantics

Table 4 shows the results for the complete semantics.

Rank	Solver	Score	Runtime
1	PYGLAF	107	283014.802853
2	$\mu$ -TOKSIA	107	288298.145252
3	FUDGE	107	288718.386884
4	ASPARTIX-V	107	288844.420419
5	ConArg	106	206539.159533
n/a	$\mu$ -TOKSIA-parallel	105	287803.928120
6	A-Folio-DPDB	80	224853.575249
7	MatrixX	57	240888.182199

(a) CE-CO

Rank	Solver	Score	Runtime
1	FUDGE	587	931.517987
2	A-Folio-DPDB	587	5706.033478
3	$\mu$ -TOKSIA	587	7540.231083
n/a	$\mu$ -TOKSIA-parallel	587	12981.315567
4	MatrixX	587	22954.152336
5	ASPARTIX-V	587	30644.831330
6	PYGLAF	586	56647.132293
7	ConArg	107	199182.937802

(b) SE-CO

Rank	Solver	Score	Runtime
n/a	$\mu$ -TOKSIA-parallel	587	21217.410508
1	A-Folio-DPDB	584	19690.986367
2	PYGLAF	557	75189.484641
3	$\mu$ -TOKSIA	522	62550.500735
4	ASPARTIX-V	506	107024.507674
5	FUDGE	414	127037.443418
6	ConArg	108	201311.092994
7	MatrixX	58	240710.947344

Rank	Solver	Score	Runtime
1	FUDGE	587	928.161398
2	A-Folio-DPDB	587	6614.674043
3	$\mu$ -TOKSIA	587	8540.708610
n/a	$\mu$ -TOKSIA-parallel	587	13997.941173
4	ASPARTIX-V	587	30766.565703
5	PYGLAF	585	56820.945178
6	ConArg	107	199660.489916
7	MatrixX	57	242088.917976

(d) DS-CO

Rank	Solver	Score	Runtime
n/a	$\mu$ -TOKSIA-parallel	1866	336000.595368
1	A-Folio DPDB	1838	256865.269137
2	PYGLAF	1835	471672.364965
3	$\mu$ -toksia	1803	366929.58568
4	ASPARTIX-V21	1787	457280.325126
5	FUDGE	1695	417615.509689
6	MatrixX	759	746642.199855
7	ConArg	428	806693.680245

(e) Overall Results for CO

Table 4: Result for the Exact Solvers on the Complete Semantics

A.2 Preferred Semantics

Table 5 presents the results for the preferred semantics.

Rank	Solver	Score	Runtime
1	FUDGE	107	284741.528303
2	$\mu$ -TOKSIA	107	285297.039682
3	ASPARTIX-V	107	288914.836731
4	PYGLAF	107	289391.458361
5	ConArg	107	295524.859508
n/a	$\mu$ -TOKSIA-parallel	105	287455.555046

(a) CE-PR

Rank	Solver	Score	Runtime
1	$\mu$ -TOKSIA	305	209598.856572
2	FUDGE	298	206903.872679
n/a	$\mu$ -TOKSIA-parallel	283	219758.945737
3	ASPARTIX-V	266	231154.499826
4	PYGLAF	210	255406.861456
5	ConArg	107	291867.461181

(b) SE-PR

Rank	Solver	Score	Runtime
n/a	$\mu$ -TOKSIA-parallel	587	21132.196079
1	PYGLAF	557	75277.596631
2	$\mu$ -TOKSIA	523	62541.727318
3	ASPARTIX-V	506	107016.876147
4	FUDGE	414	127127.327070
5	ConArg	108	199312.234514

Rank	Solver	Score	Runtime
1	PYGLAF	425	143255.632741
2	FUDGE	371	159974.358059
3	$\mu$ -TOKSIA	275	223662.448184
n/a	$\mu$ -TOKSIA-parallel	220	242186.574490
4	ASPARTIX-V	173	265994.171780
5	ConArg	107	292285.817332

(d) DS-PR

Rank	Solver	Score	Runtime
1	PYGLAF	1299	763331.549189
2	$\mu$ -toksia	1210	781100.071756
n/a	$\mu$ -TOKSIA-parallel	1195	770533.271352
3	FUDGE	1190	778747.086111
4	ASPARTIX-V21	1052	893080.384484
5	ConArg	429	1078990.372535

(e) Overall Results for PR

Table 5: Result for the Exact Solvers on the Preferred Semantics

A.3 Semi-Stable Semantics

Table 6 presents results for the semi-stable semantics.

Rank	Solver	Score	Runtime
1	ConArg	107	201218.617540
2	ASPARTIX-V	107	288935.416848
3	$\mu$ -TOKSIA	107	289033.884355
4	PYGLAF	106	228421.036070
n/a	$\mu$ -TOKSIA-parallel	105	291565.135040

(a) CE-SST

Rank	Solver	Score	Runtime
1	PYGLAF	442	149977.311142
2	$\mu$ -TOKSIA	285	222654.386562
n/a	$\mu$ -TOKSIA-parallel	271	236882.931798
3	ASPARTIX-V	215	245358.997303
4	ConArg	107	198299.550176

(b) SE-SST

Rank	Solver	Score	Runtime
1	PYGLAF	485	114847.031276
2	$\mu$ -TOKSIA	481	83988.743044
n/a	$\mu$ -TOKSIA-parallel	476	90649.801238
3	ASPARTIX-V	210	250516.541457
4	ConArg	107	200332.750174

Rank	Solver	Score	Runtime
1	PYGLAF	482	115633.357949
2	$\mu$ -TOKSIA	230	242028.695258
3	ASPARTIX-V	212	246004.720807
n/a	$\mu$ -TOKSIA-parallel	156	272773.151090
4	ConArg	107	199271.215080

(d) DS-SST

Rank	Solver	Score	Runtime
1	PYGLAF	1515	608878.736437
n/a	$\mu$ -TOKSIA-parallel	1008	891871.019166
2	$\mu$ -toksia	1103	837705.709219
3	ASPARTIX-V21	744	1030815.676415
4	ConArg	428	799122.13297

(e) Overall Results for SST

Table 6: Results for the Exact Solvers on the Semi-Stable Semantics

A.4 Results for the Stable Semantics

Table 7 presents results for the stable semantics.

Rank	Solver	Score	Runtime
1	PYGLAF	176	262076.166538
2	FUDGE	171	261123.826755
3	$\mu$ -TOKSIA	164	263995.703902
3	ASPARTIX-V	163	266764.981322
n/a	$\mu$ -TOKSIA-parallel	159	272437.936664
4	A-Folio-DPDB	125	247167.250461
5	ConArg	107	143824.774571
6	MatrixX	62	248857.004617

(a) CE-ST

Rank	Solver	Score	Runtime
1	A-Folio-DPDB	577	57664.945319
2	PYGLAF	508	116669.565521
3	FUDGE	457	135396.148397
4	ASPARTIX-V	399	172533.607683
5	$\mu$ -TOKSIA	387	167510.314348
n/a	$\mu$ -TOKSIA-parallel	371	190641.447195
6	ConArg	107	141595.187820
7	MatrixX	71	243684.820216

(b) SE-ST

Rank	Solver	Score	Runtime
1	A-Folio-DPDB	585	35566.107761
2	PYGLAF	548	89570.851193
3	FUDGE	505	92693.283381
4	$\mu$ -TOKSIA	504	94875.690027
5	ASPARTIX-V	472	118986.052239
n/a	$\mu$ -TOKSIA-parallel	463	126633.414211
6	ConArg	108	142981.220887
7	MatrixX	64	247636.605310

Rank	Solver	Score
1	A-Folio-DPDB	575	58912.571800
2	PYGLAF	511	115965.465025
3	FUDGE	452	139597.720683
4	ASPARTIX-V	395	172828.848761
5	$\mu$ -TOKSIA	386	166901.037702
n/a	$\mu$ -TOKSIA-parallel	373	189986.356391
6	ConArg	107	141538.881464
7	MatrixX	62	247830.428679

(d) DS-ST

Rank	Solver	Score	Runtime
1	A-Folio-DPDB	1862	399310.875341
2	PYGLAF	1743	584282.048277
3	FUDGE	1585	628810.979216
4	$\mu$ -toksia	1441	693282.745979
5	ASPARTIX-V21	1429	713113.490005
n/a	$\mu$ -TOKSIA-parallel	1366	779699.154461
6	ConArg	429	569940.064742
7	MatrixX	259	988008.858822

(e) Overall Results for ST

Table 7: Results for the Exact Solvers on the Semi-Stable Semantics

A.5 Stage Semantics

Table 8 shows the results for the stage semantics. PYGLAF was removed from the ranking because of wrong results on CE-STG.

Rank	Solver	Score	Runtime
1	$\mu$ -TOKSIA	107	287359.108885
2	ASPARTIX-V	107	288840.295997
3	ConArg	105	81427.551066
n/a	$\mu$ -TOKSIA-parallel	105	290706.406160

(a) CE-STG

Rank	Solver	Score	Runtime
1	PYGLAF	504	112541.710700
2	ASPARTIX-V	271	225125.628443
3	$\mu$ -TOKSIA	236	236304.243330
n/a	$\mu$ -TOKSIA-parallel	180	262418.192826
4	ConArg	107	79497.777484

(b) SE-STG

Rank	Solver	Score	Runtime
1	ASPARTIX-V	245	235698.489668
2	$\mu$ -TOKSIA	219	242756.715179
n/a	$\mu$ -TOKSIA-parallel	166	267034.642346
3	PYGLAF	149	276541.935306
4	ConArg	106	81097.743104

Rank	Solver	Score	Runtime
1	ASPARTIX-V	256	230874.248362
2	$\mu$ -TOKSIA	226	240280.159325
n/a	$\mu$ -TOKSIA-parallel	176	265249.960730
3	PYGLAF	164	270734.066210
4	ConArg	107	79347.232359

(d) DS-STG

Rank	Solver	Score	Runtime
1	ASPARTIX-V21	879	980538.66247
2	$\mu$ -toksia	788	1006700.226719
n/a	$\mu$ -TOKSIA-parallel	627	1085409.202062
3	ConArg	425	321370.304013

(e) Overall Results for STG

Table 8: Results for the Exact Solvers on the Stage Semantics

A.6 Ideal Semantics

Table 9 shows the results for the ideal semantics.

Rank	Solver	Score	Runtime
1	FUDGE	234	242189.672977
n/a	$\mu$ -TOKSIA-parallel	149	279337.091744
2	PYGLAF	118	291221.721682
3	$\mu$ -TOKSIA	108	287179.393261
4	ConArg	107	300671.098342
5	ASPARTIX-V	104	279290.787518

(a) SE-ID

Rank	Solver	Score	Runtime
1	FUDGE	258	222757.487117
2	ASPARTIX-V	202	263712.239254
n/a	$\mu$ -TOKSIA-parallel	151	278761.114962
3	PYGLAF	120	290737.838592
4	$\mu$ -TOKSIA	108	287582.446341
5	ConArg	107	300511.844717

(b) DS-ID

Rank	Solver	Score	Runtime
1	FUDGE	492	464947.160094
2	ASPARTIX-V21	306	543003.026772
n/a	$\mu$ -TOKSIA-parallel	300	558098.206706
3	PYGLAF	238	581959.560274
4	$\mu$ -toksia	216	574761.838951
5	ConArg	214	601182.943059

Table 9: Results for the Exact Solvers on the Ideal Semantics

Appendix B Results of the Approximate Track

B.1 Complete Semantics

Table 10 shows the results for the complete semantics.

Rank	Solver	Score	Runtime
1	AFGCN	291	15273.326580
2	HARPER++	160	677.187391

(a) DC-CO

Rank	Solver	Score	Runtime
1	HARPER++	587	958.272555
2	AFGCN	377	18882.239960

(b) DS-CO

Rank	Solver	Score	Runtime
1	HARPER++	747	1635.459946
2	AFGCN	668	34155.56654

Table 10: Results for the Approximate Solvers on the Complete Semantics

B.2 Preferred Semantics

Table 11 shows the results for the preferred semantics.

Rank	Solver	Score	Runtime
1	AFGCN	288	15253.015020
2	HARPER++	159	674.846465

(a) DC-PR

Rank	Solver	Score	Runtime
1	HARPER++	279	68.114114
2	AFGCN	279	3138.994710

(b) DS-PR

Rank	Solver	Score	Runtime
1	AFGCN	567	18392.00973
2	HARPER++	438	742.960579

Table 11: Results for the Approximate Solvers on the Preferred Semantics

B.3 Semi-Stable Semantics

Table 12 shows the results for the semi-stable semantics.

Rank	Solver	Score	Runtime
1	AFGCN	290	13470.044680
2	HARPER++	119	621.545732

(a) DC-SST

Rank	Solver	Score	Runtime
1	HARPER++	232	50.005328
2	AFGCN	232	2292.188910

(b) DS-SST

Rank	Solver	Score	Runtime
1	AFGCN	522	15762.23359
2	HARPER++	351	671.55106

Table 12: Results for the Approximate Solvers on the Semi-Stable Semantics

B.4 Stable Semantics

Table 13 shows the results for the stable semantics.

Rank	Solver	Score	Runtime
1	AFGCN	300	14610.356710
2	HARPER++	142	642.296641

(a) DC-ST

Rank	Solver	Score	Runtime
1	HARPER++	337	289.741214
2	AFGCN	315	9290.415250

(b) DS-ST

Rank	Solver	Score	Runtime
1	AFGCN	637	23900.77196
2	HARPER++	457	932.037855

Table 13: Results for the Approximate Solvers on the Stable Semantics

B.5 Stage Semantics

Table 14 shows the results for the stage semantics.

Rank	Solver	Score	Runtime
1	AFGCN	164	2228.788830
2	HARPER++	121	48.270998

(a) DC-STG

Rank	Solver	Score	Runtime
1	HARPER++	228	49.946634
2	AFGCN	228	2301.554530

(b) DS-STG

Rank	Solver	Score	Runtime
1	AFGCN	392	4530.34336
2	HARPER++	349	98.217632

Table 14: Results for the Approximate Solvers on the Stage Semantics

B.6 Ideal Semantics

Table 15 shows the results for the ideal semantics.

Rank	Solver	Score	Cumulated Runtime
1	HARPER++	108	9.848397
2	AFGCN	108	470.655630

Table 15: Results for the Approximate Solvers on the Ideal Semantics (DS-ID)