SuMo: A Mutation Testing Strategy for Solidity Smart Contracts

The Ethereum platform is based on the account model. Each account is an object identified by a 20-byte address and associated with a state. There are two types of accounts: (1) Externally Owned Account (EOA): is controlled by a private key. The owner of the key can perform transactions and manage the funds associated with the account. (2) Contract Account (CA): is controlled by the code. When triggered by a transaction, a CA can perform operations on its storage and fire calls to other contracts. The Account State comprises the following information: nonce: a counter that stores either the number of transactions executed from an EOA or the number of generated contract instances for a CA; balance: the amount of Ether held by the account; storageRoot: the Merkle-Patricia trie root that encodes the storage contents of the account; It is only relevant for a CA. codeHash: hash of the code associated with a CA.

Ethereum was defined in the Ethereum Yellow Paper [10] as a transaction-based state machine. Unlike Bitcoin, Ethereum has a global state that enables the management of real account balances. Each Account State is stored inside a World State trie data structure. Every time transactions take place, the global state of the blockchain is updated. The Ethereum blockchain can be seen as a state transition system, where a state transition takes a state and a transaction and outputs a new state as a result. The last block of the chain represents the current world state. This block holds information about the balances of all the Ethereum accounts after the most recent transactions have been executed.

A transaction is a digitally signed message issued by an EOA. Three types of transactions can be executed with different purposes: Monetary: A monetary transaction is a simple transfer of funds between EOAs. Contract Deployment: A contract deployment transaction allows the sender to create a CA on the blockchain. The transaction must contain the compiled contract code as a payload. Contract Execution: A contract execution transaction runs a function defined within a deployed contract. Upon execution, the smart contract can perform operations on its own state and send message calls to other CAs.

The execution and confirmation of a transaction require several steps. First, the sender must create a transaction and broadcast it to the Ethereum network. The miner nodes are responsible for including new transactions into a candidate block. Each miner selects the transactions to include in the block from a transaction pool and starts validating them. This process involves checking the digital signature and the output of any contract code execution. Once the candidate block is built, the miner starts working to find a valid Proof of Work solution. When a valid hash is found the candidate block is broadcast to the rest of the network. Upon reception of the block, each full node repeats the validation process for each transaction. The nodes do not compare the results among themselves, but they rely solely on the consensus mechanism; if at least one transaction is invalid every honest node will reject the block. A valid block cannot be considered part of the main chain until it is confirmed. The distributed nature of the blockchain can cause the nodes to receive blocks in a different order. When separate parts of the network follow distinct solutions the main chain forks. Ethereum selects the only valid branch using the GHOST (Greedy Heaviest Observed Subtree) protocol [11]. If a valid block is part of a discarded fork, all the included transactions go back to the transaction pool. Otherwise, the transactions will be permanently committed to the blockchain. In this case, users must wait for a confirmation which is generally issued after 12 blocks.

Each node of the Ethereum network includes the Ethereum Virtual Machine (EVM), a stack-based execution environment for running smart contract code. The EVM allows each Ethereum node to participate in the verification protocol. Since each contract is locally executed by the whole network the validation process can become computationally expensive. Moreover, non-terminating program executions could potentially freeze the participating nodes. Since the EVM supports Turing complete languages, predicting the amount of required computational resources is not simple. Ethereum implements a gas mechanism that serves two main purposes: It limits the usage of resources, and it compensates miners for their work. The issuer of a transaction must pay a gas fee to the client that commits the transaction to the blockchain. This fee is determined by the amount of computation and data storage required by the transaction execution. Each transaction object contains a gasLimit and a gasPrice field. The gas price indicates the market price in Wei of a unit of gas. The gas limit is the maximum amount of gas that can be burnt for executing the transaction. The amount of gas needed for executing a smart contract depends on the number and type of instructions run by the EVM.

Solidity is the most widely used object-oriented language for developing smart contracts. When the high-level Solidity code is compiled to bytecode, it can be run on the EVM to enforce the terms described within the contract. The syntax of Solidity is mainly influenced by JavaScript, but it is statically and strongly typed. Solidity supports inheritance, polymorphism, and user-defined data types. Moreover, it introduces several novel constructs and keywords dedicated to smart contracts development.

A function is a unit of code that can be executed within a contract as a result of a transaction or a message call. Solidity provides several standard modifiers that can be added to a function to alter its behavior. In Solidity, it is possible to return data from a function using two different syntaxes. Moreover, each function can have multiple return values. These peculiarities can be a source of confusion, and lead to mistakes in the implementation of a function. RVS (Return Values Swap) is a novel operator included in SuMo. In case a function returns multiple values, the RVS operator swaps them among themselves. RVS targets both types of return structure supported by Solidity. In particular, each return value is swapped with a single compatible return value within the return statement. This allows to limit the number of generated mutants in case of lengthy return structures. Two values are only considered compatible if they belong to the same data type. This allows to prevent the generation of stillborn mutants and speed up the mutation process. Additional operators belonging to such a class are: PKD (Payable Keyword Deletion), and RSD (Return Statement Deletion).

Solidity allows to attach visibility modifiers to functions and variables. These alter how the function (or variable) can be accessed by other contracts. Ensuring the correct usage of visibility keywords is essential to avoid the deployment of faulty code. Using a visibility keyword that is too restrictive might cause integration issues with other contracts. This type of fault does not necessarily lead to a software failure. However, depending on the internal logic of the affected function, an external party could be able to make unauthorized or unintended state changes. For instance, the affected function could perform critical operations such as withdrawing funds (SWC-105 – SWC-n can be accessed at https://swcregistry.io/docs/SWC-n) or self-destructing the contract (SWC-106). State variables are permanently stored in the blockchain to maintain the current state of the contract. Functions can perform operations on these variables to read or update the contract state. SuMo implements two mutation operators that simulate faults concerning both function and variable visibility. The FVR (Function Visibility Replacement) operator replaces a function visibility keyword with a different one. There are some exceptions to this rule that we introduced to limit the possible generation of stillborn mutants. FVR does not mutate the receive Ether function and the fallback function, as they can only be declared as external. Moreover, it does not change the visibility of payable functions to internal or private. Lastly, FVR does not delete the visibility keyword, as the function default visibility is no longer allowed. The VVR (Variable Visibility Replacement) operator replaces the visibility keyword of a state variable with a different one. It also adds a visibility keyword to variables with default visibility.

Solidity supports the definition of modifiers for altering the behavior of functions. When a modifier is attached to a function declaration it can have different effects on its execution. Function modifiers are commonly used for implementing access control mechanisms. However, a modifier can also define additional logic that “extends” the behavior of the decorated function. Errors in the usage of modifiers can introduce faults of varying severity into the contract. For instance, attaching the wrong modifier to a function might apply unnecessary restrictions to the transaction execution. This prevents the function from running every time the modifier conditions are not respected (SWC-123). Similarly, omitting a modifier might prevent a precondition check from taking place. In the worst-case scenario, this will lead to the unintended or malicious execution of critical operations, such as withdrawing funds or destroying the contract. The MOD (Modifier Deletion), MOI (Modifier Insertion) and MOR (Modifier Replacement) operators implemented by SuMo simulate errors concerning the wrong usage of modifiers. These operators were inspired by Deviant [16] and ContractMut[18]. MOC (Modifier Order Change) is a novel operator that simulates the erroneous usage of multiple modifiers. If a function signature is attached with multiple modifiers, the MOC operator alters their order. As a result, the modifiers will be run in a different order upon the execution of the function. This mutation can have different effects on the contract, such as the function becoming unreachable. The OMD (Overridden Modifier Deletion) is a novel operator that targets the modifier overriding mechanism. The operator removes an overridden modifier from a derived contract. When the function decorated with the deleted modifier is called, it will be forced to use a modifier of a higher level in the inheritance hierarchy. Executing a different modifier version than the one intended can cause the contract to behave incorrectly. Moreover, the base modifier might apply different restrictions from the overridden one.

A constructor is an optional function that is run upon contract creation. The constructor allows initializing the state variables of a contract before the code is deployed on the blockchain. SuMo implements a novel CCD (Contract Constructor Deletion) mutation operator that targets the contract constructor. This operator was inspired by the JDC operator implemented by muJava [20, 21], which forces a class to execute the default constructor. The strategy described by Andesta et al. [15] is the only one that proposes dedicated mutation operators for altering a contract constructor. However, from Solidity version 0.4.22 the constructor is defined with the new constructor keyword, which requires the definition of a novel mutation. The CCD operator removes the constructor from a contract by commenting it out. As a result, the user-defined constructor will not be executed before deployment, and the contract state will not be correctly initialized. This mutation allows determining whether the tests ensure the correctness of the contract state after deployment.

Solidity contracts can leverage the EVM logging facilities by firing events. Whenever an event is emitted its arguments are memorized in the transaction log, a special data structure that resides in the blockchain. Events are fired from a contract so that external parties can catch them and trigger a response. Even though the log is not accessible to contracts, events are often used within DApps (Decentralized Applications) to monitor the smart contract behavior and trigger code execution. The EED (Event Emission Deletion) operator implemented by SuMo comments out an event emission statement to prevent its execution. This mutation encourages the definition of test cases that include conditions on events. Mutations that replace event invocations are omitted because they can generate many redundant mutants. Since events often log the arguments of the function from which they are emitted, swapping events also has a high chance of generating a stillborn mutant.

Solidity provides two variable units: Ether Units and Time Units. A literal number can take a suffix of wei, gwei or ether to specify a sub-denomination of Ether. Confusing the Ether units can have many dangerous side effects, such as transferring the wrong amount of funds. Using the wrong time unit can cause anomalous contract behavior as well. If the affected value is a trigger for performing actions, this fault can cause a piece of code not to execute, or to execute at the wrong time. SuMo implements the VUR (Variable Unit Replacement) mutation operator to simulate the incorrect usage of variable units.

Solidity provides many special variables and functions that allow the retrieval of information about the blockchain. Confusing the available global variables can have many unexpected effects on the contract executions. Moreover, the improper usage of certain global variables, such as block.timestamp, can introduce dangerous dependencies into the contract. Variables that are subject to the miner’s influence should never be used as a source of entropy (SWC-120), or to trigger time-dependent events (SWC-116). The novel GVR (Global Variable Replacement) operator implemented by SuMo is designed to avoid the incorrect usage of the global variables and functions available in Solidity. It can also prevent the developer from inserting dangerous dependencies into the contract. The replacement rules were designed to limit the generation of stillborn mutants as well as the total number of mutants. GVR only performs replacements of variables that represent likely programming mistakes, and that return a compatible data type. TOR is similar to GVR, but it targets a subset of global variables for retrieving the sender of a transaction.

Solidity provides many global functions for performing different types of operations. These include the addmod() and mulmod() mathematical functions, which are used for modulo addition and multiplication respectively. Since they take in the same parameters, a developer could easily confuse them without noticing the error. Using the incorrect function leads to the computation of wrong values, as well as potential overflow conditions. Solidity also provides the keccak256(), sha256() and ripemd160() global cryptographic functions. These can be used for a variety of tasks, including hash calculation and public key retrieval. Lastly, Solidity provides a special selfdestruct function to remove a smart contract from the blockchain. When executed, the selfdestruct(address payable recipient) method sends all the Ether stored in the contract to the designated address. The contract’s data is then cleared from the state. The incorrect implementation of selfdestruct can be extremely dangerous, as there is no way of recovering the data or the Ether held by a destroyed contract. SuMo implements several operators that target global functions. The MCR (Mathematical and Cryptographic function Replacement) is a novel operator that helps to spot mistakes in the usage of mathematical and cryptographic functions. In particular, it replaces a mathematical or cryptographic function with a different function of the same type. Other operators in this class are SFD (Selfdestruct Function Deletion), and SFI (Selfdestruct Function Insertion).

Solidity provides a special address type for handling contracts. Since addresses are required for performing a wide variety of operations, an address error can propagate in many unexpected ways. More in general, a call to the wrong address will either target a different contract than intended or a non-existent contract. If the faulty address exists, the invoked function will likely not match any identifier of the recipient contract. This will either cause the transaction to fail, or the recipient fallback function to execute (Call To the Unknown). If the target address is orphaned, the transaction to the non-existent contract will fail. This is not a big issue in the case of normal function calls. However, sending Ether to an orphaned address results in the permanent loss of funds. SuMo implements several operators that target contract addresses and their members. AVR (Address Value Replacement) helps in avoiding mistakes regarding the usage of addresses. The SCEC (Switch Call Expression Casting) operator was imported from Deviant for ensuring the correct casting of address types.

Solidity provides several methods for transferring Ether between contracts: transfer(), send(), call(), staticcall() and delegatecall(). They differentiate among each other concerning the gas limit and the returned value in case of error. The ETR (Ether Transfer function Replacement) operator replaces an ether transfer function with a different one. ETR is similar to operators designed for other proposals. However, in SuMo it was adapted to target the new syntax for the call() method, and it was augmented to mutate the delegatecall() and staticcall() methods.

Every reference type in Solidity is associated a data location keyword that specifies where the variable is stored, where memory is used for storing temporary variables and storage is used for persistently storing state variables. Confusing the data location keywords alters the behavior of the affected variable. Replacing storage with memory limits the lifetime of the variable to the execution of the function in which it is declared. Conversely, swapping memory with storage extends the lifetime of the variable. The gas cost is affected as well because storage variables are more expensive than memory variables. The DLR (Data Location Replacement) operator swaps the memory and storage data location keywords.

The delete keyword is a Solidity-specific operator that assigns the default value to a variable. SuMo implements the DOD mutation operator which was imported by MuSC. This removes any instance of the delete keyword from a contract, one at a time.

Solidity manages exceptions with three different state-reverting functions. Since transactions are atomic, reverting all changes is the only way of preventing unsafe contract executions. When triggered, these functions revert all modifications to the state in the current call, including all sub-calls, and flag an error to the caller. require() is used to validate conditions that can only be checked at run-time. If the condition is not met, an exception is thrown and the unused gas is refunded to the sender. assert() is only used for invariants and internal error checking. Upon error the transaction fails and the state changes are reverted without refunding gas to the sender. revert() is similar to require but it is used for handling business logic errors. Errors in the usage of exception-handling statements can have different effects. require() and revert() should not be used for internal error checking, while assert() should not be used for checking external inputs (SWC-110). A buggy or missing assert() statement does not ensure the proper validation of the contract. Likewise, if a require() or revert() statement contains a fault, the run-time conditions will not be properly checked. The EHC (Exception Handling statement Change) operator removes any instance of exception handling statement from a contract, one at a time. This mutation prevents the condition check from taking place. It also replaces an instance of exception handling statement with a different one. In particular, the run-time require() statement is swapped with assert() and vice versa. This mutation simulates the usage of an exception handling statement in the wrong context.

SuMo implements the novel SFR (SafeMath Function Replacement) operator that targets the widely used SafeMath library. SafeMath¹¹1https://docs.openzeppelin.com/contracts/2.x/api/math#SafeMath provides arithmetic functions that automatically revert the transaction in case of overflow. Developers often use them in place of the base arithmetic operators to increase code reliability. The SFR operator replaces each SafeMath function call with a different one, causing the calculation of the wrong value.

The operators supported by Solidity are similar to the ones available in JavaScript. Binary mutation operators target binary expressions, which consist of two operands and one operator. Unary mutation operators target unary expressions, which consist of one operand and one operator. Lastly, assignment mutation operators target the assignment of a value to a variable, which can either be another variable or a literal. BOR (Binary Operator Replacement): The BOR mutation operator targets all binary arithmetic, relational, conditional, bitwise and shift operators. In particular, BOR replaces a binary operator with a single binary operator of the same class to simulate typographical errors. UORD (Unary Operator Replacement or Deletion): The UORD mutation operator targets all unary arithmetic, bitwise and conditional operators. Unary operators are either removed or replaced with a different operator of the same class. AOR (Assignment Operator Replacement): AOR replaces a compound assignment operator with a different one. AOR was designed following the same pattern as PIT’s Math operator. This allows reducing the number of redundant mutants as well as the total number of mutants. ICM (Increments Mirror): SuMo implements a modified version of the Increments Mirror operator. The usage of this operator for Solidity was first introduced by the Vertigo[17] tool, which imported it from PIT. If a developer accidentally types =- instead of -=, the statement is interpreted in the wrong way. In fact, =- is an assignment followed by the unary -, while -= is a shortcut assignment that performs a decrement operation.

Solidity supports most of the known control structures. These are: if, else, while, do, for, break and continue. The return statement was discussed in detail in section III-A1. Solidity includes try/catch block for catching exceptions and reacting to the failure. In Solidity a faulty loop statement could execute the wrong number of iterations and also lead to out-of-gas exception at run time. SuMo implements many mutation operators for control structures. In particular, we included the BCRD (Break and Continue Replacement and Deletion) operator, the CBD (Catch Block Deletion) operator, the CSC (Conditional Statement Change) operator, and finally the LSC (Loop Statement Change) operator.

Literal mutation operators replace hard-coded values within a contract with a different fixed value. In SuMo we included the BLR (Boolean Literal Replacement) operator that simply replaces true with false and false with true. The ILR (Integer Literal Replacement) operator replaces any integer literal with a different value. In particular, each value is incremented and decremented by one. The HLR (Hexadecimal Literal Replacement) operator replaces a hexadecimal literal with a zero or non zero hexadecimal value. The SLR (String Literal Replacement) operator replaces any string literal with the empty string.

Types and conversions mutation operators target data types that are not unique to Solidity. The novel ER (Enum Replacement) operator implemented by SuMo helps to ensure the correct usage of Enums. Enums are broadly used in smart contract development, as they allow the creation of user-defined types. Each Enum can contain one or more members, where the first member represents the default value. If a developer is unaware of the default Enum behavior, it might lead to wrong assumptions about the value stored in a variable. It is also possible that an Enum variable is mistakenly assigned the wrong value, which can cause unexpected contract behavior. The ER operator swaps the first and second members of an Enum definition. This mutation forces the second member to become the default value of the Enum. It also replaces the member of an Enum assignment with a different member. This causes the affected Enum variable to assume a different value than intended. SuMo also implements the novel ECS (Explicit Conversion to Smaller type) operator that helps to avoid dangerous mistakes during explicit conversions. ECS targets explicit integer conversions to simulate truncation faults. In particular, it forces a conversion to the smallest integer type to cause the loss of higher-order bits. Similarly, it forces explicit byte conversions to a smaller type than intended. This can cause the right-most bits to be truncated.

Solidity’s overriding mechanism allows child contracts to redefine the behavior of certain derived functions, modifiers, and constructors. The overriding mechanism must be enabled on each public or internal function with two distinct keywords: A virtual function can be overridden by a derived contract; An override function has been overridden in the current contract. Function modifiers and contract constructors can be overridden in the same manner, although they do not support the usage of the super keyword (or the contract name). SuMo implements several operators that target the overriding mechanism. The ORFD (Overridden Function Deletion), SKD (Super Keyword Deletion) and SKI (Super Keyword Insertion) operators ensure the correct implementation and invocation of overridden methods. These operators were inspired by Deviant, which is the only other framework that implements overriding mutation operators. In addition, SuMo implements the OMD operator that targets the modifier overriding mechanism (section III-A3).

Function overloading allows a smart contract to implement multiple functions with the same name, but with a different number (or type) of parameters. The most problematic aspect of overloading is that it can increase human confusion, which is the primary source of bugs. If a function is heavily overloaded, a developer could forget which version of the method does what. Moreover, when many versions of the same method are available, it is more likely to call an unintended method. Overusing overloading can also lead to the implementation of trivial methods which are not needed. This can cause further confusion, as well as an increased amount of gas for contract deployment. The novel OLFD and ACM operators implemented by SuMo were designed to test different aspects of method overloading in Solidity. The OLFD (Overloaded Function Deletion) operator removes the declaration of each overloaded method within the contract, one at a time. This operator is useful in two ways. First, it allows identifying errors in the invocation of overloaded methods. If the mutant contract still works as expected without the deleted method, it means that such a method is not being called correctly. This can happen, for instance, when the developer invokes the wrong method version. Besides, OLFD allows to ensure coverage of overloaded methods. The test suite can only notice the mutation if the missing method is actually being called. The ACM (Argument Change of Overloaded Method Call) operator alters the invocation of overloaded methods within the contract. In particular, it swaps an overloaded method call with one different, existing overloaded method call. The mutated invocation must have different arguments to match a different overloaded method. This causes a different method version to be invoked.