Improving the Accuracy of Transaction-Based Ponzi Detection on Ethereum

Ethereum is the second most popular blockchain after Bitcoin in terms of market capitalization [15]. It is also the largest platform that provides a decentralised virtual environment (i.e., Ethereum Virtual Machine or EVM for short) to execute smart contracts [46]. In 2022, the Ethereum chain reached 15 million blocks with over 1.5 billion transactions [48]. Smart contracts on Ethereum are executable programs that run automatically when their trigger conditions are met. Those contracts can be implemented using an object-oriented and high-level language called Solidity [22]. Contract source codes are then compiled into bytecodes, which can be represented as low-level human-readable instructions - opcodes [55]. After that, the bytecodes are launched onto EVM. Once a contract is deployed, it cannot be modified by anyone. Moreover, any activity in the life cycle of a contract must be triggered by a transaction. Therefore, any interaction ‘from’ or ‘to’ a contract is recorded as a transaction and stored on the blockchain permanently. In other words, in Ethereum, a transaction is a key unit that involves all activities of a contract.

The blockchain technology, although has the potential to revolutionise the way traditional businesses work [36], also creates a golden opportunity for cybercriminals, resulting in the migrations of many financial scams to the blockchain platforms [9]. Among blockchain scams, Ponzi schemes [2] were the most popular from 2017 to 2020. In hindsight, this is not a surprise because Blockchain’s inherent properties, i.e., automation, transparency, immutability, and anonymity create an ideal environment for this scam to grow [35].

In layman term, Ponzi schemes are scams often camouflaged as high-return investment programs that use the funds from new investors to pay existing ones. With no real project behind and no intrinsic value, a Ponzi scheme will collapse when there are not enough new investors joining and/or the payment commitment can no longer be fulfilled. A more official and authoritative definition of Ponzi schemes is given by the U.S. Securities and Exchange Commission [43]. At the heart of each Ponzi scheme is a money redistribution mechanism. Bartoletti et al. [3] classified Ponzi schemes on Ethereum into four different categories based on their redistribution mechanisms, including Chain-shaped, Tree-shaped, Handover, and Waterfall schemes (see Appendix 0.A).

Existing Ponzi detection models can be divided into two groups, depending on whether they rely on smart contract codes or on the transactions.

Contract-Code-Based Approaches: Bartoletti et al. [3] first proposed four criteria to detect a Ponzi application by inspecting their contract source codes. However, it turns out that the Solidity source codes of 77.3% contracts on Ethereum are not available [60]. To tackle this drawback and to detect Ponzi automatically, many researchers built Ponzi detection tools based on the frequency distribution of operation codes (opcodes), which are always available on the Ethereum chain. Chen et al. [12, 13] proposed an automatic detection tool on opcode features using machine learning models. Their experimental results showed that the detection models using opcode features achieved greater performance than those using account features, which were aggregated from transactions. Fan et al. [23] pointed out that an imbalanced dataset caused an over-fit in previous works [12, 13]. To improve data quality and the detection accuracy, they proposed a data enhancement method that expanded the dataset and eliminated the imbalance using data sampling techniques. Wang et al. [51] adopted a deep learning technique to build a more accurate detection tool and also used oversampling techniques (Smote and Tomek) to deal with an imbalanced dataset. Jung et al. [32] and Sun et al. [44] focused more on crafting better representative features than improving data quality.

A common drawback of all previously mentioned studies is the lack of robustness. As pointed out by Chen et al. [14], scammers can use code obfuscation techniques [6] to counter those detection models that rely on opcode features (see [14, Section 7.2.1]). For example, a contract code can be manipulated or modified to change the opcode occurrence frequency. Chen et al. [14] also proposed in their work a new detection tool called SADPonzi, which was built upon a semantic-aware approach and achieved 100% Precision and Recall. SADPonzi was proven to be more robust than the current opcode-based method when facing code obfuscation techniques. More specifically, it can detect a Ponzi contract by comparing the extracted semantic information of its bytecode and the predefined semantics of four known Ponzi schemes. However, the approach of SADPonzi requires a domain expert to analyse a Ponzi application’s operational logic to build the corresponding semantic pattern, which can be costly to put into practice. On top of that, as also mentioned by the authors, SADPonzi can only effectively detect known Ponzi types with predefined semantics, and may fail to detect a new Ponzi variant (see [14, Section 8]).

Transaction-Based Approaches: Transactions are records that save historical activities between an application and its participants. Transaction data was used in some existing works [3, 12, 13] to capture the differences between Ponzi and non-Ponzi applications. Detection tools based on transaction data are more resilient to scammers’ countermeasures because transaction information cannot be modified or deleted from the chain. Although scammers can add transaction records, they cannot manipulate transaction data as freely as they can with smart contract’s source code and opcodes for two reasons. First, any participant, not just the creator, can create transactions, which is not under the control of the contract creator. Second, the cost to create a transaction on the chain is expensive (approximately $14.26 on average per transaction [47]). These factors prevent the Ponzi scammer from manipulating their transaction data arbitrarily to elude detection, e.g. by flooding the system with fake transactions.

Despite several advantages, existing transaction-based models [12, 13, 14] suffered from low classification accuracy, achieving F1-scores around 44%-69% only. The key reason for their mediocre performance, based on our analysis, could be due to the fact that existing works have completely missed the time dimension when building their models. We note that some studies [12, 13, 32, 51], in order to improve the detection accuracy, integrated account features with opcode features. However, such a hybrid approach also inherits the shortcomings of the contract-code-based approach. In this work, we explored the temporal behaviour of Ponzi applications and introduced time-series features alongside existing account features, aiming for both robustness and accuracy in detection.

We refined the dataset of labeled Ponzi and non-Ponzi addresses provided in the SADPonzi paper [14] by first downloading and extracting transaction histories of these contracts from the XBlock-ETH repository [59, 56]. We then filtered out unsuccessful transactions which failed for various reasons such as insufficient gas (a required fee to successfully conduct a transaction) or errors in the contract codes. We also discarded applications with no transactions or having lifetime (the period from the recreation time to the last transaction) shorter than one day. These are outliers, whose behaviours do not resemble the whole group. Even if such an application was a Ponzi, it was also a failed scam. Therefore, removing those applications is important to build a clean dataset, especially for a transaction-based approach. Our refined dataset contains 1182 non-Ponzi and 79 Ponzi applications. The Ponzi types statistics are displayed in Table 1.

In this section, we investigate how Ponzi and non-Ponzi applications work differently with respect to their temporal behaviours. To this end, we chose to analyse the historical transaction data of DynamicPyramid, a representative Ponzi contract²²20xa9e4e3b1da2462752aea980698c335e70e9ab26c (DynamicPyramid’s address). This is a chain-shaped scheme, the most popular type, which constitutes 86% of all known Ponzi contracts. In general, different types of applications have different transaction behaviours, and understandably, Ponzi applications have unique behaviours that are different from non-Ponzi ones. In our analysis, we compare the representative applications of the two groups to demonstrate their potential differences regarding temporal behaviours. A comprehensive comparison between different types of Ponzi and all different types of non-Ponzi, while valuable, would be an overkill for our purpose, and hence, out of the scope.

Transaction volumes. We start our analysis by comparing the transaction volumes of a Ponzi application (DynamicPyramid) and a non-Ponzi application³³30xb2c3531f77ee0a7ec7094a0bc87ef4a269e0bcfc (a non-Ponzi contract address). The transaction volume of an application measures the daily number of associated transactions. As observed in Fig. 1, DynamicPyramid had a shorter lifespan with a peak transaction volume concentrating in the first month followed by almost no activities. In comparison, the non-Ponzi application had more regular activities throughout its long lifespan. The reason is that a Ponzi application, although often introduces itself as a potential project with a high investment return promise, has no actual project behind. Thus, participants of Ponzi applications often participated actively at the beginning as early investors got paid regularly. However, as time goes by, fewer investors got paid and more will start leaving, leading to the unavoidable collapse.

Investment versus payment activities. Pushing our analysis one step further, we break down transactions into two different types, namely, investments and payments. An investment refers to a transaction sending ETH from an investor to an application, whereas a payment refers to a transaction from an application that pays ETH to an investor. As demonstrated in Fig. 2, payments (orange dots) and investments (blue dots) of the Ponzi application concentrated only in the first month. Moreover, each orange dot was preceded by some blue dots of smaller ETH amounts. This is because the examined Ponzi application, a chain-shaped scheme, must gather sufficient new investment funds before making a payment to a single participant. After this intensely active period, the number of payments decreased and finally disappeared. This happened because the application’s balance was no longer enough to make any new payment despite having a few new investments coming in. On the contrary, both investment and payment activities spread out over the lifespan of the non-Ponzi application.

Application balance. The balance of an application is the amount of ETH in the application at a time. How the balance varies as time goes by can indicate the type of application. As demonstrated in Fig. 3, the balance of the Ponzi application (Dynamic Pyramid) often rose gradually (investments), and after a while, dropped dramatically, generating a “cliff” (payment). The reason is that the balance was gradually accumulated from the investments until reaching the amount that the application had committed to pay to a particular investor when they joined the application. Once the desired balance was reached, the promised profit was immediately paid to the corresponding investor.

We classify transaction-based features into two types: account features and time-series features. Transaction-based detection models proposed so far in the literature only used account features [12], which capture general statistics of the transactions associated with the application, e.g. the total/average investment amount, the final balance of the contract, or the maximum number of payments to an investor. As mentioned in Section 3.2, using only account features led to rather poor prediction performance (low F1-scores). To improve transaction-based Ponzi detection models, it is essential to employ both account features and time-series features. We discuss in detail below how to extract these features, especially the new time-series ones.

Account features: This type of features captures general information about the contract of interest and has been widely used in previous studies [12, 13, 32, 51]. More specifically, general statistic metrics such as average, count, sum, standard deviation, and Gini coefficient [54] can be extracted from the set of all relevant transactions to aggregate account features. Although account features are insufficient to capture all behaviours of the Ponzi scheme, they are still very useful in revealing Ponzi’s working logic. For example, the Gini coefficient of the number of payments can demonstrate an inequality in money distribution, or the final balance of the application indicates whether the investment funds have been distributed all to investors. Therefore, we still include in our list 29 account features introduced earlier in the literature [13, 12, 32, 51] (listed in Appendix 0.B).

Time-series features: As discussed earlier, time-series features play an important role in identifying Ponzi applications. Unlike account features, they capture the behaviours and activities throughout the application’s lifetime. To aggregate time-series features, we first partitioned our transactions into several time intervals (with interval length of either 12, 24, or 48 hours) and built 43 time series (see Appendix 0.C for the complete list), which measure various aspects of the transactions. These times series form three dimensions, namely, the contract address, the interval, and the data value for that contract in that interval (e.g. account balance). We then use a dimensionality-reduction technique to capture various characteristics (e.g, mean, entropy, spikiness) of aggregated time-series and mapped the 3-D data into the 2-D space to produce the final time-series features. The creation steps of time-series are depicted clearly below:

(1) For a fixed time duration $T$, we split our transaction data into $N$ time intervals of length $T$ each where $N\triangleq\lceil\texttt{life\_time}/T\rceil$. In our study, we have chosen 3 different values of T to study, which are 12, 24, and 48 hours.

(2) Based on the timestamp field, we assigned each transaction to its corresponding interval.

(3) We created a comprehensive list of 43 time-series to represent all activities that occur during the application’s lifetime. Thus, for each application, the time-series can be represented as a 2-D matrix of size $N\times 43$. Lastly, if the dataset has $M$ applications, the time-series data extracted from the whole dataset can be represented as a 3-D array with size $M\times N\times 43$.

Finally, to generate time-series features, we applied a dimensionality-reduction technique [29] to compress the time-series data. In particular, we employed a finite set of 12 statistical measures (see Appendix 0.D) proposed in [18, 25, 29, 53] to capture the global information of the 43 time-series and compressed the 3-D time-series data down to a 2-D $M\times 516$ matrix (note that $516=43\times 12$). The Python codes for time-series features aggregation are available at [1].

To measure the effectiveness of our proposed set of features (account and time-series features), we reused classification methods employed in the previous studies  [32, 12], namely, Random Forest (RF) [45] and XGBoost (XGB) [11]. In addition, other well-known classification methods such as K-nearest neighbour (KNN) [17], Support vector machine (SVM) [27], and LightGBM (LGBM) [33] were also included in our experiments in order to find the most suitable classification model for the problem. RF uses the bootstrap resampling technique to generate different training decision trees from the original dataset and a prediction is made by aggregating the predictions from these trees. This algorithm works effectively in several domains [42] including fraud detection [5]. XGB is a gradient-boosting based algorithm that creates gradient-boosted decision trees in sequential form and then groups these trees to form a strong model. Unlike RF, the result of XGB is the prediction of the last model, which addressed data misclassified from previous models. KNN is a non-parametric classifier that uses proximity to estimate the likelihood of a data point belonging to one group. SVM is a classic algorithm that has been widely applied in binary classification or fraud detection problems, which performs classification by establishing a hyperplane that enlarges the boundary between two categories in a multi-dimensional feature space. LGBM is also a gradient-boosting-based algorithm similarly to XGB. However, LGBM grows a tree vertically (leaf-wise) while XGB grows trees horizontally (level-wise). With leaf-wise algorithms, LGBM is often more accurate and faster than other gradient-boosting-based algorithms. The Scikit-learn python library⁴⁴4https://scikit-learn.org/stable/ was used for RF (default parameters), KNN (default parameters), SVM (default parameters with “poly” kernel), and XGB (default parameters without using label encoder). The Microsoft LightGBM Python library⁵⁵5https://github.com/microsoft/LightGBM was used for LGBM (with default parameters).

To evaluate the performance of our models, we use standard prediction metrics including Accuracy, Precision, Recall, and F1-score, which are calculated based on the numbers of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) as follows. $\textbf{Accuracy}\triangleq(\texttt{TP}+\texttt{TN})/(\texttt{TP}+\texttt{FP}+\texttt{TN}+\texttt{FN})$ is the fraction of correct predictions, $\textbf{Precision}\triangleq\texttt{TP}/(\texttt{TP + FP})$ is the fraction of the actual Ponzi applications out of all the predicted Ponzi by the method, $\textbf{Recall}\triangleq\texttt{TP}/(\texttt{TP + FN})$ is the fraction of detected scams among all actual scams, and $\textbf{F1-score}\triangleq(2\cdot\texttt{Precision}\cdot\texttt{Recall})/(\texttt{Precision + Recall})$ is the harmonic mean of Precision and Recall.

After account features and time-series features were produced, these two feature groups were used individually and as a combination in different models. Our overall transaction-based detection workflow is designed as follows.

Train-test split: the dataset and their feature groups were split into a training set (80%) and a test set (20%). The former is used for training a detection model, while the later is used to evaluate the trained model.

Data sampling: the Ponzi applications only occupy 6% of total applications in our dataset. Therefore, we applied data sampling techniques to balance our dataset. We adopted the well-known oversampling method Borderline-SMOTE [26] to generate new Ponzi instances that have more than half of the K nearest neighbours being non-Ponzi applications. That helps to enhance the existence of Ponzi applications that are more likely to be misclassified as they are located near the border of the two classes.

Model training: the $K$-fold cross-validation method was used to train a classifier on the training set. In our experiment, we only set $K$ = 5, which is lower than common practice in the literature because our dataset is small.

Model evaluation: a trained model was used to classify the applications in the unseen test dataset. To guarantee the reliability of our experiment, we repeated the experiment process 50 times, and the final result was obtained by taking the average. It is worth mentioning that the same hyperparameters were used for the same models for a fair comparison.

We conducted three experiments to demonstrate the advantages of our proposed time-series features.

Experiment 1 (Feature sets and detection models evaluation). In this experiment, we aimed to evaluate the effectiveness of our proposed feature list while applying it across diverse machine learning models. As already mentioned, most of the previous studies used either opcode features or both opcode features and account features to build their detection models. Only a few works attempted a transaction-based approach (without using opcode features) separately [12, 32]. To show advantages over previous studies, we rerun their approaches on our dataset as the baselines. However, their codes and feature data have not been released to the community, so we re-implemented those models based on the descriptions in their papers, including feature lists and machine learning models.

In this experiment, we first evaluated our feature sets corresponding to different time intervals ($T$ = 12, 24, 48 hours). We also tested with three feature sets: ACC consists of account features only, TS consists of time-series features only, and ACC-TS consists of both. A comparison of various metrics between our feature sets and the baselines including the feature set from Chen et al. [12] (Appendix 0.B.1) and from Jung et al. [32] (Appendix 0.B.2) is provided in Table 2. Finally, we evaluated the detection performance of different machine learning models using our ACC-TS feature sets (see Table 3).

We note that the F1-score we obtained for XGBoost is close to what was reported in [12] (49.9% versus 44%). However, we were unable to reproduce the very high scores reported in [32] for Random Forest. The authors of [14] also encountered the same issue: they re-implemented the approach in [32] and achieved similar F1-score (69.4%) as ours (68.8%). This could be due to the fact that in both [14] and our paper, we started from the same dataset of 1395 Ponzi and non-Ponzi schemes, while in [32], the authors used a different dataset that includes 3203 non-Ponzi addresses. Unfortunately, their paper doesn’t provide enough detail on how these addresses were collected and hence, we were not able to recreate their dataset. We also noticed that although the bytecode size (size_info) was created from the smart contract bytecode, it was listed among the top eight important transaction-based features listed in [32, Table 2]. This makes their detection model depend on the contract code as well and therefore is susceptible to opcode obfuscation techniques [14, 6]. In our experiment, to reproduce their transaction-based model, we ignored this irrelevant (contract-code-based) feature size_info.

As shown in Table 2, the detection models using our ACC-TS 24 Hrs feature set improved the F1-score of the models by Jung et al. [32] and Chen et al. [12] by 5.7% and 30.3%, respectively. It is also clear that using both account and time-series feature lead to better Accuracy, Precision, Recall, and F1-score compared to using individual type of features. Note that when using time-series features (TS) alone, these models already yielded higher F1-scores than the baselines. From Table 3, we observed that tree-based classifiers were more efficient in Ponzi detection than other algorithms. More specifically, RF, XGB, and LGBM achieved better Accuracy, Precision, and F1-score values than other classifiers across different ACC-TS feature sets. Among tree-based models, LGBM with the ACC-TS 12 Hrs features achieved the best F1-score, Accuracy, and Precision. Last but not least, our models achieved 11% higher F1-score for RF compared to [32] and 30% higher F1-score for XGB compared to [12].

Experiment 2 (Contribution of time-series features). Next, we investigate how much the newly proposed time-series features have contributed to LGBM’s performance, which was the best performing model. To do this, we first retrieve the list of feature importance from LGBM in the previous experiment. The importance of a feature in the LGBM model is defined to be the number of times this feature is used to split the data across all decision trees. In LGBM, an effective feature selection technique, namely Exclusive Feature Bundling (EFB), has been adopted to reduce the number of features without affecting the model’s performance. We find that only 205/545 features (516 time-series features and 29 account features) had been used at least once to build a tree in the LGBM detection model. More specifically, these 205 important features consist of 176 time-series features and 29 account features. We sorted these 205 features in descending order of importance. After that, we conducted a detection with the LGBM model using only the $5,10,15,\ldots,205$ most important features among the 205. The experimental results shown in Fig. 4 demonstrate how the F1-score values of the prediction were increased as more time-series features were added in the model. We can also observe in the bottom sub-figure that from the top 5 onward, time-series features start to appear. For example, the top 30 contains 15 account features and 15 time-series features.

As can be seen from Fig. 4, the F1-score sharply increases when we increase the number of features, especially at the beginning. According to the F1-score in Fig. 4, the value exceeds 0.8 when at least 20 features out of 205 most important features are used, reaching a peak at 0.83 with the top 85 features which consist of 63 time-series features and 22 account features. Then, the F1-score values fluctuate around 0.81 as the number of features further increases. As shown in the bottom sub-figure of Fig. 4, the percentage of time-series features in the important feature list increases in the same direction as F1-score. It proves that our proposed time-series features have significantly contributed to the best-performing model (LGBM) in Experiment 1.

According to above experiment, we refined the ACC-TS 12 Hrs feature set by selecting top 85 important features of the best model (see Appendix 0.E), labelled as RF-ACC-TS 12 Hrs. To improve the detection performance, we ran all detection models with selected features instead of the full set of features. Table 4 show the detection performance results between models that use ACC-TS 12 Hrs and RF-ACC-TS 12 Hrs feature sets. Remarkably, the models using RF-ACC-TS 12 Hrs features show an average improvement of 2.1%, 5.5%, 1.6%, 5.3% for Accuracy, Precision, Recall, and F1-score, respectively.

Experiment 3 (Detecting a new type of Ponzi). To verify whether our classification model using the proposed feature list can detect a new Ponzi type, we conducted the third experiment using the LGBM model as follows. The key idea is to train our model on some types of Ponzi schemes and test it on another type of Ponzi schemes to see if it can still accurately detect these schemes.

We first removed all applications for each known type of Ponzi schemes mentioned in Section 2.2 from our training set. The removed applications were then used in a test set for testing the trained model’s new Ponzi detection ability. However, we only removed each of the three Ponzi types (waterfall schemes, tree-shaped schemes, and handover schemes) or all three and not the chain-shaped schemes, which account for 86% of all the Ponzi schemes in the dataset. If we remove all chain-shaped schemes, the number of Ponzi samples becomes too small to learn the scam’s behaviours. Furthermore, various test sets with different scam rates were used to test our model in different situations, e.g., with a full-scam test set (100% scams), a balance test set (50% scams), and a few-scam test set (6% scams, similar to our entire dataset’s scam rate). Due to the lack of Ponzi (P) applications, we can only decrease the scam rate by increasing the number of non-Ponzi (non-P) applications in test sets.

The results, demonstrated in Table 5, indicate that the detection model can detect over 89% of actual new Ponzi applications in a given test set (greater than 89% of Recall value in most cases). Moreover, the Precision value is approximately 80% even in test sets with very few scams, and the model also achieved F1-score at least 90% in all cases. Analysing the top 30 important features exported from the trained model, we notice that more than 80% of features in the list are aggregated from transaction volume, investment and payment activities, and application balance with 50% of them being time-series features. It is not surprising since those features help discriminate between Ponzi and non-Ponzi applications, as clearly shown in Section 3.2. That confirms the ability of the transaction-based approach to detect a new Ponzi and the importance of the time factor. Although the dataset we use (from [14]) is not ideal in the sense that there are very few Ponzi applications of types other than the chain-shaped, which may affect the reliablity of our third experiment, the outcome still gives strong evidence that a completely new Ponzi type can be detected.