Protecting Retail Investors from Order Book Spoofing using a GRU-based Detection Model

Illicit activity, due to its nature, is generally hard to define so rather than attempting a formal definition we will be giving a description of a very common scenario. A fraudulent investor could act as follow:

• •

  He will first place one or multiple large orders on one side of the order book, as close as possible to the current bid-ask to maximise the impact on the market [4]. This would be the fake order that he is not willing to execute.

• •

  If the spoofing succeeds, the orders will trigger a move on the real market price.

• •

  Then the trader will cancel the previously placed orders, usually just before someone starts executing it.

• •

  He will finally place orders on the other side of the order book, thus taking advantage of the market shift.

A more detailed description of the phenomenon can be found in Figure 3, Section D of Leangarun et al. paper [5].

In order to be able to detect this type of activity, the data processing algorithm will have to have access to: 1. the volume of the orders, 2. the distance of the price of orders to the current market bid-ask and 3. the volatility of prices.

The market data currently used by this project can be divided in two levels: 1. Level 1 data is the most basic: it contains Opening/Highest/Lowest/Closing prices and the volume traded in a given period. 2. Level 2 data contains more order book information, which is usually composed of all bid and ask orders that have not been yet matched by the exchange.

For this research project, we are using primarily level 2 data for several pairs of cryptocurrency tokens such as: BTC/USD, DAI/USD, BTC/DAI from several exchanges including Bitfinex and Kraken. The data has been gathered over several months (November 2019-May 2020) on a milliseconds frequency, which totals to 337GB of raw uncompressed data.

The problem of price manipulation spotting can be seen as the detection of anomalies in trading signals. In an unsupervised framework, the time series are not labelled. The model will first learn a representation of the data. If future data points are too far from the representation, the model will consider it as an anomaly.

In [6], the authors try to detect anomalies in the rating of products and services on websites such as Amazon, TripAdvisor, or Yelp. They create a model that learns the base behaviour of users by considering it as a latent multivariate auto-regressive process (Robust Latent Autoregression or RLA). The observation set being already polluted with anomalies, it is not possible to directly infer the base behaviour of users. The authors, therefore, consider that the observed time series are the mix of the base and abnormal behaviour. Then, if the deviation between newly seen observation and the base behaviour of users is too large, an anomaly is detected.

A major advantage of this method is that it does not require labelled data. However, clustering data implies the loss of temporal information which is critical to our problem. Furthermore, spoofing is a well documented type of fraud. As a consequence, labelling our time series based on the definition of spoofing is possible to do without major effort.

The anomaly detection problem can also be seen as a classification task where time series needs to be separated into two categories based on whatever the price is manipulated or not.

In [5], the authors focus on a way to detect spoofing and pump-and-dumping (buy stock to push its price up before dumping all the stock). They use a simple feed-forward neural network on labelled level 1 data from NASDAQ companies (Amazon, Intel, and Microsoft). Unfortunately, the model was not able to detect spoofing. The reason is that level 1 data does not contain enough information, thus preventing the neural network to learn the manipulation mechanism.

In time series analysis, Recurrent Neural Networks (RNN) are widely used because they can grasp temporal dependencies in sequences of data. State of the art recurrent neural network such as GRU (Gated Recurrent Units) [7] can therefore show strong performances in time series classification problems. Elsayed et al. [8] shows that GRU mixed with Fully Convolutional Neural Networks can yield good performances on the classification of different types of time series. This architecture is still outperforming state-of-the-art models in many univariate time series classification problems.

In our current setup, the detection model is developed as a supervised learning problem. The time series are labelled and the model will try to predict the category of the time series, either manipulated or not. We follow the same approach as Leangarun et al. [5] to label the L2 data using thresholds.

Based on the previously established definition of spoofing, the following conditions are defined on the state of the order book to detect price manipulation:

Condition 1 :

As seen in Section 1.1, the volume of the cancelled order needs to be large enough. We can compare the volume of each cancelled order to the aggregated volume of the order book for the 25 top price levels.

• •

  $V^{bid/ask}_{cancelled}(t)\ \ \$ : Volume of the cancelled order

• •

  $V^{cumul(bid/ask)}_{25}$ : Cumulative volume of the 25 top price levels (bid or ask side).

Condition 2:

The price level of the cancelled order needs to be close enough to the current bid-ask. The distance signal needs therefore to be under a certain threshold:

Condition 3: The successful spoofing manipulation leads to an increase in price volatility. As a consequence, the third condition is based on the $\Delta_{\sigma}$ signal:

A spoofing order must therefore meet all these conditions:

The following time series is an example of an alleged spoofing attempt detected thanks to the previously established conditions on the 9th of March 2020 on the ETH/USD market of the Kraken exchange.

In Figure 1, four consecutive suspicious orders are flagged, all containing a very large volume of ETH (1500 ETH worth approx. 150 000 USD at the time). These large orders are added then cancelled several times on the bid side. The aggregated volume of the bid side of the order book is therefore experiencing sudden changes. The regularity in the patterns seems to indicate that this is the work of one wealthy individual or entity (ETH whale) using a trading bot with very high frequency trading capability. In this case the added orders were cancelled immediately after placing them.

It also appears that the manipulation is not restricted to the 4 detected orders. The algorithm (described in Section 3.1) misses suspicious orders at the beginning and at the end of the time series. Therefore, the start and the end of the price manipulation have been here manually annotated, with the full spoofing event visible between the green lines on Figure 1.

The price level shown on the figure is very close to the current bid, at about 1% price move. Therefore, these orders are likely to have an impact on the current ETH price. This can be verified by looking at Figure 2 showing the volatility variation of the prices.

As expected, the volatility variation is substantially increasing and stays above the threshold for several seconds. This can be seen also by looking at the mid price for the same period shown in Figure 3:

The large orders are placed on the bid side, attempting to artificially drive up the price. The plots above show that the price is increasing by 20 cents (USD) in the suspicious 30 seconds periods.

The end of the manipulation, where prices move the most, is unfortunately not detected automatically. This is because no cancellation of orders is sent to the exchange. As we can see on Delta Volume shown on Figure 1, the two last orders are sent to the exchange without being cancelled. However, changes in the aggregated volume of the order book can still be seen. This can be potentially explained by a wash-trading attempt, which is not studied by the current paper. This makes other investors believe that a real trade has been successfully executed, whereas in reality nothing happened.

The price manipulation mechanisms are reflected in time series by complex temporal patterns, but Recurrent Neural Networks (RNN) are powerful models that allow grasping such temporal dependencies. The model used in this paper will be based on Gated Recurrent Units (GRU) [3] as they share the same performance of classic Long Short-Term Memory Networks (LSTM) [9] but are less complex, thus resulting in lower training time. This is a key feature important for our detection framework as it needs to be able to be trainable and deploy-able as quick as possible [3].

The GRU output is used to train a feed-forward neural network. The final layer of the neural network will be a scalar value corresponding to the probability of the time series being manipulated or not. Figure 5 shows a graphical representation of our model¹¹1The full model is available at: BlindedForReview..

The model has been trained using the Adam optimiser due to its computational efficiency on large datasets [10]. A learning rate decay was also implemented to improve the stability of the Binary Cross-entropy Loss. The hyperparameters (i.e. number of GRU layers, learning rate, dropout, number of training epochs) were tuned using a random search strategy.

The chosen network architecture is flexible and allows the integration of other signals and features, such as prices of other assets, other exchange specific information (traded volumes, average prices among other currencies, etc.). To prove this flexibility claim, we experimented with different architectures for additional features, such as cryptocurrency tokens being sent to an exchange at their network address, which can be potential strategy of arbitrageurs. The details of this architecture are in the Appendix, Figure 6. Ultimately, for our experiments we selected the architecture presented in Figure 5 as a good balance between the accuracy, pre-processing complexity and training time.

The dataset used for all our experiments consists in 4 annotated samples following the conditions described in Section 3.1. Each sample describes a different style of price manipulation, varying in term of distribution of the features, length of the event and sampling frequency. The features used for these experiments are standard L2 signals, such as: 1. current bid-ask, 2. delta volume, 3. price of the current price level update, 4. price volatility, 5. cumulative volume (bids), and 6. cumulative volume (asks). Each sample is split into training, validation, and test set (65%, 15%, and 20%). The validation set is used to fine tune the GRU hyperparameters. Table 1 shows each sample characteristics, including the sampling frequency.

Given that all the samples have different data distributions, for all the experiments, the model will be trained and validated on one sample, and then tested on everything else (e.g. train the model on sample 1 and test on samples 1, 2, 3, 4). This should also show the model’s potential to generalise across samples. To compensate for any training bias, the models are trained 5 times and the average accuracy is reported. We also use a weighted accuracy to compensate for class imbalance. Table 2 shows these results. The proposed model could be evaluated against a rule-based system (i.e. Section 3.1) or any other system that detects spoofing events after the fact, but it would be unfair as these systems are not capable or early detection.

It can be noted that, when the train and test data distributions are the same, the test accuracy is higher, except sample 3. All models, except the one trained on sample 3, reach the highest accuracy when tested on sample 1. This may be because the order execution pattern from other samples generalised well for sample 1 as well.

This experiment shows that the performance of the model depends on two things : 1. The training set: model trained with sample 2 has the best overall accuracy whereas the one trained with sample 3 has a very poor predictive power. 2. The test set: all models tested on sample 2 and 3 reach poor accuracy.

To smooth these results and hopefully yield better accuracy, the next experiment will consider the concatenation of all 4 training sets and validation sets. As performed in the previous experiment, the model will then be tested on each separate sample. The results of the experiment are shown in Table 3.

The average performance on all samples is better than the first experiment. The model manages, almost all the time, to reach a better accuracy per sample. The model reaches good performances for sample 1 and sample 4 (almost 90% for sample 1). However, for sample 2 the false positive rate is very high compared with other samples. The same thing can be seen in the false-negative rate for sample 3. This leads to an average accuracy of 75% which remains a good result.

All these experiments show that it is possible to detect with good confidence a price manipulation two seconds before it ends. Unfortunately, there are still scenarios where the model still struggles to predict price manipulations. When testing on sample 2, the False Positive Rate is still too large (68%) and may lead to many false alarm triggers. In scenario 3, the False Negative Rate is also quite high, leaving investors exposed to manipulations.