QLAMMP: A Q-Learning Agent for Optimizing Fees on Automated Market Making Protocols

• •

  Uniswap: The now popular, Uniswap protocol, introduced in 2018, had only one pair of tokens per smart contract, and both tokens had the same value. Under these assumptions, the protocol maintained a constant product invariant to facilitate exchanges on the Ethereum Blockchain [10].

• •

  Curve Finance: The Curve Finance protocol, formerly called StableSwap, had liquidity pools consisting of two or more assets with the same peg, for example, USDC and DAI. This approach allows Curve to use more efficient algorithms and feature the lowest levels of fees, slippage, and impermanent loss of any DEX on the Ethereum Blockchain [11].

Our system consists of a central AMM protocol that acts as a counter-party to all swaps. The system also has users who request swaps. The users’ decision to swap is based on modeled demand based on the value they get for their tokens. These users query the protocol to check the bid/ask price and the expected gain/loss on the desired swap. The swaps are modeled as a queue, with the front being serviced first. The AMM protocol aims to maximize its collected fee within the system.

We design our proposed Q-Learning Agent to optimize for a hybrid automated market-making protocol. This protocol is inspired mainly by the whitepaper for Curve Finance, primarily designed to facilitate stablecoin trades. The protocol’s underlying exchange function can dynamically shift from a constant-product invariant to a constant-sum invariant. This shift is accomplished by interpolating between the two invariants, controlled by a Leverage Coefficient, $\mathcal{A}$. When $\mathcal{A}$ → 0, the exchange function emulates a constant-product one (like Uniswap); when $\mathcal{A}$ → +$\infty$, the exchange function is essentially a constant-sum one. This behavior is captured by the following invariant proposed in [11]:

Our software implementation of the protocol is based on the implementation by the recent survey on AMMs [2]. We modify their implementation by adding fee mechanics to the Curve class. For each successful swap, the protocol collects a set percent of fee, $\mathcal{R}_{f}$, from the input tokens, $\mathcal{T}_{i}$, to get the final tokens, $\mathcal{T}_{f}$.

We define a getLoss() function, allowing users to query the protocol for the current loss, or price imapact, $\varphi$, resulting from slippage and fees. Additionally, we define new accessor methods to make it easier for objects of the user class to interact with the curve protocol.

We have designed our simulation to closely emulate a real user interacting with a financial exchange, making swapping decisions based on empirical factors such as price and slippage and a subjective component - urgency. This modeling is random for each swap, with each parameter of the swap being sampled from carefully chosen random distributions. Assuming there are $\mathcal{N}$ users in the system, we let $U=\{U_{1},U_{2},...,U_{\mathcal{N}}\}$, denote the set of users. Each user, $i$ $(i\in U)$, has a non-negative balance of the two tokens, $\{x_{k},y_{k}\}$ - initialized to 1,000 each, and a number identifying them, $k$.

The environment generates swaps (mechanics discussed in Section VI-C) and assigns them to random users. Upon receiving such a swap order, the user randomly chooses a token index, either 0 or 1. If the balance of the chosen index is less than a given fraction (set to 20% by default) of the other token, indices are inverted. This mechanism prevents the drying up of either token’s balance. Once the swap parameters are finalized, the user queries the protocol for the expected $\varphi$. The swap goes through if the returned $\varphi$ is less than the urgency-modified tolerance. If not, the swap is canceled based on a 40% chance modified by the urgency factor. If the swap is urgent, it is less likely to get canceled and vice-versa. In addition, if the swap is not canceled, the urgency is increased by 1%, and the “holding” swap status is returned to the environment.

A swap object, $\mathcal{E}_{i}$, as described by us, consists of a slippage tolerance value, $s_{i}$, an urgency value, $u_{i}$, and a user identification number, $k$ $(k\in\{1,2,...\ \mathcal{N}\})$. The environment generates a set number of swaps based on the number of users in the system. The environment samples normal distributions for each swap’s tolerance and urgency value. Each of these swaps is randomly assigned to users in the system and pushed into a queue, ready to be executed. If the swap goes through or is canceled by the user, it is popped from the queue.

These swap objects also hold an amount, token index, and a new flag for swaps that were returned with a “holding” status by the user - the object is then pushed back $k$ places (by default $k$ = 10) places in the swap queue and goes through again after the protocol’s state has changed over the ten transactions. If the swap is tried a predetermined number of times (by default set to 15) times and fails with a “holding” status, it is automatically canceled by the environment and popped from the queue.

The swap method finally returns either,

(1) status code 1 → amount; “success”,

(2) status code 0 → 0; “holding”,

(3) status code -1 → -1; “canceled”.

At each decision epoch, $t$, the state space for the system can be characterized by $\mathcal{S}_{t}=(U,\ \mathcal{E}_{t},\ \mathcal{L}_{t},\ \mathcal{A},\ \mathcal{R}_{f})$. The meaning of each variable is as follows.

• •

  $U=\{U_{1},U_{2},...,U_{\mathcal{N}}\}$, denotes the set of users in the system, each user, $i$ $(i\in U)$, has their current non-negative balance of the two tokens, $x_{i},y_{i}$.

• •

  $\mathcal{E}_{t}=current\ swap$ denotes the swap at the head of the swap queue, it consists of a slippage tolerance value, $s_{i}$, an urgency value, $u_{i}$, and a user identification number, $k$ $(k\in\{1,2,...\ \mathcal{N}\})$.

• •

  $\mathcal{L}_{t}=slippage$ denotes the protocol predicted slippage for the given swap, $\mathcal{E}_{t}$. Since this is a continuous - real value, we discretize it.

• •

  $\mathcal{A}=leverage\ coefficient$ that controls the bonding curve’s curvature.

• •

  $\mathcal{R}_{f}=fee\ rate$ denotes the amount of fee collected by the protocol on each swap.

This state space is shared across the aforementioned agents. Upon getting a swap instruction from the environment, the user generates a token index and an amount, which are used to query the protocol for the expected $\varphi$. $\varphi$ consists of two components, (1) the current fee collected, $\mathcal{F}$, and (2) the current slippage. As the agent chooses actions to adapt the fee rate, $\mathcal{R}_{f}$, and the leverage coefficient, $\mathcal{A}$ - it attempts to optimize $\varphi$ to allow for maximum fee collection and minimum trade cancellations.

The reward mechanism is shared across the three agents. We model the system by defining three different reward types that are awarded under different conditions.

We utilize the Q-Learning algorithm to train the agent on the given problem. It is a model-free off-policy RL method where the agent’s goal is to obtain an optimal action-value function $Q_{*}(s,a)$ by interacting with the environment. It maintains a state-action table $Q[S,A]$ called a Q-table containing Q-values for every state-action pair. At the start, Q-values are initialized to all zeroes. Q-learning updates the Q-values using the Temporal Difference method [6].

where $\alpha$ is the learning rate and $\gamma\in[0,1]$. $Q(s_{t},a_{t})$ is the actual Q-value for state-action pair $(s_{t},a_{t})$. The target Q-value for state-action pair $(s_{t},a_{t})$ is {$R_{t+1}+\gamma\max_{a}Q(s_{t+1},a_{t})$} i.e. immediate reward plus discounted Q-value of next state. The table converges using iterative updates for each state-action pair. To efficiently converge the Q-table, an $\epsilon$-greedy approach is used.

The $\epsilon$-greedy approach: At the start of the training, all Q-values are initialized to zero. This uniformity implies that all actions for a state have the same chance to be selected. So to enable the convergence of the Q-table using iterative updates, the exploration-exploitation trade-off is employed. The explorative update for the Q-value of random state-action pair $(s_{t},a_{t})$ is carried out by randomly selecting the action. The exploitative update selects a greedy action $(a_{t})$, with maximum expected rewards, for the state $(s_{t})$ from the Q-table.

So, to converge the Q-table from the initial condition (all Q-values are set to zero), the agent initially prioritises exploration and later it prioritises exploitation. It uses probability $\epsilon$, to choose random action, or it chooses an action from the Q-table using probability $1-\epsilon$. In the beginning, the value of $\epsilon$ is one, and it decays with time, tending to zero as the Q-table converges.

where $\epsilon_{max}\ \&\ \epsilon_{min}$ are the maximum and minimum values for $\epsilon$, predefined by the environment; and $\eta$ is the factor by which $\epsilon$ is reduced every epoch, $\chi$.

We consider a dynamic exchange function as the underlying curve to facilitate trades on our Automated Market Maker (AMM). The AMM is initialized with 20,000 units of the two tokens. We populate the system with 20 users and 400 swaps per epoch. Under normal swap conditions, each user is given 1,000 units of each token at the beginning of the epoch and under high-liquidity swap conditions the users are initialized with 18,000 units of each token. Each simulation is run over 3,000 epochs.

The Q-Learning algorithm requires discrete values of the observation. Therefore the continuous - real state value is discretized by placing it into 1 of 500 buckets over slippage values ranging from -20% to 20% (negative slippage indicates more tokens received than spent). The environment is reset to a random state, along with random fee rate & leverage coefficient values at the beginning of each epoch.

Users are generated as a list of objects, $U$, from the User class. Each user is assigned a balance of 1,000 units of each token when the environment is reset. The users are assigned an identification number - equal to their 0-indexed position in the list. There are two additional member variables that a user object has, namely, slippage tolerance and urgency.

We simulate the three agents individually and observe their performance across the different slippage tolerance modes against their respective baseline performances. We expect the agents to adapt to the user’s demands in a way that maximizes profit by extracting the maximum fee possible while ensuring that canceled transactions are kept at a minimum.

We hypothesize that our agents will excel at different components of the given optimization problem. Agent 1, who controls the rate of fees, is hypothesized to perform better when the fee rate primarily governs the slippage. This case is likely to occur when the amounts being swapped are a small fraction of the total liquidity in the pool - the invariant mostly stays close to the “center” of the curve, where the curve is ‘flatter.’ Hence the slippage is dictated by the fees being levied on the swap. Agent 2, on the other hand, will be more effective at optimizing fees when users make swaps of amounts that represent more significant fractions of the total liquidity in the system - the invariant is pushed closer to the axes, where the curve starts deviating from its constant sum characteristic. This means the behavior of the curve primarily dictates the slippage, and Agent 2 can optimize for this by adjusting the curve’s leverage coefficient. QLAMMP makes actions that change both the fee rate and the leverage coefficient. Thus it is expected to combine the expertise of Agents 1 & 2 and outperform both across the given tasks.

We also expect our proposed agent, QLAMMP, to be able to adapt to time-varied user behavior. To accomplish this, one may modify the users’ slippage tolerance from loose to normal and the other way around – halfway through the epochs. This shift would mean that the agent would adapt to optimize for a particular type of user but would face a completely different user after the halfway mark. We hypothesize that an effective agent would adapt to this change quickly and extract maximum fee by changing its policy.

We assess the agents’ performance against a system of users with normal and loose slippage tolerances, respectively. Row 1 of Fig. 2 shows the agents’ behavior against users with a normal slippage tolerance. Row 2 of Fig. 2 shows the agents’ behavior against users with a loose slippage tolerance. Both conditions corroborate the hypothesis that Agent 1 cannot make a significant difference in total rewards because the price impact, $\varphi$, is primarily governed by the fee rate, $\mathcal{R}_{f}$.

The hypothesis is further concretized by Agent 2’s performance, which is significantly better than the baseline and Agent 1. This boost in rewards can directly be attributed to the control Agent 2 has over the fee rate, $\mathcal{R}_{f}$. QLAMMP performs significantly better than the baseline, and we can ascribe its success to the $\mathcal{R}_{f}$-modifying subset of its action set. The slight improvement of QLAMMP over Agent 2 is likely due to its additional control over the leverage coefficient, $\mathcal{A}$.

We also assess the agents’ performance in conditions where each user swaps a relatively large fraction of tokens compared to the liquidity in the pool. Fig. 3 shows the agents’ performance when these users comprise the environment. Compared to the earlier simulation, the users now make trades of up to $18,000 in initial pools of $20,000. As hypothesized, Agent 1 outperforms the baseline and Agent 2 because as the pool state moves to a point closer to the axes, the price impact, $\varphi$, increases and is governed primarily by the leverage coefficient, $\mathcal{A}$. Agent 2 lacks control over $\mathcal{A}$ and thus cannot adapt sufficiently to the users’ demands. Once again, QLAMMP significantly outperforms the baseline, Agent 1, and Agent 2 due to its larger action space - controlling both $\mathcal{A}$ and $\mathcal{R}_{f}$.

QLAMMP also performs well when user behavior changes from a normal slippage tolerance to a loose slippage tolerance and vice-versa. We observe that QLAMMP can quickly adapt to the new market conditions and update the protocol parameters to extract fees from the users optimally. Fig. 4 shows QLAMMP adapting its policy to suit the new market conditions.