Analysis of E-commerce Ranking Signals
via Signal Temporal Logic

We begin with properties that predicate over the initial days of products. We are interested in determining if products are ranked at a stable position (flat start) or if they gain (cold start) or loose (warm start) positions in the days after launch:

where $w\in\mathbb{N}$ and $\epsilon\in\mathbb{R}_{\_}{\geq 0}$ are tunable parameters that define the length of the initial time window and noise tolerance.

The $flat\_start$ specification (Eq. 3) forces the first derivative $x^{\prime}$ to be always $\epsilon$-close to zero (we include the $\epsilon$ tolerance to account for noise). The ranking signals that satisfy this specification are those that found their steady ranking position on launch day, i.e., those that experienced a flat start. Similarly, we define $cold\_start$ (Eq. 4) and $warm\_start$ (Eq. 5) specifications by requiring the ranking signals to always decrease/increase. The right-most conjunct ($F_{\_}{[0,w]}$) forces the signals to strictly grow/decrease at least once.

Fig. 1 shows some examples of trajectories that do and do not satisfy (green and red, respectively) the flat and cold start STL specifications with parameters $w=3$ and $\epsilon=1$ for $flat\_start$ and $w=3$ and $\epsilon=0$ for $cold\_start$.

We now increase the scope of our specifications by predicating over entire ranking signals.

In learning to rank, we often assume that a document reaches a steady state ranking position after an initial period during which behavioral data is collected. The following STL formula characterizes the existence of a steady state:

where $w\in\mathbb{R}_{\_}{\geq 0}$ defines the maximum stabilization time and $\epsilon\in\mathbb{R}_{\_}{\geq 0}$ the tolerance to noise. The $steady\_state$ formula holds if there is a day in $[0,w]$ after which $|x^{\prime}|$ is always smaller than $\epsilon$, i.e., the trajectory maintains a steady state with $\epsilon$ tolerance. Fig. 1(c) shows two trajectories evaluated on $steady\_state$ with parameters $w=3$ and $\epsilon=1$. The green signal satisfies the specification because it stabilizes after day $3$ at position $2$. On the other hand, the red trajectory does not satisfy the specification because on day $4$ it experiences a drop in positions and thus it is not stable after day $3$.

Next, we define a liveness property that checks if a product that reaches a certain ranking interval eventually hits a critical position:

with $s,r\in\mathbb{R}$. For instance, with parameters $s=10$ and $r=1$ we check whether products that reached the top $10$ positions eventually rank in first position too. Fig. 2(a) depicts two signals that satisfy the $reach$ specification and one that does not with parameters $s=10$ and $r=1$. The upper green signal satisfies the specification because it enters the $[1,10]$ ranking interval and eventually reaches position $1$ on day $12$. The lower green signals also satisfies the requirement since it never enters the $[1,10]$ range. However, the red signal does not satisfy the specification because its values become smaller than $10$ but never reach the first position.

Finally, we analyze the stability of products. We define two specifications that capture signals with large bouncing drops or rises in a short time frame:

The parameters $d,w\in\mathbb{R}_{\_}{>0}$ determine the signal drop/rise width and amplitude, respectively. These two formulas check if at any point in time there is a drop/rise of at least $d$ positions followed by a rise/drop of at least $d$ position within $w$ days. Fig. 2(b) shows some signals evaluated on $ditch$ with parameters $d=10$ and $w=2$. The green signal satisfies the specification since it experiences a ditch of at least $10$ positions on day $3$. The red signal does not satisfy the property since its ditch is not deep enough.

Finally, we hypothesize a scenario where some data might miss, i.e., the ranking position might be unknown. Let $-1$ denote the ranking position of a product on a day for which data is missing. The atomic predicate that holds if the ranking position is unknown is $miss:=x=-1$.

In learning to rank systems, the first days after a product launch are crucial for the collection of behavioral data and the subsequent correct ranking. Hence, we do not want too much missing data in the days after a product launch:

where $w\in\mathbb{N}$ defines the initial time window. $no\_init\_miss$ ensures that there are no $w$ consecutive initial days of missing data. We can also extend this requirement to any part of a ranking signal and not just to its prefix. We define a specification that ensures that are no windows with too many consecutive days with missing data:

where $w\in\mathbb{N}$. $no\_long\_miss$ ensures that if there is a missing data day, then eventually within $w$ days there will be non missing data day. In Fig. 2(c), the red signal does not satisfy $no\_long\_miss$ for $w=3$, since after day $4$ there are $4$ consecutive days of missing data. On the other hand, the green signal satisfies the specification since it never has $3$ consecutive days of missing data.

Before proceeding with the evaluation of our STL specifications, it is worth noticing how STL is both succinct and efficient when compared with standard logics.

For instance, we ran the clustering algorithm k-means (with $k$=10) on our dataset of product signals with the goal of mining the most representative signals. Fig. 3 shows the centroid signals of the clusters obtained by running k-means (with $k$=10) on product ranking signals dataset. From this analysis it seems most products have smooth trajectories. However, as we will later discover in Sec. 4 , our STL-based analysis reveals that 50% of all the analyzed products in this test experience a shift in ranking position over two consecutive days. This shows how clustering techniques might fail in isolating important signal patterns.

We also compared STL with classic logics and query formalisms. For instance, the translation of the $ditch$ property (Eq. 8) in propositional logics is $\bigvee_{\_}{i=0}^{T}(xgd_{\_}i\wedge\bigvee_{\_}{j=0}^{w}xld_{\_}j)$ where $xgd_{\_}i:=x^{\prime}(i)>d$ and $xld_{\_}i:=x^{\prime}(i)<d$. For this formulation, we must know in advance the length $T$ of the signal. In addition, it involves $T(1+w)$ operators against the $3$ operators of Eq. 8.

The encoding of Eq. 8 in a first-order logic, e.g., semi-algebraic logics, would result in a formula with less operators $\exists t(x^{\prime}(t)>d\wedge\exists t^{\prime}(t\leq t^{\prime}\leq t+w\wedge x^{\prime}(t^{\prime})<d))$ which is still less compact than Eq. 8 and less efficient to evaluate. The evaluation of semi-algebraic formulas is generally doubly exponential in the number of quantifier alternations while the evaluation of an STL formula is linear in the signal length [7].

For completeness, we also include the translation of Eq. 8 into a pandas [16] query, a Python library for data manipulation and analysis. Let $df$ be the dataframe with our signals where $pos_{\_}i$ is the column with the ranking position on day $i$. The pandas query that isolates the signals that satisfy the $ditch$ property is $df[((df.pos_{\_}0>d)\ \&\ (df.pos_{\_}1<d\ |\ \dots\ |\ df.pos_{\_}w<d))\ |\dots|\ ((df.pos_{\_}{T-w}>d)\ \&\ (df.pos_{\_}{T-w+1}$ $<d\ |\ \dots\ |\ df.pos_{\_}{T}<d))]\$. The structure of this query is similar to the propositional encoding. Also in this case, we are dealing with a long query, we need to know in advance the length of the signal, and any parametric change of our requirement affects the query’s structure. Finally, note that the encoding formula length explosion occurs for any specification that involves temporal operators and not just this particular case.

Do satisfaction rates of our STL specifications vary across categories? To address this question we compute the percentage of product signals that satisfies a given STL specification for each product category and STL specification combination. Results, reported in Fig. 4, highlight how satisfaction rates are not equally distributed across different categories.

c9 and c4 are the categories less affected by cold start with a 2% satisfaction rate opposed to c8, c7, and c5 with satisfaction rates 5%. c4 and c9 have also the highest flat start and steady state rates (5% and 8%, respectively) in contrast to the other categories. This means that c4 and c9 are the categories whose products most frequently find their natural ranking position from launch day and keep it constant over time.

Interestingly, c9, c4, together with c0, are the most affected by warm start (1.50%) suggesting that products from these categories might rank too high after launch or might include a high number of low engagements products (e.g., spam).

Finally, spike and ditch satisfaction rates are almost equally distributed. Remarkably, 60% of all the analyzed products experience a drop or ditch of at least 10 ranking position within two consecutive days.

We now analyze how impressions, clicks, and purchases are distributed across different specifications. We compute the average number of impressions, clicks, and purchases for query-documents tuples that satisfy a given STL specification. Fig. 4 shows the obtained distributions.

The signals within this test that garner the highest and lowest average number of impressions (400 and 250, respectively) are those that satisfy the warm start and cold start specifications, respectively. The most clicked products satisfy flat start and steady start (30 clicks) while the least clicked ones satisfy warm start, reach, ditch, and spike (5 clicks). The products that collect highest purchases satisfy flat start and steady state (90) followed by cold start (35), while the products with lowest purchases are those affected by warm start (5).

This analysis suggests that products affected by warm start, despite receiving the highest number of impressions, tend not to be clicked and consequently gather low purchases. We could speculate that warm start products are either not relevant to customer’s searches or are low quality products that do not attract customer’s attention. cold start products show the symmetric phenomenon. They receive a low number of impressions, they are clicked twice as much as warm start and gather 5x purchases compared to warm start. This might mean that cold start products are eventually discovered and purchased by customer despite being initially poorly ranked.

Outliers are flat start and steady state specifications. They receive 3x clicks (30 vs 10) and 4x purchases (100 vs 25) compared to the second mostly clicked and highest purchases cold start, no init miss, and no long miss. Products that have early flat and steady positions tend to rank very high and consequently collect the highest amount of clicks and purchases.

Note how both the product categories and performance metrics STL analyses revealed interesting behaviors of our learning to rank model. The obtained insights can be used, for instance, to rebalance our training sets by focusing on particular product segments or redesign relevance scores. Note also that these are just demonstrative examples of how STL can be used to reason over ranking signals. Nothing prevents STL from being applied to more complex temporal properties or more sophisticated analyses.