Joint Modelling of Cyber Activities and Physical Context to Improve Prediction of Visitor Behaviors

Knowledge about consumer behavior is critical for retailers to make personalized recommendations in targeted marketing, improving services, or conduct location prediction. The operators of large indoor shopping malls wish to better understand consumer’s behaviors to compete with online retail. Currently, physical retailers primarily gather customer insights by analyzing point-of-sale data. The path a customer took when visiting a mall, how much time they spent at a particular location, or whether they looked for a specific item is information that is typically not available. In contrast, online retailers benefit from rich information about customer activities, including knowledge of Web interaction such as page visits and dwell times. Combined with sales information, such data provides actionable insights that can help retailers improve the online shopping experience of customers. The activities inferred from the data can be exploited to recognize user intent during online shopping. Such an understanding has not been previously explored in physical retail environments.

Malls, museums, galleries, and transport hubs are large heterogeneous environments offering a range of different services: retail, entertainment, information, catering, etc. Increasingly, Wi-Fi networks and Bluetooth$\copyright$ beacons are being introduced into these spaces allowing the logging of movement and information behavior of visitors to such environments. Coupled with an understanding of the functions of the different locations in a space (i.e. physical contexts) one can ground and classify user behaviors or predict future movements. Such an understanding allows the creation and eventual delivery of improved services to visitors.

A person’s behavior within a physical space is represented by heterogeneous data, both cyber and physical. In the context of our study, the cyber domain captures a user’s interest in the form of queries issued. The physical, associations with Wi-Fi Access Points (APs), captures information related to an area of interest to the user. We hypothesize that users with contextual intent exhibit similarity between their physical contexts and their cyber behavior, i.e. users issue queries related to the context of the physical space. Their cyber-physical behavior reflects what they are interested in.

To illustrate: consider user $A$ who intends to buy a laptop and compares products online while in the vicinity of a computer store; user $B$, who enters the mall searching for a particular store and follows a trajectory that ends in the store’s vicinity; and user $C$, who checks an online footwear size chart while in a store selling shoes. User $A$ is interested in computers, user $B$ is interested in a specific store, and user $C$ is interested in footwear. Such interests can be inferred from the physical context and the combined cyber and physical activity of users.

We present an approach to formulate a correlation between user physical and cyber behavior from heterogeneous data i.e. the Web Query Logs (Cyber) and the Wi-Fi AP association logs (Physical) in order to identify users’ interests specific to the physical location. There are number of challenges.

1. (1)

   The Semantic Labeling of a physical space. In a mall, this can be done by assigning the category of shops (e.g. Cosmetics, Footwear, Clothing etc) that are in the range of an AP. However, these categories are broad and may not correlate well with a user query.

2. (2)

   Therefore, we employ Semantic Category Expansion to expand the categories to cover the range of sub categories and products.

3. (3)

   To discover the semantic similarity between queries and a physical space we also create a Contextual Similarity to map a user query to the representation of categories and relevant sub-categories.

The cyber physical contextual similarity shows the users’ interests across different semantic categories related to the physical environment and can be used in various applications that involves understanding of user behavior. In our work, we show the use of cyber-physical similarity features in two different applications: Classification of User Visiting Intent and Future Location Prediction. We hypothesize that contextual similarity is a strong indicator of what a user is interested in. It can be helpful in identifying whether a user exhibits high contextual intent with the physical space or is just browsing the area; and, which places the user will visit next.

For behavior classification, the aim is to identify shoppers with high contextual intent, such as users $A$, $B$, and $C$. There are many visitors to a mall for whom their Web behavior and indoor context are contextually intentless. Consider user $D$, who visits a mall searching the Web for information about a particular festival occurring in the city, and interleaving these searches with queries about “lost luggage” and “baggage claim”. This user is likely a tourist more focused on the free Wi-Fi than the primary services provided by the mall. While such users clearly have an intent, from the point of view of the mall operator, their visit can be classed as intentless. We also place in this category ‘window shoppers’, or shoppers with a high-level shopping intent (e.g., ‘I need to get a present for my brother’) that cannot be tied to a particular retailer or category of retailers. While all visitors are potentially of great interest to indoor retailers, we focus our work on detecting contextually intentful customers.

Previous intent recognition relied on examining either physical behavior from Wi-Fi signals, mobile phone sensors, mobile proximity sensors (Martella et al., 2016; You et al., 2014; Zeng et al., 2015), or exploiting cyber behavior from online Web browsing and searching logs (Moe, 2003). To the best of our knowledge, this is the first time a user’s contextual intent in an indoor space is inferred from both physical and cyber behavior.

We also employed user’s cyber-physical semantic similarity for future location prediction. Past work (Noulas et al., 2012) studied the effect of different features on such prediction exploiting Location Based Social Network data. The researchers reported that category of location visited by a user has high impact on prediction accuracy. However, the same is not studied for an indoor setup where movements of a user are captured by Wi-Fi traces. Therefore, we experimented to see if a user’s future locations can be predicted accurately by using the semantics of indoor locations visited by the user and query context.

The main contributions of this work are:

• •

  Semantic Categorization used to semantically label a physical space and find the correlation between open text queries and physical semantics;

• •

  A Cyber-Physical Contextual Similarity model, used to extract contextual features, including Physical and Cyber activities captured by Wi-Fi AP associations and Web Query logs;

• •

  A shopping intent recognition system for user intent recognition, used to classify two broad categories: intentful or intentless.

• •

  An evaluation on the effect of Semantic Context on Future Location Prediction.

We categorize our description of past work into five areas. As mentioned earlier, our main goal is to find cyber-physical semantic similarity from user’s cyber behavior captured by Web logs and physical context represented by semantic categories. We then show the application of this similarity in two different applications, User behavior classification and Future Location Prediction.

Semantic Labeling of Contexts: Context is an influential factor in analyzing both human behaviors (Pejovic et al., 2015) and user intent from mobile information access (Church and Smyth, 2009). Context is defined as ’any information that can be used to characterize the situation of an entity, where an entity can be a person, place, or physical or computational object.’ (Abowd et al., 1999). Semantic labelling of a location context is an important step to identify intent. Krumm and Rouhana proposed Placer, which treats semantic labelling as a classification problem based on time, user demographics, and nearby businesses (Krumm and Rouhana, 2013). They found that the demographic information and nearby businesses was helpful in semantic labelling of places, e.g. school, home, and work. Later, they proposed an advanced version, called Placer++, which utilizes two more features, the labelled visitors from others and the visit sequences, and found higher accuracy was achieved with these two new features (Krumm et al., 2015). Elhamshary et. al. proposed CheckInside, a fine-grained indoor location-based social network, which utilized check-in data collected from crowd source workers to associate a location with its name and semantic fingerprint. The researchers claimed CheckInside provides more accurate localization and better coverage (Elhamshary and Youssef, 2014).

Indoor Behavior Analysis: To support real-world, mobile-centric behavioral research, Misra and Balan presented LiveLabs, which is a large-scale mobile testbed for in-situ experimentation (Misra and Balan, 2013). They also investigated user behaviors when considering whether users are in a group or alone. The researchers found people’s mobility patterns, app usage, and propensity to communicate over phones are significantly different across these two scenarios (Jayarajah et al., 2015). Martella et. al. (Martella et al., 2016) studied the relationship between indoor visitors and the objects in the case of museum exhibition. Specifically, they deployed energy-efficient mobile proximity sensors to measure the face-to-face proximity between people and objects, and achieved high accuracy of identifying which exhibit a user is facing at short distance (Martella et al., 2016).

Shopping Behavior Recognition: Zeng et. al. (Zeng et al., 2015) studied how to determine a shopper’s physical behaviors based on channel state information of Wi-Fi signals. They focused on behaviors near shop entrances or within a store, The researchers found the channel information of Wi-Fi signals were a good source to classify these different physical behaviors (Zeng et al., 2015). Radhakrishnan et. al. presented how to use a smartphone and a smartwatch to segment fine-grained user shopping behaviors: e.g. putting an item in the cart (Radhakrishnan et al., 2016). Ren et. al. analyzed how people use Wi-Fi to access the Web in indoor retail spaces while navigating through a mall. They found temporal patterns in shoppers’ visits and determined that physical context influences user’s cyber behaviour (Ren et al., 2017b). Based on these findings, Ren et. al. developed a tripartite location-query-browse graph for contextual recommendations of query, Web content and location, inferred from searching, accessing, and moving behaviors (Ren et al., 2018b). Using only such behaviours inferred from Wi-Fi logs, Ren et. al. also found strong correlations between behaviours and user demography (e.g. age, gender, income, parental status, visitor types) (Ren et al., 2018a). The work in this paper will build on the contributions from the previous works by Ren et al. (Ren et al., 2017b; Ren et al., 2017a; Ren et al., 2018a; Ren et al., 2018b) and extends on our previous work on modelling cyber-physical contextual similarity (Kaur et al., 2018).

User Intent Recognition: Jansen et. al. studied the user intent of Web queries focusing on determining the informational, navigational, and transactional intents (Jansen et al., 2008). Other work investigated intent based on diary studies by focusing on user mobile information needs (Church and Smyth, 2009). The researchers suggested two additional intents: geographical and personal information management. Chuang-Wen You et. al. proposed a phone-based system to monitor shopping time in stores classifying user trajectories as either shopping or non-shopping. The researchers utilized spatial and temporal features extracted from both Wi-Fi signals and the accelerometer and digital compass of a phone (You et al., 2011). Duan and Zhai studied the intent representation problem in the field of entity search, e.g. product retrieval. They proposed a coordinated intent representation by linking the query and entity space collectively. The researcher’s focus was on utilizing query terms and product attributes (Duan and Zhai, 2015). Little research reveals whether shopping intent is detectable in user movement (physical) and query (cyber) behaviors.

Location Prediction: Exploiting user check-in data from location based social networks to predict/future check-in locations is a well studied topic (Noulas et al., 2012). Features used included information on types of places, mobility flows between venues, and spatio-temporal characteristics of user check-in patterns. The work proposed a supervised prediction model, based on linear regression and an M5 model tree. Another study predicted locations, stay duration, and contact from Wi-Fi and Bluetooth traces (Vu et al., 2011). For location prediction, the authors use Wi-Fi traces and cluster AP information by exploiting regularity of movement. The study was based on traces collected from fifty participants. The former study used the characteristics of venues in terms of categories, the latter did not. Recent work that introduced continuous trajectory prediction problem (Sadri et al., 2018) also presented a solution based on information-gain to segment multivariate temporal sensor data (Sadri et al., 2017). The study focused on predicting the continuation of user movements, i.e. trajectories, including the sequence of location and departure times for the remainder of the day. Also introduced were two types of trajectories: geographical and labelled.

Check-in prediction: Noulas et. al. (Noulas et al., 2012) analyzed various features from user social network check-in data and found that the types of places users tend to visit (cinema, nightclub, coffee shops etc.) can be highly informative about user mobility preferences. In Location Based Social Networks, venues that user checks in to are labeled with well defined categories, which is not the case with the physical location. Users’ physical movements are captured by Wi-Fi AP connections. A location prediction model using Wi-Fi traces was presented in (Vu et al., 2011). We hypothesize that Physical Context (i.e.semantic categories assigned to Wi-Fi APs in a physical location) and Cyber Context (i.e user queries context in an indoor space such as shopping centers or museums) can further enhance prediction results as found in studies on check-ins prediction.

Gaps addressed by this research: In our study, we use trajectories, location traces from Wi-Fi APs, labelled with semantic categories of the surroundings. Context information from a user’s web query logs can be used to predict future locations. Our evaluation results in successful prediction of less popular locations with higher accuracy then using a model entirely based on mobility flows between locations.

We define the following:

In order to identify user intentions, we need to find if their physical trajectories and cyber activities correlate with the physical surroundings. The main challenge is to map open text queries to a small set of category terms that have little or no lexical similarity with each other. For example, a user query may be product name or brand (e.g. Mascara and Ugg Shoes) which are not in the mall-defined shop category list.

Hence, we propose a system that uses structured information to find intent signals from user queries with respect to physical context, by extending the text of both queries and categories. We gather additional information from DBPedia concepts (Auer et al., 2007), extracting categories related to each concept. The extended content representations are then compared.

We first describe preliminaries, then approach the first task i.e Modelling Physical Space using extended categorical information as an Enrichment of Semantic Categories problem. The second task, query extension, we consider it as an Entity Search problem where given a query we try to identify Wikipedia concepts from query text and generate a document of categories related to each entity identified.At last in Section 4.4, we compare category document generated in Step 2 with each category document in step 1 to generate a vector of similarities to physical context signaling user interests.

To review, given an AL, QL, Shop Categories, and Voronoi cells we create an Intent Recognition Model as shown in Figure 12. The first step is to enrich shop categories, provided by the mall operator, with categories from DBPedia. With such enriched categories (stored as documents), we then label each AP with the categories based on the shops within range of the AP’s Voronoi cell. After semantic labeling, we determine Physical Activity (trajectories) and Cyber Context for user sessions as recorded in the logs. We then calculate a Cyber-Physical Contextual similarity, which allows us to form Contextual Similarity features that act as input to our Intent Recognition Model along with Physical and Cyber features derived from the AL and QL.

We focus our experiments on a subset of complete user trajectories with associated user queries. A complete trajectory is one where the start and end points correspond to entry/exit points of the mall. Such trajectories must connect at least three APs. Out of $6784$ total trajectories, we identified 176 that are complete. Four annotators, without in-depth knowledge of the experiment, manually categorized the trajectories into intentful (48) and intentless (128), with 100% inter-annotator agreement. The annotators inspected the queries and marked them as relevant if the content was deemed related to the environment of the shopping centre. A session was labelled as intentful if at least one of the queries was relevant, and intentless otherwise.

To evaluate the classification models we used: $Accuracy\%$, the percent of correct classifications; $Precision$, the percent of correct positive classifications; $Recall$, the percent of positive instances correctly classified; and $F-Score$, a weighted harmonic mean of Precision and Recall.

To evaluate location prediction, we used: Accuracy@k, the number of correct locations predicted over $k$, which is the total no. of locations predicted; and MRR, the ranking of first correct location, $MRR=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{R_{i}}$, where $n$ is the no. of prediction results and $R_{i}$ is the rank of first correct predicted location for trajectory $i$.

For behavior recognition, we first performed an experiment to classify users’ shopping intent as intentful and intentless, by using users’ Cyber-Physical-Contextual activities captured by a Wi-Fi APs association log and a Web query log. We proposed a Shopping Intent Recognition System, which includes Semantic Categorization to semantically label a physical space and find the correlation between open text queries with the physical semantics. We also described Cyber-Physical Contextual Similarity model to extract contextual features including Physical and Cyber activities captured by Wi-Fi AP and Web Query logs. Finally, we detailed a User intent recognition system to classify a user’s intent.

We showed that the proposed contextual features significantly improved the accuracy of intent recognition models where we used Decision Tables, Naïve Bayes and a Decision Table Naïve Bayes hybrid model. The models were applied to a set of 176 real user trajectory sessions in an indoor shopping mall. The DTNB achieved the best performance, compared to DT and NB over all feature sets. The maximum classification accuracy of 81.25% was achieved by using DTNB on all feature sets (Cyber-Physical-Contextual). We also note that in the entire query log dataset for the target environment, only 8% of the queries belong to the broad shopping semantic categories from Table 1. If this set of queries was larger, the similarity detected may not be conclusive. But because this category of queries represents a small subset of the overall query activity of indoor mall users, the high level of similarity detected between the semantics of the physical context and those of the query activity are strong indicators of intentful activity.

We further performed an experiment to study the effect of Cyber-Physical Contextual Similarity on location prediction using Collaborative Filtering. Using contextual features as a weight to the probability likelihood calculated from collaborative filtering model improves the accuracy of prediction of less popular locations.

Limitations. This paper has largely focused on the semantic expansion of spatial features, but more can be done on embedding temporal features. The data evidently reveals that some trajectory behaviours are specific to certain visiting intent (e.g. going for a lunch) on different temporal contexts, such as shown on Figure 17, where there are direct movements between the first floor to fifth floor during noontime and lunch break period. The fifth floor is where the food court is located. If we also extract the temporal features and expand the semantic representation, this could potentially boost the predictability of the intent.

Further, these features could also be used further for profiling users (Ren et al., 2018a), useful for providing a personalised model to predict the visiting intent and next location indoors.

In addition, sequential or continuous behaviours of the whole trajectory in a session is not yet incorporated in the CPS model of this paper. From our earlier paper, we have observed repeating and habitual behaviours of returning visitors are observed across the history (Ren et al., 2017b), which could be used for continuous trajetory prediction (Sadri et al., 2018) for example, or intelligent notification, shopping assistant, or recommendation (Ren et al., 2018b) purposes.

Finally, a complete semantic expansion and embedding of cyber, physical, social behaviours can be done in the future. Recent work on graph embedding, especially node and relation embedding (Wang et al., 2020) can be used to deal with the data sparsity and cold start problems in this data (or similar datasets) when high-dimensional features are combined across multiple sources or domains. Such an embedding could also generate more effective recommendation results.

We proposed a semantic enrichment and contextual similarity model that deals with the major challenge of mapping semantic similarity across two different domains: cyber and physical behaviours. We show that using this combined contextual similarity further improves the accuracy of both intent recognition and location prediction with respect to the use of cyber/physical features in isolation.

There are some limitations to our work that can be improved in the future. Firstly, we performed the Shopping Intent Recognition on a small dataset, as manual labeling of trajectories and the respective query sets was required. Future experiments with crowdsourced labelling of much larger datasets are envisaged. Secondly, we only studied the effect of contextual features on location prediction. Future works should investigate the effect of other features, such as time spent at an AP captured through other distance metrics (e.g., cosine similarity or Pearson correlation). Finally, as we characterize users based on their detailed cyber activities and physical contexts, the construction of group profiles based on demography and visiting patterns will be further investigated.

We acknowledge the support of Australian Research Council (ARC) Linkage Project LP120200413. Flora Salim would also like to acknowledge the support of Alexander von Humboldt Foundation. We also want to thank the referees of the previous versions of this paper for their useful suggestions.