Privacy-Preserving Personalized Fitness Recommender System $(P^{3}FitRec)$: A Multi-level Deep Learning Approach

With the advent of artificial intelligence, many innovative enterprises have provided various personalized products and services to individuals. Personalization is generally made possible by mining multidimensional data from individuals so that the products and services can be custom-made to people’s inclinations. In particular, the research of personalization on improving individuals’ well-being has attracted a lot of attention. For instance, smartphones and smartwatches can sense the dynamic changes of users, such as heart rate, blood pressure, sleep patterns, etc., to realize real-time and non-invasive monitoring of human health [16]. A study found that smartwatches can effectively monitor the instantaneous changes in the user’s heart rate, and can accurately help diagnose atrial fibrillation associated with ischemic stroke (asymptomatic or paroxysmal) [25]. On the other hand, other products focus on providing personalized suggestions to users, such as customizing exercise plans for users, understanding their reactions to the plans, and analyzing exercise results to continuously improve user experiences. For example, a framework named PRO-Fit (Personalized Recommender and Organizer Fitness assistant) was introduced, which uses collaborative filtering on user profile and activity data to generate personalized fitness schedules according to user availability and fitness goals [11]. Traditionally, people can achieve comparative goals by hiring a human trainer, but machine learning-based applications have certain advantages in terms of quality and affordability. More specifically, machine learning algorithms can integrate various information sources from thousands or even millions of users to develop products or services, which is far beyond the knowledge range of a human trainer. In addition, recommender systems as software products usually have a lower marginal cost to provide services to new users. However, despite the convenience brought by technology, societies are paying more attention to the security and privacy issues in data mining and prediction. In 2018, the European Union issued General Data Protection Regulation (GDPR) to regulate the protection of citizens’ personal data and privacy. In this sense, Tan et al. [31] and Bouhenguel et al. [7] discussed the Bluetooth security threats of wearable devices, and share their knowledge and insights on how to prevent devices and the networks they are connected to from being attacked. Other researchers such as Cyr et al. [10] analyzed the potential security problems of Fitbit devices when collecting and utilizing users’ data, such as unnecessarily collecting information from nearby devices, or withholding all collected data to the device owners. Unfortunately, most service or product providers are inherently motivated to collect as much data as possible, especially personal data, from users to cultivate machine or deep learning models to enhance user experience.

Although most of the existing work in the literature focuses on building accurate artificial intelligence models to make personalized fitness recommendations, little consideration is given to protecting the privacy of users. For instance, Dharia et al. [11] proposed a PRO-Fit framework to proactively push notifications recommending fitness-related activities to users, which is based on their multivariate data, including their fitness preferences, calendar data, and social network data. Although the model has achieved good performance, the framework inevitably collects an intensive amount of sensitive data from users. Likewise, the “TweetFit” framework which was proposed in [13] profiles people’s wellness by taking advantage of data from sensors such as speed or heart rate measurements during exercises and multiple social media sources such as Twitter tweets and Instagram image captions and comments. On the other hand, with specific attention to GDPR compliance, Sanchez et al. [26] presented a fitness data privacy model that learns people’s privacy preferences for fitness data sharing and processing collected by the Internet of things (IoT) devices. They studied user privacy permission settings of fitness trackers such as Fitbit and smartwatches with the supplement of a questionnaire to learn users’ privacy profiles by applying machine learning modeling. They further developed a rule-based personal data manager (PDM) framework to provide privacy advice to users based on their machine learning models. However, this work mainly focuses on the general privacy settings of IoT devices, not their product functions. Loepp et al.[21] proposed a prototypical smartphone app that recommends personalized running routes based on analyzing multivariate data of the workout routes users have run, e.g., length of the route, uniqueness of the route, the shape of the route, the light of the route, and so on. The framework makes personalized recommendations based on running routes following a rule-based approach and users’ preferences which were manually set by users. Compared with the rule-based approach, machine or deep learning-based recommender systems have the advantage of learning user attributes intelligently and require minimum manual input from the users.

In this paper, we address the above challenges by proposing a novel privacy-aware personalized fitness recommender system. In our proposed approach, we introduce a multi-level deep learning framework that learns important features from a large-scale real fitness dataset collected from wearable IoT devices to derive intelligent fitness recommendations. Unlike most existing approaches, our approach achieves personalization by inferring the fitness characteristics of users from sensory data and thus minimizing the need for explicitly collecting user identity or biometric information, such as name, age, height, and weight.

Our proposed approach consists of two key components. We first build a model that learns user embedding and workout route embedding from a real-world Fitbit dataset. The user and workout route embeddings are then fed as input features to our proposed deep learning models to create personalized recommendations. Second, we develop a workout profile prediction model that suggests personalized workout recommendations that can guide the user based on their choice of workout distance, route and speed sequence, sport type, and target calories to consume. The goal of our exercise distance prediction is to provide personalized guidance for users to achieve the target exercise calorie. Similarly, our predicted speed sequence aims to guide the user to adjust the exercise speed according to the conditions of the selected target exercise calories and the selected exercise route. Meanwhile, our predicted heart rate sequence aims to provide the user with an important indicator of the expected health status in the upcoming exercise.

In our approach, we also propose a three-dimensional tensor of ”users – workout routes – contextual features” based on the historical workout data. The goal of this tensor is to capture the underlying structures inherited in users and workout routes using the Tensor Decomposition method CANDECOMP/PARAFAC (CP) [20]. The two resultant matrices related to the latent characteristics of the users and workout routes are then combined with other contextual features (choice of sport, target calories, etc.) as input to two models. The first model is a Multi-Layer Perceptron (MLP) which is used to predict the total distance of a future workout. The second model is a Long Short-Term Memory (LSTM) which is used to predict the speed sequence and heart rate sequence of a future workout.

The main contributions of our proposed approach are threefold:

1. 1.

   An approach to building privacy-aware personalized fitness recommendation systems, by inferring the fitness characteristics of users from sensory data instead of collecting multidimensional private data from users. This is complementary to the personalized fitness recommendation systems that do not consider privacy preservation, and the privacy preservation approaches running on independent encryption protocols.

2. 2.

   A model that learns user embedding and workout route embedding from real workout datasets collected from Fitbit devices. This includes gender, sport type, calories, workout duration, workout distance, workout speed, workout heart rate, and workout route geographical data. Our user and workout route embeddings are further utilized as the input features for our proposed privacy-aware and personalized fitness recommender.

3. 3.

   A workout profile prediction model that suggests personalized recommendation of the necessary workout distance, the rhythm of speed, the change of heart rate of a future exercise based on the user’s choice of workout route, sport type, and target calories to consume.

The rest of this paper is structured as follows. In section 2, we present background information relevant to the topics of our proposed approach. We also, discuss various related studies in the literature and practice. The details of our proposed approach, including our methods and algorithms, are introduced in section 3. The experimental evaluation and result analysis are discussed in section 4. In section 5, we discuss the results in the context of our research goals and contributions. We draw key conclusions and future work in section 6.

In this section, we first briefly review the technologies for sequential data modeling and technologies used in building recommender systems. Then we discuss related studies on different fitness recommendation topics, e.g., Fitness Activity Detection and Recommendation, Sequential Fitness Profile Recommendation, and Privacy Preservation in Fitness Recommendation.

Recurrent Neural Networks (RNNs). In recent years, RNNs and particularly LSTM networks have been widely used in processing time-series data for sequential modeling tasks, for example, speech recognition [15], machine translation [30], image captioning [9], etc. Ilya Sutskever et al. proposed an end-to-end approach to address sequence to sequence mapping problem by constructing a multi-layer LSTM network, which maps the input sequence to a fixed-dimensional vector through LSTM, and then uses another LSTM to output the target sequence [30]. They put forward several innovative techniques to improve the model, such as using stacked LSTM structure and reversing input vectors, etc. In English-French translation tasks, their model shows better performance over single forward LSTM models. Moreover, Bi-directional LSTM is an extension of traditional LSTMs that improves model performance on many sequential modeling tasks. This is because a unidirectional LSTM network only propagates information from past to future, while a Bi-directional LSTM network manages inputs in two ways, one from past to future and one from future to past [28]. Therefore, it consolidates the context of each step in the input sequence. Similarly, in our work, we consider a Bi-directional stacked LSTM architecture to solve the sequence-to-sequence modeling task.

Technologies of Recommender Systems. Recommender systems generally follow two basic approaches: collaborative filtering or content-filtering [1]. Collaborative filtering holds the belief that people would continue the same experience in the future if they liked something in the past. The K-nearest Neighboring (KNN) method is often used to be the most favored approach to conducting collaborative filtering, which conducts finding similar users’ profiles to one certain user to compute likeness and dislikes for an item [27]. In contrast, content-based filtering methods are useful in situations where user information is sufficient but not item information. It works as a classifier to model the likes and dislikes of the users to evaluate an item [27]. In collaborative filtering-based recommender systems, a two-dimensional data matrix associated with users and items is usually constructed. The 2-D matrix can be factorized into two matrices, namely a user matrix and an item matrix that contain the latent characteristics describing the user preferences and item profiles [17]. One disadvantage of the matrix factorization approach is that the context is only a scalar, representing the user’s rating of the item. To overcome this shortcoming, Karatzoglou et al. discussed a multidimensional equivalent method to the 2-D matrix factorization approach named Tensor Decomposition [17]. Contrary to the single rating feature in the matrix decomposition method, a multidimensional contextual feature vector is established between each user and item. This method allows flexible integration of context information when learning entity embeddings to provide context-aware recommendations. In our work, we consider establishing a rich context feature vector between each user and workout route to derive user embeddings and workout route embeddings.

Fitness Activity Detection & Recommendation. Guo et al. proposed FitCoach, which is a virtual fitness coach built upon data sensed by IoT devices [14]. It aims at detecting people’s workout statistics such as exercise types with a lightweight support vector machine (SVM) classifier and providing fine-grained feedback on exercise form scores, i.e., motion strength and performing period, to assist users to maintain proper exercise postures and avoid injuries. Similarly, Zhao et al. introduced a fitness recommender system designed to generate personalized and gamified content to promote daily physical activities [36]. They collected various types of user data and built separate sub-models for user profile prediction with a non-machine learning approach. The sub-models work individually, but their results are jointly input into a decision tree-based recommendation engine to create personalized recommendations, such as extending an existing exercise or suggesting a different type of activity. Yong et al. proposed an IoT-based intelligent fitness system that monitors people’s health with data collected by IoT devices, recognizes people’s actions using a convolutional neural network (CNN) based model, and provides fitness-related recommendations, such as reminding users to attend fitness courses or going to gyms based on user predefined exercise plans [34]. They explicitly collected users’ scores on the exercise items to build a collaborative filtering based recommender system to realize personalization. Similarly, Saumil Dharia et al. presented the PRO-Fit framework that collects users’ multivariate data, including fitness preferences, calendar data, and social network data [11]. It applies machine learning algorithms to classify users’ activities into specific types, which are then used to establish user profiles reflecting their current lifestyle (sedentary vs. active). These user profiles are further fed into a collaborative filtering-based recommender system for personalized fitness activity or fitness partner recommendations. Unlike most studies that target the general public, Mogaveera et al. introduced a health monitoring and fitness recommendation system using machine learning targeting patients [23]. The system collects data from both patients (body details, disease & health records) and doctors (disease categories) to monitor patients’ condition based on some predefined rules and to provide personalized recommendations of diet and exercise plans through a decision tree based machine learning model.

Compared with the existing research, our focus is on recommending dynamic changes of workout speed and heart rate of an exercise, which are complementary dimensions in the field of fitness recommendation. In addition, the construction of these exercise activity recommendation systems usually boils down to recommending some predefined fitness categories. In contrast, we propose a deep learning framework to solve a rarely studied sequence-to-sequence regression task to model workout speed and heart rate. Moreover, as discussed above, most researchers have achieved personalization with rule-based or collaborative filtering-based methods. In contrast, we propose learning user profiles based on tensor decomposition. Compared with the rule-based method, this method is considered to be more scalable and compared with the collaborative filtering method, it can realize context-aware user profiles [17].

Sequential Fitness Profile Recommendation. Berndsen et al. proposed a recommender system that predicts the target finish time of Marathon with an XGBoost model, followed by a collaborative filtering & K-Nearest-Neighbours (KNN) based framework to generate pacing recommendations for runners to achieve the target finish time [6]. More specifically, historical training data of athletes, such as GPS data, completion time, and pacing information, are used to predict the target completion time. Meanwhile, a runner’s user profile can be inferred by applying collaborative filtering and further used to find the successful marathon finishers with the most similar user profiles applying K-Nearest-Neighbours (KNN) algorithm. Finally, pacing strategy recommendation is achieved by using the pacing strategies from these successful marathon finishers with similar user profiles. Compared with their work, the exercises we focus on are less competitive than Marathon, such as jogging, biking, etc. However, our task of predicting workout distance is similar to the prediction of target finish time in [6], except that we propose a multi-layer perceptron model to solve this task. In addition, their approach to workout pacing recommendation is through looking up historical exercises and following existing pacing records. We propose an LSTM based model to generate personalized new pacing sequences.

Jianmo et al. proposed a personalized fitness recommendation system named FitRec to solve two tasks: predicting the heart rate and speed at all time steps of a future workout, and short-term prediction of heart rate and speed at a specific time step during an ongoing workout [24]. The former predicts a future workout profile to show users the anticipated performances in terms of heart rate and speed while the latter predicts transient heart rate and speed during an ongoing exercise, so as to facilitate tasks like anomaly detection or real-time decision-making. FitRec was primarily designed based on LSTM modules. More specifically, they used LSTMs to project the sequence of the most recent workout measurements of the user into a dense vector, which forms the user embedding vector. Then, the user embedding vector is concatenated with other input attributes of the new workout sequence as the combined contextual sequence input for two different LSTM networks corresponding to the two tasks and adopt a 2-layer stacked LSTM architecture and an LSTM-based encoder-decoder architecture respectively.

Compared with FitRec, our work differs in many ways, although we study the same dataset as FitRec, and its task of predicting a future workout profile has a similar purpose to ours. First, we suggest adding caloric expenditure to the input features, so that our system can provide different recommendations according to users’ inputs of target caloric expenditure, which enhances the practical value of our research. Furthermore, FitRec predicts the speed of a workout using distance and time as inputs, which arguably may be seen as a trivial issue solvable by simple math. Therefore, we propose removing the time sequence feature from the inputs to avoid potential information leakage. Besides, we improve the derivation of entity embeddings by learning them from all historical workout records of the users, instead of learning only from the latest one. Our tensor decomposition method for learning entity embedding is also different from theirs.

Privacy Preservation in Fitness Recommendation. In privacy-aware recommendation systems, privacy preservation is usually achieved through some independent encryption protocols in the process of data collection or data exchange [26]. [5] proposed a reversible data transform algorithm based privacy-preserving data collection protocol for mobile app recommendation systems. According to this protocol, a user’s data is sent through a user group with encryption, which avoids direct communication between a user and a data collector. Likewise, Arijit Ukil addresses the privacy preservation problem through a random security key pre-distribution method [32]. According to the proposed scheme, private data can be collected from various sources and aggregated by the service provider or the server securely. [35] studies the challenge of privacy preservation in a more specific scenario when the recommendation systems of two independent business entities are merged. In this scenario, both a homomorphic encryption approach and a scalar product approach are proposed to encrypt raw data before data exchange takes place between the two systems. Badsha et al. also proposed a homomorphic encryption approach to enforce privacy preservation in building a recommender system [3]. More specifically, they collected encrypted ratings on items from users and sent ciphertexts to recommender servers to calculate the similarity among the rated items homomorphically. The similarity scores were subsequently decrypted by the users without revealing any private information and then used to build a recommender system by using content-based filtering and collaborative filtering methods.

In contrast, our work focuses on developing the recommendation algorithm itself with minimum private data, which is complementary to the data encryption approach discussed above. Sanchez et al. conducted a study on users’ preferences on privacy permission in the fitness domain and recommended a series of strategies for users to set permissions according to the collected and shared data [26]. They found that users have the highest acceptance rates of privacy permissions in terms of gender and fitness types, while the users are reluctant to share their height, weight, age, and social network information. The results of this study confirm our hypothesis about privacy protection and are consistent with our attempt to construct a personalized fitness recommendation system by using minimum private information from the users. To the best of our knowledge, most existing work in the fitness recommendation domain attempted to collect multidimensional demographic parameters from the users to improve the performances of their recommendation algorithms. In this paper, we demonstrate that a recommendation algorithm can be developed using minimum private data from users.

In this section, we first introduce an overview of our proposed framework ($P^{3}FitRec$), and then we introduce the details of each component of the proposed framework.

Our $(P^{3}FitRec)$ is composed of three-dimensional tensor data analysis and two deep learning models (MLP and multi-layer Bi-LSTM), which are used to predict the total workout distance, and the speed and heart rate sequences respectively. The main inputs of the framework are Target Caloric Expenditure, Sport Type, User ID, and Workout Route ID. Based on the User ID, User Embedding can be found by looking up the pre-trained user embedding tensor using the Tensor Decomposition method. Similarly, based on the Workout Route ID, Workout Route Embedding can be found by looking up the pre-trained workout route embedding tensor, as well as the Total Workout Route Distance, Altitude Sequence and Distance Sequence associated with the chosen route. In summary, the contextual input features consist of Target Caloric Expenditure, Sport Type, User Embedding, Workout Route Embedding, and Total Workout Route Distance. The sequential input features encompass Altitude Sequence and Distance Sequence.

Our $(P^{3}FitRec)$ can provide personalized fitness recommendations by applying the models mentioned in section 3. The system allows users to specify the type of sport (run, bike, or mountain bike) and select a specific route, and then input target caloric expenditure. According to these inputs, our $(P^{3}FitRec)$ will provide various recommendations in the aspects of distance as well as speed and heart rate at each time step. Table 5 shows a typical running workout from the dataset, whose original record has 6.2 km of distance, 592 kcal of caloric consumption, and average speed and heart rate of 8.8 km/h and 149 beats per minute respectively. Given the same caloric input, our $(P^{3}FitRec)$ predicts similar distance, speed, and heart rate. Furthermore, for higher target caloric inputs, we observe the greater distance, speed, and heart rate are predicted, and vice versa. Moreover, Fig. 8 shows the change in speed and heart rate with respect to time stamps. The blue line represents the original workout record, while the dotted lines represent the recommended speed and heart rate according to various input target caloric expenditures. We observe that the model can properly predict the fluctuation of speed and heart rate with relatively low MAE. By comparing three different predicted workout speed and heart rate sequences, the system can properly predict the correlation between distance, calories, and heart rate.

Our $(P^{3}FitRec)$ has a high degree of freedom in choosing sport type, route, and target calories, and the personalized recommendations of speed and heart rate change dynamically according to the workout route. Therefore, the system provides users with greater flexibility to plan and predict exercises, and to modify the speed and pace during exercises.

The proposed models, datasets and experimental data will be made available at the project repository for further extension and reproducibility studies. ³³3https://github.com/BasemSuleiman/Personalized_Intelligent_Fitness_Recommender..

In this paper, we propose $(P^{3}FitRec)$, a Multi-layer MLP and Multi-layer Bi-LSTM based framework that provides personalized fitness recommendations with privacy preservation. Our $(P^{3}FitRec)$ employs Tensor Decomposition to infer entity embeddings from historical workout data and has achieved satisfactory results on predicting workout distance, workout speed sequence, and workout heart rate sequence. We demonstrate that personalized fitness recommendations can be achieved using minimum identity information from the users.

For further studies, we propose extending the work in three aspects. First, the cold-start problem is an interesting topic in building recommender systems, for which in our case is on new sport types and new users. Users playing new sports will have to exercise without personalized recommendation and contribute an adequate amount of workout data before a model can be trained to cover these sport types. Likewise, for new users who have absolutely no workout history, it is a challenge to learn user embeddings using the Tensor Decomposition method. Moreover, our $(P^{3}FitRec)$ is built on a two-model pipeline, instead of an end-to-end model structure to realize the two related tasks respectively. Though the models share the same contextual input features and the prediction of the first model feeds into the second model, one can argue that the accuracy of the application can be further improved if an end-to-end structure is applied, such as an encoder-decoder structure. Lastly, the Transformer architecture [33] has become dominant in sequence prediction tasks, especially in the field of natural language processing. Although our experiment on the attention mechanism shows little performance enhancement, it is possible to improve the performance by trying the Transformer architecture.