Detecting Abnormal Health Conditions in Smart Home Using a Drone

Although our current solution only solves the fall detection aspect, we envisioned our project to have an end-to-end solution based on person detection and fall classification.

As seen in Figure 1 below, our solution accounts for all possible outcomes of person detection and fall classification. The drone will constantly be scanning the area and taking images and videos and if a person is detected we move on to feature extraction to gather all relevant information on the human’s movement. If a fall is detected the user will be asked whether or not they need help. If the human answers yes or does not give an answer at all, an alarm will be triggered, contacting emergency personnel along with family members programmed into the system. However, if a person or a fall is not detected, the drone will go back to the beginning phase to continue its analysis. Lastly, if there is any uncertainty, the drone will re-evaluate by moving to a new location to reassess the situation.

As previously mentioned, our current solution does not offer an end-to-end solution, but this serves as the framework we envisioned our project to follow.

Previous studies have presented fall detection systems used to detect patient falls. While other studies have designed a UAV to monitor patients’ behavior in order to alert emergency personnel for victims of falls. This section covers previous work related to UAVs and other fall detection systems such as robots.

We used a Ryze Tello drone as a low-cost hardware platform to perform our study. The Ryze Tello drone costs approximately 100 USD and is a small and light weight drone that weighs approximately 80 g (with propellers and battery). The full specifications are given below:

• •

  Dimensions: 98×92.5×41 mm

• •

  Propeller: 3 inches

• •

  Built-in Functions: Range Finder, Barometer, LED, Vision System, 2.4 GHz 802.11n Wi-Fi, 720p Live View

• •

  Port: Micro USB Charging Port

• •

  Detachable Battery: 1.1Ah/3.8V

The drone can be operated both indoors and outdoors. A single camera, an IMU with a 3-axis gyroscope, 3-axis accelerometer, and 3-axis magnetometer, and a pressure and IR-based altitude sensors are all incorporated into the drone. The front-facing camera features an 8-megapixel sensor and can record video at a resolution of $1280\times 720$ at 30 frames per second.

The Tello drone is a ready-to-use device that requires no additional setup. In order to control the drone, a compatible mobile phone is required along with the ’Tello’ app and wifi. The drone is a battery charged device, lasting approximately 12-13 minutes.

As previously mentioned, our study consisted of two male participants, aged 12 and 16 years old. This study took place in a single-family home with approximately 10 inches ceilings. The images and videos were captured in two different rooms within the home, one with more open space and one with less space. The study was carried out over a span of two weeks, with sessions ranging from 15 minutes to an hour depending on drone complications, participant poses and assessments, and retakes.

The instructions that our participants followed were straightforward. Using our own intuition, we first determined what positions people usually are in after a fall and took note of it. We then showed our participants an image of the position and instructed them to reenact it. Once our participants were in position, we powered up the drone, captured the videos and images, assessed them and then asked our participants to move on to the next position.

We propose a deep learning-based approach for detecting whether or not a person has fallen using RGB photographs. For feature extraction, we employ a CNN algorithm¹¹1The proposed idea is taken from [11] with 6 blocks and a header. Conv2D Layer with ReLU activation and MaxPooling2D are included in each block. We begin with 16 troops and increase by 2 units every 2 blocks:

We flatten the result after feature extraction, add a Dense layer of 32 units, and utilize sigmoid activation for classification.

The CNN model uses visual representations based on the RGB image and segmentation and learns high-level embedding features for fall recognition. We explain the individual components of the conceptual structure in detail below.

Initialize the weights of the nodes in the FallNet network model at the start. The weights for the convolutional layer are started in this model using the weights from the following embedding layer, which have a zero-mean gaussian distribution (SD= 0.01 and bias=0) [8]. We utilize the Adam optimizer with a learning rate of 0.0001 and a loss function of Binary Crossentropy to train our model. The proposed system is built on the basis of Torch, Keras libraries[9, 10]. Figure 6 depicts the convolution layers, which range from 1-6 convolutions.

At first, we utilized this publicly available dataset data to train our model for fall detection. The acquired data was then used to validate the pre-trained model. The accuracy and loss plot was obtained after 5 epochs.

The loss is the value that a neural network tries to minimize in deep learning: it’s the difference between the ground truth and the forecasts. The neural network learns to decrease this distance by altering weights and biases in a way that minimizes the loss.

In regression problems, for example, you have a continuous target, such as height. The discrepancy between your predictions and the actual height is what you wish to minimize. You can use mean absolute error as a loss to tell the neural network what it should be minimizing.

It’s a little more complicated in terms of classification, but it’s still comparable. Probability is used to predict classes and the neural network minimizes the possibility of assigning a low probability to the real class in classification. Typically, the loss is categorical crossentropy.

The difference between loss and validation loss is that the former is applied to the train set, while the latter is applied to the test set. As a result, the latter is a useful indicator of how the model performs on data that hasn’t been seen before. Use validation data=[$x_{test},y_{test}$] or validation split=0.2 to get a validation set.

To avoid overfitting, it’s advisable to rely on the validation loss. When the model fits the training data too closely, overfitting occurs, and the loss continues to decrease while the val loss remains stale or grows. When the validation loss stops lowering, you can use EarlyStopping in Keras to halt training.

After performing 5 epochs we get the following accuracy and loss plot from training and testing data.

Table I shows that the model achieves complete accuracy after 5 epochs. Our model performs better on the unseen data, as seen in figures 7 and 8. On testing data, our model is more accurate. Furthermore, the loss of model is minimal. Figure 9 also displays the prediction rate of $100\%$. That means our model accurately detects the fall.

During the duration of our study, we faced some challenges and had some limitations along the way. This may be due to the inexpensive nature of the Tello drone, however during the initial stages of understanding how to fly the drone, the propellers would often fall off on it’s own. After flying the drone several times, we encountered takeoff complications, where the drone could not takeoff from the ground. Even after re-calibrating the device multiple times, we could not successfully address the issue, causing us to utilize a secondary Tello drone.

Secondly, many of the training datasets available publicly required permission to use and given the time constraint of the project, we could not rely on waiting to be approved for access.

Also, for a number of poses we wanted our participants to re-enact, we could not achieve certain angles based on the height limitations of the drone in order to ensure safe flying.

Lastly, the drone’s camera quality often gave us issues when testing the images with the CNN model, sometimes being incapable of determining what the image consisted of.

We used a small drone to detect falls in this experiment. We wanted to test if the Tello drone could be effective in monitoring smart health on a broader scale. Based on our findings, we can safely conclude that we can deploy a Tello drone to monitor the residence. There are some drawbacks as well, because the CNN model requires a large dataset with uniformly distributed classes. Also, the drone’s range is limited, and the video resolution is only 720p with an 8-megapixel sensor. The weather circumstances contribute to hazy photos, and we are unable to fly it in strong winds. It is, nevertheless, quite inexpensive and convenient. We feel that this drone can still be used to monitor the residence in an interior scenario.

As machine learning and smart home devices, such as a drone are new topics that we had not previously explored, we learned a great deal throughout the duration of the project. First and foremost, it was critical to work together as a team from start to finish. We played on our strengths and assisted the other when in need of help. Furthermore, we learned the importance of trouble shooting because although technology is revolutionary, there will still be some challenges along the way. Being able to turn to one another and other resources when needed allowed us to achieve the best results that we could.

In terms of the drone, we determined that a drone can be an effective device to detect falls in the home if created in a robust fashion. Although drones are initially thought of to be used in the entertainment industry to capture footage in areas where humans cannot reach, it can be applied to many other disciplines such as healthcare. A drone can be used in the home to detect falls, minimize the need for an in-home aid, and reduce unwanted stress.

This project serves as an eye opener that smart home technology can be adapted to make the world a more safer place.

In the future, we hope to explore whether the drone is capable alerting appropriate officials in the event of an incident. Furthermore, we would be interested in exploring the possibilities of having an autonomous drone or autonomous vehicle, taking away the burden of having to fly the drone manually. We aim to monitor the system for potential attacks and devise strategies to mitigate them, recognizing that autonomous drones or vehicles are susceptible to security threats[12, 13]. Furthermore, we would like to introduce the idea of having the drone dock itself when it needs to be charged. Lastly, since we did not include mask R-CNN in our analysis, comparing mask R-CNN and CNN algorithms in the future can prove beneficial to the effectiveness of our system.