Online Action Representation using Change Detection and Symbolic Programming

Action representation is important for many tasks like surveillance, human-computer interface, rehabilitation monitoring, sports analysis, behavior analysis, autonomous navigation, etc. In most of the existing methods, processing is done on fixed-sized segments of an incoming video, and an output will be produced for each segment. Some methods do segmentation to get meaningful windows as a pre-processing step, but they are mostly offline methods, i.e., they require the whole video to find the segments. But for tasks like rehabilitation feedback, surveillance, etc., it’s crucial that the methods work online. Setting a fixed window size is not optimal, as different motions have different dynamics. Certain approaches use image-by-image methods, but they don’t capture the temporal dynamics of actions. We identify this gap in the existing works and propose an online action representation method, where we use change detection to segment out action sequences and then use primitive fitting to represent the motion segments at a higher level. The segmentation works online as it uses an online change detection algorithm. Thus, we get meaningful segments of actions as soon as they happen in the streaming video without needing future video frames. Also, since representation is derived over time periods, they capture temporal information.

As an introduction, we intend to provide an intuition to the method. The motivation for this representation comes from the idea that an activity(e.g., walking) is composed of elementary sub-actions like moving left leg forward, moving right leg forward, etc. Also, these sub-actions can be detected from the increasing and decreasing changes in the joint angles that occur along with the action. So, we can extract the regions of such sub-actions and then make higher-level representations of them. Such a representation will be both discriminative and generalizable because of its compositionality. In order to segment the sub-actions, we first identify the regions of change. Next, we fit each segment to symbolic primitives to create more semantic representations. Symbolic programming uses domain knowledge to predefine generative symbols and fit the data optimally to those symbols. Here, we fit the trajectories to simple shapes like lines, circles, etc. These symbols are generative as they can be used for synthesis as well. So, with such fits, we will be able to state something like: “the arm is rotated in this time segment.” This method can work sequentially on an incoming video by the choice of an online change detection algorithm. Fig. 2 shows the complete pipeline of our proposed method.

We explain each step in the method in more detail. Later on, we use repetition detection as an example downstream task to show the effectiveness of the representation. We also compare the results of our method with the existing offline counterpart.

Existing works that attempt to do rehabilitation and exercise monitoring include [1, 2, 3], but they are not online. In [4], an image-based approach is used that gives feedback on each frame. As it is image-based, it fails to capture the temporal information. Being data-driven makes these methods less feasible in specific domains where data is sparse, such as rehabilitation.

In [5], a motion programming idea of fitting primitive motions to action segments is introduced. That method uses a dynamic programming approach to automatically segment the videos based on an optimal error of fitting. But this method only works offline. We improve on this by replacing the segmentation part with a change detection algorithm and then fitting a single primitive to the segmented part.

Change detection is the process of identifying meaningful changes in a time series over time. They can be classified into online and offline algorithms. Though we use a classical change detection algorithm, it can be replaced with any online change detection method.

Loop/repetition detection is the task of counting the number of repetitions of approximately the same action in an input video sequence. Some of the existing works on loop detection are [6, 7]. In [6], online loop detection is done, but it uses windows of a predetermined fixed size.

Here we provide a comprehensive overview of the entire pipeline, followed by detailed explanations of relevant parts. The primary objective of our work is to make the task of action recognition online. The input is an incoming sequence of frames.

First, we employ an off-the-shelf pose estimator to extract the keypoints from the frames. Here we use the mediapipe pose estimator[8]. We calculate the angles at several joints and also the length of bones using vectors constructed from the keypoints. Joint angles refer to angles subtended at body joints, such as the shoulder angle and the elbow angle, while bone lengths refer to the lengths of body parts between two joints, such as the calf length and the torso length. Consequently, we have several sequences of joint angles and bone lengths as illustrated in Fig.1(a). These values are computed frame by frame, and hence they are online incoming time series signals. We use the CuSum change detection algorithm[9] to detect change in each of these signals. Once a change is triggered in any one of the signals, that particular signal is utilized to determine the starting and ending times for temporal segmentation. While CuSum provides the time of change and the starting time of change, it does not provide the ending time. Therefore, a piece-wise line-fitting method is devised to estimate the ending time after waiting for a few more frames from the point of change. The number of extra frames to take is decided based on a heuristic. We observe that the clip between the starting and the ending time of the detected change forms a meaningful action segment. The keypoint sequences, the joint angle sequences, and the bone length sequences are trimmed according to those time points. In order to obtain a higher-level representation of the action, we use a motion primitive fitting method. This method adopts a symbolic programming approach, wherein the data is matched to a set of predefined symbols. We fit the trajectories of each keypoint in the trimmed time span to primitive shapes(circle, line, stationary)(Fig. 1(b)). Note that the change detection is done on angles and lengths signals, whereas the primitives are fit on 2D coordinates of each keypoint. This symbolic representation brings us one step closer to the temporal semantics of the motion. Thus, they are better features for downstream tasks. Fig. 2 provides a block diagrammatic depiction of the process.

We use this action representation for class agnostic repetition detection, which is one example of a downstream task.

We propose a consecutive match search algorithm for repetition detection. Match is defined using an error metric between the start, mid, and end points of two primitives. The error is given by $|\overline{s}_{i}-\overline{s}_{j}|+|\overline{m}_{i}-\overline{m}_{j}|+|\overline{e}_{i}-\overline{e}_{j}|$, where $\overline{s}_{i},\overline{m}_{i},\overline{e}_{i},\overline{s}_{j},\overline{m}_{j},\overline{e}_{j}$ are start, mid and end coordinates of two primitives $i$ and $j$ respectively of one keypoint, and the norm is L2 norm. This error is defined for all keypoints, and the total error is taken as the sum of all. Two motion primitives are said to match if the error is less than a threshold.

A buffer of the recent motion primitives is maintained. When a new motion primitive is encountered, it is matched with the existing primitives in the buffer. If a match is found, then consecutive matches are searched for by drawing more primitives and checking matches with adjacent primitives. This process is continued until a mismatch. A group of consecutive primitives that are matched belong to the same loop. The buffer is updated by removing the primitives which are matched. Counts get updated as the instances occur, as opposed to giving a total count of a given video offline.

We demonstrate change detection and primitive fitting in 4 different exercise examples(Fig. 6, Fig. 7, Fig. 8, Fig. 9). The segmentation window found using change detection is indicated by the red arrows in the angle/length sequences. Corresponding primitive fit(for each keypoint) is overlaid on the skeletons below. The trajectories that appear curvy are those which were fitted as circles, those which appear straight were fitted as lines and those which are single points were fitted as stationary. It can be observed that majority of the extracted primitives are meaningful. Fig. 9 shows an example of false positives, where primitives are detected though not meaningful. From the figure, it can be seen that the middle two primitives are segmented based on change in elbow-angle signal, which is irrelevant in crunches. Ideally, we want two primitives to be detected(the up motion and the down motion), but two additional noisy primitives were added.

In [5], a loop detection method that uses similar start, mid, and end points of extracted primitives is proposed. However, those primitives are extracted offline from videos. We compare our method with that method in terms of per-class percentage accuracy on repetition counting. Percentage accuracy is defined as $(1-\frac{\left|\text{Predicted count}-\text{True count}\right|}{\text{True count}}*100)$. The validation is done on moticon dataset from [11] after manually labeling the number of counts of actions in the videos. We validate the methods only on videos where the count is more than 1, as the algorithms are based on finding matches and hence require at least two instances. Table I shows the per-class percentage accuracy of our online method and the aforementioned method. We managed to get either comparable or better results in each class. Note that this was achieved by operating online, which is the key contribution. The offline method failed to do prediction on bulk of arm-cross-swings instances, and hence it was left out. Meanwhile, the proposed method gives consistent results for all classes.

Further, we collected more exercise videos and evaluated the method using those. The data collection involved 3 subjects, each performing 4 exercises(push-ups, crunches, arm raises, arm rotations). Each participant performed 5 instances of varying counts for each exercise, totaling 60 videos of different counts for different exercises. We annotated each video with the actual count and then calculated the percentage accuracy of the proposed method on this data. The exercises were performed in different view orientations w.r.t the camera in order to check the performance of the method in such cases. Table II shows per class percentage accuracy on repetition counting using the proposed method. It can be observed that the accuracy is high for simple motion exercises like arm raises and push-ups, where the movements remain mostly regular. There is a higher chance of irregularity in the repeated motion for exercises like crunches and arm rotation. This factor has an effect on the performance because the method relies on similarity matching.

The main advantage of the proposed method is that it operates online. Furthermore, the choice of symbolic representation makes it compositional, generalizable, and interpretable. The compositionality here is with respect to body parts. A human activity can be composed of elementary actions of each part. It can be argued that these elementary primitives are finite in number as human motion is constrained. Then, any activity can be represented as a composition of those finite primitives. This makes it generalizable. A part-based representation also proves to be advantageous in scenarios where the view is incomplete, such as when the camera is positioned such that just a portion of the body is visible. Since we explicitly know which symbol a part is fit to, the representation is interpretable. Also, this method doesn’t require any data for training and instead relies on domain knowledge for good choices for primitives.

Noisy involvement of other body parts when performing an exercise will compromise the effectiveness of the method. For e.g., if a person is randomly moving his arms while performing a knee raise, those hand motions might get triggered as changes. In that case, the segmentation done is not meaningful (Fig. 9). One way to address this drawback is to use a consensus of change detections over multiple time series instead of considering only a single series. Usage of adaptive thresholds for different signals may also be a promising approach.

We use bone lengths along with joint angles because we are working with 2D coordinates. We expect bone lengths to capture surrogates of some of the 3D motions. If we use a 3D pose estimation method directly, then we need only use angle sequences for change detection, and further we can directly fit 3D curves as motion primitives. However, most of the 3D pose methods are noisy along the depth dimension.

We address the problem of online action representation. We use a change detection algorithm, which has been shown to give meaningful segments for further symbolic fitting. Using a symbolic programming approach allows us to use this method in sparse data scenarios. Additionally, the nature of segmentation and primitive fitting captures the compositional aspect of actions as a consequence. This representation can effectively work for many downstream tasks, and more importantly, it can be operated online, which is the primary gap that is addressed.