Can viewer proximity be a behavioural marker for Autism Spectrum Disorder?

The objective of the wheel task is to present a ‘rotating’ disk (refer below) on the tablet screen to a child who is being screened for ASD.Autistic children tend to look intently [6] at a spinning wheel as well as come closer to the wheel. While the child is engaged in this task, the front camera of the device records the child’s movement.

In the current work, we investigate the relative movement of the child with respect to the screen. If a child is fascinated by the spinning object they tend to often move closer to the object and thus closer to the front camera. In this paper, we posit that a measure of the relative motion of the child with respect to the front camera can be used as an indicator of their interest in the spinning object. We then utilise this observation for predicting the diagnostic classification of the child.

Challenges: Although cues such as relative face size can be considered for computing the distance of the child from the screen, we found in our experiments that these proxies for distance are unreliable. The key computational challenge stems from our assumption that different children could be seated at different distances – indeed some children may even run the test in a supine position. Also, the head orientation of a subject can inordinately impact the perceived face size in a two-dimensional image.

It is to be noted that while we administered the tests in an unconstrained home environment, such tests are usually administered in controlled laboratory settings. Also, the tests were conducted by low-skilled health workers in LMIC households. The workers were not provided any special training, unlike the clinicians who usually conduct the tests in controlled settings. Hence, our approach addresses this constrained context and aims to capture the viewer’s proximity using images captured from an uncalibrated low resolution camera. Please see the supplementary video on the data capture.

Tablets recording viewers is becoming increasingly common, to say the least, and face identification is standard. The key technical contributions in this work are

• •

  A method that computes in metric units the distance between the front camera of a tablet and the face of a human being. This measure is often a requirement in such diagnostic analyses. (The source code will be provided)

• •

  A method that works

  • –

    for children as they tend to get distracted quickly and may not remain stationary in front of the tablet device during the task.

  • –

    in poorly lit conditions as the study is aimed at children from less privileged backgrounds. (Methods that work in well-lit homes may not generalise across low resolution, poorly lit environments. In this paper, we demonstrate evidence that current computer vision methods are quite robust in home settings.)

• •

  Quantitative evidence that viewer proximity in the wheel task can be a marker for classification in child development. (All earlier work on this task were subjective lab-based assessments. We report the first classifier which utilises this measure for diagnostic classification).

The rest of this paper is organised as follows. We first discuss related work, and follow it up with the method suggested and its validation (Section II). Results are presented in Section III.

Since ASD is a spectrum of developmental disorders, most of the diagnostic and screening assessments revolve around the analysis of behavioural symptoms. These can be analysed in dyadic interaction between the subject and others. For instance, in the case of young children, by observing the interaction between mother and child, psychologists can devise an appropriate intervention plan (i.e., therapy) for better parental care and child development.

In such assessments [7] the aim is to capture important behavioural cues such as shared mutual attention, imitating other’s actions, etc. Such play interactions are also analysed for screening and diagnostics [8] purposes and for longitudinal impact assessments [9] of the selected intervention therapy.

Researchers have tried to employ machine vision techniques to automate such analysis but they are either dependent on a multimodal setup or analyse a very structured set of actions under a constrained paradigm. For instance, Rehg et al.[10] have analysed behaviour of children by creating an independent Rapid-ABC paradigm. In contrast, Marinoiu et al.[11] have attempted both action and emotion recognition tasks in an unstaged environment. However, they have focused on robot assisted therapy and not on the diagnostic aspect of autism which is the focus of our paper.

The analysis of head movement, while children perform autism specific assessments, has gained traction in the computer vision community in recent years. More specifically, Martin et al.[12] contrasted head movements between ASD and non ASD groups of children when watching videos of social vs non-social stimuli. They have studied group differences vis-à-vis pitch, yaw and roll of the head movements. Ogihara et al.[13] studied specific temporal patterns of head movements to understand their diagnostic potential.

The studies based on head movement [12] and [13], discussed above, have been carried out in laboratory settings. Children are seated in front of a monitor, with a mounted camera, which records their movement while they view various stimuli on the screen. In contrast, our work relies solely on videos captured at the home of participant children. The videos are captured from the relatively weak front camera ($0.35$ Megapixels) of Android tablet devices while the children view the stimuli presented on the tablet screen.

As mentioned, several tasks are administered in determining markers for autism. Typically, the battery of incorporated tasks are carried out in controlled settings in a laboratory with expensive setups, sensors and trackers. In contrast, we are limited in our approach due to the requirements of a low-resource setting. Due to this requirement the data collection was performed using an economical tablet device with additional limitations of:

• •

  No user dependence is permitted (For example, no task or user specific calibration was allowed).

• •

  No separate recording – the recording was only via a low resolution RGB front camera that could be triggered with the task.

• •

  No additional user sensors or wearable setup was allowed.

These limitations are to be contrasted with prior work. The tasks were conducted by non-specialist community health workers on young children at their homes. This was essential for the desired scalability and for collecting data on autism related symptoms in LMIC. In summary, the task was administered in as natural a setting as possible. In this paper, we have focused only on one of the several tasks which have been incorporated for screening of autism risk.

The data is a set of videos of children performing the wheel task. Prior to the data collection the children were categorised in one of the three diagnostic groups by experts in a prestigious medical school-cum-hospital-cum-medical research university that sees a daily footfall of over thousand patients of all ages. In contrast, the video data set is collected by minimally trained workers who have administered these tasks in semi-urban households. A corpus of $121$ videos has been created (with $195000$ image frames). The videos have been captured across the diagnostic categories or classes of TD (Typically Developing), ASD and ID (Intellectual Disability) children as shown in Table I.

These videos were manually verified for usability in our task and some videos were observed to be unusable due to insufficient length of relevant footage. Finally, after removing such videos a total of 115 videos were used for this analysis with the respective distribution across the three diagnostic classes as shown in Table I. We have included a sample video from this dataset in the supplementary material.

As its input, the algorithm receives a video of a user watching a tablet screen as captured from the front camera. The intrinsic (or extrinsic) camera parameters are unknown. As its output the algorithm predicts the user’s distance (in metric units) from the tablet screen in each video frame. There is no expectation that the user will be in a seated position, or in any fixed position. It is conceivable that the user may not be visible while the task is going on since children can be easily distracted.

First we detected the face, and then landmarks within the face using a state-of-the-art deep neural network-based face detector (FAN) [15]. This detector provides, for each frame in the video, a reference 3D value for each landmark in a local coordinate system. However, these 3D coordinates, while valid for each frame, are incomparable across frames. Next, a reference frame is therefore desirable, and we used the frame corresponding to the “most frontal”¹¹1Details provided in the supplementary material face amongst all faces in the video.

The unifying factor across all these frames is, of course, the camera. Since the 2D landmark for each corresponding 3D landmark is available (indeed, that’s how the 3D is estimated in the first place), we were able to use these correspondences to find the camera centre. Once the camera centre is computed, it is relatively straightforward to compute the metric distance to one of the landmarks, such as the left or the right eye. The data from this step is used in ablation Scheme 2 described later.

As communicated by domain experts, some child development studies rely on actual Euclidean distance values. Since from a single camera, we cannot obtain Euclidean (metric) values, we use the concept of “bitragion breadth” – the width between the two tragions (cartilaginous notches at the front of the ear). Once the head pose is calculated in pixel units, we obtain the average depth in real units using the scale factor from the bitragion breadth of children. The benchmark bitragion breadth is taken to be 10.6 cm corresponding to the $50^{th}$ percentile bitragion breadth for children, both males and females (obtained from [14], page 454). While the head sizes do vary across children, the variation is insignificant in children in our data set who primarily belong in a narrow age range.

In this section, we describe the process of analysing the viewer movement data in order to predict the diagnostic classification. We utilise the data output by our proximity estimation algorithm as described in Section II-B.

Preprocessing: As is evident from Figure 3 and Figure 3 the movement data is now reduced to a vector containing the estimated distance of the child from the tablet for each frame of the recorded video. The size of the vector depends on how long the task was undertaken (a stop button is provided on the screen for the child). It can range from a few hundred data points to a maximum of $2400$ data points (recall that videos in the task are displayed for no more than 80 seconds).

Since the whole process is natural and free format, there are frames which do not depict the child, i.e., the child has moved out of the screen. These could be a deliberate action on the part of the child (“disinterest”) or could be a momentary glitch (“the tablet is swung out”). One way to handle this is to apply a windowed median filter followed by Gaussian smoothing. Any other kind of preprocessing, e.g. applying transforms, was not considered as the neural network was expected to learn from the raw data.

For larger gaps, there may have been some intent, and we wish to seek to preserve this information. Therefore, if after filtering, the distance estimate is still unavailable, a marker value (‘0’) is substituted with the intention of letting the network learn this event. (Any resulting string of $0$s in the vector has been replaced with a single $0$ value.) We observed that this replacement stabilised the subsequent learning process.

Since the input vector sizes are of variable length, we extracted contiguous chunks of smaller vectors of size $100$ from each input vector. This helped in providing a uniform input size as well as, in essence, boosting the training set size.

The resulting equi-sized vectors are provided as input to a 1D convolutional neural network to predict one of the three diagnostic classes. The network details are shown in Figure 4. Cross entropy loss function was employed to the last layer’s output to arrive at the final predictions. A data split of $7:1:2$ was used for training, validation, and test sets respectively.

Ablation schemes: To further investigate the impact of various factors on the classification accuracy, we have divided the analysis into three broad schemes as described below:

• •

  Scheme 1: In this scheme we analyse the movement data as generated by our viewer proximity algorithm (refer Figure 3). The equi-sized chunks of vectors are input as is to the network model shown in Figure 4(a).

• •

  Scheme 2: In this scheme we normalise the movement data before providing it as input to the network model as shown in Figure 4(a). Since the movement data is estimated based on an average head size (derived using the bitragion width of an average child), this scheme mitigates any inadvertent impact of this assumption. The movement data for each child subject is normalised by mean shifting and scaling it down by the respective standard deviation.

• •

  Scheme 3: As mentioned earlier the output video duration in our data set differ across child subjects. This duration indicates how long the child was actually engaged in the wheel task. In this scheme we analyse the impact of supplementing the input with video duration (expressed as the number of frames in the video).

  To analyse this modified input we have altered our network model. Now, there are two separate inputs provided:

  • –

    vector representing the movement data. This part is unaltered and remains consistent with schemes 1 and 2 described above. The output of the penultimate layer is extracted before final inference and concatenated as shown in Figure 4(b)

  • –

    number of video frames for the corresponding video. This input is parsed through a fully connected layer and concatenated with the output of the penultimate layer mentioned above

  The concatenated vector is finally parsed through fully connected layers for making the diagnostic classification prediction.