DFKI Cabin Simulator: A Test Platform for Visual In-Cabin Monitoring Functions

The OpenDS simulator software [6] is utilized and provides a near-realistic driving environment. There are over 20 different driving scenarios and driving tasks in urban and rural environments including other traffic participants like cars and pedestrians as well as traffic lights. The software is also capable of simulating different environmental and weather conditions.

Three projectors (Fig. 2(b)) are used to project the simulated scene on a wide-angle screen covering almost the driver’s entire field of vision (Fig. 2(a)). This leads to a realistic driving experience and, most importantly, to realistic driver’s movements (e.g. head movement in turning maneuvers). Together with the simulation, the software also comprises a drive analyzer to gather the data of the USB Controller that senses the driving actions as steering, braking, and acceleration.

Of course, it is also possible to install other simulation software. This might come in handy to cover special needs for testing e.g. commercial vehicles, agricultural vehicles.

The test platform has been equipped with a wide-angle 2D/3D camera system for monitoring the entire interior of the vehicle mock-up of the simulator. The mounting position of the camera corresponds to the overhead module of the car close to the rear mirror (see Fig. 2(c) and 2(d)). In order to be capable of monitoring both driver and passenger from that position, a wide field of view to monitor the full vehicle cabin, including the driver and the front-seat passenger is crucial. The metal fixation bar also allows mounting several cameras in parallel in similar positions. Section 3 discusses several 2D/3D camera systems that had been mounted and evaluated in this test setup.

Due to the multi-purpose aluminum profile framework, the mounting is, however, also flexible enough to mount additional cameras and other sensors anywhere according to various specific needs.

This test platform is supplemented with a ground truth sensor system that allows to track and record the occupant’s movement at high frame rates, synchronously with the 2D and 3D video stream of the camera system monitoring the cabin of the driving simulator. The 3D tracking system chosen for that purpose is OptiTrack [12]. OptiTrack is a motion capture system that works with IR cameras detecting small reflective markers on the subject’s body joints. It has often been used as a reference system in the past (e.g. [22]). Fig. 2(e) and 2(f) show the OptiTrack cameras mounted around the driving simulator in a way that each tracking point on the subject is visible at all times. With this tracking mechanism, it is possible to record the position of each joint for every point in time with sub-millimeter accuracy. This leads to an easy and precise automated testing of algorithms. Fig. 3 shows a skeleton preview of the OptiTrack software.

A metal frame holding the simulator projectors is rigidly fixed to the vehicle mock-up and defines the so-called world coordinate system. Fig. 2(g) shows the designated calibration board that can be mounted on the simulator frame. It offers a checkerboard for camera calibration as well as a calibration square for the ground truth sensor system. This setup enables extrinsic calibration of both the installed camera and the ground truth sensor system in relation to the world coordinate system. The precisely mounted board solution guarantees precise calibration and position verification. The calibration board can be unmounted from the frame to make way for undisturbed data recording.

Synchronization between the camera and ground truth sensor system is realized via a synchronization wire. The ground truth system sends periodic pulses with exposure frequency as soon as data recording is started. The camera is triggered with these pulses and gets re-synchronized to avoid drift.

The positioning of the driver and passenger seat in the mock-up is controlled via the CAN bus system of the seats which originate from a real automobile (see Fig. 2(h)). This access via CAN guarantees the full traceability of the seat position while testing and also enables a precise seat adjustment according to predefined positions. Another very important aspect of the testing or development of deep learning algorithms is to test and train them with a wide variability of data. For depth imagery, the most interesting and realistic way to add data variability to the depth data is by changing the seat position. The CAN control of the seats provides the full range of seat positions with a set of four degrees of freedom: seat height, seat position, backrest position, and seat tilt. Fig. 4 schematically shows these variables. In addition to that, the steering wheel position is also alterable, though not controlled via the CAN-interface. With that variability, it is possible to cover the full range of possible real-world seat positions by a comprehensive pre-defined set of parameter combinations. The influence of the seat pose variation on the in-cabin scenery recorded by the selected 2D/3D-camera is discussed further below in Section 3 (see also Fig. 6).

In order to monitor the entire interior of the vehicle mock-up from a mounting position corresponding to the overhead module of the car close to the rear mirror (see Section 2.2), a camera field of view of at least 120° is required. The depth-sensing range has to cover a range of 25cm to 200cm. Also, the camera needs to capture data at a high framerate of at least 30Hz to be capable to track person’s movements.

Several depth cameras available on the market have been evaluated against these requirements. Tab. 1 summarizes the main optical characteristics of the reviewed cameras.

Fig. 5 shows example RGB-D images of the various generations of Kinect cameras. The Kinect cameras are consumer cameras also commonly used in the research community, e.g. to record benchmark datasets [4]. The cameras have been mounted on the rear mirror position as described in section 2.2 (see Fig. 2(c) and 2(d)).

One recognizes in Fig. 5 that only the latest generation Azure Kinect provides a complete view of the cabin interior from the selected rear mirror perspective. The field of view of the Kinect v1 is too small to monitor the driver fully: The left arm is not visible. The Kinect v2 has a larger field of view, but the driver’s hands are too close to the camera for valid depth measurement. The other available depth sensors show limitations according to the first two generations of the Kinect, as one can already recognize from their datasheets (see Tab. 1), and were therefore not further considered for the in-cabin test setup.

With these results, we decided that the Azure Kinect is the favorable camera to use. Although it has some limitations, it is the only camera that is capable of recording the whole driver’s body. On the downside, the IR transmitter is not strong enough to provide sufficient signal everywhere on the low-remitting black leather seat for valid depth measurement.

As discussed in Section 2.5, the position of the mock-up seats can be varied over the full range of four degrees of freedom. In order to demonstrate the influence of the seat positioning on the depth image, we have recorded a scene with a driver and an empty seat with a set of pre-defined seat positions (see Fig. 6). For that purpose, each of the four parameters was set to both extreme positions while the others were kept at mean positions. One recognizes that the backrest inclination (top row in Fig. 6) has a large influence on the depth image. The measured depth values, as well as the scale and shape of both driver and empty seat, change significantly when under variation of the backrest inclination. Varying the height or position of the seat changes mainly the depth and thus the scale of the objects in the image, while a change of the tilt of the seat base has almost no influence on the depth image. These observations are important for designing an optimal test matrix covering the full range of scene variations with a manageable amount of test cases.

By monitoring the entire vehicle cabin, the DFKI test platform is supporting the development of multi-seat occupancy detection, including occupant classification, driver recognition, and object detection functions. The robustness of novel deep learning approaches for occupancy classification based on augmented and fully synthetic data will be tested against the variability of the scene in driving scenarios.

We use SVIRO [2], a large-scale synthetic dataset that provides multi-modal image data for scenarios within the passenger compartment of ten different vehicles. Fig. 7 shows an example depth image from the dataset. To show the promise of our test platform, we have trained a simple object detector for the task of detecting people on SVIRO synthetic depth images and evaluated it on real depth images captured in our driving simulator. Our model is based on popular Faster R-CNN [14] architecture with all training parameters the same as in the original paper. We used 10k synthetic images for training and let the model train until no further decrease in the loss was observed. To test our model, we recorded a driving scenario with one person in the driving seat in our driving simulator. We achieve a mean AP score of 88.6% for class ’person’ with our simple detector. Fig. 7(b) shows the results of our trained detector on an example image captured with our test platform.

The introduced driving simulator was used in the acquisition of the AutoPOSE dataset [17]. AutoPOSE is a new head pose and eye gaze targets dataset. An IR camera was located at the driver’s dashboard, giving almost a frontal view of the driver’s face. A Kinect v2 was placed at the location of the center mirror (rear mirror) of the car providing 3 image types, IR, depth, and RGB images. The subjects total number was 21 (10 females and 11 males). The dashboard IR camera subset consists of 1,018,885 IR images and the Kinect subset consists of 316,497 synchronized RGB, depth, and IR images. As mentioned in Section 2.3, we used the submillimeter accurate OptiTrack motion capturing system for accurate and reliable tracking data acquisition that can be used as ground truth. The driver’s head coordinate system and the cameras were calibrated and synchronized with the tracking system. Besides having the driver’s head pose, also the eye gaze targets were acquired. The subjects were asked to gaze at reflective markers placed at driving-related locations, for example, side mirrors, center rear mirror, dashboard, road view, and media center. Fig. 8 shows samples of the dataset from the IR camera perspective and the Kinect camera perspective. Please refer to the dataset paper for further details.

3D body pose tracking is the basis for a comprehensive activity recognition of the vehicle occupants, and also supports a robust analysis of the driver’s state and availability. Since it is possible to equip the subject with as many tracking markers as needed, a broad range of applications can be tested with individual joint positioning.

A first deep learning approach towards a full 3D body pose tracking in a car based on a spherical camera was published in [10] and [3] and was already demonstrated in real-time in the driving simulator (Fig. 9). Comprehensive recording of driving scenarios with ground truth data will allow to quantitatively evaluate these algorithms.

Another recent research activity has been the development of novel deep-learning-based hand pose and gesture recognition in a car [20]. It was shown that special recurrent neural networks allow the recognition of more complex hand gestures than what is currently available as a state-of-the-art sensor system. The setup for this study was a time-of-flight camera installed in the rear mirror of a vehicle. The test platform allows an extension of this study towards the recognition of arm gestures and activities. The test platform offers moreover the possibility to investigate approaches to fuse the visual information with other sensor modalities, as, e.g. vehicle driving parameters, for a robust intention recognition of the driver.

In the future, the DFKI in-cabin test platform will be used to record and annotate a large-scale database of in-cabin scenes under realistically simulated driving scenarios. The database will support the benchmark testing of a range of monitoring functions as those described above, in the first line seat occupancy classification, occupant size, and 3D-pose estimation, as well as driver activity and intention recognition. Apart from the ground truth data recording for the pose estimation, the data needs to be annotated with various, application-specific attributes, including also a 2D- and 3D- labeling of the object location, size, and orientation. Therefore, semi-automated annotation tools are under development.