AutoDRIVE: A Comprehensive, Flexible and Integrated Digital Twin Ecosystem for Enhancing Autonomous Driving Research and Education

AutoDRIVE’s native vehicle, named Nigel (refer Fig. 2-A and Fig. 2-B) offers realistic driving and steering actuation, comprehensive sensor suite, high-performance computational resources, and a standard vehicular lighting system.

AutoDRIVE offers a modular and reconfigurable infrastructure development kit (refer Fig. 3) for rapidly designing and prototyping custom driving scenarios. This kit includes a range of environment modules, traffic elements and surveillance elements, along with several preconfigured maps.

The vehicle is jointly modelled (refer Fig. 4-A) as a rigid-body and a collection of sprung masses ${}^{i}M$, such that the total mass of the rigid-body is $M=\sum{{}^{i}M}$. The rigid-body center of mass $X_{COM}=\frac{\sum{{{}^{i}M}*{{}^{i}X}}}{\sum{{}^{i}M}}$ is what links the said representations; where ${}^{i}X$ are the sprung mass coordinates.

The suspension force acting on each of the sprung masses is ${{}^{i}M}*{{}^{i}{\ddot{Z}}}+{{}^{i}B}*({{}^{i}{\dot{Z}}}-{{}^{i}{\dot{z}}})+{{}^{i}K}*({{}^{i}{Z}}-{{}^{i}{z}})$; where, ${}^{i}Z$ and ${}^{i}z$ are the displacements of sprung and unsprung masses, and ${}^{i}B$ and ${}^{i}K$ are the damping and spring coefficients of $i$-th suspension, respectively.

The wheels of the vehicle are also modelled as rigid-bodies of mass $m$, acted upon by gravitational and suspension forces: ${{}^{i}m}*{{}^{i}{\ddot{z}}}+{{}^{i}B}*({{}^{i}{\dot{z}}}-{{}^{i}{\dot{Z}}})+{{}^{i}K}*({{}^{i}{z}}-{{}^{i}{Z}})$.

The tire forces are computed based on the friction curve for each tire: $\left\{\begin{matrix}{{}^{i}F_{t_{x}}}=F(^{i}S_{x})\\ {{}^{i}F_{t_{y}}}=F(^{i}S_{y})\\ \end{matrix}\right.$; where, ${}^{i}S_{x}$ and ${}^{i}S_{y}$ are the longitudinal and lateral slips of $i$-th tire, respectively. Here, the friction curve is approximated by a two-piece cubic spline $F(S)=\left\{\begin{matrix}f_{0}(S);\;\;S_{0}\leq S<S_{e}\\ f_{1}(S);\;\;S_{e}\leq S<S_{a}\\ \end{matrix}\right.$; where, $f_{k}(S)=a_{k}*S^{3}+b_{k}*S^{2}+c_{k}*S+d_{k}$ is a cubic polynomial function. The first segment of the said spline starts at zero $(S_{0},F_{0})$ and reaches the extremum point $(S_{e},F_{e})$, while the other segment starts at the extremum point $(S_{e},F_{e})$ and saturates at the asymptote point $(S_{a},F_{a})$, as shown in the inset of Fig. 4-A.

Now, the tire slip is itself affected by various factors including tire stiffness ${}^{i}C_{\alpha}$, steering angle $\delta$, wheel speeds ${}^{i}\omega$, suspension forces ${}^{i}F_{s}$, and rigid-body momentum ${}^{i}P$, all of which affect the longitudinal/lateral/both components of the linear velocity of the vehicle. Longitudinal slip ${}^{i}S_{x}$ of $i$-th tire is computed by comparing the longitudinal components of surface velocity of $i$-th wheel (i.e., longitudinal linear velocity of vehicle) $v_{x}$ with the angular velocity ${}^{i}\omega$ of $i$-th wheel: ${{}^{i}S_{x}}=\frac{{{}^{i}r}*{{}^{i}\omega}-v_{x}}{v_{x}}$. Lateral slip ${}^{i}S_{y}$ of $i$-th tire is dependent on the direction the tire is pointing in and the direction it is moving in, commonly called the slip angle $\alpha$. It is computed by comparing the longitudinal $v_{x}$ (a.k.a. forward velocity) and lateral $v_{y}$ (a.k.a. side-slip velocity) components of vehicle’s linear velocity: ${{}^{i}S_{y}}=tan(\alpha)=\frac{v_{y}}{\left|v_{x}\right|}$.

The simulated vehicle is provided with the same sensing modalities as its real-world counterpart. Particularly, the throttle ($\tau$) and steering ($\delta$) sensors are simulated through a simple feedback loop.

The incremental encoders are simulated by measuring the rotation of rear wheels (i.e., output shaft of driving actuators): ${}^{i}N_{ticks}={{}^{i}PPR}*{{}^{i}GR}*{{}^{i}N_{rev}}$; where, ${}^{i}N_{ticks}$ represents ticks measured by $i$-th encoder, ${}^{i}PPR$ is the base resolution (pulses per revolution) of $i$-th encoder, ${}^{i}GR$ is the gear-ratio of $i$-th motor, and ${}^{i}N_{rev}$ represents the number of revolutions of output shaft of $i$-th motor.

The IPS and IMU are simulated based on temporally-coherent rigid-body transform updates of the vehicle $\{v\}$ w.r.t. the world $\{w\}$: ${{}^{w}\mathbf{T}_{v}}=\left[\begin{array}[]{c | c}\mathbf{R}_{3\times 3}&\mathbf{t}_{3\times 1}\\ \hline\cr\mathbf{0}_{1\times 3}&1\end{array}\right]\in SE(3)$. The IPS provides 3-DOF positional coordinates $\{x,y,z\}$ of the vehicle, whereas the IMU provides linear accelerations $\{a_{x},a_{y},a_{z}\}$, angular velocities $\{\omega_{x},\omega_{y},\omega_{z}\}$ and 3-DOF orientation of the vehicle as Euler angles $\{\phi_{x},\theta_{y},\psi_{z}\}$ or quaternion $\{q_{0},q_{1},q_{2},q_{3}\}$.

The LIDAR is simulated using iterative ray-casting raycast{${}^{w}\mathbf{T}_{l}$, $\vec{\mathbf{R}}$, $r_{max}$} $\forall\;\theta\in\left[\theta_{min}:\theta_{res}:\theta_{max}\right]$ at $\sim$7 Hz update rate; where, ${{}^{w}\mathbf{T}_{l}}={{}^{w}\mathbf{T}_{v}}*{{}^{v}\mathbf{T}_{l}}\in SE(3)$ is the relative transform of LIDAR {$l$} w.r.t. vehicle {$v$} w.r.t. world {$w$}, $\vec{\mathbf{R}}=\left[r_{max}*sin(\theta)\;\;r_{min}*cos(\theta)\;\;0\right]^{T}$ is the direction vector of each ray-cast $R$, $r_{min}=$ 0.15 m and $r_{max}=$ 12 m are respectively the minimum and maximum linear ranges of LIDAR, $\theta_{min}=0^{\circ}$ and $\theta_{max}=360^{\circ}$ are respectively the minimum and maximum angular ranges of LIDAR, and $\theta_{res}=1^{\circ}$ is the angular resolution of LIDAR. The laser scan ranges are recorded by checking the ray-cast hits and thresholding the minimum linear range of LIDAR: ranges[i]$=\begin{cases}\texttt{hit.dist}&\text{ if }\texttt{ray[i].hit}\text{ and }\texttt{hit.dist}\geq r_{min}\\ \infty&\text{ otherwise}\end{cases}$; where, ray.hit is a Boolean flag that checks if a ray-cast hits any colliders in the scene and hit.dist$=\sqrt{(x_{hit}-x_{ray})^{2}+(y_{hit}-y_{ray})^{2}+(z_{hit}-z_{ray})^{2}}$ is the Euclidean distance from source of the ray-cast $\{x_{ray},y_{ray},z_{ray}\}$ to the hit-point $\{x_{hit},y_{hit},z_{hit}\}$.

The simulated physical cameras are parameterized by their focal length ($f=$ 3.04 mm), sensor size ($\{s_{x},s_{y}\}=$ {3.68, 2.76} mm), target resolution (default = 720p) as well as distance to near and far clipping planes ($N=$ 0.01 m and $F=$ 1000 m). The viewport rendering pipeline for simulated cameras works in 3 stages. First, the camera view matrix $\mathbf{V}\in SE(3)$ is computed by taking the relative homogeneous transform of the camera $\{c\}$ w.r.t. the world $\{w\}$: $\mathbf{V}=\begin{bmatrix}r_{00}&r_{01}&r_{02}&t_{0}\\ r_{10}&r_{11}&r_{12}&t_{1}\\ r_{20}&r_{21}&r_{22}&t_{2}\\ 0&0&0&1\\ \end{bmatrix}$; where, $r_{ij}$ and $t_{i}$ denote rotational and translational components, respectively. Next, the camera projection matrix $\mathbf{P}\in\mathbb{R}^{4\times 4}$ is computed by projecting the world coordinates to image space coordinates: $\mathbf{P}=\begin{bmatrix}\frac{2*N}{R-L}&0&\frac{R+L}{R-L}&0\\ 0&\frac{2*N}{T-B}&\frac{T+B}{T-B}&0\\ 0&0&-\frac{F+N}{F-N}&-\frac{2*F*N}{F-N}\\ 0&0&-1&0\\ \end{bmatrix}$; where, $N$ and $F$ respectively denote distances to near and far clipping planes of the camera, and $L$, $R$, $T$ and $B$ respectively denote the left, right, top and bottom offsets of the sensor. The camera parameters $\{f,s_{x},s_{y}\}$ are related to the projection matrix terms through the following relations: $f=\frac{2*N}{R-L}$, $a=\frac{s_{y}}{s_{x}}$, $\frac{f}{a}=\frac{2*N}{T-B}$. The perspective projection from the simulated camera’s viewport is given by $\mathbf{C}=\mathbf{P}*\mathbf{V}*\mathbf{W}$; where, $\mathbf{C}=\left[x_{c}\;\;y_{c}\;\;z_{c}\;\;w_{c}\right]^{T}$ represents the image space coordinates and $\mathbf{W}=\left[x_{w}\;\;y_{w}\;\;z_{w}\;\;w_{w}\right]^{T}$ represents the world coordinates. Finally, this camera projection is converted into normalized device coordinates (NDC) by performing perspective divide (i.e., dividing throughout by $w_{c}$), obtaining a viewport projection by scaling and shifting the result and then using the rasterization process of the graphics API (e.g., DirectX for Windows, Metal for macOS and Vulkan for Linux). Additionally, a post-processing step simulates lens and film effects of the camera such as lens distortion, depth of field, exposure, ambient occlusion, contact shadows, bloom, motion blur, film grain, chromatic aberration, etc.

The vehicle is actuated using two driving actuators and a steering actuator, the response delays and saturation limits of which are matched with their real-world counterparts by tuning their torque profiles and actuation limits, respectively.

The driving actuators drive the rear wheels by applying a torque: ${{}^{i}\tau_{drive}}={{}^{i}I_{w}}*{{}^{i}\dot{\omega}_{w}}$; where, ${{}^{i}I_{w}}=\frac{1}{2}*{{}^{i}m_{w}}*{{}^{i}{r_{w}}^{2}}$ is the moment of inertia, ${}^{i}\dot{\omega}_{w}$ is the angular acceleration, ${}^{i}m_{w}$ is the mass and ${}^{i}r_{w}$ is the radius of $i$-th wheel. Additionally, the holding torque of the driving actuators is simulated by applying an idle motor torque equivalent to the braking torque: ${{}^{i}\tau_{idle}}={{}^{i}\tau_{brake}}$.

The front wheels are steered using a steering actuator, which produces a torque proportional to the required angular acceleration: $\tau_{steer}=I_{steer}*\dot{\omega}_{steer}$. The individual turning angles, $\delta_{l}$ and $\delta_{r}$, for left and right wheels, respectively, are calculated based on the commanded steering angle $\delta$, using the Ackermann steering geometry defined by wheelbase $l$ and track width $w$, as follows: $\left\{\begin{matrix}\delta_{l}=\textup{tan}^{-1}\left(\frac{2*l*\textup{tan}(\delta)}{2*l+w*\textup{tan}(\delta)}\right)\\ \delta_{r}=\textup{tan}^{-1}\left(\frac{2*l*\textup{tan}(\delta)}{2*l-w*\textup{tan}(\delta)}\right)\end{matrix}\right.$

Simulated environments can be set-up in one of the following ways:

• •

  AutoDRIVE IDK: The modular and reconfigurable infrastructure development kit (IDK) can be used to create custom scenarios and maps by setting up the terrain modules, road networks, obstruction modules and traffic elements. These assets are present within the simulator source files. Particularly, preconfigured maps depicted in Fig. 3-F-(i), Fig. 3-F-(iii) and Fig. 3-F-(iv) have been constructed using the AutoDRIVE IDK.

• •

  Plug-In Scenarios: AutoDRIVE Simulator supports third-party tools (e.g., RoadRunner [25]) and modular open-source architecture (MOSA) standards (e.g., OpenSCENARIO [26], OpenDRIVE [27], etc.) that enable extensibility. Additionally, users can import a wide array of plugins, packages and assets in a variety of industry-standard formats (e.g., FBX, OBJ, SKP, 3DS, USD, etc.) for developing or customizing driving scenarios. Furthermore, graphics textures designed for AutoDRIVE Testbed can be first imported into the simulator before large-scale printing and setup in real-world. Particularly, preconfigured map depicted in Fig. 3-F-(ii) has been designed using a third-party graphics editing software, imported in AutoDRIVE Simulator and finally printed and setup using AutoDRIVE Testbed.

• •

  Unity Terrain: Being built atop the Unity game engine, AutoDRIVE Simulator natively supports scenario design and development using Unity Terrain [28]. Users can define the terrain mesh, texture, heightmap, vegetation, skybox, wind, etc. to design on-road/off-road scenarios and perform variability testing.

At every time step, the simulator performs mesh-mesh interference detection and computes the contact forces, frictional forces and momentum transfer, along with linear and angular drag acting on the rigid-bodies (refer Fig. 4-B).

AutoDRIVE Simulator is developed atop Unity [29] game engine, which employs PhysX [30] for simulating multi-threaded framerate-independent kinematics and dynamics of all the physical entities, and exploits High Definition Render Pipeline (HDRP) [31] along with Post-Processing Stack [32] for rendering photorealistic graphics.

The simulator features an interactive graphical user interface (GUI) consisting of Menu Panel on left-hand side and Heads-Up Display (HUD) on right-hand side. Fig. 4-C depicts the simulator’s GUI with both the panels enabled. The Menu Panel hosts input fields and buttons to configure and control various features of the simulator (refer Fig. 4-D). This includes controls for the communication bridge, along with a series of buttons for (a) toggling between manual and autonomous driving modes for the ego vehicle; (b) switching between the available scene cameras, each providing a distinct view; (c) altering the graphics quality to match the quality-performance trade-off; (d) toggling the scene light to simulate day and night driving conditions; (e) resetting the scene to initial conditions; and (f) quitting the simulator application. The HUD Panel, on the other hand, displays prominent simulation parameters along with vehicle status and sensory data in real-time. It also hosts a time-synchronized data recording functionality, which can be used to export vehicle as well as infrastructure data for a specific run, thereby fostering data-driven approaches aimed at autonomous driving and smart city management.

The simulator natively supports C# scripting, which can be leveraged to customize existing and/or introduce new features, functionalities, modules, behaviors, physics, graphics, communication bridges, APIs and even setup co-simulation frameworks with other simulation tools.

Finally, it is worth mentioning that the simulator, in its source form, has been integrated with various plugins and packages such as the Unity ML Agents Toolkit [33], a machine learning framework for developing and deploying deep imitation/reinforcement learning-based applications directly from within the simulator.

The autonomous driving software stack (ADSS) aids in development of autonomy algorithms specifically targeting the vehicle. It can be used to develop single as well as multi-agent autonomous driving algorithms.

The smart city software stack (SCSS) aids in development of autonomy algorithms specifically targeting the infrastructure. It can work in tandem with ADSS to develop smart city applications pertaining to traffic management.

This demonstration leveraged AutoDRIVE’s ROS-enabled capabilities to demonstrate autonomous parking (refer Fig. 5-A). First, the vehicle mapped its surroundings using Hector SLAM algorithm [35] (refer Fig. 5-B). It could then localize itself against this known static map using range-flow-based odometry [36] (refer Fig. 5-C) and adaptive particle filter algorithm [37] (refer Fig. 5-D). For autonomous navigation, the vehicle planned a feasible global path from its current pose to parking pose using A* algorithm [38], while also re-planning its local trajectory for dynamic collision avoidance using timed-elastic-band approach [39]. A proportional controller generated driving (throttle/brake) and steering commands for the vehicle to track the local trajectory (refer Fig. 5-E). Future work in this direction can include benchmarking of various SOTA and/or novel algorithms for mapping, localization, path planning and motion control.

This demonstration was based on [40], wherein the objective was to employ a convolutional neural network (CNN) [41] for cloning the end-to-end driving behavior of a human. As such, AutoDRIVE Simulator was exploited to record 5-laps worth of temporally-coherent labeled manual driving data, which was balanced, augmented and pre-processed using standard computer vision techniques, to train a 6-layer deep CNN (refer Fig. 6-A). Post training for 4 epochs, with a learning rate of 1e-3 using the Adam optimizer [42], the training and validation losses converged stably without over/under-fitting. The trained model was then deployed back into the virtual world to validate its performance through activation and prediction analyses (refer Fig. 6-B). Further, the same model was transferred to AutoDRIVE Testbed for validating sim2real capability of the ecosystem through activation and prediction analyses (refer Fig. 6-C). In order to have a zero-shot sim2real transition: (i) the physical and visual aspects of virtual world were setup as close to the real world as possible, and (ii) the exhaustive data augmentation pipeline implicitly performed domain randomization. However, further investigation is required to comment on and improve the robustness of sim2real transfer considering sensor simulation, vehicle modeling and scenario representation. Finally, although this work adopted a coupled-control law for vehicle motion smoothing, a potential improvement could be to investigate independent actuation smoothing techniques like low-pass filters or proximal bounds.

Inspired from [43], this work demonstrates single and multi-agent (refer Fig. 7-A) intersection traversal using deep reinforcement learning (refer Fig. 7-B). Each agent collected a vectorized observation $o_{t}^{i}=\left[g^{i},\tilde{p}^{i},\tilde{\psi}^{i},\tilde{v}^{i}\right]_{t}$, including its relative goal location $g_{t}^{i}$, along with relative location $\tilde{p}_{t}^{i}$, relative yaw $\tilde{\psi}_{t}^{i}$ and velocity $\tilde{v}_{t}^{i}$ of its peers obtained through V2V communication. The action space of each agent, $a_{t}^{i}$, included discretized steering $\delta_{t}^{i}\in\left\{-1,0,1\right\}$ and constant throttle ($\tau_{t}^{i}=80\%$). An extrinsic reward function $r_{t}^{i}=\begin{cases}r_{goal}=+1\\ r_{collision}=-0.425*\left\|g_{t}^{i}\right\|_{2}\end{cases}$ kept agents in check while training a 3-layer fully-connected neural network based policy, $\pi_{\theta}\left(a_{t}|o_{t}\right)$, using PPO algorithm [44] (refer Fig. 7-C). This ultimately resulted in all agents being able to traverse the intersection safely (refer Fig. 7-D). Although we could not implement this application in real-world owing to monetary constraints, investigating the sim2real transition of this application would be a natural progression of this work. Additionally, application of actuation smoothing techniques could be a potential improvement.

This novel use case of smart city traffic management was possible due to AutoDRIVE’s V2I and IoT capabilities. As depicted in Fig. 8-A), the SCM server hosted a database to keep track of all the traffic elements, and acted as a high-level behavior planner for the ego vehicle. It switched the vehicle behavior upon detecting respective traffic signs and lights by setting the appropriate throttle and steering trims, which were then passed on to proximally optimal predictive (POP) controller coupled with adaptive longitudinal controller (ALC) [45]. Finally, SCM server teleoperated the vehicle to achieve the mission objective (refer Fig. 8-B). A natural progression of this work would be to implement a multi-agent scenario, preferably within a mixed-reality digital-twin setting, to investigate different strategies for efficient smart city traffic management.