Embodied AI-empowered Low Altitude Economy: Integrated Sensing, Communications, Computation, and Control (ISC3)

LAE can be a key position in the modern economic system, and its development will further promote the highly-productive and sustainable development of the economy. According to the report by CCID Consulting⁴⁴4https://www.chinadaily.com.cn/a/202404/09/WS6614977da31082fc043
c0e2b.html, LAE in China surged to 69.9 billion dollars in 2023, a 33.8% year-on-year increase. The market is projected to surpass 1 trillion yuan by 2026. Taking an example, electric vertical take-off and landing (eVTOL) aircraft revenue jumped 77.3% to 980 million yuan in 2023, with wider commercial use expected in 2024. The eVTOL market is forecast to reach 9.5 billion yuan by 2026, driven by accelerated airworthiness certifications. Meanwhile, in Germany, the air taxes investment has raised roughly 13 billion dollars since 2019. The eVTOL makers have raised 2.3 billion dollars so far in 2024, compared with 1.5 billion dollars in 2023⁵⁵5https://money.usnews.com/investing/news/articles/2024-11-25/analysis-liliums-fall-throws-spotlight-on-air-taxi-cash-crunch. Furthermore, according to Amazon company⁶⁶6https://www.aboutamazon.com/news/transportation/amazon-prime-air-drone-delivery-mk30-photos, the newest Prime Air drone, MK30, will deliver to customers in three US locations as well as cities in Italy and the UK by the end of 2024. All above and other statistical data corroborate the importance of LAE. As shown in Fig. 1, LAE primarily involves the use of manned and unmanned civil aircrafts, encompassing activities such as passenger and cargo transportation, as well as related operations and activities⁷⁷7https://www.scmp.com/economy/china-economy/article/3275949/whats-buzz-chinas-low-altitude-economy-and-what-propelling-its-development. Generally, based on the primary functions, LAE’s applications can be diverse such as transportation, agriculture, telecommunication, and public services and safety⁸⁸8https://www.aci-asiapac.aero/media-centre/perspectives/low-altitude-economy-to-fly-high-china-makes-great-strides-in-advanced-air-mobility-development:

• •

  LAE for transportation. Transportation is a crucial element of modern economies. LAE can realize express delivery, provide navigation and location services, transport passengers, and disaster relief. Specifically, within the LAE context, UAVs can provide navigation and localization services and deliver merchandise to customers within 120-300 meters. For longer distances, an eVTOL aircraft can be deployed for transportation within 1000-3000 meters, opening new horizons for urban air mobility. Likewise, helicopters can also assist in disaster rescue, search, and transportation of patients.

• •

  LAE for agriculture. Agriculture is a vital field, providing precious raw materials for food. Other activities include pest control and farm management. In this regard, LAE offers precision farming solutions that can optimize resource use and minimize environmental impact. Specifically, below 120 meters, UAVs can seed, monitor crop health, and perform cost-effective pesticide spraying, contributing to sustainable and productive farming practices.

• •

  LAE for telecommunication. Telecommunication infrastructure is essential for modern society, but it is often lacking or disrupted in remote areas, during natural disasters, or in emergency situations. In this regard, LAE can provide a flexible and adaptable communication solution based on its valuable data and communication capabilities. Specifically, in telecommunication settings, UAVs and aerial platforms can provide data collection, aerial computing, and communication relay services within 120-300 meters⁹⁹9https://www.mskyeye.com/what-is-low-altitude-economy/.

• •

  LAE for public service and safety. LAE can enhance public service and safety¹⁰¹⁰10https://enterprise.dji.com/public-safety through surveillance, security, and emergency response. Specifically, traditional ground-based methods may be limited or inaccessible in disaster, while drones equipped with thermal imaging can aid in locating missing persons in challenging terrains.

Evidently, within the LAE context, aircrafts will execute complicated tasks to provide highly-efficient, cost-effective, and safe operations. These tasks require four major modules, information perception (sensing), information transmission (communication), data processing for decision-making (computation), and flying and landing (control) [9]. Existing techniques, ISAC, integrated sensing and computation (ISC) and integrated communications and computation (ICC), can only support partial above-mentioned complicated tasks with loose performance requirements. This may lead to failures in several real-time tasks with robust and high precision demands. Therefore, it is necessary and important to realize the ISC3 in LAE. To evaluate the importance the of ISC3, we present relationships between LAE applications and its supporting techniques in Fig. 1.

Fig. 2 presents the architecture of EAI-enabled aircrafts. Generally, the EAI-enabled aircraft comprises two main components: an MMLLM-based agent and an embodied entity [5]. The environment consists of both virtual and physical components, interconnected through modeling and sim2real¹¹¹¹11Sim2real is a technique in robotics, computer vision, and AI that aims to connect simulated environments and the real world, enabling the efficient training and development of AI agents and robots. operations. The virtual environment provides training and simulation services for the agent, while the physical environment influences the embodied entity. The embodied entity interacts with the physical environment, gathering data and providing feedback to the agent. The key functions of EAI-enabled aircrafts are organized as follows:

• •

  Perception. Perception acts as the data collector and processor for the EAI-enabled aircraft, enabling it to understand the external environment and support reliable decision-making and taking actions. This is achieved by fusing multi-modal sensing data from various sensors equipped on the embodied entity. For example, with RGB/depth/thermal cameras, LiDAR, radar, GPS, acoustic sensors and magnetometer sensors, EAI can process above multi-modal sensing data by MMLLM, providing navigation, obstacle avoidance, and target tracking services.

• •

  Decision. The decision module acts as the command center of EAI-enabled aircraft. Following perception, it executes advanced task planning and reasoning analysis, generating decision-making instructions to control the embodied entity. This process typically leverages the capabilities of the MMLLM. For example, by imputing multi-modal sensing data to MMLLM, EAI can derive strategies for flying path, transmission power allocation, and processing using machine learning models for UAVs with its inference capacity.

• •

  Action. Action is the execution component of EAI, responsible for receiving decision-making instructions and executing specific actions through the mechanical activity devices of the embodied entity. For the UAV/drone, EAI can control rotor speed, angle, alierons, elevator, and rudder, and also decide the UAV’s data transmission/receiving rates, and onboard computers working model (e.g., CPU’s working power).

Based on above three major functions, EAI can be further applied in UAV swarm systems. It can facilitate better coordination and cooperation between UAVs in a swarm, where UAVs share sensory data, learn from each other¡¯s experiences, and dynamically adjust their roles and tasks based on the needs of the swarm. Furthermore, different from disembodied AI-enabled aircrafts, EAI-enabled aircraft places a strong emphasis on the embodied entity and its interactions with the environment. Through perceiving and actively changing environments with physical interactions, EAI can continuously learn and adapt, leading to improved performance in sensing, communications, computation, and control. For example, perception in EAI enhances sensing capabilities, decision-making guides communication and computation strategies’ making, and action provides feedback for control optimization. Therefore, leveraging EAI techniques to enhance sensing, communications, computation, and control capabilities is of paramount significance and importance, which can further revolutionize the way we utilize the low-altitude airspace and unlock new possibilities for economic growth and innovation.

The framework of EAI-enabled aircraft is composed of an MMLLM and an embodied entity. Specifically, the MMLLM component excels in decision-making and action planning, while the embodied entity facilitates perception through sensor-based data collection. Therefore, this section explores EAI-enabled advancements in sensing, communications, computation, and control issues that could be further applied in LAE. We examine these issues from the perspectives of: 1) MMLLM-based sensing, communications, and computations. How MMLLM components enhance data acquisition, information transfer, and processing. 2) Embodied entity-based control. How the physical embodiment of the aircraft contributes to robust and adaptable control systems. This structured approach allows us to comprehensively analyze how EAI empowers LAE systems, highlighting the synergistic interaction between the AI and physical components.

In summary, EAI can offer significant advancements to LAE across sensing, communications, computing, and control aspects. A comprehensive comparison of traditional approaches and EAI methods in these four areas is presented in Table I. Despite the numerous benefits that EAI can bring to LAE by optimizing sensing, communications, computing, and control performances, its introduction into LAE still faces the following challenges:

1. 1.

   Model pre-training requirements: For ease of deployment, EAI models require large amounts of high-quality data to realize pre-training. Ensuring sufficient data availability and addressing data biases are crucial for effective model development and implementation.

2. 2.

   Malicious and rogue aircrafts concerns: Unauthorized operation of EAI-enabled UAVs poses significant security risks, requiring robust detection and mitigation strategies. This includes measures to prevent unauthorized access and control malicious commands.

3. 3.

   Law and regulation issues: EAI must follow law and regulation strictly. This imposes new constraints to the ISC3 systems. For example, limitations on flight altitudes, operational areas, data collection practices, and communication protocols may be imposed to ensure safety and privacy.

Express delivery in LAE refers to utilizing UAVs to transport goods and packages rapidly, especially for short-distance and time-sensitive deliveries. Compared with traditional ground express delivery, UAVs offer advantages including faster delivery times, greater accessibility to remote areas, and reduced traffic congestion. However, large-scale UAV deployment for express delivery presents unique challenges. These include sensing capacity (e.g., integrating data from multiple sensors), communication quality (e.g., real-time interaction with customers and ground stations), computing efficiency (e.g., processing fused multi-source and multi-modal data), and control accuracy (e.g., obstacle avoidance and precise flight path design). To address these challenges, we need to consider a holistic approach integrating sensing, communications, computation, and control. This accounts for base station distribution, weather and terrain conditions, refining UAV flight paths, and implementing effective multi-source data fusion. By integrating these aspects, we can further enhance UAV-based express delivery services, leading to a more efficient and reliable transport system within LAE.

With the help of EAI, ISC3 can be realized in LAE. Specifically, as shown in Fig. 3, based on the MMLLM[14], we propose an EAI-enabled ISC3 framework for LAE. This framework can be implemented with the following five major steps:

Step 1: Demand generation for ISC3-based LAE. Drawing upon the specific LAE application scenario, we establish performance indicator demands for the sensing, communications, computation, and control aspects. For instance, this involves defining requirements for object detection and localization accuracy, throughput, latency, computing data rate, and flight trajectory parameters.

Step 2: MMLLM-based task cognition. The MMLLM-based agent, informed by the defined LAE scenario, performs cognitive and reasoning operations to derive detailed tasks. These tasks encompass trajectory design, resource allocation, flight control, edge computing, and object tracking.

Step 3: Multi-modal sensing information collection. UAVs equipped with multiple sensors gather multi-modal and multi-source sensing data. This data is fed into the MMLLM for fusion. For example, temperature, humidity, and wind speed sensors provide weather data, visual cameras capture terrain and transportation details, and radar sensors detect obstacles and distances.

Step 4: MMLLM-driven decision making. After receiving ISC3 demands, detailed tasks, and fused multi-modal data, the MMLLM makes corresponding decisions. Generally, image, video, and audio-based multi-modal data are input to the modality encoder of the MMLLM, while performance indicator demands are input as text[14]. Through reasoning and inference, the MMLLM outputs strategies in text format or generates image, video, or audio-formatted results. To be specific, based on the five-layer architecture, MMLLM can make ISC3 demand-based decisions with the following implementation details.

• •

  At first, the input layer is tasked with receiving and preprocessing data from various modalities. For example, for the text data on ISC3 performance demands, the input layer can perform text tokenization and embedding. Image/video/audio-fused multi-modal data is preprocessed by input layer, such as performing scaling, normalization operations.

• •

  Secondly, the feature extraction layer is responsible for extracting high-dimensional feature representations from the preprocessed inputs. For example, for ISC3 performance demands-based text data, pre-trained language models (e.g., BERT or RoBERTa) are commonly used to extract text features, outputting text feature vectors. Besides, for image/video/audio-fused multi-modal data, pre-trained convolutional neural networks (e.g., ResNet or VGG) are used to extract features, outputting feature vectors.

• •

  Then, the multi-modal fusion layer integrates feature vectors from different modalities to generate a unified multi-modal representation. Fusion methods can be categorized into early fusion, late fusion, and intermediate fusion. For example, early/late fusion occurs before/after the feature extraction layer and is suitable for scenarios where modalities have strong relationships. Additionally, the intermediate fusion happens during the middle stages of feature extraction and is suitable for scenarios requiring gradual integration of information from different modalities.

• •

  Next, the context understanding layer further processes the fused multi-modal representation to capture contextual information and interactions between modalities. For example, within an MMLLM, Transformer layers are usually adopted, which employ multi-head self-attention mechanisms to process the fused multi-modal representation, outputting context-aware multi-modal representations.

• •

  Finally, in the output layer. The specific strategies can be derived based on the ISC3 performance demands. For example, in the text format, MMLLM can classify whether the current command can satisfy the ISC3 performance demands or not. Besides, MMLLM can provide newly aircraft controlling strategies to meet ISC3 performance demands by generating text (e.g., commanding code), images (e.g., flying trajectories), or other modal outputs.

Step 5: Embodied entity-oriented control. Based on the obtained strategies, the embodied UAV entity executes corresponding actions to complete the assigned task. This includes adjusting flight trajectory and altitude, selecting communication channels, transmitting at specified power levels, computing using the derived strategy, and following the optimized flight path.

Scenario and goal description. In this case study, we explore an application of EAI to UAV flight route design for package delivery in LAE. Specifically, an EAI-enabled UAV, at its starting point, can access pre-flight information about weather conditions, traffic patterns, terrain features, and base station distribution within the target delivery area. Recognizing that each package delivery point may have unique demands on qualities of ISC3, and the UAV must maintain optimal communication quality, sensing, computation, and control performance, we can formulate the ISC3 optimization problem. This optimization problem aims to minimize UAV route length while adhering to constraints on communication quality, sensing, computation, and control capabilities. The key variable to optimize is the UAV’s flight route. By strategically adjusting the UAV’s flight path, we can improve communication quality, maintain sensing effectiveness, ensure efficient computation, and optimize control accuracy, ultimately enhancing the overall efficiency of UAV-based express delivery services.

Problem formulation and basic idea. To achieve efficient express delivery using an EAI-enabled UAV, we formulate two subproblems: (1) optimizing the UAV’s trajectory based on predefined ISC3 performance constraints (e.g., e.g., data rate of 200kbps, sensing accuracy of 95%, and energy consumption of 200Wh), and (2) solving the resulting optimization problem. The basic idea is as below. The UAV, informed by ISC3 requirements, gathers real-time data (delivery locations, terrain, traffic, weather, and base station distribution) within the operational area. This data is processed to identify critical information (obstacles, delivery stations, optimal delivery counts per flight, and current weather conditions). The UAV then models the express delivery route as a Traveling Salesperson Problem (TSP), minimizing flight path length while adhering to constraints like delivery capacity and flight range. Due to energy limitations, the TSP is solved on an edge server using heuristic algorithms¹²¹²12https://www.crgsoft.com/algorithm-advantages-disadvantages-examples-and-characteristics/ to ensure real-time performance and stability, avoiding the computational overhead of learning-based or convex optimization methods. In this regard, above approach highlights the synergistic benefits of EAI: intelligent data acquisition and analysis, efficient decision-making, and offloading computationally intensive tasks to optimize performance.

Parameter settings. The adopted MMLLM is Pixtral 12B¹³¹³13https://mistral.ai/news/pixtral-12b/. Besides, the number of express stations are set as 10, the UAV can only carry 20 express one time, and the maximum flying distance of the UAV is 75 km one time. The simulation experiment is executed with Matlab 2023b, on a laptop with Intel(R) Core(TM) i7-11700 CPU @ 1.60GHz, NVIDIA GeForce RTX 3060 GPU, 32.0 GB memory and Windows 11 home operation system. More detailed simulation settings and code are available at github¹⁴¹⁴14https://github.com/liukewia/Solving-TSP-VRP.

Results evaluation. Fig. 4 presents the performance evaluation of the proposed framework in an EAI-enabled UAV express delivery scenario. As shown in Figure 4(a), the EAI-enabled UAV effectively gathers necessary sensing data within the task area, including information on delivery locations, terrain, traffic¹⁵¹⁵15https://www.amap.com/, weather¹⁶¹⁶16http://nmc.cn/publish/forecast and base station distribution¹⁷¹⁷17https://www.opengps.cn/Data/Cell/Region.aspx. Then, in Fig. 4(b), the EAI-enabled UAV demonstrates communication and computation optimization capabilities when processing the gathered sensing data through MMLLM. For instance, MMLLM can leverage its inference capacity to formulate the TSP [15], a mathematical problem, and subsequently offload the route computation task to an edge server. This enables the determination of the optimal route for express delivery. Subsequently, Fig. 4(c) illustrates the optimized route of the EAI-enabled UAV in the control aspect. Specifically, Figs 4(c)-1 to 4(c)-4 showcase the solutions obtained from applying different algorithms to the formulated TSP (UAV routing problem with travel distance and capacity constraints): ant colony optimization, hybrid particle swarm optimization, simulated annealing, and genetic algorithm, respectively. The execution times for these algorithms were 0.228723, 0.377555, 0.11304, and 0.300411 seconds, respectively. The shortest route, with a length of 198.801 km, was achieved by the hybrid particle swarm optimization, simulated annealing, and genetic algorithms. These numerical results validate the feasibility of the proposed framework, where EAI can effectively optimize UAV routes, demonstrating its potential for providing efficient express delivery services within the LAE.

Since LAE involves massive data transmission and processing, privacy and security are very important for aircraft security assurance. In this regard, intrusion detection system (IDS)–based approaches can be adopted to defend against jamming, eavesdropping, and Man-in-the-middle (MITM) attacks.

The performance of EAI models is related to the quality and quantity of training data. To address potential data scarcity and improve model robustness, data augmentation techniques are crucial. These techniques can include generative AI-based synthetic data generation, image transformations (rotation, scaling, cropping), and noise injection to create a more diverse and representative training dataset.

Developing intuitive and safe human-machine interfaces that enable effective collaboration between UAVs and human operators is crucial for ensuring safe and efficient operations. This involves designing user-friendly interfaces that allow operators to monitor UAV activities, provide guidance, and intervene when necessary.