Large Generative Model Assisted 3D Semantic Communication

First, in the transmitter, we use the UE to capture images $\mathbf{I}_{\text{M}}$ of a 3D scenario $\chi^{\text{3D}}$ from various perspectives. Then, we apply SAM to enable users to select a key 3D object from one of the multi-perspective images $\mathbf{I}_{\text{M}}$ according to their specific goals. Subsequently, we employ mask inverse rendering technology to obtain multi-perspective images of the selected 3D object, denoted as $\mathbf{I}_{\text{M}}^{\prime}$, which we can consider as the semantics of the raw 3D scenario. This process of semantic extraction is illustrated in Algorithm 2. Finally, at the receiver end, we leverage NeRF to construct the 3D scenario based on the recovered multi-perspective images of the goal-oriented 3D object in $\hat{\chi}^{\text{3D}}$.

For the multi-perspective images of the goal-oriented 3D object, we utilize the ASCM to transmit them one by one. The training and inference process of ASCM is described in Algorithm 3. With the semantic encoder of ASCM, the images are transformed into semantic information in the latent feature space. However, DL-based semantic encoders can only reduce pixel-level redundancy and are unable to reduce feature-level redundancy. Hence, our semantic encoder also outputs a mask array to mask redundant feature-level data in the semantic information. This process reduces the volume of the transmitted data, consequently reducing communication costs.

Before executing ASCM, we feed a smaller pilot sequence into the trained CGAN, thus we can estimate the CSI of the physical channel. Due to the prior constraints on CGAN, the generated CSI images often lack fine details. Therefore, an additional DM is employed to enhance the predicted details of CSI. The training and inference steps of GDCE are described in Algorithm 4. This approach helps us better perform the recovery of signals in the receiver.

For better understanding, we summarize the proposed GAM-3DSC system in Algorithm 1.

Firstly, we capture images $\mathbf{I}_{\text{M}}$ from multiple perspectives of a static 3D scenario $\chi^{\text{3D}}$ using UE. Subsequently, we define a neural network $F_{\xi}$ as:

where $\mathbf{x}=(x,y,z)$ signifies the coordinates of a point in three-dimensional space, $\mathbf{d}$ represents the observation direction (expressed in spherical coordinates), $\mathbf{c}$ indicates the color (expressed as RGB values), and $\nu$ denotes the density (expressed as a scalar).

Next, for each pixel in every image, we compute a ray $\mathbf{r}(t)$ originating from the camera based on the camera parameters:

where $\mathbf{o}$ denotes the origin of the light (i.e., the position of the camera), $t$ is the parameter on the ray (expressed as a scalar), and $\mathbf{d}$ signifies the direction of the ray (i.e., the direction corresponding to the pixel). Following this, we sample several points on the ray and input these points along with their corresponding directions into the neural network $F_{\xi}$ to obtain predictions of color $\mathbf{c}(\mathbf{r}(t),\mathbf{d})$ and density $\nu(\mathbf{r}(t))$.

Subsequently, based on the color and density predictions, we calculate the final color $C(\mathbf{r})$ of the ray via volume rendering:

where $T(t)=\exp(-\int_{t_{n}}^{t}\nu(\mathbf{r}(s))ds)$ can be interpreted as the probability that the light traveling from $t_{n}$ to $t$ is not obstructed by any object. $t_{n}$ and $t_{f}$ denote the near and far bounds of the ray, respectively.

Lastly, we calculate the loss function by comparing this value with the actual color of the image:

where $M$ represents the number of images; $P$ denotes the number of pixels in each image; $C_{m}(\mathbf{r}_{p})$ is the true color of the $p$-th pixel on the $m$-th image; $\hat{C}_{m}(\mathbf{r}_{p})$ is the predicted color of the $p$-th pixel on the $m$-th image. By implementing optimization methods such as backpropagation and the Stochastic Gradient Descent (SGD) algorithm [28], we can update the parameters $\xi$ of the neural network $F_{\xi}$ to minimize the loss function.

To allow users to freely select a 3D object of interest from the 3D scenario $\chi^{\text{3D}}$, we use SAM to perform semantic segmentation on the image from a specific perspective. Next, we will introduce the workflow of SAM [12, 29].

Firstly, we provide a specific single-perspective image $\mathbf{I}_{\text{S}}\in\mathbf{I}_{\text{M}}$ as input, along with a prompt $\mathbf{P}$. This prompt can be either text or 2D points and used to identify the objectives for segmentation. When the prompt is in text, we need to convert it into points. For example, if the text prompt is “flower”, the conversion result would be the center point coordinates of the flower.

Subsequently, we feed the input image $\mathbf{I}_{\text{S}}$ and prompts $\mathbf{P}$ into a neural network $F_{\Gamma}$:

where $\mathbf{M}$ refers to the generated mask with a shape of $(H,W)$. Each element of $\mathbf{M}$ is either 0 or 1, indicating whether the pixel is part of the target object or not. $\mathbf{S}$ is the Intersection over Union (IoU) score, which depicts the intersection ratio between the mask and the actual annotation. $\mathbf{L}$ is a category label, denoting the category of the target object. The predicted category label $\hat{\mathbf{L}}$ can be computed as follows:

where $\zeta(\cdot)$ denotes a function that generates class label predictions based on mask predictions $\hat{\mathbf{M}}$, IoU score predictions $\hat{\mathbf{S}}$, and prompt $\mathbf{P}$.

Finally, after the training process, SAM is capable of generating precise mask predictions.

Since the mask generated by SAM is 2D, we utilize mask inverse rendering to project this 2D mask into 3D space, resulting in a 3D mask. Note the 3D mask refers to the masks of the images from all perspectives. Technically, we represent the 3D mask as voxel grids $V\in\mathbb{R}^{L\times W\times H}$, where each grid vertex maintains a zero-initialized soft mask confidence score. Using these voxel grids, we render each pixel of the 2D mask from a specific single perspective [30]:

where $\omega(\mathbf{r}(t))$ is derived from the density values of the pre-trained NeRF; $\mathbf{U}(\mathbf{r}(t))$ signifies the mask confidence score at location $\mathbf{r}(t)$, obtained from voxel grids $\mathbf{U}^{2}$. We denote $\mathbf{M}_{\text{SAM}}(\mathbf{r})$ as the corresponding mask generated by SAM under ray $\mathbf{r}$. When $\mathbf{M}_{\text{SAM}}(\mathbf{r})=1$, the objective of mask inverse rendering is to enhance $\mathbf{U}(\mathbf{r}(t))$ in relation to $\omega(\mathbf{r}(t))$. In practice, we can optimize this by using the SGD algorithm. To serve this purpose, we define the mask projection loss as the negative product between $\mathbf{M}_{\text{SAM}}(\mathbf{r})$ and $\mathbf{M}_{\text{3D}}(\mathbf{r})$:

where $\mathcal{R}(\mathbf{I}_{\text{S}})$ denotes the ray set of the image $\mathbf{I}_{\text{S}}$.

We formulate the mask projection loss based on the assumption that both the geometry from the NeRF and the segmentation results of SAM are accurate. However, in real-world situations, this is not always the case. As a result, we add a negative refinement term to the loss function to optimize the 3D mask grids in line with multi-perspective mask consistency:

where $\lambda$ serves as a hyperparameter to govern the magnitude of the negative term. In each iteration, we update the 3D mask $\mathbf{U}$ by employing the SGD algorithm to minimize $\mathcal{L}_{\text{proj }}$. After obtaining the 3D mask $\mathbf{U}$, we multiply it with $\mathbf{I}_{\text{M}}$ to obtain $\mathbf{I}_{\text{M}}^{\prime}$:

For SAM, we can utilize the pre-trained weight presented in [12], thus we only need to train the NeRF in 3DSE. We summarize the 3DSE approach in Algorithm 2.

In conclusion, there are two advantages to 3DSE: (1) We employ NeRF to render the 3D scenario, requiring only a few images from different perspectives. This implies that we merely transmit some 2D images when transferring 3D scenario data, thereby reducing the cost and complexity of communications. (2) We utilize SAM to allow users to select a 3D object of interest in the 3D scenario from a specific perspective image, thereby attaining accurate semantic extraction that aligns with human perception.

To extract the semantics from the single-perspective image $\mathbf{I}_{\text{S}}^{\prime}$ ($\in\mathbf{I}_{\text{M}}^{\prime}$) of the selected 3D object, we employ a DL network as the semantic encoder of ASCM. Transformer networks [31] have proven more effective than traditional Convolutional Neural Networks (CNNs), due to their superior performance in various domains such as computer vision, natural language processing, and speech processing. Therefore, we utilize the transformer networks as the semantic encoder. Firstly, we convert $\mathbf{I}_{\text{S}}^{\prime}$ into a sequence of one-dimensional patch embeddings via a PatchEmbed layer [32]. Then, the transformer networks extract semantic features from the embeddings. The transformer networks consist of multiple stacked attention layers and utilize the following attention formula:

where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ are all obtained by linear transformation of the input embeddings; $d$ is the adjustment factor; $\operatorname{Softmax}(\cdot)$ is an activation function.

To further reduce communication costs, we propose eliminating redundant data in the semantic features. In the semantic encoder, we design two output heads, called semantic head and mask head. The semantic head output the full semantics $\mathbf{E}_{s}^{+}$, and the mask head output a mask array $\mathbf{M}_{s}=[m_{1},m_{2},..,m_{i},...,m_{S}]$, where $S$ signifies the length of $\mathbf{E}_{s}^{+}$. Here, either $m_{i}=0$ or $m_{i}=1$, and $m_{i}$ determines whether each feature in $\mathbf{E}_{s}^{+}$ is reserved. When the $i$-th semantic feature in $\mathbf{E}_{s}^{+}$ is retained, then $m_{i}=1$ otherwise $m_{i}=0$. Thus, the compressed semantic information $\mathbf{E}_{s}^{-}=\mathbf{E}_{s}^{+}\odot\mathbf{M}_{s}$.

To enable semantic information to be transmitted over wireless channels, we construct the channel encoder and decoder based on the MLP networks. The channel encoder is used to modulate and encode $\mathbf{E}_{s}^{-}$. Then, the result of the encoded semantic information can be given by:

where $\sigma(\cdot)$ represents an activation function; $\mathbf{w}_{t}$ is the weight matrix of the channel encoder; $\mathbf{b}_{t}$ is the bias.

According to Eq. (2), after transmission on the wireless channel, $\mathbf{X}$ changes into $\mathbf{Y}$. The channel decoder performs demodulating $\mathbf{Y}$ and removes the impairments from the channel. Then, the decoding results of the channel decoder can be expressed as:

where $\mathbf{w}_{r}$ is the weight matrix of the channel decoder; $\mathbf{b}_{r}$ represent the bias.

The semantic decoder performs decoding the received semantics $\hat{\mathbf{E}}_{s}^{-}$, where the semantic decoder is also constructed based on the transformer networks. The decoded result is the reconstructed single-perspective image $\mathbf{\hat{I}}_{\text{S}}^{\prime}$. When the transmission of the images from all perspectives is completed, we get the reconstructed multi-perspective images $\mathbf{\hat{I}}_{\text{M}}^{\prime}$.

To ensure that ASCM learns to compress semantic information without compromising the quality of image reconstruction, we implement an SKD-based training approach. As illustrated in Fig. 5, we consider the ASCM without semantic compression (i.e., there is no mask output head in the semantic encoder) as the “teacher”, while the ASCM with semantic compression (i.e., there is a mask output head in the semantic encoder) serves as the “student”. The task loss for the teacher can be expressed as follows:

where $\mathbf{\hat{I}}_{\text{S}}^{\prime,+}$ is the reconstructed image without semantic compression in the transmission; $\mathcal{L}_{\text{MSE}}(\cdot)$ represents the mean square error. Hence, $\mathcal{L}_{\text{task}}^{\text{tech}}$ represents the reconstruction loss of the ASCM with complete semantic information and provides task-specific supervision for the teacher. Similarly, the task loss for the student can be given by:

where $\mathbf{\hat{I}}_{\text{S}}^{\prime,-}$ is the reconstructed image with semantic compression in the transmission.

During training, the teacher transfers the learned knowledge to the student, thus directing the student to learn well. This process can be represented as:

where ${\operatorname{KL}}(\cdot)$ means the Kullback–Leibler divergence, i.e., ${\rm{KL}}(\mathbf{a},\mathbf{b})=-\sum_{i}\mathbf{a}_{i}\log\left(\mathbf{b}_{i}/\mathbf{a}_{i}\right)$; $\hat{\mathbf{E}}_{s}^{+}$ represents the reconstructed semantic information with the uncompressed semantics $\mathbf{E}_{s}^{+}$; $\hat{\mathbf{E}}_{s}^{-}$ is the reconstructed semantics corresponding to the compression semantics $\mathbf{E}_{s}^{-}$. $\mathcal{L}_{\text{KD}}$ is proposed to minimize the difference between the two reconstructed semantic information. Here, $\hat{\mathbf{E}}_{s}^{-}$ is expected to as possible as close to $\hat{\mathbf{E}}_{s}^{+}$. The $\mathcal{L}_{\text{task}}^{\text{tech}}+\mathcal{L}_{\text{task}}^{\text{stu}}$ is used to adjust the $\operatorname{KL}(\hat{\mathbf{E}}_{s}^{+},\hat{\mathbf{E}}_{s}^{-})$, adaptively.

By employing the SGD algorithm to minimize the aforementioned loss functions, we can effectively update the parameters of ASCM. Assuming that the number of training epochs is $J$, we summarize the ASCM in Algorithm 3.

The benefits of ASCM can be encapsulated as follows: (1) We design two output heads in the semantic encoder to extract semantic features while performing semantic compression, eliminating the extra feature-level semantic information and reducing the communication cost; (2) We employ an SKD-based training method to ensure that the performance of ASCM with semantic compression approximates as closely as possible to that achieved with complete semantics, thus effectively maintaining the accuracy of the semantics.

Firstly, we can denote the received signal $\mathbf{Y}$, the pilot sequence $\mathbf{\Theta}$, and channel matrix $\mathbf{H}$ as dual-channel images [34]. The two channels represent the real and imaginary parts of a complex matrix. From this perspective, we can redefine the problem of channel estimation as an image-to-image generative task.

Secondly, within the CGAN, the generator uses the pilot sequence $\mathbf{\Theta}$ and the received signal $\mathbf{Y}$ as input to estimate $\mathbf{H}$, denoted as $\hat{\mathbf{H}}$. In this case, the pilot sequence $\mathbf{\Theta}$ is used as the condition. The goal of the discriminator is to distinguish between the real and generated CSI, with the pilot sequence $\mathbf{\Theta}$ serving as the condition.

Thirdly, CGAN aims for the generator to synthesize highly realistic CSI that can deceive the discriminator. Correspondingly, the discriminator tries to improve its discernment capabilities so it is not easily tricked. To optimize this process, we apply a least-squares GAN loss function [35] that incorporates conditional information into consideration. The loss functions for both discriminator $D$ and generator $G$ are expressed below:

Additionally, to ensure the CSI generated by the generator closely aligns with the real CSI at a pixel level, we incorporate an L1 loss into the loss function of the generator:

where $\rVert\cdot\rVert$ denotes the L1 loss function.

Due to the prior constraints on CGAN, the generated CSI images often lack fine details. Therefore, an additional DM is employed to enhance the predicted details of CSI $\hat{\mathbf{H}}$. Therefore, we employ a DM to further refine the CSI and thereby enhance its accuracy.

The training of DM encompasses two procedures: a forward process (diffusion process) and a reverse process (generation process). The forward process involves gradually adding Gaussian noise $\mathbf{z}$ to the predicted CSI $\hat{\mathbf{H}}$ until it transforms into pure noise $\hat{\mathbf{H}}_{T}$. Each step adheres to the following Markov chain:

where $\alpha_{t}=1-\beta_{t}$ is a predefined decreasing sequence that satisfies $\beta_{1}<\beta_{2}<\cdots<\beta_{T}$; $\hat{\mathbf{H}}_{t}$ is the intermediate sample at the $t$-th step. This ensures that each step has the same noise diffusion amplitude.

The reverse process is the procedure of progressively reconstructing the original image $\mathbf{H}$ from the pure noise $\hat{\mathbf{H}}_{T}$. Each step involves a neural network predicting a denoising function $p(\hat{\mathbf{H}}_{t-1}|\hat{\mathbf{H}}_{t})$, designed to fit the true posterior distribution $q(\hat{\mathbf{H}}_{t-1}|\hat{\mathbf{H}}_{t},\hat{\mathbf{H}}_{0})$.

We regard the refined CSI is denoted as $\hat{\mathbf{H}}_{\text{ref}}$, while the mean of the denoising function $p(\hat{\mathbf{H}}_{t-1}|\hat{\mathbf{H}}_{t})$ is represented as $\mu_{\theta}(\hat{\mathbf{H}}_{t},t)$, where $\theta$ represents the parameters of DM. $\mathbf{z}$ represents Gaussian noise. We assume the training epoch of GDCE is $O$. Finally, we summarize the training and inference stages of GDCE in Algorithm 4.

The GDCE has two benefits as follows: (1) We employ a CGAN to estimate the CSI based on pilot sequences and the received signals. This process achieves the transition of channel effects from multiplicative noise to additive noise and thus help the recovery of signals in the receiver. (2) We utilize a DM to refine the CSI produced by CGAN, with an aim to enhance CSI image details. This further improves the accuracy of the obtained CSI.

We implement 3DSE on the LLFF and mip-NeRF360 datasets [30], comprised of various images of 3D scenarios captured from different angles. Subsequently, we employ two forms of prompts (i.e., points and text) to evaluate the performance of 3D semantic extraction.

Fig. 7 presents partial results of semantic extraction by 3DSE. The first box on the left demonstrates two forms of prompts: one is selecting the 3D object of interest using several points, and the other is providing the name of the 3D object in text form. The second box displays the 2D mask corresponding to the user’s prompts. The third box showcases the 3D mask derived from mask inverse rendering, illustrating that 3DSE can obtain masks of all multi-perspective images based solely on a mask from a single-perspective image. The final box presents the multi-perspective images of the selected 3D object, which constitute the transmitted image data.

In summary, the results demonstrate that the implemented 3DSE can accurately extract the intended 3D object (i.e., key semantics) based on the user prompts. The utilization of NeRF and SAM can provide robust capabilities for 3D object processing and image segmentation.

Firstly, we utilize the multi-perspective images extracted from the LLFF, mip-NeRF360, and LERF datasets by 3DSE to train the ASCM.

Secondly, to highlight the advantages of the transformer architecture in ASCM, we compare it with several other architecture-based SC methods. These include Convolutional Autoencoder-based SC (CAE-SC), Variational Autoencoder-based SC (VAE-SC), and Vision Transformer-based SC (ViT-SC). Apart from ASCM, all other methods transmit complete semantic information and employ the SGD algorithm for updates. Except for ASCM, the rest models adopt the Deconvolution Networks (DCNNs) [36] architecture as the semantic decoder. For a more comprehensive understanding, we have summarized the architectures of these methods in Table I.

Thirdly, the evaluation metrics include loss, Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index Measure (SSIM). PSNR serves as a measure of the quality of a reconstructed or compressed image. It is typically expressed in decibels, with higher values denoting superior image quality. The definition of PSNR is as follows:

where $\mathrm{MAX}_{I}$ denotes the maximum possible pixel value of the image, which is typically 255 for an 8-bit image. $\mathrm{MSE}(\cdot)$ represents the average squared difference between the original image $\mathbf{I}_{\text{S}}^{\prime}$ and the reconstructed image $\mathbf{\hat{I}}_{\text{S}}^{\prime}$. Similarly, SSIM is a metric that gauges the perceived similarity between two images, factoring in three key components - luminance, contrast, and structure. The definition of SSIM is outlined as follows:

where $\varphi_{\mathbf{I}_{\text{S}}^{\prime}}$ and $\varphi_{\mathbf{\hat{I}}_{\text{S}}^{\prime}}$ are their means; $\phi_{\mathbf{I}_{\text{S}}^{\prime}}^{2}$ and $\phi_{\mathbf{\hat{I}}_{\text{S}}^{\prime}}^{2}$ are their variances; $\phi_{\mathbf{I}_{\text{S}}^{\prime}\mathbf{\hat{I}}_{\text{S}}^{\prime}}$ is their covariance; $c_{1}$ and $c_{2}$ are two constants used to avoid division by zero.

Finally, we consider two types of channels, namely AWGN and fading (Rician channel is used in this paper) channels in the data transmission. Signal-to-Noise Ratio (SNR) range is from 0 dB to 25 dB. The training epoch is set to 40.

Fig. 8 and 9 illustrate the evaluation results of various SC models under AWGN and fading channels, respectively. Initially, as depicted in Fig. 8(a) and Fig. 9(a), the final convergence results of the proposed ASCM outperform the CAE-SC and VAE-SC schemes, but are slightly inferior to the ViT-SC scheme. Subsequently, Fig. 8(b) and Fig. 9(b) demonstrate that the SC models attain a comparable PSNR under both channels when the SNR is sufficiently high. It is observable that the CAE-SC and VAE-SC schemes perform poorly, while the ASCM and ViT-SC schemes perform better. Additionally, the performance gap between the ASCM and ViT-SC schemes is narrower on the fading channel than on the AWGN channel. Lastly, Fig. 8(c) and Fig. 9(c) indicate that the various SC schemes obtain similar results to those in Fig. 8(b) and Fig. 9(b) when the metric is SSIM. This illustrates that the ASCM can ensure consistency of the sent and received images at the pixel level.

The superior performance of the ViT-SC and ASCM schemes results from the benefits of the transformer architecture, which extracts more precise semantic information compared to the CNN and MLP architectures. Since the ViT-SC scheme does not perform semantic compression, it slightly outperforms ASCM. However, as Table I shows, ASCM reduces the amount of data to be transferred by 80%. This suggests that while ASCM sacrifices some model accuracy, it significantly reduces transmission energy consumption. The SKD-based training method plays an important role in this, as it enables ASCM to reduce the size of transmitted semantics while keeping the most critical semantic information.

Firstly, we simulate a scenario where two devices communicate at a carrier frequency of 24.2 GHz within the 28 GHz millimeter wave band. Following this, we employ the Rician fading channel model as the physical channel to simulate the path loss, utilizing the path loss model in the urban microcell street scenario as per the 3GPP TR 38.901 standards.

Secondly, we conduct the ASCM in a simulated wireless environment, where we gather the output of the channel encoder in ASCM as the pilot sequences and the input of the channel decoder as the received signals. For training the CGAN, we collect 1800 pairs of samples in total.

Then, we compare the proposed GDCE to both traditional channel estimation methods and DL methods. The traditional channel estimation contenders include the Least Squares (LS) channel estimator, Orthogonal Matching Pursuit (OMP), Approximate Message Passing (AMP), and the Minimum Mean Square Error (MMSE) channel estimator. For the DL contenders, we select different architectures-based channel estimators, including U-net, MLP, and CGAN.

Finally, we use the Normalized Mean Square Error (NMSE) as the evaluation metric, which can assess the error between the predicted and actual results. The calculation formula for NMSE is as follows:

where $\operatorname{Var}(\mathbf{H})$ represents the variance of $\mathbf{H}$. The smaller the value of NMSE, the closer the predicted results are to the real results.

Fig. 10 depicts the NMSE of different channel estimation schemes in different SNRs. As SNR improves, so does the performance of each scheme. The proposed GDCE consistently achieves the lowest NMSE across all SNRs, while the LS channel estimator continues to perform the worst. Fig. 11 demonstrates the NMSE of different DL-based channel estimators as the iterations increase. It is apparent that GDCE and CGAN outperform the other methods, highlighting the superiority of the CGAN architecture.

The superior performance of GDCE stems from two factors. Firstly, the CGAN architecture allows us to achieve more realistic CSI even with limited pilots. Secondly, the DM-based refinement strategy removes noise from the generation process of CGAN, thereby improving the accuracy of channel estimation. In conclusion, the effective utilization of the CGAN architecture and the DM-based refinement strategy accounts for the exceptional performance of the GDCE, collectively enhancing channel estimation accuracy.

We employ the LLFF and mip-NeRF360 as our experimental datasets. Specifically, we use the GAM-3DSC to transmit data and recover the 3D scenario. We then assess the difference between the raw and recovered 3D scenarios. As our primary concern is the semantics loss of the selected 3D object, we use the 3DSE to process the raw 3D scenario and generate a new 3D scenario containing only the selected 3D object. We then compare this processed 3D scenario with the recovered one. We utilize the following two evaluation methods:

• •

  Pixel-level evaluation: We adopt the evaluation method proposed in [12], in which we first obtain multiple images from the same perspective in the processed 3D and recovered scenarios respectively. Furthermore, we compare the pixel-level differences between images from the original 3D scenario and those from the reconstructed 3D scenario, captured from the same perspective. The metrics we employ are PSNR and SSIM.

• •

  Semantic-level evaluation: We use LAMs for semantic-level evaluation. Firstly, we adopt the BLIP [37], a large visual language model that unifies the tasks of visual language understanding and generation, to transform the multi-perceptive images into text. Then, we adopt the large language model such as BERT [38] to obtain the embeddings of these texts. Finally, we compare the difference between the embeddings by BLEU score and cosine similarity [39].

Fig. 12 shows the pix-level evaluation results, in which we can see the performance of the GAM-3DSC in terms of the PSNR and SSIM is increased with the improvement of SNR. However, the change of the SNR and SSIM is small between the low and high SNR, which reflects the anti-interference ability against channel noise. Furthermore, the proposed GAM-3DSC system can achieve a PSNR value of approximately 25 dB and an SSIM value of about 0.95. This indicates that, despite potential variations in pixel values arising from differences in brightness, contrast, or color within the reconstructed image, the structural similarity between the original and reconstructed images is substantial, ensuring similarity between images. Fig. 13 showcases the semantic-level evaluation results. We can see similar results in Fig. 12. The BLEU score can reach a maximum of $0.61$ and the cosine similarity can reach a maximum of $0.97$. This shows that although the contrasting texts vary in word composition, their underlying semantics are remarkably similar. These evaluation results confirm that the proposed GAM-3DSC can transmit the 3D object while preserving semantic consistency.