13 Jul 2024 | Li Qiao, Mahdi Boloursaz Mashhadi, Zhen Gao, Chuan Heng Foh, Pei Xiao, Mehdi Bennis
This paper proposes a latency-aware and channel-adaptive semantic communication framework using pre-trained foundation generative AI models. The framework enables ultra-low-rate, low-latency, and channel-adaptive semantic communications by leveraging pre-trained generative models for semantic decomposition and synthesis. The transmitter performs multi-modal semantic decomposition on the input signal, extracting a textual prompt and conditioning signals. These semantic streams are then transmitted with appropriate coding and communication schemes based on the communication intent. A re-transmission-based scheme is used for the prompt to ensure reliable transmission, while adaptive modulation/coding schemes are used for other semantic modalities to achieve robustness to the changing wireless channel. A semantic and latency-aware scheme is designed to allocate transmission power to different semantic modalities based on their importance, subject to semantic quality constraints. At the receiver, a pre-trained generative model synthesizes a high-fidelity signal using the received multi-stream semantics. Simulation results demonstrate the effectiveness of the proposed framework in achieving ultra-low-rate, low-latency, and channel-adaptive semantic communications. The framework is particularly suitable for scenarios requiring communication of huge multi-modal data with stringent latency and reliability requirements, such as the wireless metaverse, extended/mixed reality (XR/MR), holographic teleportation, and the internet of senses. The key contributions include a semantic decomposition scheme that extracts the most important semantic contents as a compact textual message or prompt, a multi-stream scheme that transmits each extracted semantic modality with appropriate coding and communication techniques, and a semantic and latency-aware scheme for power allocation and modulation order adaptation. The framework also demonstrates the effectiveness of using pre-trained models for semantic communication, reducing the need for shared knowledge bases and enabling a separation-based architecture. The results show that the proposed framework achieves better semantic quality and lower latency compared to semantic-unaware transmission benchmarks. The framework is also evaluated for computation latency, showing that it is comparable to transmission latency, emphasizing the need to minimize transmission latency for reduced end-to-end latency.This paper proposes a latency-aware and channel-adaptive semantic communication framework using pre-trained foundation generative AI models. The framework enables ultra-low-rate, low-latency, and channel-adaptive semantic communications by leveraging pre-trained generative models for semantic decomposition and synthesis. The transmitter performs multi-modal semantic decomposition on the input signal, extracting a textual prompt and conditioning signals. These semantic streams are then transmitted with appropriate coding and communication schemes based on the communication intent. A re-transmission-based scheme is used for the prompt to ensure reliable transmission, while adaptive modulation/coding schemes are used for other semantic modalities to achieve robustness to the changing wireless channel. A semantic and latency-aware scheme is designed to allocate transmission power to different semantic modalities based on their importance, subject to semantic quality constraints. At the receiver, a pre-trained generative model synthesizes a high-fidelity signal using the received multi-stream semantics. Simulation results demonstrate the effectiveness of the proposed framework in achieving ultra-low-rate, low-latency, and channel-adaptive semantic communications. The framework is particularly suitable for scenarios requiring communication of huge multi-modal data with stringent latency and reliability requirements, such as the wireless metaverse, extended/mixed reality (XR/MR), holographic teleportation, and the internet of senses. The key contributions include a semantic decomposition scheme that extracts the most important semantic contents as a compact textual message or prompt, a multi-stream scheme that transmits each extracted semantic modality with appropriate coding and communication techniques, and a semantic and latency-aware scheme for power allocation and modulation order adaptation. The framework also demonstrates the effectiveness of using pre-trained models for semantic communication, reducing the need for shared knowledge bases and enabling a separation-based architecture. The results show that the proposed framework achieves better semantic quality and lower latency compared to semantic-unaware transmission benchmarks. The framework is also evaluated for computation latency, showing that it is comparable to transmission latency, emphasizing the need to minimize transmission latency for reduced end-to-end latency.