This paper reviews representative techniques, algorithms, and theoretical foundations for trustworthy distributed AI through robustness guarantee, privacy protection, and fairness awareness in distributed learning. It discusses the inherent vulnerabilities of distributed AI systems in terms of security, privacy, and fairness, and provides a unique taxonomy of countermeasures for trustworthy distributed AI. The paper first provides an overview of alternative architectures for distributed learning, then discusses the vulnerabilities of AI algorithms in distributed learning, and analyzes why these problems are present in distributed learning regardless of specific architectures. It then provides a comprehensive review of robustness countermeasures to mitigate evasion attacks and irregular queries at inference, as well as poisoning attacks, Byzantine attacks, and irregular data distribution during federated learning. Next, it reviews representative privacy-enhancing techniques to enable privacy-preserving distributed learning against different types of unauthorized data leakages. It also discusses privacy leakage and deception problems in distributed multi-armed bandit learning systems and the representative countermeasures. Finally, it examines different sources of bias and important fairness-enhancing techniques. The paper concludes with a discussion on open challenges and future research directions toward trustworthy distributed AI, including the need for governance policy guidelines, responsibility-utility co-design, and incentives and compliance. The paper argues that robustness, privacy, and fairness are three critical pillars for ensuring and strengthening the trustworthiness of distributed AI. It also discusses the importance of data governance and model governance in ensuring robustness, privacy, and fairness in distributed AI systems. The paper provides a visualization metaphor in Figure 1 to illustrate the taxonomy of surveyed approaches in trustworthy distributed AI. The paper also discusses the challenges of safeguarding data-in-use in distributed AI systems, including the vulnerability of data-in-use to memory dumps and other malicious exploits. The paper reviews robustness countermeasures, privacy-preserving methods, and fairness-enhancing techniques, and describes how data governance and model governance can put humans in the loop of distributed model training and model deployment to strengthen the enforcement of trustworthy AI through explainable and responsible guidelines. The paper also discusses the importance of robustness, privacy, and fairness in ensuring the trustworthiness of distributed AI systems.This paper reviews representative techniques, algorithms, and theoretical foundations for trustworthy distributed AI through robustness guarantee, privacy protection, and fairness awareness in distributed learning. It discusses the inherent vulnerabilities of distributed AI systems in terms of security, privacy, and fairness, and provides a unique taxonomy of countermeasures for trustworthy distributed AI. The paper first provides an overview of alternative architectures for distributed learning, then discusses the vulnerabilities of AI algorithms in distributed learning, and analyzes why these problems are present in distributed learning regardless of specific architectures. It then provides a comprehensive review of robustness countermeasures to mitigate evasion attacks and irregular queries at inference, as well as poisoning attacks, Byzantine attacks, and irregular data distribution during federated learning. Next, it reviews representative privacy-enhancing techniques to enable privacy-preserving distributed learning against different types of unauthorized data leakages. It also discusses privacy leakage and deception problems in distributed multi-armed bandit learning systems and the representative countermeasures. Finally, it examines different sources of bias and important fairness-enhancing techniques. The paper concludes with a discussion on open challenges and future research directions toward trustworthy distributed AI, including the need for governance policy guidelines, responsibility-utility co-design, and incentives and compliance. The paper argues that robustness, privacy, and fairness are three critical pillars for ensuring and strengthening the trustworthiness of distributed AI. It also discusses the importance of data governance and model governance in ensuring robustness, privacy, and fairness in distributed AI systems. The paper provides a visualization metaphor in Figure 1 to illustrate the taxonomy of surveyed approaches in trustworthy distributed AI. The paper also discusses the challenges of safeguarding data-in-use in distributed AI systems, including the vulnerability of data-in-use to memory dumps and other malicious exploits. The paper reviews robustness countermeasures, privacy-preserving methods, and fairness-enhancing techniques, and describes how data governance and model governance can put humans in the loop of distributed model training and model deployment to strengthen the enforcement of trustworthy AI through explainable and responsible guidelines. The paper also discusses the importance of robustness, privacy, and fairness in ensuring the trustworthiness of distributed AI systems.