The paper introduces Ch³Ef, a comprehensive dataset and evaluation strategy for assessing the alignment of multimodal large language models (MLLMs) with human values. The dataset contains 1002 human-annotated samples across 12 domains and 46 tasks, based on the principles of being helpful, honest, and harmless (hhh). It also presents a unified evaluation strategy that supports assessments across various scenarios and perspectives. The evaluation results reveal over 10 key findings, deepening the understanding of MLLM capabilities, limitations, and the dynamic relationships between evaluation levels. The dataset and evaluation codebase are available at https://openlamm.github.io/ch3ef/. The paper discusses the challenges of aligning MLLMs with human values, particularly in complex visual scenarios and the difficulty of collecting relevant data. It categorizes MLLM evaluations into three levels: A1 (semantic alignment), A2 (logical alignment), and A3 (human value alignment). A3 focuses on whether models can mirror human-like engagement in the visual world while understanding human expectations and preferences. The dataset and evaluation strategy are designed to address these challenges, providing a structured methodology for assessing MLLMs' alignment with human values. The evaluation strategy includes three components: Instruction, Inferencer, and Metric, enabling varied assessments from different perspectives across scenarios ranging from A1 to A3. The dataset is manually curated based on hhh criteria, and the evaluation process involves human-machine synergy to annotate 1002 QA pairs across various visual contexts. The results show that open-source MLLMs face challenges in A3, with lower performance in helpful, honest, and harmless dimensions. The study also highlights the importance of balancing safety and engagement in AI interactions and suggests that reinforcement learning from human feedback (RLHF) can improve human-value alignment. The paper concludes that Ch³Ef provides a comprehensive dataset and evaluation strategy for assessing MLLMs' alignment with human values, contributing to a deeper understanding of MLLM capabilities, limitations, and the dynamics between evaluation levels. Future research should focus on enhancing MLLMs' alignment with human values, promoting their effectiveness and ethical integration into various applications. The study also acknowledges limitations, including the potential for the defined dimensions and annotated data to not encompass all real-world scenarios, and the reliance on probabilistic methods for option selection.The paper introduces Ch³Ef, a comprehensive dataset and evaluation strategy for assessing the alignment of multimodal large language models (MLLMs) with human values. The dataset contains 1002 human-annotated samples across 12 domains and 46 tasks, based on the principles of being helpful, honest, and harmless (hhh). It also presents a unified evaluation strategy that supports assessments across various scenarios and perspectives. The evaluation results reveal over 10 key findings, deepening the understanding of MLLM capabilities, limitations, and the dynamic relationships between evaluation levels. The dataset and evaluation codebase are available at https://openlamm.github.io/ch3ef/. The paper discusses the challenges of aligning MLLMs with human values, particularly in complex visual scenarios and the difficulty of collecting relevant data. It categorizes MLLM evaluations into three levels: A1 (semantic alignment), A2 (logical alignment), and A3 (human value alignment). A3 focuses on whether models can mirror human-like engagement in the visual world while understanding human expectations and preferences. The dataset and evaluation strategy are designed to address these challenges, providing a structured methodology for assessing MLLMs' alignment with human values. The evaluation strategy includes three components: Instruction, Inferencer, and Metric, enabling varied assessments from different perspectives across scenarios ranging from A1 to A3. The dataset is manually curated based on hhh criteria, and the evaluation process involves human-machine synergy to annotate 1002 QA pairs across various visual contexts. The results show that open-source MLLMs face challenges in A3, with lower performance in helpful, honest, and harmless dimensions. The study also highlights the importance of balancing safety and engagement in AI interactions and suggests that reinforcement learning from human feedback (RLHF) can improve human-value alignment. The paper concludes that Ch³Ef provides a comprehensive dataset and evaluation strategy for assessing MLLMs' alignment with human values, contributing to a deeper understanding of MLLM capabilities, limitations, and the dynamics between evaluation levels. Future research should focus on enhancing MLLMs' alignment with human values, promoting their effectiveness and ethical integration into various applications. The study also acknowledges limitations, including the potential for the defined dimensions and annotated data to not encompass all real-world scenarios, and the reliance on probabilistic methods for option selection.