The paper introduces MERBench, a unified evaluation benchmark for multimodal emotion recognition, and MER2023, a new Chinese emotion dataset. MERBench aims to provide a comprehensive and fair comparison of various techniques in multimodal emotion recognition, including feature selection, multimodal fusion, cross-corpus performance, robustness analysis, and language sensitivity analysis. MER2023 is designed to support multi-label learning, noise robustness, and semi-supervised learning research. The paper also discusses the importance of domain compatibility for visual encoders, language matching for acoustic encoders, and the impact of punctuation and additive noise on emotion recognition. The results highlight the necessity of fine-tuning pre-trained models and the effectiveness of attention mechanisms in multimodal fusion. The code for MERBench is available at: https://github.com/zeroQiaoBa/MERTools.The paper introduces MERBench, a unified evaluation benchmark for multimodal emotion recognition, and MER2023, a new Chinese emotion dataset. MERBench aims to provide a comprehensive and fair comparison of various techniques in multimodal emotion recognition, including feature selection, multimodal fusion, cross-corpus performance, robustness analysis, and language sensitivity analysis. MER2023 is designed to support multi-label learning, noise robustness, and semi-supervised learning research. The paper also discusses the importance of domain compatibility for visual encoders, language matching for acoustic encoders, and the impact of punctuation and additive noise on emotion recognition. The results highlight the necessity of fine-tuning pre-trained models and the effectiveness of attention mechanisms in multimodal fusion. The code for MERBench is available at: https://github.com/zeroQiaoBa/MERTools.