This paper addresses the challenges in evaluating 3D medical image segmentation, such as metric selection, multiple definitions of metrics, and inefficiency in large datasets. It presents an overview of 20 evaluation metrics selected through a comprehensive literature review, providing unified definitions for metrics with inconsistent definitions. The paper also introduces efficient algorithms for calculating these metrics, particularly for large 3D image segmentations, and proposes an open-source implementation tool named EvaluateSegmentation. This tool supports fuzzy segmentations and multiple labels, and is optimized for speed and memory usage. The paper discusses the properties of different metrics, provides guidelines for selecting appropriate metrics, and highlights the importance of standardization in medical image segmentation evaluation. The tool is designed to handle large datasets, such as whole-body MRI or CT volumes, and is available as an open-source project.This paper addresses the challenges in evaluating 3D medical image segmentation, such as metric selection, multiple definitions of metrics, and inefficiency in large datasets. It presents an overview of 20 evaluation metrics selected through a comprehensive literature review, providing unified definitions for metrics with inconsistent definitions. The paper also introduces efficient algorithms for calculating these metrics, particularly for large 3D image segmentations, and proposes an open-source implementation tool named EvaluateSegmentation. This tool supports fuzzy segmentations and multiple labels, and is optimized for speed and memory usage. The paper discusses the properties of different metrics, provides guidelines for selecting appropriate metrics, and highlights the importance of standardization in medical image segmentation evaluation. The tool is designed to handle large datasets, such as whole-body MRI or CT volumes, and is available as an open-source project.