This paper presents an experimental study on federated learning (FL) on non-IID data silos. With increasing privacy concerns and data regulations, training data are increasingly fragmented into distributed databases of multiple "data silos." FL enables multiple parties to collaboratively train a machine learning model without exchanging raw data. However, data distributions among parties are often non-IID, posing significant challenges for FL algorithms. Existing studies have focused on rigid data partitioning strategies, which are not representative or thorough. To address this, we propose comprehensive data partitioning strategies to cover typical non-IID cases and conduct extensive experiments to evaluate state-of-the-art FL algorithms. Our experiments show that non-IID data significantly affects FL accuracy, and no single algorithm outperforms others in all cases. We introduce a benchmark called NIIDBench with six non-IID data partitioning strategies, including label distribution skew, feature distribution skew, and quantity skew. We evaluate four FL algorithms (FedAvg, FedProx, SCAFFOLD, and FedNova) on nine datasets. Our results provide insights for future FL research, highlighting the importance of comprehensive benchmarks and the need for algorithms that can handle mixed types of skew. We also find that FedProx and SCAFFOLD perform well in certain settings, while FedNova is less effective. The study emphasizes the importance of considering non-IID data distributions when designing FL algorithms and highlights the challenges of achieving stable and efficient learning in distributed data silos.This paper presents an experimental study on federated learning (FL) on non-IID data silos. With increasing privacy concerns and data regulations, training data are increasingly fragmented into distributed databases of multiple "data silos." FL enables multiple parties to collaboratively train a machine learning model without exchanging raw data. However, data distributions among parties are often non-IID, posing significant challenges for FL algorithms. Existing studies have focused on rigid data partitioning strategies, which are not representative or thorough. To address this, we propose comprehensive data partitioning strategies to cover typical non-IID cases and conduct extensive experiments to evaluate state-of-the-art FL algorithms. Our experiments show that non-IID data significantly affects FL accuracy, and no single algorithm outperforms others in all cases. We introduce a benchmark called NIIDBench with six non-IID data partitioning strategies, including label distribution skew, feature distribution skew, and quantity skew. We evaluate four FL algorithms (FedAvg, FedProx, SCAFFOLD, and FedNova) on nine datasets. Our results provide insights for future FL research, highlighting the importance of comprehensive benchmarks and the need for algorithms that can handle mixed types of skew. We also find that FedProx and SCAFFOLD perform well in certain settings, while FedNova is less effective. The study emphasizes the importance of considering non-IID data distributions when designing FL algorithms and highlights the challenges of achieving stable and efficient learning in distributed data silos.