This study presents the largest EEG-based Brain-Computer Interface (BCI) reproducibility analysis using open-source datasets, aiming to assess existing solutions and establish open, reproducible benchmarks for effective comparison in the field. The need for such benchmarks arises from the rapid industrial progress that has led to proprietary solutions and the dense, often hard-to-reproduce scientific literature. The study re-implements and evaluates 30 machine learning pipelines across 36 publicly available datasets, including motor imagery (MI), P300, and SSVEP. Statistical meta-analysis techniques are used to assess results, considering execution time and environmental impact. The results show that Riemannian approaches using spatial covariance matrices outperform deep learning techniques, emphasizing the need for large data volumes. The comprehensive results are openly accessible, supporting future research to enhance BCI reproducibility. The study contributes to establishing a rigorous and transparent benchmark for BCI research, offering insights into optimal methodologies and highlighting the importance of reproducibility. The paper also addresses the environmental impact of machine learning, emphasizing the need for sustainable models. The methodology includes open-source tools for analysis, standardized evaluation procedures, and statistical analysis. The study evaluates three common BCI paradigms: MI, ERP, and SSVEP, and considers various machine learning pipelines, including raw signal, Riemannian, and deep learning approaches. The results show that Riemannian pipelines outperform others across all datasets and paradigms, demonstrating their superior performance. The study provides a detailed analysis of benchmark results, offering guidelines for proposing new machine learning pipelines and datasets. The results are summarized in tables and figures, highlighting the effectiveness of Riemannian approaches in BCI tasks.This study presents the largest EEG-based Brain-Computer Interface (BCI) reproducibility analysis using open-source datasets, aiming to assess existing solutions and establish open, reproducible benchmarks for effective comparison in the field. The need for such benchmarks arises from the rapid industrial progress that has led to proprietary solutions and the dense, often hard-to-reproduce scientific literature. The study re-implements and evaluates 30 machine learning pipelines across 36 publicly available datasets, including motor imagery (MI), P300, and SSVEP. Statistical meta-analysis techniques are used to assess results, considering execution time and environmental impact. The results show that Riemannian approaches using spatial covariance matrices outperform deep learning techniques, emphasizing the need for large data volumes. The comprehensive results are openly accessible, supporting future research to enhance BCI reproducibility. The study contributes to establishing a rigorous and transparent benchmark for BCI research, offering insights into optimal methodologies and highlighting the importance of reproducibility. The paper also addresses the environmental impact of machine learning, emphasizing the need for sustainable models. The methodology includes open-source tools for analysis, standardized evaluation procedures, and statistical analysis. The study evaluates three common BCI paradigms: MI, ERP, and SSVEP, and considers various machine learning pipelines, including raw signal, Riemannian, and deep learning approaches. The results show that Riemannian pipelines outperform others across all datasets and paradigms, demonstrating their superior performance. The study provides a detailed analysis of benchmark results, offering guidelines for proposing new machine learning pipelines and datasets. The results are summarized in tables and figures, highlighting the effectiveness of Riemannian approaches in BCI tasks.