This study conducts a comprehensive reproducibility analysis of Brain-Computer Interface (BCI) systems using open electroencephalography (EEG) datasets. The primary objective is to assess existing solutions and establish open and reproducible benchmarks for effective comparison within the field. The study involves re-implementing and evaluating 30 machine learning pipelines across 36 publicly available datasets, including motor imagery (MI), P300, and Steady State Visually Evoked Potential (SSVEP). The analysis incorporates statistical meta-analysis techniques to assess execution time and environmental impact. Key findings include: - Riemannian approaches using spatial covariance matrices exhibit superior performance, highlighting the need for significant data volumes to achieve competitive outcomes with deep learning techniques. - The comprehensive results are openly accessible, facilitating future research to enhance reproducibility in the BCI domain. - The study contributes to establishing a rigorous and transparent benchmark for BCI research, offering insights into optimal methodologies and emphasizing the importance of reproducibility. The paper also discusses the challenges in BCI research, such as the lack of standardized evaluation protocols and the environmental impact of machine learning techniques. It provides detailed methodologies for the benchmark, including the inclusion of analysis pipelines, evaluation methods, and statistical analysis. The study covers different BCI paradigms and evaluates various machine learning pipelines, providing a detailed overview of the datasets and pipelines considered. The results demonstrate the dominance of Riemannian approaches and the importance of reproducibility in BCI research.This study conducts a comprehensive reproducibility analysis of Brain-Computer Interface (BCI) systems using open electroencephalography (EEG) datasets. The primary objective is to assess existing solutions and establish open and reproducible benchmarks for effective comparison within the field. The study involves re-implementing and evaluating 30 machine learning pipelines across 36 publicly available datasets, including motor imagery (MI), P300, and Steady State Visually Evoked Potential (SSVEP). The analysis incorporates statistical meta-analysis techniques to assess execution time and environmental impact. Key findings include: - Riemannian approaches using spatial covariance matrices exhibit superior performance, highlighting the need for significant data volumes to achieve competitive outcomes with deep learning techniques. - The comprehensive results are openly accessible, facilitating future research to enhance reproducibility in the BCI domain. - The study contributes to establishing a rigorous and transparent benchmark for BCI research, offering insights into optimal methodologies and emphasizing the importance of reproducibility. The paper also discusses the challenges in BCI research, such as the lack of standardized evaluation protocols and the environmental impact of machine learning techniques. It provides detailed methodologies for the benchmark, including the inclusion of analysis pipelines, evaluation methods, and statistical analysis. The study covers different BCI paradigms and evaluates various machine learning pipelines, providing a detailed overview of the datasets and pipelines considered. The results demonstrate the dominance of Riemannian approaches and the importance of reproducibility in BCI research.