What Happens When 168 Teams Analyze the Same EEG Dataset? EEGManyPipelines

May 7
3 min read

The reproducibility crisis has become one of the biggest discussions in modern neuroscience and psychology. Over the last decade, researchers have increasingly recognized that scientific conclusions can sometimes depend not only on the data itself, but also on the analytical decisions made along the way.

Electroencephalography (EEG) is a perfect example of this challenge.

From preprocessing choices and artifact rejection strategies to statistical models and feature extraction methods, EEG analysis offers substantial analytical flexibility. Different researchers can apply different — yet entirely reasonable — pipelines to the exact same dataset and still arrive at different conclusions.

This is precisely the question behind the ambitious international initiative known as EEGManyPipelines.

A Large-Scale Community Experiment in EEG Reproducibility

The EEGManyPipelines project was created to investigate how much variability in EEG research outcomes can be explained by differences in data analysis pipelines.

Rather than trying to standardize every methodological choice, the project embraced analytical diversity as a scientific object of study.

The idea was simple but powerful:

Provide the same unpublished EEG dataset to many independent analyst teams
Ask them to test the same predefined hypotheses
Allow each team to use their own preferred analysis pipeline
Compare how methodological decisions influence the final results

The scale of participation was remarkable:

168 independent analyst teams
396 researchers
37 countries

All teams analyzed the same dataset consisting of:

EEG and behavioral recordings from 33 participants
A visual long-term memory task
8 predefined hypotheses involving both ERP and time-frequency analyses

Importantly, participating teams did not only submit final statistical results. They also shared:

Processed EEG data
Analysis scripts
Pipeline descriptions
Meta-scientific questionnaires about decision-making and methodology

This makes EEGManyPipelines one of the richest openly shared resources ever created for studying analytical variability in EEG research.

Figure 1. Inclusion flowchart and demographic overview of the 168 participating analyst teams in the EEGManyPipelines project, including expertise, publication history, EEG data collection experience, geographical distribution, gender, and professional roles. From Cesnaite et al. (draft preprint available on OSF/MetaArXiv), which itself partially adapted panel C from Trübutschek et al. (2024). — **Figure 1.** Inclusion flowchart and demographic overview of the 168 participating analyst teams in the EEGManyPipelines project, including expertise, publication history, EEG data collection experience, geographical distribution, gender, and professional roles. From Cesnaite et al. (draft preprint available on OSF/MetaArXiv), which itself partially adapted panel C from Trübutschek et al. (2024).

Why This Matters for EEG Research

EEG is an incredibly powerful tool for studying brain function, but it is also highly sensitive to methodological choices.

Even small differences in preprocessing can influence outcomes:

Filtering parameters
Referencing strategies
Artifact rejection methods
Time-window selection
Electrode selection
Statistical thresholds
Frequency decomposition methods

None of these choices are necessarily “wrong.” In many cases, multiple approaches are scientifically justified.

However, this flexibility creates an important question:

How robust are EEG findings across different valid analytical pipelines?

Projects like EEGManyPipelines help move this discussion from theory to evidence.

By systematically mapping how analysts make decisions, researchers can better understand:

Which findings are highly robust
Which analyses are sensitive to methodological variation
Which parts of EEG processing need stronger standardization
How reporting practices can improve transparency and reproducibility

Ultimately, this contributes to more reliable neuroscience.

Figure 2. ERP analyses corresponding to four predefined hypotheses in the EEGManyPipelines project, performed by the project steering committee for data validation purposes. The figure compares ERP responses across experimental conditions including man-made vs. natural scenes, old vs. new items, remembered vs. forgotten items, and subsequently remembered vs. forgotten items. Shaded grey regions indicate the temporal windows used for mean amplitude topographies, while ERP traces are averaged across the electrodes highlighted in white. From Cesnaite et al. (MetaArXiv preprint draft, OSF Preprint). — **Figure 2.** ERP analyses corresponding to four predefined hypotheses in the EEGManyPipelines project, performed by the project steering committee for data validation purposes. The figure compares ERP responses across experimental conditions including man-made vs. natural scenes, old vs. new items, remembered vs. forgotten items, and subsequently remembered vs. forgotten items. Shaded grey regions indicate the temporal windows used for mean amplitude topographies, while ERP traces are averaged across the electrodes highlighted in white. From Cesnaite et al. (MetaArXiv preprint draft, OSF Preprint).

The Dataset Is Already Publicly Available

Although the main results paper is still under preparation, the EEGManyPipelines dataset itself has already been released to the scientific community.

The dataset opens the door to many future applications, including:

Reproducibility studies
EEG methodology benchmarking
Meta-science research
Large-scale multicenter analyses
Machine learning and pipeline comparison studies
Educational and training purposes

Because the project includes both raw and processed data alongside analysis scripts and questionnaires, it offers a uniquely comprehensive view of how EEG research is actually conducted “in the wild.”

Our Contribution

We were very happy to contribute to this initiative as one of the participating analyst teams.

Projects like EEGManyPipelines align strongly with our vision of advancing EEG research through:

methodological transparency,
robust analytical practices,
reproducible neuroscience,
and open scientific collaboration.

As EEG technologies continue to expand into clinical applications, biomarkers, neurotechnology, and AI-driven analytics, understanding the impact of analytical variability becomes increasingly important.

Reliable neuroscience requires not only better tools and algorithms, but also better scientific processes.

Learn More

You can explore the project and publications here: