The multiverse analysis can be used as a way of assessing the influence of different analysis choices that could reasonably be made by researchers, instead of only presenting the result of one...Show moreThe multiverse analysis can be used as a way of assessing the influence of different analysis choices that could reasonably be made by researchers, instead of only presenting the result of one research ‘path’ as is often done in studies. While the multiverse analysis increases transparency about the results, it is still unclear how researchers can best summarize the results of this analysis more formally. Moreover, as far as we are aware, no previous studies have examined how the multiverse analysis performs under different research conditions. In this study, we simulated data under different research conditions. In addition, we built a generic multiverse analysis that was used to analyze this data. Two methods were used to summarize the results of this analysis, namely the mean p-value and the harmonic mean p-value (HMP). The results of this study showed that the mean p-value may be the preferred summarization method, as it provides a more conservative estimate of the different paths in the multiverse and has less false-positive results than the HMP in a situation where data was simulated under the null hypothesis. In addition, our study shows that the summarization methods of our multiverse analysis are robust against variations regarding the number of variables that are part of the analysis, the amount of missing data in a dataset and changes in the correlation between variables. However, the summarization methods in our multiverse were not robust against underpowered data. Only if the different research paths in our multiverse analysis had adequate power, the HMP was generally able to find a significant result in at least 90% of cases. However, future research is needed to see if these results can be replicated, since the definition of a generic multiverse analysis may differ depending on the research field.Show less
The science of psychology is facing a so-called replication and reproduction crisis. This crisis calls for increased transparency in the way that statistics are reported, in order to make findings...Show moreThe science of psychology is facing a so-called replication and reproduction crisis. This crisis calls for increased transparency in the way that statistics are reported, in order to make findings in psychology more reliable, more likely to be interpreted correctly, and easier to verify and replicate. Undisclosed flexibility in data collection and analysis can severely inflate the false positive rate in psychology (and other disciplines). This issue can be tackled by performing a multiverse analysis. Recent literature proposes that this method guarantees an unprecedent level of transparency for research papers. Building on this stream of research, the present thesis examines the value of using a multiverse analysis to examine the robustness of a non-significant effect. Multiverse analysis refers to a method which involves performing the analysis of interest across the complete set of data sets that arise from several defensible data processing and analysis choices. The demonstration of the multiverse focuses on data collected by Dieleman et al. (2020). The reported results were first reproduced using the R software environment, and then, alternative analysis pathways were examined. The results proved to be reproducible, yet alternative pathways revealed some fluctuation. This thesis contributes to a better understanding of how a multiverse analysis can improve the transparency of psychological science.Show less
