Prior research has compared Bayes factors and p values within Hypothesis testing using t tests (Wetzels et al., 2011). The current research expanded on this comparison to include both t tests and...Show morePrior research has compared Bayes factors and p values within Hypothesis testing using t tests (Wetzels et al., 2011). The current research expanded on this comparison to include both t tests and various forms of Analyses of Variance. Further, we conducted maximal n sequential analyses, following the design as proposed by Schönbrodt and Wagenmakers (2018). We conducted two forms of sequential analysis: The fixed order sequential analysis, in which the order the data was presented in the dataset dictated the order by which it was added, and the replicated random order sequential analysis in which the order was randomized, and the procedure repeated 100 times. For both the comparison and Sequential analyses we used Bayesian alternatives to Classical Significance tests with real-world data that were reproduced with author’s assistance by Hardwicke et al. (2018). We found that Bayes factors and p values covary in both t tests and Analyses of Variance. However, we observed Bayes factors that underemphasized the perceived effect by p values, as well as Bayes factors that overemphasized when compared to p values even after performing a sensitivity analysis. We also found that most Sequential analyses produced Bayes factors exceeding a threshold prior to the maximal n, with most analyses exceeding more. We also contextualized our fixed order sequential analysis using the percentage of Bayes factors across the 100 replications of each sequential analysis that exceeded thresholds. We evaluate these findings and propose measures researcher may take based on our findings to utilize optional stopping in a way that is efficient, reasonable and accurate.Show less
Objective: In this study I investigated the p-curves of 250 social psychology papers to see if they showed evidence of p-hacking. I also divided the papers into categories based on types of...Show moreObjective: In this study I investigated the p-curves of 250 social psychology papers to see if they showed evidence of p-hacking. I also divided the papers into categories based on types of justifications for exclusions (strongly justified, grey area, poorly justified) to describe how pcurves of these categories differ from each other. Methods: From a larger sample of 1000 random social psychology papers, 250 were randomly selected and coded by a larger group of coders. The papers were coded for main finding, main result, statistics sentence and the number and justification of exclusions. The statistics sentences were then pasted into a software that graphed the p-curves and analysed them for skewness and the presence of evidential value. Results: Two studies were mistakenly added in the data file but were kept for further steps. About 118 studies were excluded for missing or inadequate statistics sentences and 16 reported non-significant results. A total of 15 papers were left un-coded. The data analyses were done with the 104 remaining papers. The p-curve for the total sample was right-skewed and had adequate evidential value. The p-curve for the grey area justifications for exclusions was almost identical to the p-curve of the total sample. The p-curve of strongly justified exclusions followed similar pattern but also had a small peak around p = .05. The p-curve for poorly justified exclusions had two large peaks at .01 and .04. Conclusion: The sample of social psychology papers shows no evidence of p-hacking and has adequate evidential value. There seems to be some differences in the p-curves for different categories of justifications for exclusions. However, difficulties in coding the papers show that there is a need for more meta research into the standardization of reporting findings and exclusions in social psychology papers.Show less
