In the past decade, the importance of replication studies has gained attention in the field of psychology. This led to an increased interest in estimating the rate of replication studies in the...Show moreIn the past decade, the importance of replication studies has gained attention in the field of psychology. This led to an increased interest in estimating the rate of replication studies in the field. We explore an alternative approach to estimating the prevalence of replication studies. We constructed a code that counts the occurrences of search terms. To determine which search terms are useful for estimating a replication rate, we used this code on a dataset of 325 publications that were manually coded on whether they were considered replication studies. We were also able to use the word counts to investigate the replication rates per search term, and the role that the frequency of these search terms plays. We found that looking at the frequency of search terms does not help finding replication studies more accurately. However, a single occurrence of a reference, combined with the term “replicat*”, does give a more accurate search when looking for replication studies. In addition, the terms “note”, “reapprais*”, “revis*” and “verif*” seem beneficial to include in a search profile, but our sample data for these terms was too small to be conclusive. To accommodate future research on this topic, we present a search profile and provide a code that can be used in R to determine which search terms can contribute to finding replication studies accurately.Show less
A study performed by Makel et al. (2012) looked at the prevalence of replications in the field of Psychology in a time-period from 1950-2012. At the time they found an increase in replication...Show moreA study performed by Makel et al. (2012) looked at the prevalence of replications in the field of Psychology in a time-period from 1950-2012. At the time they found an increase in replication studies over time. Besides the increase they also found a possibility of experimenter bias when there was an overlap in authors. This thesis is a conceptual replication study of Makel et al. (2012) focussing on the possible prevalence of experimenter bias in replication studies based on an overlap of authors. It was expected that there would be more successful results when there is an overlap in authors. A difference of 19.04 was found on the percentage of successful results between an overlap in authors and unique authors. This indicates that there is a relation between overlap in authors and successful results, however to determine if this relation is caused by experimenter bias, more research is needed.Show less
This research was conducted as a conceptual replication of the study by Makel et al. (2012), in order to compare the success rate of two different types of replication (direct and conceptual)....Show moreThis research was conducted as a conceptual replication of the study by Makel et al. (2012), in order to compare the success rate of two different types of replication (direct and conceptual). Additionally, Hüffmeier et al. (2016)’s typology was used (exact, close, constructive, conceptual in lab, and conceptual in field types of replication), in order to achieve a more detailed understanding of the different types of replication. The expectation of the study was to find a higher success rate of lower replication types: close, constructive, conceptual in lab, conceptual in field (Hüffmeier et al., 2016) and conceptual (Makel et al., 2012), in comparison to the higher replication types: exact (Hüffmeier et al., 2016) and direct (Makel et al., 2012). After an advanced search on Web of Science for articles in the psychological field published between 2000 and 2020 using the term “replicat*”, a random sample of 5000 articles was selected, that was manually coded by 14 people. The main findings showed that conceptual replications had a lower success rate than direct replications. However, the results based on the replication types in Hüffmeier et al. (2016) indicated that conceptual in lab and conceptual in field replications had the highest success rates, followed by exact replications. This demonstrated the importance of the order of studies in yielding research quality. Additionally, looking at the success rates, it can be concluded that confidence in findings can still be achieved in lower types of replication, despite of changing the methodology of the original article.Show less
Flexibility in the decisions researchers make during their research can lead to false positive findings. Due to low transparency in published papers in the field of psychology, the amount of...Show moreFlexibility in the decisions researchers make during their research can lead to false positive findings. Due to low transparency in published papers in the field of psychology, the amount of flexibility authors had is often unclear. In the current thesis, in a first step a quantitative measure of Replication Value is applied to a random set of studies (n = 1257) from Social Psychology, using citation count as a proxy for impact and sample size as a proxy for uncertainty. This Replication Value has been suggested as an indicator of how worthwhile it is to replicate a study (see Isager et al., in press), and can be applied to a large number of studies due to its quantitative approach. However, Replication Value is based on solely on quantitative proxies. Therefore, it is necessary to also manually examine papers. In a second step of the current thesis, it is manually explored whether the uncertainty that researchers have when making choices during their research can become clearer by mapping them. Therefore, the studies with the highest Replication Values (n = 10), with the median Replication Values (n = 10), and with the lowest Replication Values (n = 10) were examined on their reporting transparency and potential Researcher Degrees of Freedom. A detailed analysis of the first results indicated that the qualitative analysis of the Researcher Degrees of Freedom of original researchers is helpful to in selecting which study to replicate after making a larger selection based on RV. The findings from this exploratory research are discussed in the context of the field of Social Psychology, with an emphasis on how researchers looking to select a target for replication can use our DFS to map the uncertainty of original work.Show less
Making unplanned decisions during data analysis can increase the likelihood of finding evidence that unjustly supports a hypothesis. To reduce this bias, Data blinding hides certain aspects of the...Show moreMaking unplanned decisions during data analysis can increase the likelihood of finding evidence that unjustly supports a hypothesis. To reduce this bias, Data blinding hides certain aspects of the data in order to allow researchers to make data-dependent decisions (e.g., checking assumptions to ensure the right analysis is chosen) without knowing the influence of these decisions on results of a hypothesis test. In this study, we examined the suitability of different blinding techniques for three common analyses in psychology: independent samples Student’s t-test, analysis of variance (Type III sum of squares and F-test), and linear least squares regression. For each analysis type, we evaluate whether data blinding techniques interfere with assumption checks and whether they constrain p-hacking. Based on these criteria, we recommend data blinders to center group means for the independent samples Student’s t-test and ANOVA, and to scramble predictor names for regression. We provide guidelines for researchers and data blinders for each step in the data blinding process and introduce a tool that allows data blinders to easily apply the data blinding techniques to a dataset (i.e., the “blindData” R-package). It is discussed how data blinding, combined with preregistration, benefits the integrity of research output.Show less
Objective: In this study I investigated the p-curves of 250 social psychology papers to see if they showed evidence of p-hacking. I also divided the papers into categories based on types of...Show moreObjective: In this study I investigated the p-curves of 250 social psychology papers to see if they showed evidence of p-hacking. I also divided the papers into categories based on types of justifications for exclusions (strongly justified, grey area, poorly justified) to describe how pcurves of these categories differ from each other. Methods: From a larger sample of 1000 random social psychology papers, 250 were randomly selected and coded by a larger group of coders. The papers were coded for main finding, main result, statistics sentence and the number and justification of exclusions. The statistics sentences were then pasted into a software that graphed the p-curves and analysed them for skewness and the presence of evidential value. Results: Two studies were mistakenly added in the data file but were kept for further steps. About 118 studies were excluded for missing or inadequate statistics sentences and 16 reported non-significant results. A total of 15 papers were left un-coded. The data analyses were done with the 104 remaining papers. The p-curve for the total sample was right-skewed and had adequate evidential value. The p-curve for the grey area justifications for exclusions was almost identical to the p-curve of the total sample. The p-curve of strongly justified exclusions followed similar pattern but also had a small peak around p = .05. The p-curve for poorly justified exclusions had two large peaks at .01 and .04. Conclusion: The sample of social psychology papers shows no evidence of p-hacking and has adequate evidential value. There seems to be some differences in the p-curves for different categories of justifications for exclusions. However, difficulties in coding the papers show that there is a need for more meta research into the standardization of reporting findings and exclusions in social psychology papers.Show less
