A novel relative density-based probability outlier detection algorithm, merging two previous techniques, is proposed. The resulting algorithm has a probabilistic score that is robust to low-density...Show moreA novel relative density-based probability outlier detection algorithm, merging two previous techniques, is proposed. The resulting algorithm has a probabilistic score that is robust to low-density pattern problems. The technique is extended to the context of proximity mapping, weighting the least squares loss function to mitigate the damaging effects of proximity outliers. Experiments on synthetic and real data show performance gains of the proposed algorithm for outlier correction as well as advantages in the interpretability of outlier scores in proximity mapping.Show less
A challenge in clinical neuroscience is to identify systematic differences between patients’ resting state networks (RSNs) that may relate to phenotypes and/or disease subtypes. Recently, Durieux...Show moreA challenge in clinical neuroscience is to identify systematic differences between patients’ resting state networks (RSNs) that may relate to phenotypes and/or disease subtypes. Recently, Durieux et al. (2022) proposed a novel unsupervised learning method for multi-subject rs-fMRI data, called Clusterwise Independent Component Analysis (C-ICA), which enables automatic clustering of subjects based on differences and similarities in subjects’ RSNs. A drawback of C-ICA, however, is that the method assumes that subject clusters show large difference in the RSNs that characterizes them. This, however, is a quite restrictive assumption as it can be expected that there exists important RSNs that are common to all subjects (across clusters). Not accounting for these common RSNs may negatively affect C-ICA’s ability to identify the subject clustering and the RSNs for each cluster. Therefore, in order to tackle this problem, two extensions of C-ICA (CCD1 and CCD2) are presented and compared to each other that allow to distinguish between (and extract from the data) both common and distinctive components. Both extensions differ in the procedure to identify the common components, with CCD2 selecting the common components out of a wider pool of possible components than CCD1. In a simulation study, the two C-ICA extensions are compared to C-ICA while manipulating the following factors: number of clusters, number of common and distinctive components, degree of overlap among clusters and amount of noise. Performance was determined by computing cluster recovery and component recovery, where component recovery was studied separately for the common components, the distinctive components and all components. Both extensions clearly outperformed C-ICA, with CCD2 performing the best. A mixed analysis of variance (ANOVA) showed that, the degree of overlap and the amount of noise in the data were the most influential factors, with large amounts of overlap and noise decreasing cluster and component recovery. Moreover, both extensions showed a slower deterioration of cluster and component recovery -with increasing overlap and noise- than C-ICA. Specifically for intermediate amounts of noise and overlap the differences between both extensions and C-ICA were the largest.Show less
Prior research has compared Bayes factors and p values within Hypothesis testing using t tests (Wetzels et al., 2011). The current research expanded on this comparison to include both t tests and...Show morePrior research has compared Bayes factors and p values within Hypothesis testing using t tests (Wetzels et al., 2011). The current research expanded on this comparison to include both t tests and various forms of Analyses of Variance. Further, we conducted maximal n sequential analyses, following the design as proposed by Schönbrodt and Wagenmakers (2018). We conducted two forms of sequential analysis: The fixed order sequential analysis, in which the order the data was presented in the dataset dictated the order by which it was added, and the replicated random order sequential analysis in which the order was randomized, and the procedure repeated 100 times. For both the comparison and Sequential analyses we used Bayesian alternatives to Classical Significance tests with real-world data that were reproduced with author’s assistance by Hardwicke et al. (2018). We found that Bayes factors and p values covary in both t tests and Analyses of Variance. However, we observed Bayes factors that underemphasized the perceived effect by p values, as well as Bayes factors that overemphasized when compared to p values even after performing a sensitivity analysis. We also found that most Sequential analyses produced Bayes factors exceeding a threshold prior to the maximal n, with most analyses exceeding more. We also contextualized our fixed order sequential analysis using the percentage of Bayes factors across the 100 replications of each sequential analysis that exceeded thresholds. We evaluate these findings and propose measures researcher may take based on our findings to utilize optional stopping in a way that is efficient, reasonable and accurate.Show less
The multiverse analysis can be used as a way of assessing the influence of different analysis choices that could reasonably be made by researchers, instead of only presenting the result of one...Show moreThe multiverse analysis can be used as a way of assessing the influence of different analysis choices that could reasonably be made by researchers, instead of only presenting the result of one research ‘path’ as is often done in studies. While the multiverse analysis increases transparency about the results, it is still unclear how researchers can best summarize the results of this analysis more formally. Moreover, as far as we are aware, no previous studies have examined how the multiverse analysis performs under different research conditions. In this study, we simulated data under different research conditions. In addition, we built a generic multiverse analysis that was used to analyze this data. Two methods were used to summarize the results of this analysis, namely the mean p-value and the harmonic mean p-value (HMP). The results of this study showed that the mean p-value may be the preferred summarization method, as it provides a more conservative estimate of the different paths in the multiverse and has less false-positive results than the HMP in a situation where data was simulated under the null hypothesis. In addition, our study shows that the summarization methods of our multiverse analysis are robust against variations regarding the number of variables that are part of the analysis, the amount of missing data in a dataset and changes in the correlation between variables. However, the summarization methods in our multiverse were not robust against underpowered data. Only if the different research paths in our multiverse analysis had adequate power, the HMP was generally able to find a significant result in at least 90% of cases. However, future research is needed to see if these results can be replicated, since the definition of a generic multiverse analysis may differ depending on the research field.Show less
There is a need for replication studies in psychology, yet resources are scarce. Study selection strategies are required that can guide researchers in which studies to prioritise for replication....Show moreThere is a need for replication studies in psychology, yet resources are scarce. Study selection strategies are required that can guide researchers in which studies to prioritise for replication. The goal of this paper was to examine potential selection strategies and to identify possible issues with these strategies. Therefore a quantitative method for Replication Value (RV), inspired by Isager (2019), was proposed. RV determines the relative importance of replicating a study and was defined as impact over uncertainty. The studies in this paper formulated and compared different operationalizations of RV. Web of Science (WoS) was used to extract relevant data on a random sample of papers from WoS’s social psychology category. The first study examined a RV formula using minimal information, with yearly citations as a measure for impact and sample size as a measure for uncertainty. Study 1 also introduced Statcheck as a method to examine potential relations between RV-ranking and erroneous reporting. Study 2 elaborated on study 1, combining p-values with sample size as a measure for uncertainty. As part of this study, p-curve analysis was conducted to find relations between evidential value and paper ranking. Study 3 elaborated further, adding Altmetric score, a measure for societal influence of a paper, as a measure for impact. For all studies, similarity between RV-rankings was examined using Rank-Biased Overlap (RBO). Results tentatively indicate that sample size and citations are measures that can be useful when creating RV-formulas. Adding p-values to the RV-equation wasn’t beneficial, because it hardly changed the ranking of higher ranking papers. The addition of Altmetric score did change the RV-ranking and might be of interest to researchers interested in emphasising societal impact. Overall, this paper lays a groundwork for future RV research, mainly by exploring possible metrics involved in RV equations, but also by pointing out potential issues when using RV equations.Show less
