Psychological Research From a Different Angle: Implementation of Predictive Modelling Using Cross-validation Alongside Explanatory Modelling Using P-values

Psychological research is often based on null hypothesis significance testing (NHST) using explanatory modelling. In many cases, additional information or even more reliable information can be gained if predictive modelling is also used. Cross-validation (CV) is a very useful statistical procedure that can generate predictive outcome measures. This study will compare the use and results of CV with the results of traditional NHST analyses using two case studies, one explanatory theory-driven - comparing means - question and one predictive data-driven - forwards stepwise logistic regression - question. In the case of explanatory questions, CV is able to generate similar conclusions and the outcome measure is more intuitive to interpret. Regarding the predictive data-driven question, the final models from the CV procedure have slightly lower out-of-sample prediction errors than the final model based on the traditional NHST procedure. Moreover, CV proves useful in evaluating explanatory...Show morePsychological research is often based on null hypothesis significance testing (NHST) using explanatory modelling. In many cases, additional information or even more reliable information can be gained if predictive modelling is also used. Cross-validation (CV) is a very useful statistical procedure that can generate predictive outcome measures. This study will compare the use and results of CV with the results of traditional NHST analyses using two case studies, one explanatory theory-driven - comparing means - question and one predictive data-driven - forwards stepwise logistic regression - question. In the case of explanatory questions, CV is able to generate similar conclusions and the outcome measure is more intuitive to interpret. Regarding the predictive data-driven question, the final models from the CV procedure have slightly lower out-of-sample prediction errors than the final model based on the traditional NHST procedure. Moreover, CV proves useful in evaluating explanatory formulated models with respect to out-of-sample prediction accuracy and overfitting. It is recommended to implement predictive modelling in contemporary psychological research complementary to explanatory modelling in order to make psychological science more reliable and replicable.Show less

Leiden University Student Repository

Documents

In Collections

Psychological Research From a Different Angle: Implementation of Predictive Modelling Using Cross-validation Alongside Explanatory Modelling Using P-values