Exploring and Communicating the Pruning Variability of Decision Trees Caused by Cross-Validation

Decision trees are popular statistical methods because they are both intuitive and easy to implement through R packages, such as rpart and metacart. They work by iteratively dividing the data using decision rules depending on specific predictors, resulting in a tree-like structure that is easy to interpret. However, one of the main challenges of building a decision tree, is finding the optimal tree size. A tree that is too small can miss important patterns in the data. In contrast, a tree that is too big can overfit the data by including noise and, therefore, might not generalize well to new data. As a solution to this challenge, pruning methods are often used. Pruning simplifies the tree by removing excessive splits that provide little predictive value, or may reflect noise in the data. By removing these splits and nodes, while keeping the main structure and important splits at the top, the risk of overfitting is reduced. However, the pruning process depends on cross-validation,...Show moreDecision trees are popular statistical methods because they are both intuitive and easy to implement through R packages, such as rpart and metacart. They work by iteratively dividing the data using decision rules depending on specific predictors, resulting in a tree-like structure that is easy to interpret. However, one of the main challenges of building a decision tree, is finding the optimal tree size. A tree that is too small can miss important patterns in the data. In contrast, a tree that is too big can overfit the data by including noise and, therefore, might not generalize well to new data. As a solution to this challenge, pruning methods are often used. Pruning simplifies the tree by removing excessive splits that provide little predictive value, or may reflect noise in the data. By removing these splits and nodes, while keeping the main structure and important splits at the top, the risk of overfitting is reduced. However, the pruning process depends on cross-validation, which requires random partitioning of the data. This random nature of data splitting generates variability, which can result in inconsistent pruning outcomes. The focus of this thesis is on pruning variability in random-effects meta-CART models. These models apply decision tree methods to meta-analytic data. In the meta-analytic context, effect sizes are used as the response variable, while moderators are the predictor variables that may explain differences in effect sizes across studies, such as the study characteristics. The aim of this thesis is to investigate how moderator type, effect size strength, moderator correlation, and the pruning strictness, contribute to the pruning variability. Pruning was repeated across 1000 iterations using different seeds on various simulated datasets. Results showed that the pruning outcomes varied significantly and depended on interactions between moderator type, moderator correlation, effect size strength, and pruning strictness. Additionally, three visualization tools were developed, and other suggestions were provided to visualize and communicate the pruning variability.Show less

Leiden University Student Repository

Documents

In Collections

Exploring and Communicating the Pruning Variability of Decision Trees Caused by Cross-Validation