Machine learning algorithms are frequently deployed for predictive classification problems. However, as implied by the No Free Lunch (NFL) theorem, not every algorithm is destined to perform well...Show moreMachine learning algorithms are frequently deployed for predictive classification problems. However, as implied by the No Free Lunch (NFL) theorem, not every algorithm is destined to perform well on a given data set. One is often interested whether predictive capacity of an algorithm is significantly better than chance level (better than chance) for a choice of the hyperparameter(s). Machine learning algorithms generally lack the intrinsic statistical framework of statistical learning algorithms to make statements about better than chance performance. In supervised binary classification problems, multiple methods have been proposed to do so. These have shown to be flawed. Arguably, this can be attributed to the idea that the NFL also applies to these better than chance methods, or in general to any tests, suggesting that performance of the test depends on the type of signal. Therefore, in this current project, we propose novel global test (GT) based tests that are in accordance with the signal detected by their respective learning algorithm. To do so, we reformulated two popular machine learning algorithms, k-nearest neighbors (kNN) and random forest, as empirical Bayesian linear models. It turned out we can not only construct tests for specific (combinations of) hyperparameters but also for sets of hyperparameters. Properties of these tests have been explored in simulated linearly and nonlinearly separable alternatives as well as in real world data. Results from our simulation studies indicated that our novel tests had competitive power characteristics compared to existing methods. Moreover, we demonstrated their applicability to real world data. Our finding indicated that our novel tests for kNN and random forest can be readily used to assess better than chance performance. Equally important, the exploited GT framework can be applied to construct tests for other learning algorithms. Ultimately, our tests and possible future GT based tests add to list of existing methods that each serve a niche in the detection of better than chance signal for a learning algorithm.Show less
Statistical analysis of the data arising from RNA-Sequencing (RNA-Seq) experiments is compli-cated. We face difficulties in testing, as we cannot rely on asymptotic properties of tests giventhe...Show moreStatistical analysis of the data arising from RNA-Sequencing (RNA-Seq) experiments is compli-cated. We face difficulties in testing, as we cannot rely on asymptotic properties of tests giventhe small sample sizes. Furthermore, we face a huge multiple testing burden. In this thesis, wewill explore the theoretical and empirical properties of permutation tests for RNA-Seq data andcompare them to their parametric alternatives. This will be done on both simulated data and areal dataset. Additional to two classical permutation methods, the applicability of a novel per-mutation test based on sign flipping score contributions will be analysed (Hemerik et al., 2020).We demonstrate that permutation tests are an attractive alternative for simple two-group com-parisons, as they have comparable power to parametric tests, while having stricter control overthe Type I error rate. Furthermore, our parametric simulation indicates that they have superiorpower to many parametric methods when covariates are included in the model.Show less
Master thesis | Statistical Science for the Life and Behavioural Sciences (MSc)
open access
Food-borne disease outbreaks constitute a large, ongoing public health burden worldwide (Hald et al., 2016). Early identification of contaminated food products plays an important role in reducing...Show moreFood-borne disease outbreaks constitute a large, ongoing public health burden worldwide (Hald et al., 2016). Early identification of contaminated food products plays an important role in reducing health burdens of food-borne disease outbreaks (Jacobs et al., 2017). Case-control studies together with logistic regression analysis are primarily used in food-borne outbreak investigations. However, the current methodology is associated with problems including response misclassification, missing values and ignoring small sample bias. Jacobs et al. (2017) developed a formal Bayesian variable selection method which deals with the problems of missing covariates and misclassified response. The re-analysis of Dutch Salmonella Thompson 2012 outbreak data (Friesema et al., 2014) has illustrated that this Bayesian approach allows a relatively easy implementation of these concepts and performs better than the standard logistic regression analysis in the identification of responsible food products. The complete Bayesian variable selection model is composed of three different parts, namely, misclassification correction, missing value imputation and Bayesian variable selection. In this thesis, we are interested in how these different parts affect the performance of Bayesian variable selection models in scenarios with (i) the same response misclassification rate and missingness rate in an assumed responsible food product covariate as in the original food-borne disease outbreak dataset, (ii) different response misclassification rates, (iii) different missingness rates in an assumed responsible food product and (iv) the combination of different response misclassification rates and missingness rates. We answer this research question by designing and executing a simulation study. Our results indicate that for the four different versions of Bayesian variable selection models studied in this thesis, the increase in the response misclassification rate or the missingness rate in the assumed responsible food product covariate or the increase in both results in a decrease in model performance. Bayesian variable selection, misclassification correction and missing value imputation all contribute positively to the model performance. Although missing value imputation is most computationally expensive, it contributes the most to the model performance among these three components.Show less
We present a quadratic difference penalty on logistic regression as a solution for the high dimensional data problem and spatial correlation in the classification of genetic copy number data. The...Show moreWe present a quadratic difference penalty on logistic regression as a solution for the high dimensional data problem and spatial correlation in the classification of genetic copy number data. The quadratic difference penalty is the L2 norm of the first order difference penalty matrix times the coefficient vector, and thereby shrinks adjacent regression coefficients to each other. We propose an L2 fused lasso, a logistic lasso with an extra quadratic difference penalty; and a smoothed logistic regression, a logistic regression with only a quadratic difference penalty. We construct algorithms for both penalized regressions. We explain the connection between our smoothed logistic regression and ridge regression. We demonstrate the challenges in fitting a lasso, and adapt the gradient ascent. The L2 fused lasso and smoothed logistic regression are applied on genetic copy number data to classify the grade of bladder tumors.Show less
The increasing dimensionality of data, for example in genetics, requires additional assumptions on the regression models in order to obtain estimates of the regression coefficients, known as...Show moreThe increasing dimensionality of data, for example in genetics, requires additional assumptions on the regression models in order to obtain estimates of the regression coefficients, known as regularization. Lasso, one of the regularization methods available, imposes an L1 constraint on the vector of coefficients, leading to the desirable feature of setting some coefficients to zero. This results in sparse, and more importantly, estimable models. However, setting some coefficients becomes problematic when one would want a group of variables to be in or out of the model, e.g. a factor coded as several dummy variables. This problem is solved by the group lasso. This paper presents an extension to the algorithm presented by Goeman for optimizing the penalized log likelihood under the proportional hazards model. This algorithm combines the gradient ascent algorithm and the Newton-Raphson algorithm. This methodology is then applied to data obtained from the Carema case-cohort study, which aim is to assess the additional predict 10-year risk of coronary heart disease.Show less
