Introduction

The motivation for this package is to provide functions which help with the development and tuning of machine learning models in biomedical data where the sample size is frequently limited, but the number of predictors may be significantly larger (P >> n). While most machine learning pipelines involve splitting data into training and testing cohorts, typically 2/3 and 1/3 respectively, medical datasets may be too small for this, and so determination of accuracy in the left-out test set suffers because the test set is small. Nested cross-validation (CV) provides a way to get round this, by maximising use of the whole dataset for testing overall accuracy, while maintaining the split between training and testing.

In addition typical biomedical datasets often have many 10,000s of possible predictors, so filtering of predictors is commonly needed. However, it has been demonstrated that filtering on the whole dataset creates a bias when determining accuracy of models (Vabalas et al, 2019). Feature selection of predictors should be considered an integral part of a model, with feature selection performed only on training data. Then the selected features and accompanying model can be tested on hold-out test data without bias. Thus, it is recommended that any filtering of predictors is performed within the CV loops, to prevent test data information leakage.

This package enables nested cross-validation (CV) to be performed using the commonly used glmnet package, which fits elastic net regression models, and the caret package, which is a general framework for fitting a large number of machine learning models. In addition, nestedcv adds functionality to enable cross-validation of the elastic net alpha parameter when fitting glmnet models.

nestedcv partitions the dataset into outer and inner folds (default 10x10 folds). The inner fold CV, (default is 10-fold), is used to tune optimal hyperparameters for models. Then the model is fitted on the whole inner fold and tested on the left-out data from the outer fold. This is repeated across all outer folds (default 10 outer folds), and the unseen test predictions from the outer folds are compared against the true results for the outer test folds and the results concatenated, to give measures of accuracy (e.g. AUC and accuracy for classification, or RMSE for regression) across the whole dataset.

A final round of CV is performed on the whole dataset to determine hyperparameters to fit the final model to the whole data, which can be used for prediction with external data.

Variable selection

While some models such as glmnet allow for sparsity and have variable selection built-in, many models fail to fit when given massive numbers of predictors, or perform poorly due to overfitting without variable selection. In addition, in medicine one of the goals of predictive modelling is commonly the development of diagnostic or biomarker tests, for which reducing the number of predictors is typically a practical necessity.

Several filter functions (t-test, Wilcoxon test, anova, Pearson/Spearman correlation, random forest variable importance, and ReliefF from the CORElearn package) for feature selection are provided, and can be embedded within the outer loop of the nested CV.

Installation

install.packages("nestedcv")
library(nestedcv)

Examples

Importance of nested CV

The following simulated example demonstrates the bias intrinsic to datasets where P >> n when applying filtering of predictors to the whole dataset rather than to training folds.

## Example binary classification problem with P >> n
x <- matrix(rnorm(150 * 2e+04), 150, 2e+04)  # predictors
y <- factor(rbinom(150, 1, 0.5))  # binary response

## Partition data into 2/3 training set, 1/3 test set
trainSet <- caret::createDataPartition(y, p = 0.66, list = FALSE)

## t-test filter using whole test set
filt <- ttest_filter(y, x, nfilter = 100)
filx <- x[, filt]

## Train glmnet on training set only using filtered predictor matrix
library(glmnet)
## Loading required package: Matrix
## Loaded glmnet 4.1-7
fit <- cv.glmnet(filx[trainSet, ], y[trainSet], family = "binomial")

## Predict response on test set
predy <- predict(fit, newx = filx[-trainSet, ], s = "lambda.min", type = "class")
predy <- as.vector(predy)
predyp <- predict(fit, newx = filx[-trainSet, ], s = "lambda.min", type = "response")
predyp <- as.vector(predyp)
output <- data.frame(testy = y[-trainSet], predy = predy, predyp = predyp)

## Results on test set
## shows bias since univariate filtering was applied to whole dataset
predSummary(output)
##          Reference
## Predicted  0  1
##         0 18  2
##         1  6 24
## 
##               AUC            Accuracy   Balanced accuracy   
##            0.9567              0.8400              0.8365

## Nested CV
fit2 <- nestcv.glmnet(y, x, family = "binomial", alphaSet = 7:10 / 10,
                      filterFUN = ttest_filter,
                      filter_options = list(nfilter = 100))
fit2
## Nested cross-validation with glmnet
## Filter:  ttest_filter 
## 
## Final parameters:
##    lambda      alpha  
## 0.0001329  0.7000000  
## 
## Final coefficients:
## (Intercept)      V11509       V1268       V2922       V6846       V4000 
##    0.466532    1.097199   -1.033871    1.026569   -1.004862    1.001580 
##      V14381      V19635       V3141      V15105      V15188       V3081 
##    0.995284   -0.983195   -0.908433   -0.873061    0.817486   -0.814375 
##       V7811       V1929       V3124      V17710      V19851      V15476 
##    0.731193   -0.730123    0.689375   -0.685563    0.638867   -0.638166 
##       V1743       V6883      V19532      V14602      V12275      V19856 
##    0.629755    0.625354    0.623573    0.584462   -0.566232    0.555612 
##       V5815       V7041       V2239       V9006        V205      V10042 
##    0.546187   -0.537333    0.532352    0.530197    0.471562   -0.463651 
##      V11503       V8290      V16228        V913      V19053       V8607 
##   -0.461969   -0.457329    0.447494   -0.435442   -0.432575    0.432356 
##       V2141      V12559      V18438       V9590      V18478      V16914 
##    0.432049   -0.431831    0.421018   -0.409842    0.405738   -0.398580 
##      V12987       V3095       V3873       V5503       V1505      V12490 
##    0.396404   -0.383457   -0.377814   -0.359859   -0.356195    0.355862 
##      V11097       V6436       V6594       V8124       V1216      V16383 
##   -0.334628   -0.334575   -0.323585    0.321870    0.295178   -0.273920 
##      V18283       V4944       V1977       V9429      V10395      V11807 
##   -0.262283    0.243465   -0.219724   -0.216404   -0.204326   -0.203771 
##      V15145      V15704       V8633      V17475        V439       V6757 
##    0.195918   -0.195729    0.193722    0.191865    0.189348    0.177567 
##      V17596      V18013       V6767        V706       V8489       V4242 
##   -0.172518   -0.143679    0.137639    0.132834   -0.125381   -0.124519 
##       V5759       V7335       V8166       V6292       V6726       V7249 
##    0.089154    0.086103    0.064558    0.051628    0.040848    0.034939 
##      V10526        V938       V6370       V7316       V9162      V19467 
##   -0.033547    0.025457    0.016354    0.012457   -0.008686    0.004896 
##      V14945 
##    0.004686 
## 
## Result:
##          Reference
## Predicted  0  1
##         0 32 33
##         1 39 46
## 
##               AUC            Accuracy   Balanced accuracy   
##            0.4619              0.5200              0.5165

testroc <- pROC::roc(output$testy, output$predyp, direction = "<", quiet = TRUE)
inroc <- innercv_roc(fit2)
plot(fit2$roc)
lines(inroc, col = 'blue')
lines(testroc, col = 'red')
legend('bottomright', legend = c("Nested CV", "Left-out inner CV folds", 
                                 "Test partition, non-nested filtering"), 
       col = c("black", "blue", "red"), lty = 1, lwd = 2, bty = "n")

In this example the dataset is pure noise. Filtering of predictors on the whole dataset is a source of leakage of information about the test set, leading to substantially overoptimistic performance on the test set as measured by ROC AUC.

Figures A & B below show two commonly used, but biased methods in which cross-validation is used to fit models, but the result is a biased estimate of model performance. In scheme A, there is no hold-out test set at all, so there are two sources of bias/ data leakage: first, the filtering on the whole dataset, and second, the use of left-out CV folds for measuring performance. Left-out CV folds are known to lead to biased estimates of performance as the tuning parameters are ‘learnt’ from optimising the result on the left-out CV fold.

In scheme B, the CV is used to tune parameters and a hold-out set is used to measure performance, but information leakage occurs when filtering is applied to the whole dataset. Unfortunately this is commonly observed in many studies which apply differential expression analysis on the whole dataset to select predictors which are then passed to machine learning algorithms.

Figures C & D below show two valid methods for fitting a model with CV for tuning parameters as well as unbiased estimates of model performance. Figure C is a traditional hold-out test set, with the dataset partitioned 2/3 training, 1/3 test. Notably the critical difference between scheme B above, is that the filtering is only done on the training set and not on the whole dataset.

Figure D shows the scheme for fully nested cross-validation. Note that filtering is applied to each outer CV training fold. The key advantage of nested CV is that outer CV test folds are collated to give an improved estimate of performance compared to scheme C since the numbers for total testing are larger.