Factor Adjusted Robust Model Selection

Goal of the package

This R package implements a consistent model selection strategy for high dimensional sparse regression when the covariate dependence can be reduced through factor models. By separating the latent factors from idiosyncratic components, the problem is transformed from model selection with highly correlated covariates to that with weakly correlated variables. It is appropriate for cases where we have many variables compared to the number of samples. Moreover, it implements a robust procedure to estimate distribution parameters wherever possible, hence being suitable for cases when the underlying distribution deviates from Gaussianity, which is commonly assumed in the literature. See the paper on this method, Fan et al.(2017) https://arxiv.org/abs/1612.08490, for detailed description of methods and further references.

Let there be \(p\) covariates and \(n\) samples. Let us model the relationship between the response vector \(Y\) and the covariates \(X\) as \(Y = X\beta+\epsilon\). Here \(\beta\) is a vector of size \(p\). Non-zero values in this vector \(\beta\) denote which covariates truly belong in the model. For the covariates, assume the approximate factor model: \(X = FB^T+U\), where \(F\) are the \(K\) underlying factors, \(B\) are the factor loadings and \(U\) are the errors.

Installation

You can install FarmSelect from github with:

install.packages("devtools")
devtools::install_github("kbose28/FarmSelect")
library(FarmSelect)

Getting help

Help on the functions can be accessed by typing “?”, followed by function name at the R command prompt.
For up to date information, help with installation issues and questions/comments see the project page at https://kbose28.github.io/FarmSelect/.

Functions

There are four functions available.

farm.select: The main function which carries out the entire model testing procedure.
farm.adjust: Adjusts the data for latent fators.
farm.mean: Multivariate mean estimation with Huber’s loss.
farm.cov: Multivariate covariance estimation with Huber’s loss.

Also see the farm.scree function in the FarmTest package for how to generate diagnostic plots and output to evaluate the factor adjustment step.

Main function example: model selection

Here we generate data from a factor model with 3 factors. We have 50 samples of 100 dimensional data. The model is of size 5, where the first 5 covariates model coefficients being non-zero and the rest zero. The factors, loadings, errors are all generated from a normal distribution.

library(FarmSelect)
set.seed(100)
P = 100 #dimension
N = 50 #samples
K = 3 #nfactors
Q = 5 #model size
Lambda = matrix(rnorm(P*K, 0,1), P,K)
F = matrix(rnorm(N*K, 0,1), N,K)
U = matrix(rnorm(P*N, 0,1), P,N)
X = Lambda%*%t(F)+U
X = t(X)
beta_1 = 3+3*runif(Q)
beta = c(beta_1, rep(0,P-Q))
eps = rnorm(N)
Y = X%*%beta+eps 
output = farm.select(Y,X)

## Call:
## farm.select(Y = Y, X = X)
## 
##  Factor Adjusted Robust Model Selection 
## loss function used: mcp
## 
## p = 100, n = 50
## factors found: 3
## size of model selected:
##  5

names(output)

## [1] "beta.chosen" "coef.chosen" "nfactors"    "X.res"       "Y.res"

output$beta.chosen

## V1 V2 V3 V4 V5 
##  1  2  3  4  5

output$coef.chosen

##       V1       V2       V3       V4       V5 
## 3.706534 3.498892 4.566415 3.725966 3.297019

The values X.res and Y.res are the covariates and responses after adjusting for latent factors. The formulas for these are \(Y.\text{res} = (I_n-P)Y\) and \(X.\text{res} = (I_n-P)X^T\), where \(P = \hat{F}(\hat{F}^T\hat{F})^{-1}\hat{F}^T\).

Now we use a different loss function for the model selection step.

output = farm.select(Y,X, loss = "lasso" )

## Call:
## farm.select(Y = Y, X = X, loss = "lasso")
## 
##  Factor Adjusted Robust Model Selection 
## loss function used: lasso
## 
## p = 100, n = 50
## factors found: 3
## size of model selected:
##  5

We may also use robust estimates of all the parameters. This may take more time, depending upon the dimension size of the problem.

output = farm.select(Y,X, robust = TRUE )

## Call:
## farm.select(Y = Y, X = X, robust = TRUE)
## 
##  Factor Adjusted Robust Model Selection 
## loss function used: mcp
## 
## p = 100, n = 50
## factors found: 3
## size of model selected:
##  5

Other functions

The function farm.adjust adjusts the dataset for latent factors. The number of factors is estimated internally by using the method in (Ahn and Horenstein 2013).

output = farm.adjust(Y,X)
names(output)

## [1] "X.res"    "nfactors" "Y.res"

If known, we can provide this function (or the main function) the number of latent factors. Providing too large a number results in a warning message. The maximum number of factors possible is \(\max(n,p)\) but a much smaller number is recommended.

output = farm.adjust(Y,X, K.factors  = 30)

## Warning in farm.adjust(Y, X, K.factors = 30): Number of factors supplied is
## > min(n,p)/2. May cause numerical inconsistencies

We see a warning telling us that it is not a good idea to calculate 30 eigenvalues from a dataset that has only 50 samples.