Average relative accuracy indices

Daniele Girolimetto

2022-02-17

Let \[ \hat{e}_{i,j,t}^{[k],h} = y_{i,t+h}^{[k]} - \hat{y}_{i,j,t}^{[k],h}, \quad \begin{array}{l} i=1,\ldots,n, \\ j=0,\ldots,J, \end{array} \; t=1,\ldots, q , \; \begin{array}{l} k \in {\cal K}, \\ h=1,\ldots,h_k , \end{array} \] be the forecast error, where \(y\) and \(\hat{y}\) are the actual and the forecasted values, respectively, suffix \(i\) denotes the variable of interest, \(j\) is the forecasting technique, where \(j=0\) is the benchmark forecasting procedure, \(t\) is the forecast origin, \({\cal K}\) is the set of the time frequencies at which the series is observed, and \(h\) is the forecast horizon, whose lead time depends on the time frequency \(k\).

Denote by A\(_{i,j}^{[k],h}\) the forecasting accuracy of the technique \(j\), computed across \(q\) forecast origins, for the \(h\)-step-ahead forecasts of the variable \(i\) at the temporal aggregation level \(k\). For example, A\(_{i,j}^{[k],h} = MSE_{i,j}^{[k],h}\) (type=“mse” in score_index() function, default), otherwise we might have A\(_{i,j}^{[k],h} = MAE_{i,j}^{[k],h}\) (type=“mae” in score_index() function) or A\(_{i,j}^{[k],h} = RMSE_{i,j}^{[k],h}\) (type=“rmse” in score_index() function), where \[ \begin{array}{rcl} MSE_{i,j}^{[k],h} & = & \displaystyle\frac{1}{q}\displaystyle\sum_{t=1}^{q} \left(\hat{e}_{i,j}^{[k],h}\right)^2 \\ MAE_{i,j}^{[k],h} & = & \displaystyle\frac{1}{q}\displaystyle\sum_{t=1}^{q} \left|\hat{e}_{i,j}^{[k],h}\right| \\ RMSE_{i,j}^{[k],h} & = & \sqrt{\displaystyle\frac{1}{q}\displaystyle\sum_{t=1}^{q} \left(\hat{e}_{i,j}^{[k],h}\right)^2} \end{array} \]

In any case, we consider the relative version of the accuracy index \(A_{i,j}^{[k],h}\), given by: \[ r_{i,j}^{[k],h} = \displaystyle\frac{A_{i,j}^{[k],h}}{A_{i,0}^{[k],h}} , \quad i=1,\ldots,n, \quad j=0,\ldots, J, \quad k \in {\cal K}, \quad h=1, \ldots, h_k, \] and use it to compute the Average relative accuracy index of the forecasting procedure \(j\), for given \(k\) and \(h\), through the geometric mean: \[ \text{AvgRelA}_{j}^{[k],h} = \left(\displaystyle\prod_{i=1}^{n} r_{i,j}^{[k],h} \right)^{\frac{1}{n}} , \quad j=0,\ldots,J . \]

We may consider the following average relative accuracy indices for selected groups of variables/time frequencies and forecast horizons:

Average relative accuracy indices for a single variable at a given time frequency, for multiple forecast horizons \[ \label{AvgRelAikh1h2} \text{AvgRelA}_{i,j}^{[k],q_1:q_2} = \left(\prod_{h=q_1}^{q_2} r_{i,j}^{[k],h}\right)^{\frac{1}{q_2 - q_1 + 1}}, \; \begin{array}{l} i=1,\ldots,n, \\ j=0,\ldots,J, \end{array} \; \begin{array}{l} k \in {\cal K}, \\ 1 \le q_1 \le q_2 \le h_k \end{array} . %i=1,\ldots,n , \; j=0,\ldots, J, \; k \in {\cal K}, \; 1 \le q_1 \le q_2 \le h_k. \] Average relative accuracy indices for a group of variables (either all, or selected groups, e.g. a: uts, b: bts) at a given time frequency, either for a single forecast horizon or across them \[ \begin{array}{rcll} \text{AvgRelA}^{[k],h}_j & = & \left(\displaystyle\prod_{i=1}^{n} r_{i,j}^{[k],h}\right)^{\frac{1}{n}}, & j=0,\ldots,J , \; k \in {\cal K}, \; h=1,\ldots,h_k \\[.5cm] \text{AvgRelA}^{[k],h}_{a,j} & = & \left(\displaystyle\prod_{i=1}^{n_a} r_{i,j}^{[k],h}\right)^{\frac{1}{n_a}}, & j=0,\ldots, J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k],h}_{b,j} & = & \left(\displaystyle\prod_{i=n_a+1}^{n} r_{i,j}^{[k],h}\right)^{\frac{1}{n_b}}, & j=0,\ldots, J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k]}_j & = & \left(\displaystyle\prod_{i=1}^{n} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n h_k}}, & \; j=0,\ldots,J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k]}_{a,j} & = &\left(\displaystyle\prod_{i=1}^{n_a} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_a h_k}}, & j=0,\ldots, J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k]}_{b,j} & = &\left(\displaystyle\prod_{i=n_a+1}^{n} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_b h_k}}, & j=0,\ldots, J, \; k \in {\cal K} \end{array} \]

Average relative accuracy indices for a single variable or for a group of variables (all, a: uts, b: bts), across all time frequencies and forecast horizons \[ \begin{array}{rcll} \text{AvgRelA}_{i,j} & = & \left(\displaystyle\prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{k^*+m}}, & \begin{array}{l} i=1,\ldots,n \\ j=0,\ldots,J \end{array} \\[.15cm] \text{AvgRelA}_j & = & \left(\displaystyle\prod_{i=1}^{n} \prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n(k^*+m)}} , & j=0,\ldots,J \\[.15cm] \text{AvgRelA}_{a,j} & = & \left(\displaystyle\prod_{i=1}^{n_a} \prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_a(k^*+m)}} , & j=0,\ldots,J \\[.15cm] \text{AvgRelA}_{b,j} & = & \left(\displaystyle\prod_{i=n_a+1}^{n} \prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_b(k^*+m)}} , & j=0,\ldots,J \end{array} \]

library(FoReco)
data(FoReco_data)

# Cross-temporal framework
oct_recf <- octrec(FoReco_data$base, m = 12, C = FoReco_data$C,
                   comb = "bdshr", res = FoReco_data$res)$recf
oct_score <- score_index(recf = oct_recf,
                         base = FoReco_data$base,
                         test = FoReco_data$test, m = 12, nb = 5)

# Cross-sectional framework#'
# monthly base forecasts
id <- which(simplify2array(strsplit(colnames(FoReco_data$base), split = "_"))[1, ] == "k1")
mbase <- t(FoReco_data$base[, id])
# monthly test set
mtest <- t(FoReco_data$test[, id])
# monthly residuals
id <- which(simplify2array(strsplit(colnames(FoReco_data$res), split = "_"))[1, ] == "k1")
mres <- t(FoReco_data$res[, id])
# monthly reconciled forecasts
mrecf <- htsrec(mbase, C = FoReco_data$C, comb = "shr", res = mres)$recf
# score
hts_score <- score_index(recf = mrecf, base = mbase, test = mtest, nb = 5)

# Temporal framework
data(FoReco_data)
# top ts base forecasts ([lowest_freq' ...  highest_freq']')
topbase <- FoReco_data$base[1, ]
# top ts residuals ([lowest_freq' ...  highest_freq']')
topres <- FoReco_data$res[1, ]
# top ts test ([lowest_freq' ...  highest_freq']')
toptest <- FoReco_data$test[1, ]
# top ts recf ([lowest_freq' ...  highest_freq']')
toprecf <- thfrec(topbase, m = 12, comb = "acov", res = topres)$recf
# score
thf_score <- score_index(recf = toprecf, base = topbase, test = toptest, m = 12)