Empirical Dynamic Modeling
Time Series as Observations of a Dynamic System
The essential concept behind Empirical Dynamic Modeling (EDM) is that time series can be viewed as projections of the behavior of a dynamic system. This framework requires only a few modest assumptions:
- The system state can be described as a single point in a high-dimensional space. Axes can be thought of as fundamental state variables, such as population abundances, resources, or environmental properties a la the Hutchinson Niche.
- The system state changes through time following some deterministic rules. In other words, the behavior is not completely random.
Projecting the system state to an axis then gives the value of the corresponding state variable, and sequential projections over time produce a time series. Different time series observed from the system can capture different state variables, but are more generally, some function of the system state and may convolve several different state variables.
Attractor Reconstruction / Takens’ Theorem
To reproduce this fundamental, geometric view of the system, one might suppose that time series of all the state variables are required. However, Takens’ Theorem (Takens 1981) states that a mathematically equivalent reconstruction can be created by substituting lags of a time series for the unknown or unobserved variables.
In other words, instead of representing a system state as being composed of multiple different state variables, we instead use a lagged-coordinate embedding: \[ \vec{x}_t = \langle x_t, x_{t-\tau}, \dots, x_{t-(E-1)\tau} \rangle \]
If sufficient lags are used, the reconstruction preserves essential mathematical properties of the original system. For instance, the points will map one-to-one to actual system states, and nearest neighbors in the reconstruction correspond to similar system states and behave similarly in the near future.
Nearest Neighbor Forecasting using Simplex Projection
One application of the reconstructed attractor is prediction. This can be accomplished using nearest neighbor forecasting methods, because of the similar behavior of nearby points in the reconstruction. One such method is simplex projection (Sugihara and May 1990). Simplex Projection is implemented in rEDM as the function simplex, and can be used both for prediction or to identify the optimal embedding dimension by quantifying the forecast skill of reconstructions with different dimensionality.
Example
First, we load the data and look at the format. Because this dataset is part of the rEDM package, we need to first load the package into R, before we have access to its datasets and functions.
library(rEDM)
data(tentmap_del)
head(tentmap_del)## [1] -0.0992003 -0.6012986 0.7998003 -0.7944096 0.7979992 -0.8195405We can see that the data consists of just a single vector, containing the raw data (first-differences from a tentmap). Because the simplex function can accept a single vector as the input time series, we don’t need further processing of the data. Furthermore, because the data come from a discrete-time model, we can let many of the parameters be default values (e.g., \(\tau = 1\), \(\text{tp} = 1\)). The default values for the embedding dimension, \(E\), range from \(1\) to \(10\), and so the output will allow us to determine which embedding dimension best unfolds the attractor.
We need to specify what portions of the data to use for constructing the simplex projection model, and what portions to use for testing the forecast skill. By default, simplex will use leave-one-out cross-validation over the entire time series, but because the data contain no observational noise, and are particularly long, we’d like to be more conservative.
lib <- c(1, 100)
pred <- c(201, 500)This will use the first 100 points (time = 1 to 100) in the time series as the “library” to construct the model, and 300 points (time = 201 to 500) as the “prediction set” to test the model.
Note that if the code detects any overlap in the lib and pred, it will enable leave-one-out cross-validation and return a warning message.
ts <- tentmap_del
simplex_output <- simplex(ts, lib, pred)The results are a simple data.frame with columns for each of the model parameters and forecast statistics, and rows for each run of the model. In this case, there is one run for each value of \(E\), so we can simply plot \(E\) against \(\rho\), the correlation between observed and predicted values:
par(mar = c(4, 4, 1, 1), mgp = c(2.5, 1, 0))
plot(simplex_output$E, simplex_output$rho, type = "l", xlab = "Embedding Dimension (E)",
ylab = "Forecast Skill (rho)")
Prediction Decay
An important property of deterministic chaos is that nearby trajectories eventually diverge over time (the so-called “butterfly effect”). This means that prediction is primarily limited to short-term forecasts, because over the long-term, the system state may be viewed as essentially random. This property also differentiates nonlinear systems from the equilibrium view, where the system can be expected to settle around a stable point.
Example
We can test for this property by adjusting the tp parameter in the models, which determines how far into the future the model seeks to predict:
simplex_output <- simplex(ts, lib, pred, E = 2, tp = 1:10)Here, we can simply plot \(tp\) against \(\rho\) to see how forecast accuracy changes as we predict further and further into the future:
par(mar = c(4, 4, 1, 1))
plot(simplex_output$tp, simplex_output$rho, type = "l", xlab = "Time to Prediction (tp)",
ylab = "Forecast Skill (rho)")
Identifying Nonlinearity
One concern is that many time series may show predictability even if they are purely stochastic, because they behave similarly to autocorrelated red noise. Luckily, there are additional tests that can be done to distinguish between red noise and deterministic behavior, by quantifying the degree of “nonlinearity” in the data.
Here, we use the S-map forecasting method, that is based on fitting local linear maps for prediction instead of the nearest-neighbor interpolation of simplex projection (Sugihara 1994). In addition to the parameters for simplex projection, S-map also contains a nonlinear tuning parameter, \(\theta\) that affects the weights associated with individual points when fitting the local linear map. When \(\theta = 0\), all weights are equal, and the S-map is identical to an autoregressive model; values of \(\theta\) above \(0\) give greater weight to nearby points in the state space, thereby accommodating nonlinear behavior by allowing the local linear map to vary in state-space. For autoregressive red noise, the linear model should perform better, because the S-map model can reduce observation error by averaging over many points instead of just the most nearby points.
Thus, varying \(\theta\) allows us to compare equivalent linear and nonlinear models as a test for nonlinear dynamics (after first using simplex projection to estimate the optimal embedding dimension for a time-series.)
Example
Following from the previous example, we set E = 2 based on the results from simplex projection. Again, note that we allow many of the parameters take on default values (e.g., \(\tau = 1\), \(\text{tp} = 1\)). If we had changed these for simplex projection, we would want to propagate them here. The default values for the nonlinear tuning parameter, \(\theta\), range from \(0\) to \(8\), and are suitable for our purposes.
Note also, that the default value for num_neighbors is 0. Typically, when using s_map to test for nonlinear behavior, we allow all points in the reconstruction to be used, subject only to the weighting based on distance. By using 0 for this parameter (an otherwise nonsensical value), the program will use all nearest neighbors.
smap_output <- s_map(ts, lib, pred, E = 2)Again, the results are a simple data.frame with columns for each of the model parameters and forecast statistics, and rows for each run of the model. In this case, there is one run for each value of \(\theta\), so we can simply plot \(\theta\) against \(\rho\):
par(mar = c(4, 4, 1, 1), mgp = c(2.5, 1, 0))
plot(smap_output$theta, smap_output$rho, type = "l", xlab = "Nonlinearity (theta)",
ylab = "Forecast Skill (rho)")
Generalized Takens’ Theorem
Instead of creating an attractor by taking lags of a single time series, it is possible to combine lags from different time series, if they are all observed from the same system (Sauer et al. 1991, Deyle and Sugihara 2011). The practical reality of applying EDM to systems with finite data, noisy observations, and stochastic influences means that such “multivariate” reconstructions can often be a better depiction of the true dynamics than “univariate” counterparts.
In rEDM, the block_lnlp function generalizes the simplex and s_map functions, allowing generic reconstructions to be used with either of the simplex projection or S-map algorithms. The main data input for block_lnlp is a matrix or data.frame of the time series observations, where each column is a separate time series and each row represents the variables observed at the same time. In addition to the typical arguments for simplex or s_map, block_lnlp contains arguments to specify which column is to be forecast (target_column) as well as which columns to use to construct the attractor (columns). In both cases, either a numerical index or the column name can be given. For obvious reasons, column names will work only if the input data has column names.
Note that if lagged coordinates are intended to be used, they need to be manually created as separate columns in the matrix or data.frame.
Example
We begin by loading an example dataset of time series and lags from a coupled 3-species model system. Here, the block_3sp variable is a 10-column data.frame with 1 column for time, and 3 columns for each of the variables (unlagged, t-1, and t-2 lags).
data(block_3sp)
head(block_3sp)## time x_t x_t-1 x_t-2 y_t y_t-1 y_t-2
## 1 1 -0.7418625 NA NA -1.2681036 NA NA
## 2 2 1.2448818 -0.7418625 NA 1.4888875 -1.2681036 NA
## 3 3 -1.9176852 1.2448818 -0.7418625 -0.1131881 1.4888875 -1.2681036
## 4 4 -0.9623176 -1.9176852 1.2448818 -1.1067786 -0.1131881 1.4888875
## 5 5 1.3318751 -0.9623176 -1.9176852 2.3850408 -1.1067786 -0.1131881
## 6 6 -0.8170829 1.3318751 -0.9623176 -0.6753463 2.3850408 -1.1067786
## z_t z_t-1 z_t-2
## 1 -1.8639802 NA NA
## 2 -0.4815825 -1.8639802 NA
## 3 1.5352388 -0.4815825 -1.8639802
## 4 -1.4929558 1.5352388 -0.4815825
## 5 -1.1194762 -1.4929558 1.5352388
## 6 0.7466579 -1.1194762 -1.4929558In order to correctly index into columns, block_lnlp has an option to indicate that the first column is actually a time index. When first_column_time is set to TRUE, a value of 1 for target_column now points to the first data column in the data.frame, as opposed to the time column (columns is similarly indexed).
lib <- c(1, NROW(block_3sp))
pred <- c(1, NROW(block_3sp))
block_lnlp_output <- block_lnlp(block_3sp, lib = lib, pred = pred, columns = c(1,
2, 4), target_column = 1, stats_only = FALSE, first_column_time = TRUE)We can also run the same model by referring to the names of the columns directly.
block_lnlp_output <- block_lnlp(block_3sp, lib = lib, pred = pred, columns = c("x_t",
"x_t-1", "y_t"), target_column = "x_t", stats_only = FALSE, first_column_time = TRUE)Note that we did not specify a value for the tp parameter. Here, the default value of 1 means that the program will predict the target variable 1 time step into the future (based on the row-structure of the input data). In some cases, the data may already be processed into a format where one wants to predict a different column that has already been aligned correctly. In that case, one can set tp = 0 when calling block_lnlp.
By setting stats_only to FALSE, we get back a list with the full model output. Only 1 model was run, so the output is a list with 1 element. To extract the raw predictions, we can go into the model_output variable and pull out the observed and predicted values, plotting them to see how well the model fit relative to the expected 1:1 line.
observed <- block_lnlp_output[[1]]$model_output$obs
predicted <- block_lnlp_output[[1]]$model_output$pred
par(mar = c(4, 4, 1, 1), pty = "s")
plot_range <- range(c(observed, predicted), na.rm = TRUE)
plot(observed, predicted, xlim = plot_range, ylim = plot_range, xlab = "Observed",
ylab = "Predicted")
abline(a = 0, b = 1, lty = 2, col = "blue")
Causality Inference and Cross Mapping
One of the corollaries to the Generalized Takens’ Theorem is that it should be possible to cross-predict or cross-map between variables that are observed from the same system. Consider two variables, \(x\) and \(y\) that interact in a dynamic system. Then the univariate reconstructions based on \(x\) or \(y\) alone should uniquely identify the system state and the corresponding value of the other variable. Thus, it should be possible to use one variable to cross-predict the other.
In the case of unidirectional causality, \(x\) causes \(y\), then the behavior of the causal variable (\(x\)) leaves a signature on the affected variable (\(y\)). In such cases, the reconstructed states based on \(y\) can be used to cross-predict the values of \(x\) (because the reconstruction based on \(y\) must be complete, it must include information about the value of \(x\)). Note that this cross-prediction is in the opposite direction of the causal effect. At the same time, cross-prediction from \(x\) to \(y\) will fail, because the time series of \(x\) behaves independently of \(y\), so a univariate reconstruction using only lags of \(x\) is necessarily incomplete.
Although \(x\) has incomplete information for predicting \(y\), it does affect the values of \(y\), and therefore will likely to have nonzero predictive skill. However, this cross-mapping will be limited to the statistical association between \(x\) and \(y\) and fail to improve as longer time series are used for reconstruction. In contrast, in the absence of noise, the cross-prediction of \(x\) from \(y\) will continually improve. This convergence is therefore necessary to infer causality.
For practical reasons, the sensitivity of detecting causality this way is improved if, instead of predicting the future value of another variable, we estimate the concurrent value of another variable. We refer to this modified method as cross-mapping, because we are not “predicting” the future.
Convergent Cross Mapping (CCM)
In the rEDM package, the ccm function presents an easy way to compute cross map skill for different subsamples of the time series, enabling observation of both convergence and uncertainty regarding cross map skill. In the following example, we use CCM to identify causality between anchovy landings in California and Newport Pier sea-surface temperature.
Here, we use a previously identified value of 3 for the embedding dimension. We set lib_sizes (the number of library vectors) to vary from 10 to 80 in steps of 10. Setting num_samples to 100 means that 100 different library samples will be generated, by random sampling (random_libs = TRUE by default) from the possible vectors with replacement (replace = TRUE by default).
data(sardine_anchovy_sst)
anchovy_xmap_sst <- ccm(sardine_anchovy_sst, E = 3, lib_column = "anchovy",
target_column = "np_sst", lib_sizes = seq(10, 80, by = 10), random_libs = FALSE)
sst_xmap_anchovy <- ccm(sardine_anchovy_sst, E = 3, lib_column = "np_sst", target_column = "anchovy",
lib_sizes = seq(10, 80, by = 10), random_libs = FALSE)The output from CCM is a data.frame with statistics for each model run (in thise case, 100 models at each of 8 library sizes = 800 rows). Because we cross map using multiple libraries at each library size, we’d like to aggregate the results and plot the average cross map skill at each library size. Because average cross map skill less than \(0\) is noninformative, we filter out negative values when plotting.
a_xmap_t_means <- ccm_means(anchovy_xmap_sst)
t_xmap_a_means <- ccm_means(sst_xmap_anchovy)
par(mar = c(4, 4, 1, 1), mgp = c(2.5, 1, 0))
plot(a_xmap_t_means$lib_size, pmax(0, a_xmap_t_means$rho), type = "l", col = "red",
xlab = "Library Size", ylab = "Cross Map Skill (rho)", ylim = c(0, 0.4))
lines(t_xmap_a_means$lib_size, pmax(0, t_xmap_a_means$rho), col = "blue")
legend(x = "topleft", legend = c("anchovy xmap SST", "SST xmap anchovy"), col = c("red",
"blue"), lwd = 1, inset = 0.02, cex = 0.8)
