{"id":1264,"date":"2017-02-10T04:30:42","date_gmt":"2017-02-10T04:30:42","guid":{"rendered":"http:\/\/kourentzes.com\/forecasting\/?p=1264"},"modified":"2017-02-11T02:01:48","modified_gmt":"2017-02-11T02:01:48","slug":"forecasting-time-series-with-neural-networks-in-r","status":"publish","type":"post","link":"https:\/\/kourentzes.com\/forecasting\/2017\/02\/10\/forecasting-time-series-with-neural-networks-in-r\/","title":{"rendered":"Forecasting time series with neural networks in R"},"content":{"rendered":"

I have been looking for a package to do time series modelling in R with neural networks for quite some time with limited success. The only implementation I am aware of that takes care of autoregressive lags in a user-friendly way is the nnetar<\/code> function in the forecast<\/code> package, written by Rob Hyndman<\/a>. In my view there is space for a more flexible implementation, so I decided to write a few functions for that purpose. For now these are included in the TStools<\/a> package that is available in GitHub<\/a>, but when I am happy with their performance and flexibility I will put them in a package of their own.<\/p>\n

Here I will provide a quick overview of what these is available right now. I plan to write a more detailed post about these functions when I get the time.<\/p>\n

For this example I will model the AirPassengers<\/code> time series available in R. I have kept the last 24 observations as a test set and will use the rest to fit the neural networks. Currently there are two types of neural network available, both feed-forward: (i) multilayer perceptrons (use function mlp<\/code>); and extreme learning machines (use function elm<\/code>).<\/p>\n

# Fit MLP<\/span>\r\nmlp.fit <- mlp(y.in)\r\nplot(mlp.fit)\r\nprint(mlp.fit)\r\n<\/pre>\n
This is the basic command to fit an MLP network to a time series. This will attempt to automatically specify autoregressive inputs and any necessary pre-processing of the time series. With the pre-specified arguments it trains 20 networks which are used to produce an ensemble forecast and a single hidden layer with 5 nodes. You can override any of these settings. The output of print<\/code> is a summary of the fitted network:<\/p>\n
MLP fit with 5 hidden nodes and 20 repetitions.\r\nSeries modelled in differences: D1.\r\nUnivariate lags: (1,3,4,6,7,8,9,10,12)\r\nDeterministic seasonal dummies included.\r\nForecast combined using the median operator.\r\nMSE: 6.2011.\r\n<\/pre>\n
As you can see the function determined that level differences are needed to capture the trend<\/a>. It also selected some autoregressive lags and decided to also use dummy variables for the seasonality. Using plot<\/code> displays the architecture of the network (Fig. 1).<\/p>\n
<\/a>
Fig. 1. Output of plot(mlp.fit).<\/code><\/p><\/div>\n
The light red inputs represent the binary dummies used to code seasonality, while the grey ones are autoregressive lags. To produce forecasts you can type:<\/p>\n
mlp.frc <- forecast(mlp.fit,h=tst.n)\r\nplot(mlp.frc)\r\n<\/pre>\n
Fig. 2 shows the ensemble forecast, together with the forecasts of the individual neural networks. You can control the way that forecasts are combined (I recommend using the median or mode operators<\/a>), as well as the size of the ensemble.<\/p>\n
<\/a>
Fig. 2. Output of the plot<\/code> function for the MLP forecasts.<\/p><\/div>\n
You can also let it choose the number of hidden nodes. There are various options for that, but all are computationally expensive (I plan to move the base code to CUDA at some point, so that computational cost stops being an issue).<\/p>\n
# Fit MLP with automatic hidden layer specification<\/span>\r\nmlp2.fit <- mlp(y.in,hd.auto.type=\"valid\",hd.max=10)\r\nprint(round(mlp2.fit$MSEH,4))\r\n<\/pre>\n
This will evaluate from 1 up to 10 hidden nodes and pick the best on validation set MSE. You can also use cross-validation (if you have patience…). You can ask it to output the errors for each size:<\/p>\n
        MSE\r\nH.1  0.0083\r\nH.2  0.0066\r\nH.3  0.0065\r\nH.4  0.0066\r\nH.5  0.0071\r\nH.6  0.0074\r\nH.7  0.0061\r\nH.8  0.0076\r\nH.9  0.0083\r\nH.10 0.0076\r\n<\/pre>\n
There are a few experimental options in specifying various aspects of the neural networks, which are not fully documented and is probably best if you stay away from them for now!<\/p>\n
ELMs work pretty much in the same way, although for these I have made default the automatic specification of the hidden layer.<\/p>\n
# Fit ELM<\/span>\r\nelm.fit <- elm(y.in)\r\nprint(elm.fit)\r\nplot(elm.fit)\r\n<\/pre>\n
This gives the following network summary:<\/p>\n
ELM fit with 100 hidden nodes and 20 repetitions.\r\nSeries modelled in differences: D1.\r\nUnivariate lags: (1,3,4,6,7,8,9,10,12)\r\nDeterministic seasonal dummies included.\r\nForecast combined using the median operator.\r\nOutput weight estimation using: lasso.\r\nMSE: 83.0044.\r\n<\/pre>\n
I appreciate that using 100 hidden nodes on such a short time series can make some people uneasy, but I am using a shrinkage estimator instead of conventional least squares to estimate the weights, which in fact eliminates most of the connections. This is apparent in the network architecture in Fig. 3. Only the nodes connected with the black lines to the output layer contribute to the forecasts. The remaining connection weights have been shrunk to zero.<\/p>\n
<\/a>
Fig. 3. ELM network architecture.<\/p><\/div>\n
Another nice thing about these functions is that you can call them from the thief<\/a> package, which implements Temporal Hierarchies<\/a> forecasting in R. You can do that in the following way:<\/p>\n
# Use THieF<\/span>\r\nlibrary(thief)\r\nmlp.thief <- thief(y.in,h=tst.n,forecastfunction=mlp.thief)\r\n<\/pre>\n
There is a similar function for using ELM networks: elm.thief<\/code>.<\/p>\n
Since for this simple example I kept some test set, I benchmark the forecasts against exponential smoothing:<\/p>\n
\n\r\n\r\n
Method<\/th>\nMAE<\/th>\n<\/tr>\n<\/thead>\n
MLP (5 nodes)<\/td>\n 62.471<\/td>\n<\/tr>\n\r\n
MLP (auto)<\/td>\n 48.234<\/td>\n<\/tr>\n\r\n
ELM<\/td>\n 48.253<\/td>\n<\/tr>\n\r\n
THieF-MLP<\/td>\n 45.906<\/td>\n<\/tr>\n\r\n
ETS<\/td>\n 64.528<\/td>\n<\/tr>\n<\/tbody><\/table><\/div>\n
Temporal hierarchies, like MAPA<\/a>, are great for making your forecasts more robust and often more accurate. However, with neural networks the additional computational cost is evident!<\/p>\n
These functions are still in development, so the default values may change and there are a few experimental options that may give you good results or not!<\/p>\n
Related Posts<\/H3>