My new R packageĀ **nnfor** is available on CRAN. This collects the various neural network functions that appeared in TStools. See this post for demo of these functions. In summary the package includes:

- Automatic, semi-automatic or fully manual specification of MLP neural networks for time series modelling, that helps in specifying inputs with lags of the target and exogenous variables. It can automatically deal with pre-processing (differencing and scaling) and identify the number of hidden nodes. The user can control which of these settings are left on automatic or not.
- A few options for building network ensembles.
- Plotting functions of the network topology, fit and forecast.
- All the above for ELMs (Extreme Learning Machines).
- Support for Temporal Hierarchies Forecasting, with the thief package for R.

This builds on the neuralnet package for R, and provides the code to make the networks capable of handling time series data automatically. Although that package is quite flexible, it is computationally expensive and does not permit for deep learning. The plan is to eventually implement such capabilities in the package.

There are numerous papers that support the ideas used to put together this package:

- In my new book, Ord et al., 2017, Principles of Business Forecasting, 2e, Wessex Press Publishing. Chapter 10 describes the basic logic in building MLP networks for time series forecasting. This package implements the logic described there.
- This paper demonstartes the performance of the input variable selection algorithm: Crone and Kourentzes, 2010, Feature selection for time series prediction – a combined filter and wrapper approach for neural networks. Neurocmputing, 73, 1923-1936. There is some influence from this proceedings paper as well. (These feel like really old papers!)
- This paper looks at the combination operator for the ensembles. Please move away from the average! Kourenztes et al., 2014, Neural network ensembles operators for time series forecasting. Expert Systems with Applications, 41, 4235-4244.

The neural network functions in TStools will be removed, initially pointing towards this package and latter removed completely.

There is a github repository for this, where I will be posting updates and fixes till they go on CRAN: https://github.com/trnnick/nnfor

Happy (nonlinear) forecasting!

I have been following your work and was excited to see you made progress on getting this on cran.

I am having two issues with switching from nnetar to mlp…

One, I am getting an error for non conforming arguments….

and two, how do you implement forward regressors?

In nnetar i would cbind the time series objects of forward regressors into freg, then use xreg = freg in the forecast argument.

I am looking forward to upgrading to using this and figuring how to use thief too, seems impact in what you have been working on.

and I am hopeful I will see similar results in the short term natural gas load forecasting I am doing.

Thanks for the help. Cheers!

First, I found a small bug that only appears when there are no univariate lags and the modeller wants to turn off automatic input selection. I put up a github repository here: https://github.com/trnnick/nnfor. The bug is fixed there.

I wrote a R example to show how to handle xreg inputs with mlp(). There are two options. Either input the dynamics in the xreg.lags or create an xreg where each column is the appropriate dynamic manually and set xreg.lags to zeros. However, I have not implemented leads (forward regressors) yet, so you will have to go with option two. The following example should be helpful.

Hope this helps!

Thanks! Very helpful. I’m training away. What are your thoughts on lead inputs (forward regressors) ?

Computational time is killing me, With so many networks to train for an ensemble model (84) I’ve set to train all day then save models, load then forecast the next morning. I have a rolling different dates to look at….

I should be able to utilize a gpu to speed this up right? Ive been looking at a few R packages with CUDA. I’m uncertain if I can actually utilize the hardware. I have been looking through some documentation but nothing definitive yet.

I appreciate your input you an are expert, I am still working to perfect my forecasting application with the latest and greatest.

Thank you!

The computational time is a big issue. At the core of the nnfor, for now, I am calling the neuralnets package, which unfortunately does not use the GPU, and relies on CPU. This makes training rather slow. A potential way forward might be to use ELMs with the elm() function. The argument barebone=TRUE for elm() may help as well. On my todo list for the package is to implement an alternative core that will make use of GPU to speed up processing and also allow experimenting with deep learning (though I am currently very skeptical if it is needed for time series forecasting, at least I am skeptical for now!) – when I run large neural network experiments I use the neural networks toolbox in MatLab, which is very fast, but unfortunately not free or open source.

Using regressors with leads can be an useful at times. The typical restriction is that we often do not have information about a regressor for the future periods, so that is one of the reasons you do not see this happening often in examples. I have used leads quite a bit in promotional modelling, where you know the promotional plan in advance for the future periods, and therefore this information is available and known with certainty. In that particular case leads are very useful. I would suspect that if you know your regressors with some confidence then they should be helpful as well. If your regressors are forecasted, then you carry any forecast errors in your new forecast that uses them as inputs – this may be a mixed bag, depending on how good is the forecast of the regressors.

Hi Nikos,

Thanks for your work here. I’m very interested in neural network applications in time series. I was curious if you’ve seen the work being done with LSTMs and time series? Here’s a link to some work being done at Uber regarding utilizing autoencoders to capture time series properties:

https://arxiv.org/abs/1709.01907

Also this recent Kaggle winner applied similar techniques to win a forecasting competition:

https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39367

I’m interested to see what you discover for deep learning applications.

Hi Josh,

Many thanks for the links. I will read them in detail. Quickly skimming through the first paper I noticed that there are no simple statistical benchmarks (something as simple as exponential smoothing!) in the empirical evaluation. It is a shame, as this would help make NNs and deep learning more mainstream in the forecasting literature, but also help highlight whether the gains are over NNs or more general. These two literatures need to talk more to each other š When I get the time, I will investigate more and post my findings!

I too am interested to see how LSTM can help me out. Nikos…When I use the second example you provided?

Is the model forecasting based on the forward leads I put into the model?

My leads are forecasted weather variables so yes as you mentioned the forecasted input carries into the forecast error, but I am thinking day of and tomorrow are going to be okay, where as 4 plus days out is iffy.

Thank you for your help!

That’s correct the model has fitted and forecasts on the leads that you have placed. I will need some time to get my programming hands into LSTM – end of term is a busy period at my uni :/

Thanks Nikos. Just to say another random internet reader I had similar questions about LSTM when reading your review of Prophet. (Also I wonder how STLPlus package would compare.)

Thanks