{"id":1202,"date":"2016-12-28T10:13:15","date_gmt":"2016-12-28T10:13:15","guid":{"rendered":"http:\/\/kourentzes.com\/forecasting\/?p=1202"},"modified":"2016-12-31T00:53:49","modified_gmt":"2016-12-31T00:53:49","slug":"can-neural-networks-predict-trended-time-series","status":"publish","type":"post","link":"https:\/\/kourentzes.com\/forecasting\/2016\/12\/28\/can-neural-networks-predict-trended-time-series\/","title":{"rendered":"Can neural networks predict trended time series?"},"content":{"rendered":"

Yes and… no! First, I should say that I am thinking of the common types of neural networks that are comprised by neurons that use some type of sigmoid transfer function, although the arguments discussed here are applicable to other types of neural networks. Before answering the question, let us first quickly summarise how typical neural networks function. Note that the discussion is done in a time series forecasting context, so some of the arguments here are specific to that and are not relevant to classification tasks!<\/p>\n

1. Multilayer Perceptron (MLP) neural networks<\/strong><\/p>\n

MLPs are a basic form of neural networks. Having a good understanding of these can help one understand most types of neural networks, as typically other types are constructed by adding more connections (such as feedbacks or skip-layer\/direct connections). Let us assume that we have three different inputs, (X1<\/sub>, X2<\/sub>, X3<\/sub>), which could be different variables or lags of the target variables. A MLP with a single hidden layer, with 5 hidden nodes and a single output layer can be visualised as in Fig. 1.<\/p>\n

<\/a>

Fig. 1. MLP with 3 inputs, 5 hidden nodes arranged in a single layer and a single output node.<\/p><\/div>\n

An input (for example X1<\/sub><\/em>) is passed and processed through all 5 hidden nodes (Hi<\/sub><\/em>), the results of which are combined in the output (O1<\/sub><\/em>). If you prefer, the formula is:
\nY_1 = b_0 + \\sum_{i=1}^5 { \\left( b_i f(a_{0,i} + \\sum_{j=1}^3{a_{j,i}X_{j}}) \\right) } , \u00a0\u00a0\u00a0 (1)
\nwhere b0<\/sub><\/em> and a0,i<\/sub><\/em> are constants and bi<\/sub><\/em> and aj,i<\/sub><\/em> are weights for each input Xj<\/sub><\/em> and hidden node Hi<\/sub><\/em>. Looking carefully at either Eq. 1 or Fig. 1 we can observe that each neuron is a conventional regression that passes through a transfer function f()<\/em> to become nonlinear. The neural network arranges several such neurons in a network, effectively passing the inputs through multiple (typically) nonlinear regressions and combining the results in the output node. This combination of several nonlinear regressions is what gives a neural network each approximation capabilities. With a sufficient number of nodes it is possible to approximate any function to an arbitrary degree of accuracy. Another interesting effect of this is that neural networks can encode multiple events using single binary dummy variables, as this paper<\/a> demonstrates. We could add several hidden layers, resulting in a precursor to deep-learning. In principle we could make direct connections from the inputs to layers deeper in the network or the output directly (resulting in nonlinear-linear models) or feedback loops (resulting in recurrent networks).<\/p>\n

The transfer function f()<\/em> is typically either the logistic sigmoid or the hyperbolic tangent for regression problems. The output node typically uses a linear transfer function, acting as a conventional linear regression. To really understand how the input values are transformed to the network output, we need to understand how a single neuron functions.<\/p>\n

2. Neurons<\/strong><\/p>\n

Consider a neuron as a nonlinear regression of the form (for the example with 3 inputs):<\/p>\n

H_i = f(a_{0,i} + \\sum_{j=1}^3{a_{j,i}X_{j}}) . \u00a0\u00a0\u00a0 (2)<\/p>\n

If f()<\/em> is the identity function, then (2) becomes a conventional linear regression. If f()<\/em> is nonlinear then the magic starts! Depending on the values of the weights aj,i<\/sub><\/em> and the constant a0,i<\/sub><\/em> the behaviour of neuron changes substantially. To better understand this let us take the example of a single input neuron and visualise the different behaviours. In the following interactive example you can choose:<\/p>\n