{"id":717,"date":"2014-12-17T21:02:32","date_gmt":"2014-12-17T21:02:32","guid":{"rendered":"http:\/\/kourentzes.com\/forecasting\/?p=717"},"modified":"2014-12-18T10:35:02","modified_gmt":"2014-12-18T10:35:02","slug":"the-bias-coefficient-a-new-metric-for-forecast-bias","status":"publish","type":"post","link":"https:\/\/kourentzes.com\/forecasting\/2014\/12\/17\/the-bias-coefficient-a-new-metric-for-forecast-bias\/","title":{"rendered":"The Bias Coefficient: a new metric for forecast bias"},"content":{"rendered":"

In this post I introduce a new bias metric that has several desirable properties over traditional ones.<\/p>\n

When evaluating forecasting performance it is important to look at two elements: forecasting accuracy and bias. Although there has been substantial progress in the measurement of accuracy with various metrics being proposed, there has been rather limited progress in measuring bias. A typical metrics is the Mean Error (ME):<\/p>\nME = n^{-1}\\sum_{j=1}^n{e_j},\n

where n<\/em> is the number of errors ej<\/sub><\/em>. Although in principle this is a scale dependent metric, this limitation is overcome by scaling appropriately the raw errors. Nonetheless, the ME still suffers from a number of other limitations. First, the size of errors is lost. Large positive and negative errors are lost. Although the bias information is retained we cannot infer whether an observed bias is associated with large or small errors. Second, the size of bias is very difficult to interpret, as the ME does not have any natural upper or lower bounds. Variants of ME, such as the Mean Percentage Errors (MPE) have been proposed to provide an easier to communicate bias size, expressed as a percentage:<\/p>\nMPE = \\frac{100}{n}\\sum_{j=1}^n{\\frac{e_j}{y_j}},\n

where yj<\/sub><\/em> is the actual observation at time j<\/em>. However, MPE is again unbounded and introduces additional complications. For example the MPE is not symmetric in terms of how negative and positive biases are measured. Consider the following: the forecast for a period is 90, while the observed demand is 100. The error, measured as the difference between actual and forecasted values will be 10 and the MPE = 10%. If on the other hand the forecast was 100 and the demand was 90 the error would be -10, but the MPE = -11.1%, even though the bias is actually of the same size.<\/p>\n

In this working paper<\/a> we propose a new metric, the Root Error<\/strong>, attempting to overcome these limitations. Complex number analysis is not a common tool in forecasting; however it has certain advantages that we can take advantage of here. We calculate the Root Error (RE) for each period as:<\/p>\nz_j = \\sqrt{e_j} = a + bi.\n

Since errors can be negative zj<\/sub><\/em> can be a real or imaginary number and i<\/em> is the imaginary unit that satisfies the equation i2<\/sup>=-1<\/em>. Value a<\/em> is the real part and b<\/em> is the imaginary part of the complex number. For positive errors:<\/p>\na = \\sqrt{e}\n

and b=0<\/em>, while for negative a=0<\/em> and<\/p>\nb=\\sqrt{|e|}.\n

We can summarise the RE with the Sum Root Error<\/strong> (SRE) and Mean Root Error<\/strong> (MRE) to summarise across several errors:<\/p>\n

SRE = \\sum_{j=1}^n{\\sqrt{e_j}} = \\sum_{j=1}^n{a_j} + i\\sum_{j=1}^n{b_j},
\nMRE = \\frac{1}{n}SRE = \\frac{1}{n}\\sum_{j=1}^n{a_j} + \\frac{i}{n}\\sum_{j=1}^n{b_j}.<\/p>\n

These metrics are relatively robust to outliers due to the square root involved in the calculation. Let us assume that we have three experts A, B and C. The following figure shows how ME and MRE summarise the bias and error information differently.<\/p>\n

\"rooterror.fig1\"<\/a>
\n\"rooterror.fig2\"<\/a><\/p>\n

If the negative and positive errors are equal, then a<\/em> and b<\/em> will be equal and MRE will be on the diagonal. If the positive errors are more, or the negative, then the line will be under or over the diagonal respectively. The size of the bias is represented by how distant the MRE of each expert is from the diagonal. Also note that the MRE retains the size of errors, clearly highlighting that expert B is more inaccurate, although less biased than expert C.<\/p>\n

Expressing the complex errors in their polar form allows us to achieve a more intuitive interpretation. Let us define \u03b3<\/em> as the angle of MRE:<\/p>\n \\gamma = arctan(b\/a), \\text{if } a > 0,\n

where arctan<\/em> is the inverse tangent and \u03b3<\/em> is expressed in radians. If a=0<\/em> and b>0<\/em> then \u03b3 = \u03c0\/2<\/em>. If both a, b = 0<\/em> then \u03b3 = 0<\/em>. Notice that the magnitude of MRE does not relate to the bias. Following the previous description of unbiasedness, a forecast will be unbiased for \u03b3 = \u03c0\/4<\/em>, while all forecast errors will have an angle \u03b3<\/em> between 0 and \u03c0\/2<\/em>. Instead of using \u03b3<\/em> that is expressed radians, we can define the Bias Coefficient<\/strong>:<\/p>\n\\kappa = 1 - {4\\gamma}\/{\\pi}.\n

The bias coefficient is a unit-free metric. A forecast that is always over the observed values will have a bias coefficient equal to -1, always over-forecasting, while the bias coefficient will be equal to 1 for the opposite case. Let us visualise the bias coefficient in the following figure. Assuming a large number of forecasts for different time series, the MRE per time series is calculated and subsequently the bias coefficient. This is then visualiased using a boxplot. The dot in the boxplot denotes the mean MRE.<\/p>\n

\"rooterror.fig3\"<\/a><\/p>\n

The bias coefficient:<\/p>\n