Let us assume that we have a time series we want to forecast and that we have developed a new method. This is how easy it is to produce some benchmark forecasts:<\/p>\n
> # Load the necessary libraries<\/span>\r\n> library(forecast)\r\n> library(MAPA)\r\n> library(tsintermittent)\r\n> \r\n> # In-sample data: 'y.trn'<\/span>\r\n> # Out-of sample: 'y.tst'<\/span>\r\n> # Forecasts from our new brilliant method are stored in 'mymethod'<\/span>\r\n> \r\n> # Start timer - Just to see how long this takes<\/span>\r\n> tm <- proc.time()\r\n> \r\n> # Let's produce some benchmarks<\/span>\r\n> frc <- array(NA,c(7,24)) # 7 benchmarks, forecast horizon 24\r\n> \r\n> fit.ets <- ets(y.trn)\r\n> fit.arima <- auto.arima(y.trn)\r\n> fit.mapa <- mapaest(y.trn,paral=1,outplot=FALSE)\r\n> frc[1,] <- rep(y.trn[n-24],24)\r\n> frc[2,] <- forecast(fit.ets,h=24)$mean\r\n> frc[3,] <- forecast(fit.arima,h=24)$mean\r\n> frc[4,] <- mapafor(y.trn,fit.mapa,fh=24,ifh=0,outplot=FALSE)$outfor\r\n> frc[5,] <- crost(y.trn,h=24)$frc.out\r\n> frc[6,] <- crost(y.trn,h=24,type=\"sba\")$frc.out\r\n> frc[7,] <- tsb(y.trn,h=24)$frc.out\r\n> rownames(frc) <- c(\"Naive\",\"ETS\",\"ARIMA\",\"MAPA\",\"Croston\",\"SBA\",\"TSB\")\r\n> \r\n> # Calculate accuracy<\/span>\r\n> PE <- (matrix(rep(y.tst,8),nrow=8,byrow=TRUE) - \r\n+ rbind(mymethod,frc))\/matrix(rep(y.tst,8),nrow=8,byrow=TRUE)\r\n> MAPE <- rowMeans(abs(PE))*100\r\n> \r\n> # Stop timer<\/span>\r\n> tm <- proc.time() - tm\r\n<\/pre>\nSo what we have here is forecasts from ‘mymethod’ and some benchmarks:<\/p>\n
\n- Naive: hopefully our method predicts better than the random walk.<\/li>\n
- ETS: state space exponential smoothing from the ‘forecast’ package.<\/li>\n
- MAPA: multiple aggregation prediction algorithm using ets from the ‘MAPA’ package.<\/li>\n
- Croston’s method: this is supposed to be used to intermittent data, but I just wanted to demonstrate how easy is to use this as a benchmark. This is from the ‘tsintermittent’ package.<\/li>\n
- SBA: this a variant of Croston’s method from the ‘tsintermittent’ package.<\/li>\n
- TSB: another intermittent demand method from the ‘tsintermittent’ package.<\/li>\n<\/ul>\n
Of course not all benchmarks are appropriate, but I wanted to demonstrate how easy is to use them. Accuracy is assessed using Mean Absolute Percentage Error (MAPE), for t+1 up to t+24 forecasts.<\/p>\n
> print(round(MAPE,2))\r\nmymethod Naive ETS ARIMA MAPA Croston SBA TSB \r\n 5.76 7.37 6.00 7.34 5.59 5.76 7.07 7.26<\/span> \r\n> print(tm)\r\n user system elapsed \r\n 4.24 0.00 4.86<\/span> \r\n<\/pre>\nApparently mymethod is pretty good, but in terms of accuracy MAPA seems to be doing better. Perhaps it is interesting to see that Croston’s method is not that bad, which makes sense considering that for non-intermittent data it is equivalent to exponential smoothing.<\/p>\n
The whole benchmarking took very little coding and only 4.86 seconds to run. This could be sped up using parallel processing that both ‘forecast’ and ‘MAPA’ packages support. Of course we would prefer to have used rolling origin evaluation (cross-validation), and this could be implemented easily with a loop.<\/p>\n
Here is the series and the various forecasts:<\/p>\n
> cmp = rainbow(8, start = 0\/6, end = 4\/6)\r\n> ts.plot(y.trn,y.tst,ts(t(rbind(mymethod,frc)),frequency=12,end=end(y.tst)),\r\n+ col=c(\"black\",\"black\",cmp))\r\n> legend(\"bottomleft\",c(\"MyMethod\",rownames(frc)),col=cmp,lty=1,ncol=2)\r\n<\/pre>\n![\"bench.fig1\"](\"http:\/\/kourentzes.com\/forecasting\/wp-content\/uploads\/2014\/07\/bench.fig1_.png\")
<\/a><\/p>\nThe example series is ‘referrals’ from the ‘MAPA’ package. In other posts here you can find more information about the functions in the MAPA<\/a> and tsintermittent<\/a> packages.<\/p>\nTo conclude: benchmark your forecasts, it is easy and necessary!<\/strong><\/p>\n