The idea here is to give Prophet a hard time, but also avoid using too exotic forecasting methods.<\/p>\n

I provide the mean and median MASE across all forecast origins and series for each subset in tables 2 and 3 respectively. In brackets I provide the percentage difference from the ETS’ accuracy. In boldface I have highlight the best forecast for each M3 subset. Prophet results are in blue<\/span>. I provide two MAPA results, the first uses the default options, whereas the second uses comb=”w.mean” that is more mindful of seasonality. For THieF I only provide the default result (using ETS), as in principle it could be applied to any forecast on the table.
\n

\n\r\n\r\n

Table 2. Mean MASE results<\/caption>\n

Set<\/th>\n

ETS<\/th>\n

ARIMA<\/th>\n

ES (smooth)<\/th>\n

SSARIMA (smooth)<\/th>\n

MAPA<\/th>\n

MAPA (w.mean)<\/th>\n

THieF (ETS)<\/th>\n

Prophet<\/span><\/th>\n<\/tr>\n<\/thead>\n

Yearly<\/td>\n

0.732 (0.00%)<\/strong><\/td>\n

0.746 (-1.91%)<\/td>\n

0.777 (-6.15%)<\/td>\n

0.783 (-6.97%)<\/td>\n

0.732 (0.00%)<\/td>\n

0.954 (-30.33%)<\/span><\/td>\n<\/tr>\n\r\n

Quarterly<\/td>\n

0.383 (0.00%)<\/strong><\/td>\n

0.389 (-1.57%)<\/td>\n

0.385 (-0.52%)<\/td>\n

0.412 (-7.57%)<\/td>\n

0.386 (-0.78%)<\/td>\n

0.384 (-0.26%)<\/td>\n

0.400 (-4.44%)<\/td>\n

0.553 (-44.39%)<\/span><\/td>\n<\/tr>\n\r\n

Monthly<\/td>\n

0.464 (0.00%)<\/td>\n

0.472 (-1.72%)<\/td>\n

0.465 (-0.22%)<\/td>\n

0.490 (-5.60%)<\/td>\n

0.459 (+1.08%)<\/td>\n

0.458 (+1.29%)<\/td>\n

<\/strong>0.462 (+0.43%)<\/td>\n

0.586 (-26.29%)<\/span><\/td>\n<\/tr>\n\r\n

Other<\/td>\n

0.447 (0.00%)<\/td>\n

0.460 (-2.91%)<\/td>\n

0.446 (+0.22%)<\/td>\n

0.457 (-2.24%)<\/td>\n

0.444 (+0.67%)<\/strong><\/td>\n

0.447 (0.00%)<\/td>\n

0.554 (-23.94%)<\/span><\/td>\n<\/tr>\n<\/tbody><\/table><\/div><\/p>\n

\n\r\n\r\n

Table 3. Median MASE results<\/caption>\n
Set<\/th>\n	ETS<\/th>\n	ARIMA<\/th>\n	ES (smooth)<\/th>\n	SSARIMA (smooth)<\/th>\n	MAPA<\/th>\n	MAPA (w.mean)<\/th>\n	THieF (ETS)<\/th>\n	Prophet<\/span><\/th>\n<\/tr>\n<\/thead>\n
Yearly<\/td>\n	0.514 (0.00%)<\/td>\n	0.519 (-0.97%)<\/td>\n	0.511 (+0.58%)<\/strong><\/td>\n	0.524 (-1.95%)<\/td>\n	0.520 (-1.17%)<\/td>\n	0.520 (-1.17%)<\/td>\n	0.514 (0.00%)<\/td>\n	0.710 (-38.13%)<\/span><\/td>\n<\/tr>\n\r\n
Quarterly<\/td>\n	0.269 (0.00%)<\/td>\n	0.266 (+1.12%)<\/td>\n	0.256<\/span> (+4.83%)<\/td>\n	0.278 (-3.35%)<\/td>\n	0.254 (+5.58%)<\/strong><\/td>\n	0.254 (+5.58%)<\/strong><\/td>\n	0.262 (+2.60%)<\/td>\n	0.388 (-44.24%)<\/span><\/td>\n<\/tr>\n\r\n
Monthly<\/td>\n	0.353 (0.00%)<\/td>\n	0.348 (+1.42%)<\/strong><\/td>\n	0.351 (+0.57%)<\/td>\n	0.373 (-5.67%)<\/td>\n	0.352 (+0.28%)<\/td>\n	0.351 (+0.57%)<\/td>\n	0.351 (+0.57%)<\/td>\n	0.473 (-33.99%)<\/span><\/td>\n<\/tr>\n\r\n
Other<\/td>\n	0.275 (0.00%)<\/td>\n	0.269 (+2.18%)<\/td>\n	0.270 (+1.82%)<\/td>\n	0.268 (+2.55%)<\/strong><\/td>\n	0.283 (-2.91%)<\/td>\n	0.283 (-2.91%)<\/td>\n	0.275 (0.00%)<\/td>\n	0.320 (-16.36%)<\/span><\/td>\n<\/tr>\n<\/tbody><\/table><\/div>\n Some comments about the results:<\/p>\n \n Prophet performs very poorly. The dataset does not contain multiple seasonalities, but it does contain human-activity based seasonal patters (quarterly and monthly series), changing trends and outliers or other abrupt changes (especially the `other’ subset), where Prophet should do ok. My concern is not that it is not ranking first, but that at best it is almost 16% worse than exponential smoothing (and at worst almost 44%!);<\/li>\n ETS and ARIMA between packages perform reasonably similar, indicating that although there are implementation differences, both packages have followed sound modelling philoshopies;<\/li>\n MAPA and THieF are meant to work on the quarterly and monthly subsets, where, in line with the research, they improve upon their base model (ETS).<\/li>\n<\/ul>\n In all fairness, more testing is needed on high frequency data with multiple seasonalities before one should conclude about the performance of Prophet. Nonetheless. for the vast majority of business forecasting needs (such as supply chain forecasting), Prophet does not seem to perform that well. As a final note, this is an open source project, so I am expecting over time to see interesting improvements.<\/p>\n Finally, I want to thank Oliver Schaer<\/a> for providing me with Prophet R code examples! You can also find some examples here<\/a>.<\/p>\n Related Posts<\/H3> Special issue on innovations in hierarchical forecasting<\/a><\/li>\n Automatic robust estimation for exponential smoothing: perspectives from statistics and machine learning<\/a><\/li>\n Elucidate structure in intermittent demand time series<\/a><\/li>\n<\/ul><\/div>","protected":false},"excerpt":{"rendered":" Last February Facebook open sourced its Prophet forecasting tool. Since, it had appeared in quite a few discussions online. A good thing about Prophet is that it one can use it very easily through R (and Python). This gave me the opportunity to benchmark it against some more standard – and not! – forecasting models\u2026 Read More »<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[41],"tags":[14,24,32,38,39],"_links":{"self":[{"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/posts\/1402"}],"collection":[{"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/comments?post=1402"}],"version-history":[{"count":10,"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/posts\/1402\/revisions"}],"predecessor-version":[{"id":1412,"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/posts\/1402\/revisions\/1412"}],"wp:attachment":[{"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/media?parent=1402"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/categories?post=1402"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kourentzes.com\/forecasting\/wp-json\/wp\/v2\/tags?post=1402"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}