After years of working in supply chain operations — first in industry, then as a consultant helping corporations and clients better manage their demand planning processes and technologies — it's still surprising to hear so many demand planners misuse the terms fit and error.
The concepts are relatively straightforward, yet they're quite distinct, and using one in place of the other is incorrect. Anyone in a position to improve supply chain operations by influencing demand planning should understand the difference between fit and error and be mindful of both the technical and functional implications these terms have in the world of forecasting.
By Definition...
Forecast fit describes the relative difference between actual historical data and a hypothetical forecast generated by a statistical model (or algorithm) using that same historical data as input. It's quite literally a backward-looking assessment of how closely a forecast created by any one of various statistical models would stack up against — or "fit" when compared to — actual historical demand.
Planners use forecast fit to project the suitability of one or more statistical forecasting algorithms to accurately forecast future demand (see Figure 1).
Figure 1: Forecast Fit — After running statistical forecasting models using actual historical demand data as input, planners analyze the "fit" of the resulting fitted forecasts to determine which model will likely yield forecasts that most closely match past demand patterns.
Forecast Error
Forecast error is defined by APICS as "the difference between actual and forecast demand, stated as an absolute value or as a percentage." Forecast error is a postmortem benchmark of the variance between demand that was projected and actual demand that subsequently occurred (see Figure 2).
Figure 2: Forecast Error — The variance, expressed as an absolute value or a percentage, between forecast demand and actual demand.
Opportunity: The Case for Using Fit and Error
Consider the example of one Plan4Demand client, a small business comprised of multiple shops that sell flowers, gift items and packaging supplies. Business activity is highly seasonal, with peaks around Christmas, Valentine's Day, Mother's Day and other signature occasions. At any given time, the manager of these stores oversees a total of 50,000 items in stock throughout 10 locations.
Until recently, he relied on historical sales data from prior years as a guide for how much product to buy for each upcoming season. As the company's network of stores expanded over the years, however, planning in advance for expected demand became more complex, resulting in more stockouts or overstocking — problems directly related to forecasting.
Despite spending more time planning, the manager still found himself ordering either too much or too little. To streamline operations — inventory management, cash flow, etc. — company leaders decided to purchase a demand planning (DP) tool to help produce more accurate forecasts that the manager might then simply convert to purchase orders. After loading historical sales data into the forecasting tool, however, the manager ran into trouble configuring the package.
Software documentation directed him simply to "choose a forecasting algorithm with the best fit." The software offered multiple statistical forecasting algorithms to choose from, as do most demand planning applications. Common algorithms include Holt-Winters, Fourier and Box-Jenkins, but there are many other algorithms and many more that are proprietary to specific software vendors.
In addition to offering various algorithms to generate fitted forecasts based on historical demand data, as shown in Figure 1, demand planning software also calculates various metrics regarding the fit of each model's forecast. These metrics can be used to automatically identify the best model for forecasting future demand, or planners may choose a model themselves, based on their own judgment.
Once an algorithm is selected to produce a working forecast and subsequent consumer demand is fulfilled over a period of time, the forecasting software can then be used to report on forecast error, using various statistical metrics to indicate how well actual demand stacked up against forecast demand.
A robust demand planning package can provide rich insights to help planners streamline operational efficiency, improve cost-effectiveness and increase profitability, but not all planners share the same level of expertise, and not all businesses have the luxury of employing skilled statisticians to drive planning using such tools.
That was the situation facing our client's store manager, and his case was a perfect example of how confusion about forecast fit and error can compound the task of projecting what products are likely to sell in the future. This case also illustrates the importance of understanding the difference between fit and forecast and their value in terms of demand forecasting.
Understanding Forecast Fit
When evaluating the fitted forecasts created by statistical forecasting algorithms, a planner should consider two fundamental questions:
As previously described, most DP applications first apply multiple algorithms to create forecast projections based on historical demand data and then assess the relative accuracy of the resulting fitted forecasts — the models "fitted" to the actual historical data.
Sometimes called ex-post forecasting, or forecasting after-the-fact, this exercise helps planners answer the question "What would I have forecast had I used a particular algorithm?" The next step — analyzing fit — answers the question "How accurate would my forecast have been?" Generally speaking, if one fitted forecast has lower error than another, it's reasonable to presume that the algorithm that produced the fitted forecast will yield actual forecasts with similarly low error in the future.
As shown in Figure 1, ex-post forecasting reveals at a glance that the forecast generated by Model 1 fits the actual demand history better than the forecast created by Model 2. To precisely determine the relative accuracy of fitted forecasts — especially when comparing more than one — demand planning packages automatically calculate the error of each fitted forecast. By contrasting the resulting fit measures of each algorithm, DP packages help demand planners select optimal forecasting models based on their organization's historical demand. Some demand planning applications define this automated selection capability as "best fit."
Evaluating forecast fit is a great way to determine the potential quality of future statistical forecasts if you believe that historical demand is a valid indicator of future demand for your organization.
If you have any reasonable expectation that future demand will significantly deviate from history, however — due to a projected surge in new customers, promotions or increased competition, for example — then fit has only limited utility as a metric for helping you analyze and assess the potential value of a statistical forecasting algorithm.
Understanding Forecast Error
Once you select a preferred statistical algorithm to generate a forecast and start fulfilling subsequent demand, you'll begin accruing the data you need to calculate error and gauge the accuracy of your forecast compared to actual demand, as shown in Figure 2.
Clearly, forecast error is an after-the-fact measure that business leaders may use to drive process/ technology improvements, or as a performance benchmark of a planner's abilities (or lack thereof) to accurately forecast demand.
Error revealed by this analysis is typically attributable to the degree of fit that a forecasting model is able to achieve with the historical data. Error may also reflect changes in business conditions that occur after creation of the actual forecast.
One best-practice measure often overlooked during implementation or optimization of a demand planning process is the review of forecast error. Tracking error on a regular basis can reveal problems in your forecasting process or indicate opportunities to tune or adjust the statistical forecasting engine within your demand planning software — to select different mathematical models, reconfigure forecasting time horizons or change smoothing constant parameters — in an effort to improve your forecast.
Putting it All Together: Applying Fit and Error in the Real World
Assessing forecast fit helps planners choose an optimal forecasting algorithm to use prior to forecasting future demand. Assessing forecast error suggests opportunities for tuning a forecast going forward, after actual demand has occurred.
Based on the analysis of forecast fit illustrated in Figure 1, which showed that Model 1 promised to provide a better fit for future forecasts than Model 2, Figure 3 illustrates the natural outcome of that decision — how Model 1 was used to create a forecast for future demand, how subsequent demand actually stacked up in comparison to that forecast, and how the gaps between subsequent actual demand and forecast demand reveal the forecast error.
Figure 3: Forecast Fit looks at history to determine which algorithm to use. Forecast error is the difference between actual and forecasted sales. It is used to provide a post-mortem analysis of forecast performance. Algorithm 1 promised a better "fit" and was used to generate the forecast.
Whereas forecast fit initially provided guidance for projecting future demand, forecast error serves to guide ongoing forecast quality.
Common Measures of Error and Fit
One similarity between forecast fit and error is that most of the metrics used to measure them are the same. They're simply the mathematical differences between projected demand and actual demand. Most of the metrics are automatically calculated by the forecasting algorithm(s) during the process of generating a statistical forecast.
What determines the relevance of these metrics to either fit or error depends on when they are applied within the forecasting process — during the creation of fitted forecasts or ex-post facto.
Some of the most common measures used to evaluate fit and error are:
Figure 4 shows a data set comprising actual historical sales for time periods 1-12 and forecasted sales for time periods 1-24 using two algorithms — Algorithm 1 and Algorithm 2. Periods 1-12 refer to historical time periods in the past; Periods 13-24 reference future time periods for which each of the two forecasting algorithms has generated a forecast but for which actual demand has yet to occur.
Figure 4: Forecast Data Sample
By using both the actual demand data and data from the two fitted forecasts — all from Periods 1-12 — the demand planning software package performs an ex-post forecasting analysis to evaluate the fit of the two algorithms. This analysis helps identify which of the algorithms better fits the historical data and which can potentially generate a better forecast.
The results of the mathematical analysis are summarized in the Figure 5 table of fit and error metrics. For Algorithm 1, the detailed calculations used to determine each of those metrics are detailed in Appendices A, B, C, D and E, which are located at the bottom of this article. For Algorithm 2, the results are calculated similarly but are not shown.
Figure 5: Fit and error metrics
Improved Forecasting: The Benefits of Fit and Error
Understanding forecast fit and its corresponding metrics helps planners tune their statistical models and improve forecast accuracy. Some tools provide simulation capabilities to help determine optimal configuration parameters, but clearly the ability to use such tools to make effective comparisons requires a reasonably clear understanding of fit and fit metrics in the first place — and how such insights serve to help make optimal decisions when it comes to forecasting.
Analyzing fit provides revealing insight into naturally occurring demand variation. It can also indicate the fundamental suitability of available historical data by revealing whether historical data really represents the future, or whether it is even statistically forecastable.
Error metrics provide the basis for analyzing forecast performance. They help planners pinpoint areas where there may be problems with the statistical forecast or where assumptions that were valid at the time of forecast generation may have changed in the interim, warranting attention and possible corrective action. (For example, the business picked up a new customer, a major customer was acquired, etc.)
Forecast error can also indicate possible time series problems, for example, history remapping to wrong customers, etc. Error is a key metric for exception-based forecasting; it enables planners to prioritize their efforts, work more efficiently and rapidly drive business improvements by focusing on products with high error rates. And finally, forecast error is a powerful tool for evaluating the impact of forecaster bias.
Fit and error hold great potential to help planners more fully leverage the power of their demand forecasting software; and both provide baselines and benchmarks for improving this most fundamental aspect of supply chain management. Anyone in a position to impact demand planning and forecasting has a responsibility to understand the difference between these two concepts and the value they hold.
Clearly.
About the Author: Atul Mandal is employed by Plan4Demand Solutions, a supply chain management consulting firm specializing in demand planning, fulfillment, transportation management, warehouse management, sales and operations planning, and customized training. Mandal is Plan4Demand's consulting manager, with principal expertise in demand planning, supply chain network optimization and forecasting collaboration processes. For more information, visit www.plan4demand.com
Appendices
To learn more about forecast fit and error, review the various calculation tables in the appendices below or call Plan4Demand at 866-P4D-INFO.
Appendix A: Mean Squared Error (MSE) and Standard Error (SE) Calculations
Appendix B: Coefficient of Determination (R2) Calculations
Appendix C: Mean Absolute Deviation/Error (MAD/MAE) Calculations
Appendix D: Mean Absolute Percent Error (MAPE) Calculations
Appendix E: Symmetric Mean Absolute Percent Error (SMAPE) Calculations
The concepts are relatively straightforward, yet they're quite distinct, and using one in place of the other is incorrect. Anyone in a position to improve supply chain operations by influencing demand planning should understand the difference between fit and error and be mindful of both the technical and functional implications these terms have in the world of forecasting.
By Definition...
Forecast fit describes the relative difference between actual historical data and a hypothetical forecast generated by a statistical model (or algorithm) using that same historical data as input. It's quite literally a backward-looking assessment of how closely a forecast created by any one of various statistical models would stack up against — or "fit" when compared to — actual historical demand.
Planners use forecast fit to project the suitability of one or more statistical forecasting algorithms to accurately forecast future demand (see Figure 1).
Figure 1: Forecast Fit — After running statistical forecasting models using actual historical demand data as input, planners analyze the "fit" of the resulting fitted forecasts to determine which model will likely yield forecasts that most closely match past demand patterns.
Forecast Error
Forecast error is defined by APICS as "the difference between actual and forecast demand, stated as an absolute value or as a percentage." Forecast error is a postmortem benchmark of the variance between demand that was projected and actual demand that subsequently occurred (see Figure 2).
Figure 2: Forecast Error — The variance, expressed as an absolute value or a percentage, between forecast demand and actual demand.
Opportunity: The Case for Using Fit and Error
Consider the example of one Plan4Demand client, a small business comprised of multiple shops that sell flowers, gift items and packaging supplies. Business activity is highly seasonal, with peaks around Christmas, Valentine's Day, Mother's Day and other signature occasions. At any given time, the manager of these stores oversees a total of 50,000 items in stock throughout 10 locations.
Until recently, he relied on historical sales data from prior years as a guide for how much product to buy for each upcoming season. As the company's network of stores expanded over the years, however, planning in advance for expected demand became more complex, resulting in more stockouts or overstocking — problems directly related to forecasting.
Despite spending more time planning, the manager still found himself ordering either too much or too little. To streamline operations — inventory management, cash flow, etc. — company leaders decided to purchase a demand planning (DP) tool to help produce more accurate forecasts that the manager might then simply convert to purchase orders. After loading historical sales data into the forecasting tool, however, the manager ran into trouble configuring the package.
Software documentation directed him simply to "choose a forecasting algorithm with the best fit." The software offered multiple statistical forecasting algorithms to choose from, as do most demand planning applications. Common algorithms include Holt-Winters, Fourier and Box-Jenkins, but there are many other algorithms and many more that are proprietary to specific software vendors.
In addition to offering various algorithms to generate fitted forecasts based on historical demand data, as shown in Figure 1, demand planning software also calculates various metrics regarding the fit of each model's forecast. These metrics can be used to automatically identify the best model for forecasting future demand, or planners may choose a model themselves, based on their own judgment.
Once an algorithm is selected to produce a working forecast and subsequent consumer demand is fulfilled over a period of time, the forecasting software can then be used to report on forecast error, using various statistical metrics to indicate how well actual demand stacked up against forecast demand.
A robust demand planning package can provide rich insights to help planners streamline operational efficiency, improve cost-effectiveness and increase profitability, but not all planners share the same level of expertise, and not all businesses have the luxury of employing skilled statisticians to drive planning using such tools.
That was the situation facing our client's store manager, and his case was a perfect example of how confusion about forecast fit and error can compound the task of projecting what products are likely to sell in the future. This case also illustrates the importance of understanding the difference between fit and forecast and their value in terms of demand forecasting.
Understanding Forecast Fit
When evaluating the fitted forecasts created by statistical forecasting algorithms, a planner should consider two fundamental questions:
- Are the forecasts of good quality?
- Considering the number of algorithms most packages offer, which will likely provide the most accurate forecasts going forward?
As previously described, most DP applications first apply multiple algorithms to create forecast projections based on historical demand data and then assess the relative accuracy of the resulting fitted forecasts — the models "fitted" to the actual historical data.
Sometimes called ex-post forecasting, or forecasting after-the-fact, this exercise helps planners answer the question "What would I have forecast had I used a particular algorithm?" The next step — analyzing fit — answers the question "How accurate would my forecast have been?" Generally speaking, if one fitted forecast has lower error than another, it's reasonable to presume that the algorithm that produced the fitted forecast will yield actual forecasts with similarly low error in the future.
As shown in Figure 1, ex-post forecasting reveals at a glance that the forecast generated by Model 1 fits the actual demand history better than the forecast created by Model 2. To precisely determine the relative accuracy of fitted forecasts — especially when comparing more than one — demand planning packages automatically calculate the error of each fitted forecast. By contrasting the resulting fit measures of each algorithm, DP packages help demand planners select optimal forecasting models based on their organization's historical demand. Some demand planning applications define this automated selection capability as "best fit."
Evaluating forecast fit is a great way to determine the potential quality of future statistical forecasts if you believe that historical demand is a valid indicator of future demand for your organization.
If you have any reasonable expectation that future demand will significantly deviate from history, however — due to a projected surge in new customers, promotions or increased competition, for example — then fit has only limited utility as a metric for helping you analyze and assess the potential value of a statistical forecasting algorithm.
Understanding Forecast Error
Once you select a preferred statistical algorithm to generate a forecast and start fulfilling subsequent demand, you'll begin accruing the data you need to calculate error and gauge the accuracy of your forecast compared to actual demand, as shown in Figure 2.
Clearly, forecast error is an after-the-fact measure that business leaders may use to drive process/ technology improvements, or as a performance benchmark of a planner's abilities (or lack thereof) to accurately forecast demand.
Error revealed by this analysis is typically attributable to the degree of fit that a forecasting model is able to achieve with the historical data. Error may also reflect changes in business conditions that occur after creation of the actual forecast.
One best-practice measure often overlooked during implementation or optimization of a demand planning process is the review of forecast error. Tracking error on a regular basis can reveal problems in your forecasting process or indicate opportunities to tune or adjust the statistical forecasting engine within your demand planning software — to select different mathematical models, reconfigure forecasting time horizons or change smoothing constant parameters — in an effort to improve your forecast.
Putting it All Together: Applying Fit and Error in the Real World
Assessing forecast fit helps planners choose an optimal forecasting algorithm to use prior to forecasting future demand. Assessing forecast error suggests opportunities for tuning a forecast going forward, after actual demand has occurred.
Based on the analysis of forecast fit illustrated in Figure 1, which showed that Model 1 promised to provide a better fit for future forecasts than Model 2, Figure 3 illustrates the natural outcome of that decision — how Model 1 was used to create a forecast for future demand, how subsequent demand actually stacked up in comparison to that forecast, and how the gaps between subsequent actual demand and forecast demand reveal the forecast error.
Figure 3: Forecast Fit looks at history to determine which algorithm to use. Forecast error is the difference between actual and forecasted sales. It is used to provide a post-mortem analysis of forecast performance. Algorithm 1 promised a better "fit" and was used to generate the forecast.
Whereas forecast fit initially provided guidance for projecting future demand, forecast error serves to guide ongoing forecast quality.
Common Measures of Error and Fit
One similarity between forecast fit and error is that most of the metrics used to measure them are the same. They're simply the mathematical differences between projected demand and actual demand. Most of the metrics are automatically calculated by the forecasting algorithm(s) during the process of generating a statistical forecast.
What determines the relevance of these metrics to either fit or error depends on when they are applied within the forecasting process — during the creation of fitted forecasts or ex-post facto.
Some of the most common measures used to evaluate fit and error are:
- Mean Squared Error (MSE)
- Standard Error (SE)
- Coefficient of Determination or R-Squared (R2)
- Mean Absolute Deviation (MAD) or Mean Absolute Error (MAE)
- Mean Absolute Percentage Error (MAPE)
- Symmetric Mean Absolute Percentage Error (SMAPE)
Figure 4 shows a data set comprising actual historical sales for time periods 1-12 and forecasted sales for time periods 1-24 using two algorithms — Algorithm 1 and Algorithm 2. Periods 1-12 refer to historical time periods in the past; Periods 13-24 reference future time periods for which each of the two forecasting algorithms has generated a forecast but for which actual demand has yet to occur.
Figure 4: Forecast Data Sample
By using both the actual demand data and data from the two fitted forecasts — all from Periods 1-12 — the demand planning software package performs an ex-post forecasting analysis to evaluate the fit of the two algorithms. This analysis helps identify which of the algorithms better fits the historical data and which can potentially generate a better forecast.
The results of the mathematical analysis are summarized in the Figure 5 table of fit and error metrics. For Algorithm 1, the detailed calculations used to determine each of those metrics are detailed in Appendices A, B, C, D and E, which are located at the bottom of this article. For Algorithm 2, the results are calculated similarly but are not shown.
Figure 5: Fit and error metrics
Improved Forecasting: The Benefits of Fit and Error
Understanding forecast fit and its corresponding metrics helps planners tune their statistical models and improve forecast accuracy. Some tools provide simulation capabilities to help determine optimal configuration parameters, but clearly the ability to use such tools to make effective comparisons requires a reasonably clear understanding of fit and fit metrics in the first place — and how such insights serve to help make optimal decisions when it comes to forecasting.
Analyzing fit provides revealing insight into naturally occurring demand variation. It can also indicate the fundamental suitability of available historical data by revealing whether historical data really represents the future, or whether it is even statistically forecastable.
Error metrics provide the basis for analyzing forecast performance. They help planners pinpoint areas where there may be problems with the statistical forecast or where assumptions that were valid at the time of forecast generation may have changed in the interim, warranting attention and possible corrective action. (For example, the business picked up a new customer, a major customer was acquired, etc.)
Forecast error can also indicate possible time series problems, for example, history remapping to wrong customers, etc. Error is a key metric for exception-based forecasting; it enables planners to prioritize their efforts, work more efficiently and rapidly drive business improvements by focusing on products with high error rates. And finally, forecast error is a powerful tool for evaluating the impact of forecaster bias.
Fit and error hold great potential to help planners more fully leverage the power of their demand forecasting software; and both provide baselines and benchmarks for improving this most fundamental aspect of supply chain management. Anyone in a position to impact demand planning and forecasting has a responsibility to understand the difference between these two concepts and the value they hold.
Clearly.
About the Author: Atul Mandal is employed by Plan4Demand Solutions, a supply chain management consulting firm specializing in demand planning, fulfillment, transportation management, warehouse management, sales and operations planning, and customized training. Mandal is Plan4Demand's consulting manager, with principal expertise in demand planning, supply chain network optimization and forecasting collaboration processes. For more information, visit www.plan4demand.com
Appendices
To learn more about forecast fit and error, review the various calculation tables in the appendices below or call Plan4Demand at 866-P4D-INFO.
Appendix A: Mean Squared Error (MSE) and Standard Error (SE) Calculations
Appendix B: Coefficient of Determination (R2) Calculations
Appendix C: Mean Absolute Deviation/Error (MAD/MAE) Calculations
Appendix D: Mean Absolute Percent Error (MAPE) Calculations
Appendix E: Symmetric Mean Absolute Percent Error (SMAPE) Calculations
