Improving Forecast Accuracy
Improving forecast accuracy often features the business value expected from a system implementation like Anaplan for demand planning. In reality, forecast accuracy is not a business value—but it contributes to business value in terms of better service levels and economic inventory levels. Often, customers want to improve forecast accuracy using an advanced analytical software algorithm or AI/ML (read as a magic wand!), but the hard reality is there is no magic wand. The forecast accuracy of a product is not just controlled by the manufacturer or distributor or retailer, but it's a function of the whole world around us. Competitor's events, natural disasters, government policies, product inventions, and several other factors can throw the forecast accuracy upside down. Most of the time, customers perceive that poor forecast accuracy is the only reason for underperformance, such as stock-outs and aging inventory. Here we'll discuss the finer details of forecast accuracy and its implications on business value.
What is Forecast Accuracy?
Forecast accuracy is a measure of how close the forecast is to the actuals. It can also be defined as a converse of forecast error which is the measure of how far the forecast is from the actuals. There are multiple metrics to measure forecast accuracy; they have to be carefully chosen based on the business requirements.
Why Measure Forecast Accuracy?
- Anything that is not measured cannot be managed well.
- Prioritization for demand review: It will help to categorize the relatively accurate products in terms of forecast accuracy from the ones that are consistently inaccurate. Planner's capacity is limited and allows them to review only a handful of products, so it is essential to categorize the consistently inaccurate products for their review.
- Consistently accurate products can be run in an autopilot mode requiring no review by the planner. This improves the productivity of the planners.
- Demand shaping: Consistently inaccurate products can be studied closely to identify the causes that make them consistently inaccurate. The causes could be internal, such as promotion of a related product, or the cause could be external, such as promotion by competition, pandemic, natural calamity, etc. Once a cause is identified, various mitigation actions could be taken to make it produce an accurate forecast.
- Safety stock: We need to know how far the actuals might deviate from the forecast so that there can be contingency plans. Forecast error is an important driver in determining the safety stock. The more the error, the more is the safety stock for a given service level.
How Do You Measure Forecast Accuracy?
There are infinite ways to measure forecast accuracy. There are plenty of chances to make a mistake. Often we see there are misconceptions and wrong metrics are used, or a correct metric at a wrong aggregation level is used. All the metrics have their own advantages and setbacks. Right metrics have to be chosen by paying proper due diligence, and once selected, they should be tracked for a sufficiently long period of time. Avoid choosing multiple metrics as this will only result in more confusion and no conclusion. A table of all metrics with their significance in various scenarios can be found in the attachment to this post.
Should I Use $ or Units to Measure Accuracy?
Accuracy can be measured in terms of units or in terms of currency ($). Measuring accuracy in terms of currency value ($) has some advantages over using units. Forecast error in $ value helps the demand planner to focus and prioritize on big revenue impact products/SKUs. It can be used as a tool to prepare the priority list in terms of revenue impact to the company.
What is the Role of Lead Time in Forecast Accuracy?
Consider the above supply chain where the supplier takes about 90 days to deliver to the distribution center (DC), and the DC takes about 2 days to deliver to the stores. DC uses a demand forecast and determines how much to buy from the supplier. This decision has to be made 90 days in advance. Assuming the purchase cycle is monthly, once in the current month M0, the orders are placed for M+3. So when we measure accuracy for M+3, we must use the forecast generated in the month M0. For example, to measure the accuracy for May, we must use the forecast generated in February, and to measure the forecast accuracy of June, we must use the forecast generated in March. In general, we must use the forecast snapshot offset by lead time to measure the forecast accuracy.
Forecast Snapshots
Consider the following scenario where the current period is May 2021. There are forecast snapshots in each cycle respectively on Apr 21, Mar 21, Feb 21, Jan 21, and so on. Forecast time series has two components—one is backcast/fit which is generated for the periods before the current period and the other is forecast generated for the periods on and after the current period. While measuring forecast accuracy, backcast/fit values should not be used. They are only used by software algorithms to select the best fit method and should not be confused with forecast accuracy.
To measure forecast accuracy for an actualized month, the forecast snapshot that was generated in a cycle offset by the lead time of the respective product must be used. For example, to measure the forecast accuracy of Apr 2021 in the below table, the forecast snapshot of Jan 2021 must be used. Similarly, to measure the accuracy for Mar 2021, the forecast snapshot of Dec 2020 must be used.
In What Level of Detail Can Forecast Accuracy Be Measured?
Measuring forecast accuracy at different levels produces different results. So it is important to measure the forecast accuracy at the right level. Measuring forecast accuracy on a weekly level vs measuring the same at a monthly level produces a vast difference in the accuracy metrics.
| Time Period | Actual | Forecast Snapshot | MAPE |
| Wk 1 | 100 | 150 | 33% |
| Wk 2 | 100 | 150 | 33% |
| Wk 3 | 100 | 150 | 33% |
| Wk 4 | 300 | 150 | 100% |
| Month | 600 | 600 | 0% |
The average error of the weekly measurements is 50%, while the error measured at the monthly level is 0%. So, it is important to track the right level and right error. To do this, one must understand the purpose of the forecast. The forecast could be used for multiple purposes, including monthly replenishment of a warehouse/DC, weekly replenishment of a store, etc. Measuring a forecast accuracy at week level for a forecast which is used to replenish a DC monthly once is incorrect. Similarly, measuring forecast accuracy at the store level is not needed, as the purpose is to replenish the DC. Hence, based on the purpose of the forecast, the level at which forecast accuracy measurement be fixed. This has to be done on Product - Location - Time dimensions. It is not advisable to measure forecast accuracy at multiple levels within the same hierarchy.
How Do I Aggregate Forecast Accuracy?
Measuring forecast accuracy for a product or product-location combination is an easy task. The challenge is how to aggregate forecast accuracy of 1000s of such products and represent them at an aggregate level. How to represent the forecast performance of an entire product family or an entire region, are challenging questions to answer. The use of aggregating forecast accuracy is to have one number that measures and tracks the forecast performance at the macro level. It can also be used to compare the trend of the forecasting process to know the process is improving or deteriorating from cycle to cycle.
| Product | Actual | Forecast Snapshot | MAPE |
| A | 100 | 80 | 25% |
| B | 100 | 120 | 17% |
| C | 100 | 80 | 25% |
| D | 100 | 120 | 17% |
| Prd Fmly | 400 | 400 | 0% |
In the above table, the aggregate forecast error at the product family is not representing the true errors of the underlying products. This is the reason that forecast accuracy should not be aggregated using formulas, and forecast accuracy tiers should be introduced.
If anything less than 20% MAPE is accepted as Hit, the aggregation becomes easy in terms of how many hits are there within each product family can be measured and compared. In this case, a 50% hit ratio is observed for the given product family.
| Product | Actual | Forecast Snapshot | MAPE | Hit / Miss | Hit Count |
| A | 100 | 80 | 25% | Miss | 0 |
| B | 100 | 120 | 17% | Hit | 1 |
| C | 100 | 80 | 25% | Miss | 0 |
| D | 100 | 120 | 17% | Hit | 1 |
| Prd Fmly | 400 | 400 | 2 |
Forecast Accuracy—A Thoughtful Approach
Forecast accuracy is a topic that is very specific to each industry and each customer. There is no one size fits all solution. The above questions need to be answered for each case, as well as a decision made before designing the process for forecast accuracy measurement and how it is being used.
How do you approach forecasting? Did you find this information helpful? Share your thoughts on the tips presented here in the comments below.