An Abundance of New AI-Based Forecasting Technologies
The last decade has seen a surge in the use of artificial intelligence for demand forecasting (AI). This has been driven by the rapid decline in the cost of cloud computing, new data sources, and new analytical techniques. Just about every supply chain software vendor now claims to incorporate AI into their forecasting and planning products. Facebook, Amazon, and Google now offer software aimed at expanding the use of AI-based forecasting to a broader audience.
As is so often the case with new and rapidly evolving technologies, it can be very confusing for business leaders to cut through the marketing hype and understand the technology landscape. As a supply chain software vendor using AI technology, we are often asked by customers and prospects for our viewpoint. For that reason, we decided to summarize some of our thoughts in this blog post.
Clarifying the Terminology
Part of the challenge in understanding AI is confusion over definitions. DataCamp, the data skills learning platform company, does a nice job of succinctly defining AI terms. Below are DataCamp’s definitions for four terms that often come up in the context of AI and forecasting and that we use in this blog post (Definitions from Datacamp’s “Data Science Glossary,” Kosourova, E.):
- Artificial Intelligence (AI): A branch of computer science that involves using machine learning, programming, and data science techniques that enable computers to behave intelligently. AI systems are broad and have varying degrees of complexity. They can range from rule-based systems, to machine learning based systems, and can perform abilities such as fraud detection, object recognition, language translation, stock price prediction, and so much more.
- Machine Learning: A branch of artificial intelligence (AI) that provides a set of algorithms designed to learn patterns and trends from historical data. The purpose of machine learning is to predict future outcomes, and generalize beyond the data points of the training set without being explicitly programmed. There are two main types of machine learning algorithms: supervised and unsupervised, each represented by numerous techniques applicable for different use cases.
- Artificial Neural Networks (ANN): A machine learning model that is loosely inspired by biological neural networks in human brains. Neural networks consist of up to hundreds of layers of interconnected units called neurons. Conceptually, an artificial neural network has the following types of layers: input, output, and hidden layers used to filter the data through, process it with an activation function, and make predictions at the output. ANN are the building blocks of a subset of machine learning called deep learning, which delivers complex outputs such as image or sound recognition, object detection, language translation, and more.
- Deep Learning: A subset of machine learning algorithms based on multilayered artificial neural networks (ANN) that are largely inspired by the structure of the brain. ANN are very flexible and can learn from huge amounts of data, to deliver highly accurate outputs. They are often behind some data science and machine learning use cases such as image or sound recognition, language translation, and other advanced problems.
There’s another factor that creates confusion. Some forecasting algorithms developed in an earlier era exhibit characteristics of machine learning, even though they were not called that at the time. As a result, some vendors now refer to older forecasting technologies as forms of machine learning. Some of them may be technically correct. But generally, when people talk about machine learning today, they are referring to new generation techniques that have come into widespread commercial use in the last decade or so. These techniques would not have been practical before the current era of powerful low-cost cloud computing.
Hierarchy of Approaches
As implied by the above definitions, AI is really an umbrella term for a range of approaches that enable computers to mimic the human brain. Machine learning is a specific AI technique, and it is machine learning that has really driven the surge in interest in AI-based forecasting. Deep learning is a subset of machine learning and is enabled by neural networks.
The below Venn diagram summarizes the relationship among AI, machine learning, and deep learning. Note that as in many new and rapidly evolving fields, there is not universal agreement on the above definitions or the below Venn diagram. But the viewpoint conveyed by this diagram is fairly common.
Machine Learning Can Offer Superior Predictive Accuracy
Both machine learning and traditional quantitative forecasting techniques (such as time-series forecasting and causal forecasting) look for patterns in existing data to make predictions about the future. But traditional techniques are explicitly programmed to look for specific patterns using a relatively limited set of data. For example, a causal forecasting model might be configured to forecast summer ice cream sales based on how hot the weather is.
Machine learning, on the other hand, generally considers a larger set of data and can discover correlations that a human forecaster might not even consider looking for with a traditional algorithm. It also “learns” by examining data, making predictions, comparing those predictions with actual results, and then automatically tweaking the algorithms to continuously improve predictive accuracy.
By considering a broader range of factors that may help predict the future and continuously learning and improving, machine learning can provide more accurate predictions than traditional techniques.
New Generation Software
Forecasting software has traditionally been the domain of specialty statistical modeling software companies, such as MathWorks, or has been embedded in enterprise demand planning software from companies like New Horizon, Oracle, Blue Yonder, and Kinaxis. However, with the exploding interest in data science and cloud computing over the last decade, technology giants Facebook, Amazon, and Google have created new generation forecasting technologies and have raised the visibility of AI-based forecasting. New Horizon uses Facebook Prophet as one of a number of forecasting algorithms available in our demand planning software, so we will describe it and the related NeuralProphet software in some detail.
Facebook Prophet
Facebook Prophet is a new generation open-source forecasting application which Facebook developed for internal use for a variety of applications, such as forecasting the demand for datacenter computing resources and for the inventory of advertising capacity. They made it available as open source software in 2017 so that a global community of developers could use the software and improve it over time. It has been downloaded nearly 30 million times and is considered the most popular forecasting software in the world. In addition to being used by New Horizon, it is also one of several forecasting algorithms embedded in Amazon Forecast, Amazon’s entry into the new generation forecasting software market.
Prophet was built to automate a large number of forecasts across a diverse set of situations for a large number of non-expert users. It has set an industry standard for quick and automated generation of reliable forecasts that are easy to understand for non-specialists. At the same time, it offers more advanced users the ability to configure forecasting algorithms for increased accuracy. It is particularly good at forecasting data with multiple cyclical patterns (e.g., seasonal, day of week, holiday, etc.) and sudden shifts in demand trends associated with things like new product launches or market disruptions. It is also robust in the face of missing data and data outliers.
For New Horizon’s Demand Planning application, we’ve made industry-specific enhancements to Prophet to improve its forecast accuracy.
NeuralProphet
NeuralProphet is a next generation forecasting algorithm released in 2020 by Facebook and researchers at Stanford University and Monash University. It builds upon Prophet by adding deep learning forecasting capabilities using AR-Net, a state of the art neural network algorithm developed by the Facebook AI Research group. It is particularly well suited to causal factor forecasting (for use where variables such as weather or promotions have a big impact on demand) and situations where demand data is sparse (i.e., intermittent demand). Its creators claim that for short- to medium-term forecasts, NeuralProphet improves forecast accuracy by 55 to 92% (“NeuralProphet: Explainable Forecasting at Scale,” Triebe, O. et al., 29 November 2021).
NeuralProphet is currently a beta release, and New Horizon is evaluating its use in our Demand Planning application.
There’s No Best Forecasting Algorithm
We are often asked why we use multiple forecasting algorithms, while some demand planning products use just a single one. No single forecasting algorithm is best for all situations. Highly seasonal demand calls for different models than for steady demand. Intermittent demand calls for different models than for continuous demand. And so on. Also, the nature of the data may dictate the choice of forecasting model. If you have just a year or two of monthly or weekly data you will want to use a different algorithm versus if you have years of daily data. For this reason, New Horizon uses a number of different models, including Prophet, and the best model is selected for each situation.
Facebook Prophet is particularly well-suited to data with strong periodicity and demand with several years of daily or intraday data. These attributes are typical in traditional and online retail settings. In one of the few peer-reviewed articles evaluating new generation forecasting algorithms, in the International Journal of Computer Science & Information Technology, the authors compared Facebook Prophet with Amazon’s DeepAR+ and CNN-QR algorithms (two of the other algorithms Amazon Forecast uses in addition to using Prophet). The article concludes, “The results show that Prophet gives better results for items with a longer history and frequent sales, while Amazon’s algorithms show superiority for items without a long history and items that are rarely sold.” (“Comparison Analysis of Facebook’s Prophet, Amazon’s DeepAR+ And CNN-QR Algorithms for Successful Real-World Sales Forecasting,” Žunic, E. et al., April 2021, International Journal of Computer Science & Information Technology)
Thus, Prophet performs very well in some situations, but it is not best in all situations. In working with our customers in the consumer goods and foodservice industries, for example, we sometimes work with two years of monthly data with continuous demand, for a total of 24 data points for a given item and location. In these situations, traditional time-series forecasting algorithms such as ARIMA and exponential smoothing typically give better results than Prophet. For weekly data, where two years would yield 104 data points, the choice of algorithm varies on a case by case basis. In traditional and online retail settings, where customers may have years of daily or even intraday data, Prophet is usually the algorithm of choice.
Forecasting Versus Demand Planning
While this is a post about forecasting, as a demand planning software vendor, we feel obligated to point out the important distinction between forecasting and demand planning. The two terms are often used interchangeably, and this is often a source of confusion. In fact, we’ve had discussions with prospects who have said something along the lines of, “We don’t need demand planning software; we already use XYZ,” where XYZ is some type of forecasting software, such as Prophet or MathWorks.
Forecasting is a key component of an enterprise demand planning application, but it is really just the starting point. Enterprise demand planning software builds upon the forecast in a number of important ways:
- Imports transaction data from an ERP system and automates the preparation of data for forecasting and analysis
- Offers a number of forecasting algorithms and automates the selection of the best one
- Provides access to forecast and other data in a secure and scalable way to dozens or even hundreds of users
- Allows different users in different functional areas to review and edit data (and track changes) according to different dimensions, hierarchies, and units. For instance, a product manager could review and edit the annual unit forecast for a product nationwide, while a sales manager could review and edit the dollar forecast for a product line for a specific account and quarter.
- Accommodates supply constraints and allocations of demand made to priority customers
- Enables users across sales, marketing, and supply chain operations to reach a consensus forecast for each item
- Exports the final forecast to supply chain execution systems
The above list represents just some of the table stakes functionality of a demand planning software application. If a data science team wants to do forecasting for a one-off special project, it may make sense to use Prophet or Amazon Forecast or some other option. But if they want the kind of demand planning functionality in the above list, they should look at commercially available enterprise demand planning applications. For such software, the choice of available forecasting algorithms will be just one of many considerations in making a purchase decision.
Conclusion
Artificial intelligence is being increasingly used for demand forecasting and demand planning applications. New generation algorithms such as Prophet and NeuralProphet can improve forecast accuracy and thus improve supply chain efficiency and performance. As part of a portfolio of algorithms in a demand planning application, these new generation technologies give planners powerful new options to manage a diverse variety of demand forecasting challenges.
To Learn More
If you’d like to discuss how New Horizon can help your company take advantage of AI-based demand planning, please contact us – we’d love to talk.