Today, modern supply chains are under constant pressure to reduce decision cycle times, even though huge amounts of data are being generated throughout every stage. This overload of data can lead to a high signal-to-noise ratio, which makes it difficult to discern useful signals that enable meaningful decisions from meaningless ones. Instead, businesses should consider implementing anomaly detection, so questionable data is quickly analyzed to determine anomalies or unexpected patterns for making faster, more meaningful decisions.
Complexity in an Interconnected World
Target Corporation's disastrous failure in Canada, and abrupt closing of its 133 stores less than two years after moving into this new market, is much discussed today by logistics professionals as a nightmare scenario that every supply chain manager should dread. While global supply chains are increasing in complexity, it’s becoming apparent that in an ever-increasing competitive, connected and digital world, the margin of error is shrinking, even if it brings about huge consequences. In Target’s case, the company misfired on many fronts because of a failure to recognize and appreciate distinctive Canadian attitudes about products, sales tactics, and store management and promotions.
How did such critical things go wrong? Unfortunately, this can happen all too easily when there’s vast amounts of data and computing power at hand. System and process designs overly rely on a combination of algorithms without augmenting decision-making with human intuition. To be sure, we should aim to completely eliminate risk from these kinds of Black Swan events in supply chains, while at the same time, strive to cut through the noise and better detect meaningful signals from the vast amounts of analytical data.
Finding the Anomaly Needle in the Proverbial Haystack
A typical supply chain is responsible for thousands of transactions on a global scale every day and must be able to adapt to the constant pressures of change, both internal and external. While massive investments in technology are helping to capture huge amounts of data, the question now is, given the ever-growing stock of data with the flow constantly adding to it, is there a way to catch potentially catastrophic anomalies sooner?
The answer is, yes, but businesses must first understand the three different types of anomalies:
- A point anomaly is an individual data instance that is anomalous with respect to the data, e.g., a decline in units manufactured on a particular day for a plant.
- A contextual anomaly is an anomaly that is observed in context. This can also be thought of as an anomaly resulting from a specific condition. An example of this is a rapid increase in component pricing from multiple suppliers from a specific region.
- A collective anomaly is an individual data point that may not necessarily be anomalous, but becomes an anomaly when viewed as part of a collection of related data instances.
Needless to say, identifying each of these types of anomalies requires a different approach.
Moving beyond Business Intelligence
The traditional business intelligence (BI) paradigm offers a classical approach to anomaly detection. It computes the underlying distribution of data based on history and defines thresholds. Then, for each new signal, it computes an anomaly score. In the event that the score goes beyond the threshold, an anomaly is flagged. However, this paradigm presents several real-life challenges: defining a representative distribution is often challenging as data tends to be messy in real life; the exact notion of an outlier itself is always constantly shifting; and, perhaps most importantly, there may not be enough data points available for training or validation.
This presents the need to rethink traditional paradigms by leveraging techniques that are capable of better identifying event-based exceptions in near real time. But this alone is not enough—businesses need to design a man-machine ecosystem that can create adaptive learning solutions at scale.
Anomalies and Machine Learning
As the complexity of the global supply chain continues to grow, interest in anomaly detection is also growing and many firms are looking to machine learning (ML) for solutions.
In all, point anomalies are the most common and there are broadly two types of approaches to detection that are most favored:
- The classification-based approach consists of typically supervised techniques that require training on an ongoing basis. This model takes into account historical data to classify each new event.
- The clustering-based approach assumes that normal data belong to large and dense clusters, while anomalies end up in smaller or low-density clusters, or in extreme cases, in no clusters at all. While this has the advantage of not requiring supervision, it is computationally intensive and may not work well with sparse data sets.
Spotting contextual anomalies requires the creation of a context and then identifying outliers in terms of their behavioral attributes. Once the context is understood, the impact of an indirect event on the overall delivery timelines can be quantified and treated as an anomaly. The obvious challenge is to build the relevant contexts that continue to be relevant over time.
Detecting collective anomalies, on the other hand, involves understanding the relationship between data instances over time or space. Assume that a retail chain with multiple stores sees a demand spike for a SKU from a set of stores. The algorithms can be trained on this sub-sequence of demand spikes and any changes from this pattern can be flagged as an anomaly.
We’re only discovering the tip of the anomaly iceberg
It is a truism to say that global supply chains are here to stay. As competitive pressures force firms to extract every possible ounce of cost from their global supply chains, the ability to effectively detect anomalies will only increase in importance. As we have seen, there is great potential to apply anomaly detection in multiple areas like component supplies, product demand and the like, and use these signals to drive better decisions. One thing is clear: Businesses need to move away from the paradigm of static BI and reporting to a world of adaptive, learning systems that bridge the gap between an event and response.
