Leveraging AI and Algorithms for Predictive Anomaly Detection

In an earlier blog we saw how automated detection algorithms are providing new techniques to find and explain performance anomalies in electronic trading. The big challenge, however, lies in choosing the models and algorithms that will produce the best results.

Most algorithms for detecting anomalies in time-series data work by forecasting what they expect to see in newly arriving data, based on what they have seen in the past. New data that does not match the forecast is declared to be anomalous. Success at finding genuine anomalies, therefore, is closely related to how accurate and comprehensive the forecast is for expected future behavior.

How far has data science progressed at modelling and forecasting time-series data? A good barometer is provided by the so-called M Competitions, organised by Spyros Makridakis, a leading forecasting researcher based at the University of Nicosia. The first competition was held in the Eighties and there’s been one in every decade since. Essentially, leading researchers are encouraged to send in their forecasting algos, which are tested to see which ones perform best across a large collection of datasets from different domains..

Combination models work best

The M Competitions have made several consistent findings over the years that provide important guidance for practitioners. The first is that, when applied to diverse data sets, complex statistical methods are not necessarily better at forecasting than simpler ones. A reason for this is that complex models have a tendency to ‘overfit’ to the data on which they are trained. Such models have many more parameters that can be used to capture behavior and patterns, but if these patterns are the result of chance events that do not occur again in the same way, then capturing them ends up reducing the accuracy of forecasts.

A second important finding of the M Competitions is that combinations of models typically perform better than any individual one. Exactly why this occurs is still the subject of ongoing research, but it is likely to be for similar reasons that ensemble models are found to work well in other areas of data science. Combining several models together gives a better chance of getting a good fit to training data, while the risk of overfitting can be avoided when combinations are generated using appropriate simple rules.

So the conclusion from the first three M Competitions was that using a diverse combination of simple models is a better strategy for forecasting general time-series data than attempting to fit a single highly complex model. What’s interesting is what happened in the fourth competition last year, when combinations that use deep learning algorithms were added to the mix for the first time. A significant boost in performance was achieved by combining simple classical algorithms with deep learning. While the results were exciting, it’s still very early days and an open question as to how exactly the models should be combined.

All of this will be of great interest to organisations looking for leading-edge approaches to improve their infrastructure monitoring. There is the tantalising prospect that new techniques combining deep learning with classical approaches could significantly advance the way anomalies are analysed and detected in the near future. The challenge is how to take advantage of a fast-evolving landscape to future-proof your monitoring capabilities.

Future-proof monitoring

When we began to research and develop Intelligence Hub a few years ago, we knew that information visualization, AI and machine learning capabilities were emerging as must-have functionality for helping our financial trading clients get more value, more quickly from their data. Intelligence Hub currently includes both classic anomaly detection and deep-learning based approaches. As we look to the future, guided by the latest advances in forecasting science, we will continue to extend the family of algorithms we support, and we’ll also be aiming to provide practical ways to combine models together in line with emerging best practice for robust and accurate anomaly detection.

Find out more about Intelligence Hub or schedule a demo with us.