A Practical Toolkit for Time Series Anomaly Detection, Using Python

fascinating points of time collection is the intrinsic complexity of such an apparently easy sort of information.

On the finish of the day, in time collection, you may have an x axis that often represents time (t), and a y axis that represents the amount of curiosity (inventory value, temperature, visitors, clicks, and many others…). That is considerably easier than a video, for instance, the place you may need hundreds of photographs, and every picture is a tensor of width, top, and three channels (RGB).

Nonetheless, the evolution of the amount of curiosity (y axis) over time (x axis) is the place the complexity is hidden. Does this evolution current a development? Does it have any information factors that clearly deflect from the anticipated sign? Is it steady or unpredictable? Is the common worth of the amount bigger than what we’d count on? These can all in some way be outlined as anomalies.

This text is a group of a number of anomaly detection methods. The aim is that, given a dataset of a number of time collection, we are able to detect which time collection is anomalous and why.

These are the 4 time collection anomalies we’re going to detect:

1. We’re going to detect any development in our time collection (development anomaly)
2. We’re going to consider how unstable the time collection is (volatility anomaly).
3. We’re going to detect the purpose anomalies throughout the time collection (single-point anomaly).
4. We’re going to detect the anomalies inside our financial institution of indicators, to know what sign behaves in a different way from our set of indicators (dataset-level anomaly).

We’re going to theoretically describe every anomaly detection methodology from this assortment, and we’re going to present the Python implementation. The entire code I used for this weblog submit is included within the PieroPaialungaAI/timeseriesanomaly GitHub folder

0. The dataset

With a purpose to construct the anomaly collector, we have to have a dataset the place we all know precisely what anomaly we’re looking for, in order that we all know if our anomaly detector is working or not. With a purpose to do this, I’ve created a data.py script. The script incorporates a DataGenerator object that:

1. Reads the configuration of our dataset from a config.json* file.
2. Creates a dataset of anomalies
3. Provides you the power to simply retailer the info and plot them.

That is the code snippet:

So we are able to see that we now have:

1. A shared time axis, from 0 to 100
2. A number of time collection that type a time collection dataset
3. Every time collection presents one or many anomalies.

The anomalies are, as anticipated:

1. The development conduct, the place the time collection have a linear or polynomial diploma conduct
2. The volatility, the place the time collection is extra unstable and altering than regular
3. The extent shift, the place the time collection has the next common than regular
4. A degree anomaly, the place the time collection has one anomalous level.

Now our aim will probably be to have a toolbox that may determine every one in every of these anomalies for the entire dataset.

*The config.json file lets you modify all of the parameters of our dataset, such because the variety of time collection, the time collection axis and the sort of anomalies. That is the way it seems like:

1. Pattern Anomaly Identification

1.1 Concept

Once we say “a development anomaly”, we’re in search of a structural conduct: the collection strikes upward or downward over time, or it bends in a constant method. This issues in actual information as a result of drift usually means sensor degradation, altering consumer conduct, mannequin/information pipeline points, or one other underlying phenomenon to be investigated in your dataset.

We take into account two sorts of traits:

• Linear regression: we match the time collection with a linear development
• Polynomial regression: we match the time collection with a low-degree polynomial.

In apply, we measure the error of the Linear Regression mannequin. Whether it is too giant, we match the Polynomial Regression one. We take into account a development to be “vital” when the p worth is decrease than a set threshold (generally p < 0.05).

1.2 Code

The AnomalyDetector object in anomaly_detector.py will run the code described above utilizing the next capabilities:

• The detector, which is able to load the info we now have generated in DataGenerator.
• detect_trend_anomaly and detect_all_trends detect the (eventual) development for a single time collection and for the entire dataset, respectively
• get_series_with_trend returns the indices which have a big development.

We will use plot_trend_anomalies to show the time collection and see how we’re doing:

Good! So we’re capable of retrieve the “fashionable” time collection in our dataset with none bugs. Let’s transfer on!

2. Volatility Anomaly Identification

2.1 Concept

Now that we now have a worldwide development, we are able to give attention to volatility. What I imply by volatility is, in plain English, how in all places is our time collection? In additional exact phrases, how does the variance of the time collection examine to the common one in every of our dataset?

That is how we’re going to check this anomaly:

1. We’re going to take away the development from the timeseries dataset.
2. We’re going to discover the statistics of the variance.
3. We’re going to discover the outliers of those statistics

Fairly easy, proper? Let’s dive in with the code!

2.2 Code

Equally to what we now have completed for the traits, we now have:

• detect_volatility_anomaly, which checks if a given time collection has a volatility anomaly or not.
• detect_all_volatilities, and get_series_with_high_volatility, which test the entire time collection datasets for volatility anomaly and return the anomalous indices, respectively.

That is how we show the outcomes:

3. Single-point Anomaly

3.1 Concept

Okay, now let’s ignore all the opposite time collection of the dataset and let’s give attention to every time collection at a time. For our time collection of curiosity, we wish to see if we now have one level that’s clearly anomalous. There are various methods to do this; we are able to leverage Transformers, 1D CNN, LSTM, Encoder-Decoder, and many others. For the sake of simplicity, let’s use a quite simple algorithm:

1. We’re going to undertake a rolling window strategy, the place a set sized window will transfer from left to proper
2. For every level, we compute the imply and customary deviation of its surrounding window (excluding the purpose itself)
3. We calculate how many customary deviations the purpose is away from its native neighborhood utilizing the Z-score

We outline some extent as anomalous when it exceeds a set Z-score worth. We’re going to use Z-score = 3 which suggests 3 occasions the usual deviations.

3.2 Code

Equally to what we now have completed for the traits and volatility, we now have:

• detect_point_anomaly, which checks if a given time collection has any single-point anomalies utilizing the rolling window Z-score methodology.
• detect_all_point_anomalies and get_series_with_point_anomalies, which test the entire time collection dataset for level anomalies and return the indices of collection that include at least one anomalous level, respectively.

And that is how it’s performing:

4. Dataset-level Anomaly

4.1 Concept

This half is deliberately easy. Right here we’re not in search of bizarre time limits, we’re in search of bizarre indicators within the financial institution. What we wish to reply is:

Is there any time collection whose general magnitude is considerably bigger (or smaller) than what we count on given the remainder of the dataset?

To do this, we compress every time collection right into a single “baseline” quantity (a typical stage), after which we examine these baselines throughout the entire financial institution. The comparability will probably be completed when it comes to the median and Z rating.

4.2 Code

That is how we do the dataset-level anomaly:

1. detect_dataset_level_anomalies(), finds the dataset-level anomaly throughout the entire dataset.
2. get_dataset_level_anomalies(), finds the indices that current a dataset-level anomaly.
3. plot_dataset_level_anomalies(), shows a pattern of time collection that current anomalies.

That is the code to take action:

5. All collectively!

Okay, it’s time to place all of it collectively. We’ll use detector.detect_all_anomalies() and we are going to consider anomalies for the entire dataset primarily based on development, volatility, single-point and dataset-level anomalies. The script to do that could be very easy:

The df offers you the anomaly for every time collection. That is the way it seems like:

If we use the next operate we are able to see that in motion:

Fairly spectacular proper? We did it. 🙂

6. Conclusions

Thanks for spending time with us, it means lots. ❤️ Right here’s what we now have completed collectively:

• Constructed a small anomaly detection toolkit for a financial institution of time collection.
• Detected development anomalies utilizing linear regression, and polynomial regression when the linear match isn’t sufficient.
• Detected volatility anomalies by detrending first after which evaluating variance throughout the dataset.
• Detected single-point anomalies with a rolling window Z-score (easy, quick, and surprisingly efficient).
• Detected dataset-level anomalies by compressing every collection right into a baseline (median) and flagging indicators that reside on a unique magnitude scale.
• Put every little thing collectively in a single pipeline that returns a clear abstract desk we are able to examine or plot.

In lots of actual initiatives, a toolbox just like the one we constructed right here will get you very far, as a result of:

• It provides you explainable indicators (development, volatility, baseline shift, native outliers).
• It provides you a robust baseline earlier than you progress to heavier fashions.
• It scales nicely when you may have many indicators, which is the place anomaly detection often turns into painful.

Needless to say the baseline is easy on goal, and it makes use of quite simple statistics. Nonetheless, the modularity of the code lets you simply add complexity by simply including the performance within the anomaly_detector_utils.py and anomaly_detector.py.

7. Earlier than you head out!

Thanks once more to your time. It means lots ❤️

My identify is Piero Paialunga, and I’m this man right here:

I’m initially from Italy, maintain a Ph.D. from the College of Cincinnati, and work as a Information Scientist at The Commerce Desk in New York Metropolis. I write about AI, Machine Studying, and the evolving function of information scientists each right here on TDS and on LinkedIn. When you favored the article and wish to know extra about machine studying and comply with my research, you’ll be able to:

A. Comply with me on Linkedin, the place I publish all my tales
B. Comply with me on GitHub, the place you’ll be able to see all my code
C. For questions, you’ll be able to ship me an electronic mail at piero.paialunga@hotmail