Description
Trains an anomaly detection model to learn expected behaviours based on a nominal data set.Application
Engineers often accumulates a large amount of test data that is necessary to develop a new product. However, the data needs to be trusted to be used, and sometimes it can be anomalous, whether because the whole systems is behaving in a new unexpected way, or because the test instrumentation is faulty (e.g. failing sensors). This step, combined with Anomaly Detector can enable a user have a quick and complete overview of the quality of their data, before using them further.How to use
The step will run on existing data with the following properties: Hard requirements:- Series data (most often time series with a time column)
- Uniform time step size. If some data is filtered out during the pre-processing, this might result in non-uniform steps and the model will not be able to train.
- No missing data.
- No “irrelevant data”. This mean that data that’s not useful (e.g. measurement before the test start) should be filtered out.
- 5 to 100 channels. Less might lead to poor reconstruction and more might slow down the platform and consumption of the results
| Nominal Data | This data has to be as anomaly free as possible (for example, it could be data that was checked manually in the past). |
| Time column | If you are working with time series data, select the column that contains the time values. The step can also work with other data types as long as they are series-based. In that case, enter the unit of the series (e.g. angle, cumulative charge, …). |
| Series identifier (optional) | If the nominal data is made of multiple tests, select a column that can be used to uniquely identify each test (or each series). If the nominal data is only made of one test. then this can be left empty. However, if no column is selected while training the anomaly detection model, this model will only be applicable on data sets which also are only made of one test. |
| Channels to evaluate | Select all the channels that should be evaluated together (e.g. sensor channels, etc). |
| Model Name | Give a name to the anomaly detection model. |
Compute Type | There are currently 2 main options for compute type
|
| Capacity | Size of the model. Needs to increase for more channels and larger window size. |
| Window Size | Size of the rolling window of the model in number of steps. Needs to consider sampling rate and expected length of relevant time features. For example, if the time window of interest is 3 seconds and the sampling 10Hz, then the window size should be 30. If this window size is bigger than the shortest series from the data, the model will not be able to run. |
| Number of Training Steps | Number of steps for which the model is trained. |
| Quantile | Quantile applied on the reconstruction error of the training data to define the threshold error (score of 0.5). This parameter Defines the sensitivity of the model. Lower values mean the model is more sensitive (return higher scores). |
| Steepness | Defines how quickly the score goes from 0 to 1 as the reconstruction error increases. Lower values mean a more gradual increase in the score. |
Example
Here is an example of what the step could look like once the model is fully trained: