Once models are trained, it is not always clear whether they are “good enough” to be used in production. It might be easy to identify which model is the best, but that is not always enough to make it usable in production. This step enables to define accuracy criteria and assess whether they can be achieved by different models. If an accuracy criterion is not fulfilled, the step can also be used to see how far the model is from reaching it.The step also provides an option to compare the models to a simple baseline model.
You need at least one model and one data set to use this step.
Select one or multiple Models. Make sure that these models have at least one shared output.
Select a dataset in the field Test Set. For the results to be meaningful, it is recommended to use a data set that wasn’t used to train the models (for example, a test set).
Select an Output. If no outputs are suggested, it is because models do not have any output in common.
You can then decide whether you want to compare the model to a baseline (Compare to baseline).
The baseline is the average value of the outputs in the training set of the first model.
Then, you can decide whether you want to create a Criterion. A criterion will be in the following format:
At least [F] % of the data is predicted with an error smaller than [E] [Unit].
Where F, E and Unit are choices made by the user depending on their use case. The unit of the error can be in % or in the unit of the prediction (absolute value).
Note that for prediction values close to 0, using errors in percentage might give meaningless results.
Once the step is applied, you will get a plot looking like the one below. Here are a few points on how to read such a plot:
Each model is represented by a curve. The steeper the initial slope and the faster it reaches 100%, the better a model is (see black arrow). In this example, model 1 is bad. Models 2 and 3 are good, and model 4 is perfect (which is not realistic).
A criterion is represented by a cross (+). In the example below, the criterion is that at least a fraction F of the data should be predicted with an error smaller than E.
A model will fulfill a criterion if it gets around it by the left and top (see green arrow). In this case, models 3 and 4 would fulfill the criterion.
A model will not fulfill a criterion if it gets around it by the bottom and right (see red arrow). In this case, models 1 and 2 would not fulfil the criterion.
Although this step can be extremely useful to decide whether a model can be used to make predictions, it does is not suitable to assess outliers, or large errors. For example, a model might predict 90% of the data with a certain accuracy, but we are not told anything about the accuracy on the remaining 10%. In this case, this step would be run along with the Predicted Vs Actual step which will better highlight large errors.
