Description
This function creates a heatmap of a dataset on missing data and therefore enables a quick visual check for missing data.
Application
Collecting real data often involves missing data. Data sheets are not always filled completely and consistently. Sensors fail during test and don’t send a signal. Especially with bigger datasets it can be tedious to search for missing data. With this function you can quickly check a dataset for missing data before blindly starting to remove missing data with Remove Missing.
How to use
- Select the dataset to analyse in the field Data.
- Select which Columns shall be analysed and plotted in the heatmap.
- Both numerical and categorical columns can be analysed.
- Choose how the heatmap rows are grouped under Bucket by (see Bucketing modes below): Bucket by row count, Bucket by column value, or Bucket by value ranges.
- Select the Number of buckets. This has to be an integer number between 1 and the number of rows (or distinct values) in the dataset.
- For Bucket by row count the dataset is split evenly into this many buckets, each representing the same number of rows (
#rows/#buckets).
- For Bucket by value ranges the chosen column’s distinct values are split into this many ordered ranges.
- This field is hidden for Bucket by column value (which uses one bucket per distinct value).
- Click Apply to run the step and generate the heatmap.
The heatmap layout is as follows:
- On the y-axis the buckets are plotted — row numbers, the grouping column’s values, or value ranges, depending on the selected mode.
- On the x-axis one cell for each selected column will be plotted.
- The z-value (colour) is the fraction of missing data in each cell in percent with values between 0% (no missing data, bright color) and 100% (all data in that bucket missing, dark red). Each cell is colored according to its z-value. When you hover the mouse pointer over a cell the x-, y-, and z-value are shown.
Bucketing modes (grouping missing data)
The Bucket by control determines how rows are grouped onto the heatmap’s y-axis. There are three modes.
1. Bucket by row count (default)
The original behaviour. Rows are split evenly into the chosen Number of buckets and the y-axis is the row number. You can see where gaps occur by position, but not which file or value they belong to. This mode is fully backward compatible — existing saved steps keep working unchanged.
2. Bucket by column value
Groups missingness by the exact value of a chosen column — one heatmap row per distinct value. Pick the column under Column to group by. This is ideal for categorical columns such as a source-file name: you can immediately see which file is missing data.
In the example above, grouping by file_name makes it obvious that drive_003.csv is missing 100% of its oil_pressure values, while the other files are complete.
3. Bucket by value ranges
For continuous or high-cardinality columns, the chosen column’s distinct values are split into N ordered ranges (set by Number of buckets). The y-axis shows value ranges instead of individual values, which keeps the plot readable when a column has hundreds of distinct values.
Hover any cell to read its value range, the number of distinct values it covers, and the percentage missing:
Choosing the right mode
Note: choosing a numeric column in Bucket by column value produces one row per distinct value, which can make the heatmap very tall — use Bucket by value ranges for those columns instead. A text column also works in Bucket by value ranges: its values are ordered alphabetically and split into ranges rather than raising an error.
Examples
Consider this example table with missing data:
The table has just 16 entries. If you plot the missing data with 16 buckets the heatmap would look like this, giving a clear indication in which rows data is missing:
If you plot the same data with just one bucket the heatmap looks like this:
You don’t see how much data is missing and on which position but you immediately see which columns have non-zero and where you need to look closer.
Here one last example with four buckets. This would represent the typical approach for a larger dataset for which you can’t choose one bucket per row:
More on this step
The y-axis is fixed in height. For very small datasets it can make sense to choose the maximum number of rows as the number of buckets so each cell on the heatmap represents a single column. That way you will get direct feedback on missing rows.
For larger datasets this becomes infeasible as the cells would become so small that they would not be recognisable. Use a smaller number of bins; up to 100 buckets is a reasonable choice.
The quickest way to check very big datasets is to use a single bucket and to check the z-value of this single bucket. If the data is complete the value has to be 0. Any value deviating from 0 indicates missing data.Last modified on July 23, 2026
