Purpose
Checks values in selected columns sequentially and applies corrections or rejections. Suitable for cleansing before loading and explicitly stopping when data does not meet the rules.
Inputs and outputs
One in input; out contains the corrected table without rows discarded by a skip_row rule.
Parameters
| Parameter | Settings and values | Options and limits |
|---|---|---|
rules | An ordered list of rules. Rule conditions and parameters are described below. | Count: 1–100 |
whitespace_empty | Treat a string containing only spaces as empty. | Default: enabled |
log_limit | The number of rule-match examples in the log, from 0 to 100; examples do not include data values. | Default: 50Limits: 0…100 |
Conditions and actions
| Parameter | What it configures |
|---|---|
fields | Specific fields, or every field when the list is empty. |
condition | empty, text, invalid_number, negative, positive, zero, non_finite (infinity/NaN), excel_error. |
action | replace — a typed replacement; null — clear; positive/negative — change the sign; skip_row — discard the row; error — stop. |
value | Replacement: NULL, text, number or Boolean. |
Step-by-step configuration
- Add a rule and select fields; an empty list applies the rule to every column.
- Set the condition and action. For replace, specify the replacement type and value.
- Arrange rules in execution order.
- Check the number of changes, discarded rows and match examples in the results.
Rules and diagnostics
- Rules run from top to bottom: later rules see values already changed by earlier ones.
- skip_row removes the entire row, while error stops the block with DATA_QUALITY and details of the row, field and condition.
- invalid_number does not match an empty value; add a separate empty rule for it.
- The text check matches non-empty text even when it looks like a number.
- positive/negative change the sign of a numeric value; normalise arbitrary text beforehand.
Configuration example
For Amount, use empty → replace 0 (number), followed by negative → error. For CustomerID, use empty → skip_row. Empty amounts become zero, and rows without a key do not continue.