Author: Reena Sethy is a Director, Product Management at Anaplan.
Quality data is the backbone of modern decision-making, but it’s often compromised because of "all-or-nothing" approach to data loading. With a single invalid record, it can invalidate the entire batch, and a domino effect of processing delays, data unavailability, and exhausted data integration teams.
To build trust and reliability in your data, you need the ability to validate individual records, isolate any exceptions, and have valid data flowing for down-stream processing.
Traditionally, data issues or business rule violations meant outright pipeline failures. Without a mechanism to detect and isolating exceptions on the fly, teams have had to rely on manual intervention to debug and reload data wasting lot of time.
That is why we are thrilled to introduce Data Validations in Anaplan Data Orchestrator (ADO) pipelines — powerful, easily configurable validation rules designed to detect erroneous records early, and put you in control of your data quality.
Data Validations in ADO pipelines solves this by:
- Removing invalid records: Isolate bad records while continuing to process valid ones.
- Automating data quality checks: Run robust checks (like nulls, data types, and thresholds) automatically.
- Reducing support overhead: Detect and download exceptions without needing to rerun the entire pipeline or manual debugging.
Figure 1 showing row-by-row validation
The validation node within ADO pipeline includes 16 comprehensive validation rules that can be applied to your ADO datasets and views at a column level.
1. Data format and type integrity
Figure 2 showing different rules based on data type
- Regex pattern matching: Define custom regex pattern strings to validate complex formats (e.g., ensuring emails, phone numbers..).
- Length validation: Ensure text fields contain exactly the expected number of characters, or fit within minimum/maximum limits.
- Null checks: Instantly identify null, missing, or empty fields across specified columns.
- Data type enforcement: Detect mismatching data types instantly (e.g., boolean values 1/0, TRUE/FALSE, or ensuring financial fields like Revenue are strictly numeric instead of strings).
- Threshold and range checks: Automatically flag values falling outside of designated boundaries — whether ensuring numbers don't exceed a certain value or validating that dates do not fall before or after a specific time.
2. Flexible, row-by-row execution
- Row by row check: Validation criteria are applied row-by-row to all data in the column, regardless of whether the rows are visible in preview windows.
- Multiple validations: You can seamlessly plug validations into your pipeline. Can add multiple nodes as well.
- On-demand or scheduled: Run validation checks seamlessly when triggering pipelines on-demand or via scheduling.
3. Configure actions and handle failures
Figure 3 showing actions configuration
With configurable actions, you can decide exactly how the pipeline reacts when a validation rule is not met:
- Hard stop (stop pipeline): Instantly stop execution for critical data errors where compliance or accuracy is non-negotiable.
- Log warning: Keep the pipeline running but log a warning in case of validation failures for each failed row.
- Remove bad cata: Drop bad records while allowing clean records to load seamlessly.
4. Manage risk with smart tolerance limits
With this validation node, you can define tolerance limits:
- High watermark stops: Set a threshold (e.g., a specific failure rate or volume of bad data). If the bad data exceeds this watermark, the pipeline automatically stops.
- Easy exception auditing: Seamlessly download and view the exact records that failed validation so you can quickly review.
By integrating these 16 validation rules and flexible error-handling actions directly into your ADO pipelines, you can ensure that only high-quality, trusted data makes it to your downstream Anaplan models or third party systems.
Questions? Leave a comment!
