Data Quality Orchestration : a simple example with ADO & Workflow - Part 1/2
In this article, you'll find a simple but hopefully inspiring illustration of the power of Anaplan Data Orchestrator (ADO) combined with Workflow.
This is a 2 parts article.
This Part 1 focuses on the Workflow and the steps to achieve Data Quality tasks.
The Part 2 is a deep dive on some details.
Summary
For illustration, I'll replace the Region/ Country hierarchy used by the well known Level 1 Model Building training model and replace it by the Region/Nation hierarchy from the default data provided by the Snowflake's Trial environment (See here for more on ADO Snowflake Connector).
I want this process to be automatised. Thus, I'll set up a workflow to orchestrate ADO Extracts and Links as follow:
1 ) - Connectors are extracting data from my L1 MB model (the SYS08 Employee Details module) and Regions and Nations from Snowflakes ;
2) - After some little transformations, I'm disseminating Region/Country & Employee Lists into my new Anaplan Model ;
3) - More transformations are providing me with not matching Regions & Countries between Snowflake and the Level 1 MB model. I'm moving this info into my new Model and prepare a UX Page to manually define the Data Mappings which sometimes will require to Create the missing values not existing into the Snowflake's Trial environment. Eventually, I'm providing this Data Mapping information back to ADO.
4) - Using the provided Data Mapping, I'm using a last step transformation to create the Final Employee Details which then will be disseminated to my new Model.
The Workflow Process
The screenshot above represents a partial view of the workflow process I've designed for this illustration. Please note that this is just for illustration purpose. It does not represent a real world/production example which requires more work for robustness.
Step 1 - Extract Regions/Nations from Snowflake and create a new hierarchy
I'm starting my automatic and repeatable process by extracting Regions & Nations from Snowflake using the Extract Action for the Task and selecting the corresponding Source Data (SnowRegions & SnowNations).
For the Upsert of data, the selected Action is Link and the ADO link designed to populate the Region/Country hierarchy of my new Model.
Step 2 - Extract the L1MB's "SYS08 Employee Details" and create a Department/Employee Hierarchy
Now that I've created the Region/Country hierarchy with the Snowflake's data, let's create a Department/Employee Hierarchy with the SYS08 Employee Details module from my L1 Model Builder training.
Thus, I'm defining a Task to Extract the data, and I'm using the ADO prepared Transformation Views and Links to disseminate the data into my new Model.
Step 3 - Manually define Data Mapping for Region/Country
This step involves a two way data flow between ADO and my new Model.
Inside ADO, I'm able to detect the Regions and Countries which are not appearing in the Snowflake's hierarchy and create an ADO Link to populate "to Map" Lists in my new Model.
From there, I've prepared a Module and an associated UX Page to manually enter the corresponding entry in the existing hierarchy or create a new one if it doesn't exist.
Thus, I'm able to define a Group Task in my workflow process asking for using this UX Page to Complete the Data Mapping.
When the workflow task has been completed, the Data Mappings (Region & Country) are going back to ADO using pre-defined Extracts.
The last step is to Upsert the final transformation into my new Model module SYS01 Employee Details. This final version is using the data mappings to calculate the correct Region & Country hierachy.
Conclusion
Combining ADO and Workflow is enabling us to build powerful integration and data quality scenarios including automatic or manual steps.
I hope this illustration is inspiring you and make you feel the heat of your brain starting to solve some pains you could currently face in your environment.
Feel free to visit the Anaplan Data Orchestrator and Workflow documentation.
Be trained in Academy with the ADO Training and the Workflow Training.
Comments
-
Great content, many thanks @Noel Rocher 😀
3
