Author: Alejandro Gomez Sanchez, Certified Master Anaplanner, and Principal Consultant.
Data orchestration is a remarkable capability. When done well, it is genuinely transformative: comprehensive, accurate and dependable data for decision-making is available instantly.
Beyond the satisfaction of well-orchestrated data, my own rough calculations suggest that at least twelve working days per year [1] are saved as part of the resulting ROI. As we all know, manually loading data is tedious, cumbersome, time-consuming, insecure, and highly prone to error. For these reasons — and several others — it should, wherever possible, be replaced with an automated, direct source-to-target (i.e., orchestrated) data feed. This is particularly important in today’s data-hungry, connected, collaborative, and increasingly AI-driven platforms, where more data is required, therefore more efficiencies can be realised, and more errors avoided through orchestration.
However, I have also witnessed less successful attempts at data orchestration. In these cases, while teams were determined to eliminate the well-known issues associated with manual processes, they inadvertently introduced a new set of frustrations. These challenges were rarely caused by the technology itself. More often, they stemmed from poor timing and insufficient preparation within the project team.
Technically speaking, data orchestration has become far easier in recent years, thanks to the proliferation of pre-configured, direct connectors. In many scenarios, it has become nearly “plug and play”. Yet just because vendors have simplified the technical aspects for business users does not mean that preparation or thoughtful attention to the data being transferred is no longer required.
Let’s consider the following analogy:
In the past, orchestrating data demanded far more effort, systems, and specialised roles. Today, many tasks have become simple and immediate. In the same way that our ancestors had to struggle hunting and gathering their food.
Whereas now, access to food has been facilitated to the point that we have any food we want at our fingertips. Orchestration tools with connectors and friendly visual interfaces have highly simplified the access of our planning tools to data marts.
Just as the availability of abundant food does not mean we should eat indiscriminately, the simplification of orchestration does not remove the need to consider the quality, quantity, and timing of the data flows.
Ignoring the food being fed to our bodies and the data being fed to our models lead to negative outcomes in both cases.
So how should organizations approach the process to determine the right quantity, quality, and timing of data flows? There is no single answer. Different teams will take different paths depending on their technical skills, their technology stack, the maturity of their data, and the leadership team’s risk appetite. So, the following approach is not the only viable one — but it is a method shaped by experience across several recent orchestration projects.
Here is the “unpopular opinion”: start with traditional, flat-file-based manual data loads. Just as elite athletes first learned to crawl, and the world’s most beautiful cars begin life carved out from simple clay, starting with a flat file does not mean you are not on the path towards a high-performance, state-of-the-art data flow. Means that you are taking an appropriate -albeit admittedly unglamorous- first step in developing a data orchestration system in a controlled, transparent, efficient and collaborative manner.
While is no debate that crawling before running is not a waste of time but an essential part of the learning curve, nor is it questioned why clay models remain invaluable for identifying design flaws early in automotive development. Yet, when manual and simplistic methods are proposed as the starting point for an orchestration journey, they are often poorly received by clients. Such approaches are frequently perceived either as a step in the wrong direction — manual ETLs appearing to contradict the very idea of automation, much as heavy clay seems the opposite of lightweight carbon fibre — or as an avoidable waste of time — manual data manipulation is viewed as an unnecessary interim step that will not form part of the final solution, the same way as one might incorrectly argue that crawling could be skipped simply because it is not used later in life as a means of getting around.
Four reasons to start with manual, file-based data load your orchestration journey
- It is agile: most business systems provide straightforward file-based export and import capabilities. As a result, flat-file data is typically the easiest, cheapest, and quickest way to obtain, share, analyse, and manipulate the foundational datasets required for orchestration. This approach does not demand advanced technical skills, specialist software, or additional licences.
Because of this, file-based loading is an excellent way to surface data quality issues early in the project, well before they become embedded within orchestration logic or evolve into costly problems later on. Identifying issues early prevents wasted engineering effort and ensures that automation is built on stable, predictable inputs.
In short, it helps avoid the classic “garbage in, garbage out” pitfall. It allows teams to obtain data rapidly and to identify inconsistencies, such as structural mismatches, varying formats, or missing values, at a stage when such issues are far easier and less expensive to correct.
- It is democratic: anyone involved in the project has the tools and the skills to work on said data sets. This democratises access to, and understanding of, the data across all project stakeholders, fostering conversations that early automation often obscures or restricts to the most technical members of the team.
This democratisation ensures that every stakeholder can see and “touch” the same data for themselves. It promotes a shared understanding of data quality, challenges, timelines, and required effort. It helps build trust, facilitates communication, and brings diverse perspectives into the conversation — while avoiding the classic “black box” effect.
In other words, it prevents situations where only a handful of technical specialists have visibility of the data, while others must rely on unfamiliar tools, specialist licences, or second-hand interpretations. Everyone gains direct, transparent access, improving alignment and overall project effectiveness.
- It is controllable: to a large extent, orchestration simply accelerates the data flows that already exist.
If your underlying process is unclear or your data is corrupted, automation will simply accelerate the chaos and amplify data-related issues. Even when the data is broadly accurate, introducing new automated feeds into an existing system can create unexpected consequences, unwanted changes, and confusion among users — particularly in the early stages of implementation or during development and testing. At these points, it becomes difficult to determine whether unexpected behaviour is caused by new automated data streams or by incorrect business logic.
By contrast, loading data from files that have been manually exported, reviewed, and imported significantly reduces the risk of unforeseen side effects. It allows full control over when data inputs are triggered and ensures a deeper understanding of what the dataset contains and which downstream elements may be affected. This real-world clarity is essential before writing a single line of orchestration code.
Because data requirements are driven by the purpose and expected outcomes of the target system, it makes little sense to enable orchestration capabilities during the early phases of system development. Automation should not be introduced until the system has been built, tested, and signed off using controlled manual data loads.
Ultimately, there are only two things worse than poor-quality data. The first is poor-quality data being orchestrated and fed automatically into your systems, increasing both the pace and volume of problems. The second is the time wasted trying to determine whether unclear or unexpected data behaviour is the result of poor-quality inputs, errors in the orchestration process, unintended data flows being triggered, or flaws in the underlying business logic.
- Improves business ownership: Building the orchestration process from the ground up using simple, easily understood data files enables change-management teams to develop comprehensive operational and governance documentation. This documentation can be consolidated into a data requirements register, capturing key elements such as:
a. source system and source query or saved search
b. load frequency or trigger
c. links to sample files
d. associated orchestration jobs
e. target system
f. data ownership
Remember that data orchestration is not solely a technical effort; it is also an organizational change. Business users must understand how it works and how to maintain it, particularly when third parties have been involved in implementing the orchestration.
If you have reached a clear understanding of your data requirements in your Anaplan model, the data sources are stable, the format of the data templates is not subject to changes and the impact of every new data load in your model is understood, its time to explore the many benefits of using Anaplan Data Orchestration.
Questions? Leave a comment!
……………
[1] On average, a planning solution relies on around eight separate data inputs. Assuming one ETL cycle per month to refresh forecasts or run new scenarios — and estimating one hour per input to extract, review, share, transform, and load the data — we arrive at:
8 inputs × 1 hour × 12 months = 96 hours per year, which equates to 12 working days.
For FP&A use cases, these inputs typically include:
- Chart of Accounts
- Ledgers and Trial Balances
- Employee data
- Clients list
- Vendor lists
- Cost centre structures
- Currency tables
- Sales pipeline information
For S&OP scenarios, common inputs include:
- Product catalogues
- Customer and channel data
- Stock-holding locations
- Stock-on-hand levels
- Actual sales
- Open orders
- Suppliers
- Costs and Prices
This estimation also excludes the often-substantial costs and time lost to data errors, duplications, data leaks, or other issues that frequently arise from manual data handling.
