Integration is one of the key implementations for most enterprise solutions. It involves connecting different applications and transforming a huge amount of data on a daily or ad-hoc basis.
Different iPass tools are available in today's market to make the integration easy to use without any coding. They provide a good user interface to configure and run the integrations. Most of the application connectors are readily available and easily configurable. Another possible solution would be to use look at open source integration frameworks.
Before going through technical concepts, one should be familiar with the technologies below to have a better understanding:
What is integration framework?
Integration framework helps to integrate applications in a standardized way using several integration patterns, where It reduces efforts and cost. Each developer should easily understand what you did, as long as they know the framework used.
Spring Integration, Apache Camel, and Mule ESB are widely used integration frameworks in enterprise application integration (EAI). Each framework has its own pros and cons, depending on the use case. In this article, we will focus on Apache Camel.
What is Apache Camel?
Camel is an open-source integration framework based on Enterprise Integration Patterns. Camel empowers to define routing and mediation rules in a variety of domain-specific languages, including a Java-based Fluent API, Spring, or Blueprint XML Configuration file. It makes integration easier by connecting different endpoints and transforming the data from one endpoint to another endpoint by providing support for multiple protocols and data types.
In Camel, we connect over a wide range of applications through available connectors and even perform HTTP calls. Few of the major connectors such as SalesForce, WorkDay, AWS. Currently, Camel provides over 300+ connectors. To learn more about the available connectors, click here.
Below is the Camel architecture which was taken from the Apache Camel website.
When can Apache Camel be a good fit?
As mentioned previously, below are a few points we have to consider when choosing Apache Camel as an integration approach:
The integration does not have multiple connections to send/receive data
There is not a huge data set or much complexity in the transformation layer
Sophisticated monitoring and analysis of the data are not required
Finally, Camel is free and open source. It has great community support which is necessary for any open source project.
Example - Salesforce and Anaplan
To better understand Camel, below is an example where accounts have been synced between Salesforce and Anaplan.
We used Anaplan REST API 2.0 to connect Anaplan and then performed upload and process operations. Camel provides an HTTP connector to call and consume REST SPI services. Complete Anaplan documentation on how to retrieve Actions Id Using API 2.0 can be found here.
Connecting to Salesforce, can be done in one of two ways:
Salesforce Rest API, or
Salesforce Connector provided by Camel
In this example, the Salesforce Rest API approach was used. More details on Salesforce APIs can be found here.
Data and actions setup
Before going to code explanation, let's set up the required data in Salesforce and respective actions in Anaplan.
Below is the sample account data created in Salesforce, where the Account ID will be the unique ID.
Salesforce provides Salesforce Object Query Language (SOQL) which can be used to search the organization database. It is similar to Structured Query Language (SQL) but is designed specifically for Salesforce data. We will use using SOQL to retrieve account data.
Once in Anaplan, create two Import Actions and add them to the process:
Import Account_Id to AccountId in the list
Import remaining data fields into the module
Generate the project with the required configuration and dependencies, then download it as a zip file at start.spring.io. Below is the snapshot:
Use Eclipse or Spring Tool Suite as IDE for the project. Export the file as a Maven Project.
In Camel, a 'route' is a sequence of steps, executed in order, that consume and process a message.
It is the best practice to provide credentials in the property file and use them across the application, wherever required. Replace workspace ID, model ID, file ID, process ID, export ID, and import ID in the respective places.
Here are the steps that have been executed in the route anaplansync:
Fetch token from Salesforce
Run SOQL Query and fetch the account results
Transform the JSON Payload to a CSV format
Upload CSV file into Anaplan
Run the process to import data into a list and modules
A complete sample project has been attached to this article for reference.
In this example, Salesforce was used as an external application. Depending on the situation, different Camel connectors can be used in place of the Salesforce to transform the data into a CSV format, then uploading it into Anaplan. The above implementation can be used as an alternative approach for an integration solution, instead of using an integration tool. It can be even more customized based on the requirements and can handle exceptions if anything fails in route.
The content in this article has not been evaluated for all Anaplan implementations and may not be recommended for your specific situation.
Please consult your internal administrators prior to applying any of the ideas or steps in this article.