Run your integrations on Camel

karthik_kumar
karthik_kumar
in Best Practices

Integration is one of the key implementations for most enterprise solutions. It involves connecting different applications and transforming a huge amount of data on a daily or ad-hoc basis.

Different iPass tools are available in today's market to make the integration easy to use without any coding. They provide a good user interface to configure and run the integrations. Most of the application connectors are readily available and easily configurable. Another possible solution would be to use look at open source integration frameworks.

Before going through technical concepts, one should be familiar with the technologies below to have a better understanding:

  • Java
  • Maven
  • Apache Camel

What is integration framework?

Integration framework helps to integrate applications in a standardized way using several integration patterns, where It reduces efforts and cost. Each developer should easily understand what you did, as long as they know the framework used.

Spring Integration, Apache Camel, and Mule ESB are widely used integration frameworks in enterprise application integration (EAI). Each framework has its own pros and cons, depending on the use case. In this article, we will focus on Apache Camel.

What is Apache Camel?

Camel is an open-source integration framework based on Enterprise Integration Patterns. Camel empowers to define routing and mediation rules in a variety of domain-specific languages, including a Java-based Fluent API, Spring, or Blueprint XML Configuration file. It makes integration easier by connecting different endpoints and transforming the data from one endpoint to another endpoint by providing support for multiple protocols and data types.

In Camel, we connect over a wide range of applications through available connectors and even perform HTTP calls. Few of the major connectors such as  SalesForce, WorkDay, AWS. Currently, Camel provides over 300+ connectors. To learn more about the available connectors, click here. 

Below is the Camel architecture which was taken from the Apache Camel website. 

image.png

When can Apache Camel be a good fit?

As mentioned previously, below are a few points we have to consider when choosing Apache Camel as an integration approach:

  • The integration does not have multiple connections to send/receive data
  • There is not a huge data set or much complexity in the transformation layer
  • Sophisticated monitoring and analysis of the data are not required

Finally, Camel is free and open source. It has great community support which is necessary for any open source project.

Example - Salesforce and Anaplan

To better understand Camel, below is an example where accounts have been synced between Salesforce and Anaplan.

Anaplan

We used Anaplan REST API 2.0 to connect Anaplan and then performed upload and process operations. Camel provides an HTTP connector to call and consume REST SPI services. Complete Anaplan documentation on how to retrieve Actions Id Using API 2.0 can be found here. 

Salesforce

Connecting to Salesforce, can be done in one of two ways:

  • Salesforce Rest API, or
  • Salesforce Connector provided by Camel

In this example, the Salesforce Rest API approach was used. More details on Salesforce APIs can be found here.

Data and actions setup

Before going to code explanation, let's set up the required data in Salesforce and respective actions in Anaplan.

Below is the sample account data created in Salesforce, where the Account ID will be the unique ID.

karthik_kumar_0-1615652652401.png

Salesforce provides Salesforce Object Query Language (SOQL) which can be used to search the organization database. It is similar to Structured Query Language (SQL) but is designed specifically for Salesforce data. We will use using SOQL to retrieve account data.

Once in Anaplan, create two Import Actions and add them to the process:

  1. Import Account_Id to AccountId in the listimage.png
  2. Import remaining data fields into the moduleimage.png

Project setup

Generate the project with the required configuration and dependencies, then download it as a zip file at start.spring.io.  Below is the snapshot:

image.png

Use Eclipse or Spring Tool Suite as IDE for the project. Export the file as a Maven Project.

karthik_kumar_1-1614795222839.png

Code explanation

In Camel, a 'route' is a sequence of steps, executed in order, that consume and process a message.

It is the best practice to provide credentials in the property file and use them across the application, wherever required. Replace workspace ID, model ID, file ID, process ID, export ID, and import ID in the respective places.

karthik_kumar_0-1615652496245.png

Here are the steps that have been executed in the route anaplansync:

  1. Fetch token from Salesforce
  2. Run SOQL Query and fetch the account results
  3. Transform the JSON Payload to a CSV format
  4. Upload CSV file into Anaplan
  5. Run the process to import data into a list and modules

karthik_kumar_1-1615652525651.png

A complete sample project has been attached to this article for reference.

Conclusion

In this example, Salesforce was used as an external application. Depending on the situation, different Camel connectors can be used in place of the Salesforce to transform the data into a CSV format, then uploading it into Anaplan. The above implementation can be used as an alternative approach for an integration solution, instead of using an integration tool. It can be even more customized based on the requirements and can handle exceptions if anything fails in route.

Tagged:

Categories