CloudWorks—This is how we use it, Part 3: The CloudWorks API
In Part 3 of our CloudWorks - This is how we use it series, we will introduce you to the CloudWorks API.
This is typically for an audience comfortable with REST APIs concepts and programming.
If you need a refresher on What CloudWorks is, see Part 1 or, for how to set it up, see Part 2.
CloudWorks API will allow you to create and execute connections and integrations extending integration automation capabilities between Amazon AWS S3 and Anaplan.
We will use a typical CloudWorks workflow to build and run integrations using REST API and Postman. To follow along, download the set of Jupyter Notebooks and data set that will walk you through each step of this workflow using Python (CloudWorksAPIBasic.zip and AccountDetailsS3.zip).
The following are minimum requirements to be able to perform the steps described in this blog.
- Required software
- Download and install Postman (https://www.postman.com/downloads), Python 3.8.2, Jupyter notebook
- Anaplan: Create a Model, LIST (Accounts), Module (AccountDetails), Import AccountDetailsS3.csv into LIST (Create Import Action), Import AccountDetailsS3.csv into Module (Create Import Action)
- AWS S3: AWS Account, Create Access Key, Secret Access Key, Create a bucket, upload Source File (AccountDetailsS3.csv) to S3 Bucket
- Working knowledge of Anaplan Model Building Concepts, REST API, Postman, Python, Jupyter
- Familiarity with Anaplan Authentication and Integration REST APIs
Step 1 - Anaplan setup
- Download AccountDetailsS3.csv from this blog and save it to your workstation
- Create a new Anaplan Model. Provide an appropriate model name.
- Create LIST named Accounts. Import AccountID into this LIST from AccountDetailsS3.csv
- Create a Module named AccountDetails with Accounts as a dimension
- Create the following line items:
- AccountName (Text), Industry (Text), AnnualRevenue (Number), EmployeeCount (Number)
- Import AccountDetailsS3.csv into AccountDetails Module.
- Import Action Name for LIST (Accounts): ______________________
- Import Action Name for Module (AccountDetails): __________________________
Step 2 - AWS setup
- Assumptions: AWS Access Key & Secret Access Key have been generated in AWS Console
- Create an S3 bucket named anaplandemo
- Create a folder named source
- Upload AccountDetailS3.csv to anaplandemo/source in AWS S3
Step 3 - Postman setup
- Create a collection named CloudWorks
- Create following folder structure in collection CloudWorks
- We will use variables in API requests. Create the following variables in collection CloudWorks.
Now that we have completed the required prep-work in Anaplan, AWS S3 & Postman, we will make CloudWorks API requests to build and execute integrations to load data from AWS S3 to Anaplan. We will follow steps outlined in a typical CloudWorks task, outlined below, to build this integration. CloudWorks REST API requests will be made using Postman.
Step 4 - CloudWorks API
Each Anaplan API request begins with generating an Authentication Token using Anaplan Authentication Services REST API. Generated Auth Token is then passed in the header of CloudWorks API request for Anaplan authentication. Keep in mind, Anaplan Authtoken is valid for only 30 min. Once expired, you will need to regenerate the token and update CloudWorks API header.
We will perform the following tasks in Postman to bring AWS S3 data to Anaplan.
- Create a connection: Using AWS S3 connector, we will create a CloudWorks connection to AWS S3 Bucket.
- Create an integration: Using AWS connection and Anaplan information (workspaceid, modelid, fileid, import action id), we will create a CloudWorks integration.
- Run an integration: Run integration using CloudWorks API.
- Retrieve Integration Information: Get integration run information.
Generate Authentication Token (Basic Authentication)
The first step in any Anaplan API request, including CloudWorks, is generating Anaplan Authentication Token. The authentication token can be generated using Anaplan Authentication Services API. URL for authentication services is https://auth.anaplan.com/token/authentication.
Authentication Services API to generate token has the following REST structure:
|
Method |
POST |
|
API end point |
|
|
Headers |
Authorization: Basic username:password (username:password string must be base64 encoded) |
Copy value for tokenValue to collection variable token_value. You will use variables instead of hard coding values in your API requests. Token values expire every 30 minutes. You will need to re-generate a token and update the token_value value variable if you encounter authentication failures.
Create a connection using CloudWorks API
The first step in building a CloudWorks integration is to create a connection to AWS S3 bucket. Passing AuthToken in the header of API request, we will create a connection to AWS S3 bucket in this step. Create connection API has the following REST structure.
|
Method |
POST |
|
API end point |
https://api.cloudworks.anaplan.com/1/0/integrations/connection |
|
Authorization |
No Auth |
|
Headers |
Authorization: AnaplanAuthToken {{token_value}} |
|
Body |
{ "type": "AmazonS3", "body": { "name":"AWS_CW_API", "accessKeyId":"XXXXXXXXXXXXXXXX", "secretAccessKey":"XXXXXXXXXXXXXXXXXXXXXXX", "bucketName":"anaplandemo" } }
Do not change the value for “type”. Provide name for following elements: name, accessKeyId, secretAccessKey, & bucketName |
- Select POST method and API end point. https://api.cloudworks.anaplan.com/1/0/integrations/connections.
- Select “NoAuth” for Authorization Type. We will pass the authentication token in the header of the request.
- Create the following headers: Authorization and Content-Type.
- Copy and paste the following JSON structure into the Body of your request. Replace values for accessKeyId, secretAccessKey, and bucketName with your values.
{
"type": "AmazonS3",
"body": {
"name":"AWS_CW_API",
"accessKeyId":"XXXXXXXXXXXXXXXX",
"secretAccessKey":"XXXXXXXXXXXXXXXXXXXXXXX",
"bucketName":"anaplandemo"
}
}
- Once the API request is submitted, the service should return a successful response back with HTTP code 200 and a value for connection id. You will save this connection id, to be used in the next API request in creating a CloudWorks integration. Copy the value for connectionId in a notepad and update it for collection variable connection_id.
- Log into Anaplan CloudWorks. You should see the connection you created under Connections.
Create an integration using CloudWorks API
In this second step, we will create a CloudWorks integration. In the integration, we will provide connectivity information for both AWS S3 and Anaplan. Two types of integrations can be built:
- AWS S3 (Source) Anaplan (Target): Data from a file on AWS S3 is selected as a source and Anaplan Import Action as a target.
- Anaplan (Source) AWS S3: Data from Anaplan is exported using Export Action (Source) and written to a file on AWS S3 Bucket (Target).
Create integration API has the following REST structure:
|
Method |
POST |
|
API end point |
|
|
Authorization |
No Auth |
|
Headers |
Authorization: AnaplanAuthToken {{token_value}} |
|
Body |
{ "name": "CWAPI_AWSS3_Anaplan", "jobs": [ { "type": "AmazonS3ToAnaplan", "sources": [ { "type": "AmazonS3", "connectionId": "a2fb43a9-3bc8-4bcb-9f37-dd59a75eb940", "file": "source/AccountsS3.csv" } ], "targets": [ { "type": "Anaplan", "actionId": "112000000005", "fileId": "113000000000", "workspaceId": "8a81b09d5e8c6f27015ece3402487d33", "modelId": "35A6EF893D7F47EEA5A554D5CC7DC330" } ] } ] }
Do not change the value for “type”: “AmazonS3ToAnaplan”, “type”:”AmazonS3”, “type”:”Anaplan” Provide values for following elements: connectionId, actionId, fileId, workspaceId, modelId. |
- Select POST method and API end point https://api.cloudworks.anaplan.com/1/0/integrations. Also, select “No Auth” for Authorization.
- Create the following headers: Authorization and Content-Type
- Copy and paste the following JSON structure into the Body of your request. Replace values for connectionId, file, actionId, fileId, workspaceId, and modelId. You may obtain values for your import action (actionId) and Data Source File (fileId) using Anaplan Integration API. Details on Integration APIs can be found on Anapedia. You can obtain values for workspaceId & modeled from Anaplan UX.
{
"name": "CWAPI_AWSS3_Anaplan",
"jobs": [
{
"type": "AmazonS3ToAnaplan",
"sources": [
{
"type": "AmazonS3",
"connectionId": "a2fb43a9-3bc8-4bcb-9f37-dd59a75eb940",
"file": "source/AccountsS3.csv"
}
],
"targets": [
{
"type": "Anaplan",
"actionId": "112000000005",
"fileId": "113000000000",
"workspaceId": "8a81b09d5e8c6f27015ece3402487d33",
"modelId": "35A6EF893D7F47EEA5A554D5CC7DC330"
}
]
}
]
}
- Once the API request is submitted, the service should return a successful response back with HTTP code 200 and a value for integration id. Copy & paste value for integrationId to collection variable s3_anaplan_int_id. We will use integrationId in a later step when we gather run details for an integration.
- Log into Anaplan CloudWorks. You should see the integration you created under Integrations.
Run an integration using CloudWorks API
Now that we have created an integration, we will run the integration that loads data from a file on AWS S3 bucket to an Anaplan module. Once the integration is run, we will retrieve a history of integration runs and its details. We will do this in the next step. CloudWorks APIs can be used to schedule integrations. However, in this blog, we will focus on executing integrations via API.
You will need to provide the following details in your API request:
|
Method |
POST |
|
API end point |
https://api.cloudworks.anaplan.com/1/0/integrations/<integration_id>/run |
|
Authorization |
No Auth |
|
Headers |
Authorization: AnaplanAuthToken {{token_value}} |
|
Body |
None |
- Select POST method and API end point https://api.cloudworks.anaplan.com/1/0/integrations. Also, select “No Auth” for Authorization.
- Create the following headers: Authorization and Content-Type
- Once the API request is submitted, the service should return a successful response back with HTTP code 200.
- View integration run status & details in Anaplan CloudWorks UX.
- If an integration run fails, you will also be notified via email.
Get history of integration runs
CloudWorks API provides end points to get details on integration run history. REST structure for this end point is:
|
Method |
GET |
|
API end point |
https://api.cloudworks.anaplan.com/1/0/integrations/runs/<integrationId> |
|
Parameters |
Offset = 0 |
|
Authorization |
No Auth |
|
Headers |
Authorization: AnaplanAuthToken {{token_value}} |
|
Body |
None |
- Select POST method and API end point https://api.cloudworks.anaplan.com/1/0/integrations/runs/<integrationId>. Also select “No Auth” for Authorization.
- Create two parameters offset & limit. Set their values to 0 & 2 respectively.
- Create the following headers: Authorization and Content-Type
- The service request should return a successful response back with HTTP code 200 and details for each run for selected integration.
Contributing authors: Pavan Marpaka, Scott Smith, and Christophe Keomanivong.
Comments
-
@annejulie Brilliant. Thank you.
0 -
I am uploading data to S3 from Anaplan. However, my S3 bucket is KMS encrypted and I need to specify the encryption header as follows when I make an API request to S3 to upload the data.
"ServerSideEncryption": "aws:kms"
How do I specify this?
0 -
@AnaplanOEG / @christophe_keom - reading through this might solve our issue about our IT requirements about using (or not) Cloudworks natively. If we were to create the connection using the above method would this mean that a) the authentication wouldn't be visible within Anaplan and that b) we would be able to create integrations natively in Cloudworks using that S3 bucket connection?
0 -
@andrewtye
a/ I'm not sure what you mean by not "be visible" within Anaplan.
b/ You can indeed create integrations via API using this S3 bucket connection. You could even create that connection via API if you really want to push it that far.0 -
Thanks @christophe_keom . Mean the items highlighted below:
Now I know you (as Anaplan) can't see those two items but our security teams are nervous about having those items potentially readable and giving a route into our organisation.
So thought was well rather than using APIs all the way through why not just use the Connection API and then use run integrations off of that connection. Make sense?
0 -
Thank you @andrewtye.
It makes total sense.
I understand that concern but unfortunately, even if you create this connection via API, this screen will still be created and in the UI, you'll be able to open that popup.0 -
Hey @andrewtye, some of our clients' IT security teams have had similar concerns.
We've addressed those concerns through conversations in which we articulated the approach of:
- [Most importantly] On the AWS side: Implement an S3 Bucket Policy with Allowlisted IPs. To limit access only to trusted IP addresses, you'd want to apply an S3 Bucket Policy on the AWS side. Here you'd specify the relevant Anaplan IP ranges to only allow access to contents of the AWS bucket from allowed IPs. This should help you with the IPs you'll want to add to your whitelist, but you may need to work with Anaplan Support or your consulting partner to get this configured properly: https://help.anaplan.com/cloudworks-ip-and-url-allowlist-756693d9-643c-4dab-99a1-af365da79806
- On the AWS side: Use separate credentials and AWS S3 buckets (or other simple storage containers) for master vs. fact data, such that if someone gained access to one of the buckets, they wouldn't be able to make sense of that data without its counterpart
- On the AWS side: Set up the AWS buckets such that if a third party gained access to either or both of the buckets, it would be limited to just the master and/or fact data being sent to Anaplan, not other organizational data. In other words, don't put the answer to life the universe and everything in this AWS S3 bucket unless it's relevant to what you're feeding into Anaplan
- During creation of the CloudWorks Connection object(s): During creation, whether via the web interface or via the API, keys are encrypted when submitted to Anaplan. So that means, if your IT infrastructure team is setting up the AWS S3 bucket, give them temporary access to Anaplan as an Integration Admin and have them create the connection. Or have them invoke the Anaplan API (using Postman or via any other programmatic means) such that they create the connection, and you aren't privy to the connection information, however, are aware of what the connection represents (i.e., this connection gives me access to Production data from our GL).
- After creation of the CloudWorks Connection object(s): Access key ID and Secret access key are encrypted on Anaplan's side, and end users (even tenant admins) cannot view the raw values, so after step 4 is completed, your IT folks would really just need reassurance that your approach to managing security and provisioning within Anaplan meets the security standards and protocols in place at your organization
Not sure if that's helpful? Let me know if any questions.
Some of this wasn't intuitive for clients of ours that were on the smaller side (orgs which didn't have significant internal IT resources), so depending on your resources I'd work multiple threads: documentation, forums, Anaplan Support, your Anaplan BP / CSD, and if you have an experienced partner you trust get their take.
Cheers,
Kevin
0
