[Part 2] Enhancing Anaplan Audit Log Data Extraction with a Streamlined Python Solution

AnaplanOEG

The Anaplan Audit History project, consisting of several Python modules and a configuration file, which works synergistically to streamline the process of fetching, formatting, and loading Anaplan audit log data into a preconfigured Anaplan Model. The project leverages the Anaplan REST API, the Python Pandas library for efficient data conversion from web services JSON format to tabular, and SQLite for advanced data blending and transformation capabilities.

SQLite, a highly versatile and lightweight database engine, offers substantial benefits for transforming and blending various related datasets. Its capacity to execute complex SQL queries and join operations enables users to merge datasets in diverse ways, establishing itself as a powerful tool for data integration and enrichment. Utilizing SQLite's comprehensive set of built-in functions, the project seamlessly combines multiple Anaplan datasets and performs data transformation operations such as filtering, sorting, and reshaping within the SQLite environment. The simplicity of SQLite's file-based storage system facilitates easy deployment and management directly within the Python code, making SQLite an efficient, scalable, and accessible solution for data blending, transformation, and summarization. In context to this project, this SQL code combines various Anaplan metadata, including Users, Workspaces, Models, Actions, Processes, and CloudWorks Integrations via a SQL join with the raw audit data to synthesize audit data in a reportable format. 

In order to supply all this Anaplan content, the following Anaplan REST APIs need to be leveraged:

• OAuth Service API - to authenticate and refresh the access_token based on the client_id and refresh_token.
• Audit API - to fetch the audit records.
• Integration API - to fetch metadata about Anaplan objects such as data sources, Processes, and Actions. Additionally, to refresh content to the target Anaplan Audit Reporting Model, the bulk API is used to upload the report-ready audit data, and the transaction API is leveraged for updating the latest timestamp.
• SCIM API - to fetch Anaplan user metadata.
• CloudWorks API - to fetch CloudWorks integration metadata.

This project exemplifies how Python can effectively integrate and automate Anaplan operations, using modern OAuth services for Anaplan authorization in tandem with the rich capabilities of the Anaplan REST API. Additionally, the project highlights several Python best practices, such as:

1. Organizing code into packages and modules: Segmenting the code into multiple modules based on functionality improves maintainability and simplifies the process of locating and resolving future issues.
2. Enhancing error handling: Effective error management is vital when working with external APIs. Implement try-except blocks to handle exceptions that may occur during API calls or file I/O operations.
3. Utilizing Python's logging module: Opt for the built-in logging module instead of print statements for debugging, providing better control over log verbosity and streamlining log output management.
4. Leveraging environment variables or configuration files: Avoid hardcoding sensitive information like API keys or credentials, and store this information using environment variables or configuration files instead.
5. Adding comments to the code: Include annotations in complex or non-obvious code sections to improve comprehensibility for both yourself and others.

Anaplanners can use this project as a basis for building tailored integrations.

The source code and supplementary details, such as requirements, deployment instructions, and helpful videos, are available on GitHub.

Next, learn how to use this data in an Anaplan Model by reading the third installment of this series: [Part 3] Anaplan Audit History Data in an Anaplan Reporting Model.

Author: Quin Eddy, @QuinE - Director of Data Integration, Operational Excellence Group (OEG)