Building a Data Hub

Hi Anaplan Team

I have built a data hub to store all my data in that model and later transfer the same to my main model through anaplan connect.

I am facing an issue: Lot of time is taken to load just 1 file of 200 MB in txt format, approx 20 min is taken to load such files and I have multiple files of same size & format to be loaded in system.

Suggest me the best suitable alternative on how to load the files in quick time.

@NakulAggarwal @saurabh.raheja @Rishabh4590

Regards
Ak**bleep**

Accepted answers

All comments

Ankur.Sonthalia

Hi @Aks,

Hope this could be of any help to you.

https://community.anaplan.com/t5/Best-Practices/Building-a-Data-Hub/ta-p/33593

Regards,

Ankur Sonthalia.

usman.zia

Hi @Aks

Have you tried converting the TXT files to excel CSV flat files and loading them in this way?

Also have you followed the best practice by creating a list dimension from the file that you are loading as to not load all the properties directly into the lists properties. Then using this list dimension in a module for which you are then loading the complete data set into?

Would be good to get an explanation of your process to load the data in currently.

Thanks,

Usman

Aks

Hi @usman.zia

Thanks for the prompt response! We are not able to follow that practice since we are not using code in the list but using a numbered list with combination of all the properties to identify the unique combinations of data.

Regards

Aks

usman.zia

Hi @Aks

Did you try to convert the TXT file to a CSV?

I believe the issue is due to the time to load will be due to the fact you're loading into a list and not a module.

How many rows of data are you loading in?

Thanks,

Usman

rob_marshall

@Aks ,

Why are you not using a code as it is best practice to always use a code. When using the combination of properties, that takes the system longer to load data because it is trying to figure out the unique row. If you already have that unique code, the system saves time on the load process. Since you are using a combination of properties, how many properties are you using? Is time and the transactional data part of the uniqueness? If so, this is making your list extra long where it doesn't need to be. You need to look at the data, study the data, and understand what makes it unique and have the source create a unique key to be used for the code. Are you deleting the list on every load and then reloading it? If so, this is not best practice and can be remedied with a code for the list and then the transactional data gets loaded to a SYS Transactional module dimensionalized by Time. Any changes get updated so no need to do a delete.

Can you post a portion of the list with properties?

Thanks,

Rob

dsterbakova

Hi @Ankur.Sonthalia,

I am afraid the link you provided is not working and the article on Building a Data Hub is not available anymore. Do you by any chance have a new link?

Thank you,

Dominika

Kat.Carvey

Is the time taken just for uploading the 200MB file or also running the action?

rob_marshall

@dsterbakova ,

The new link is https://community.anaplan.com/t5/Best-Practices/Data-Hubs-Purpose-and-Peak-Performance/ta-p/48866

Rob

Quick Links

Can you help?

Unanswered questions

Best way to Architect Financial Statements in Anaplan for Scalability
I have seen two methods used to build financial statements and I am wondering what others have experienced in scalability/maintenance between the two or if there is a better approach. Assume that we have a flat data source for our general ledger within Anaplan. Our line item we are focused on within the source data is the…
Curious Comparison: Forecast Modeling for Unusual Products
I've been experimenting with some forecast modeling techniques in Anaplan recently and decided to test them on a few quirky, offbeat product categories just for fun. One of the categories I played around with was love dolls—yes, you read that right. It actually turned out to be more interesting than expected. Unlike…
OData Rest API for
Hi, Does anyone know how to get an outbound OData Rest API, (OAuth2) to establish a connection between SAP and Anaplan. Thanks Khaled