Limitations on data load

PujithaB

Is there any limit on data load? i.e. If the data is importing from CSV, What could be the maximum size of the CSV File?

AlejandroGomez

Hi @PujithaB 

 

That is a simple and yet rather interesting question!

 

I would like to just add to @LipChean_Soh  answer that in case of uploading large files, these can be zipped in order to reduce the time that it takes to load them into Anaplan's server. More info here.

 

I have tested it myself  with a 2.26 Gb CSV and its zipped version (17.4 Mb) and the difference in terms of time to get the file loaded to Anaplan's server is huge (around 30 min vs a few seconds - with the same connection).

 

Finally, speaking about the risk of running out of memory when loading a large file. Assuming that the module receiving the data is not connected with any other module/list in any way (so we only have to worry about the module receiving the data itself), the same 2.26Gb CSV translates into a 3.57Gb module in Anaplan, i,e around 55% more space in Anaplan than in the CSV.  

(The test module is just a table made of one list (3,989,350 items)  + Line Items (140) across columns, all the LI formatted as text, same format and display applies for the CSV).

 

I hope this helps you out to have a better sense of the space requirements in your Workspace based upon your source files and the load times (and how to reduce them!)

 

Cheers! 

 

Alex

 

Olivehorse 

 

 

 

LipChean_Soh

Hi @PujithaB 

 

This is an interesting question.

 

I am not sure if there is a limit from the incoming CSV point of view, i have heard of 40Gb CSV files being uploaded into Anaplan, and it still went through, though it took a VERY long time.

 

And it's also possible that you can have CSV_A (with number formatted data) which is smaller than CSV_B (with text formatted data), but it ends up such that the loading of CSV_B results in an error because you've hit the 130Gb limit in Model B when you tried to load a lot of TEXT into Model B.

 

I would look at this problem from another angle, i.e. do we need to bring in so much data? I'm not saying Anaplan cannot handle large amount of data, but sometimes we need to make a challenge to the data warehouse or business side. Just because Anaplan can, doesn't mean it should.

 

Please refer to https://community.anaplan.com/t5/Best-Practices/Data-Hubs-Purpose-and-Peak-Performance/ta-p/48866

Especially the part below:

 

Thanks,

LipChean

JaredDolich

@PujithaB 

@LipChean_Soh is correct.

Also, here's another link where someone asked the same question (and it was also answered by @LipChean_Soh!)

There are some good statistics there about size limits.

 

As a matter of data integration, if you use the API or one of the connectors in Informatica, Dell Boomi, or Mulesoft, there is a recommended size of file.

I can't seem to find the link for that one but my recollection was around 20MB per chunk.

 

Lastly, here is the recommendations for large import files from Anaplan. Great suggestions, many that @LipChean_Soh recommended.

 

JaredDolich

@AlejandroGomez I've never tried a zipped file before. Not sure how I missed that.

Thank you for sharing that - will be adding that to my personal best practices!

PujithaB

Thankyou  AlejandroGomez . It helped me in reducing the Uploading time.

AlejandroGomez

I am glad it came in handy. 

 

See you around, guys