Is there any limit on data load? i.e. If the data is importing from CSV, What could be the maximum size of the CSV File?
Hi @PujithaB
That is a simple and yet rather interesting question!
I would like to just add to @LipChean_Soh answer that in case of uploading large files, these can be zipped in order to reduce the time that it takes to load them into Anaplan's server. More info here.
I have tested it myself with a 2.26 Gb CSV and its zipped version (17.4 Mb) and the difference in terms of time to get the file loaded to Anaplan's server is huge (around 30 min vs a few seconds - with the same connection).
Finally, speaking about the risk of running out of memory when loading a large file. Assuming that the module receiving the data is not connected with any other module/list in any way (so we only have to worry about the module receiving the data itself), the same 2.26Gb CSV translates into a 3.57Gb module in Anaplan, i,e around 55% more space in Anaplan than in the CSV.
(The test module is just a table made of one list (3,989,350 items) + Line Items (140) across columns, all the LI formatted as text, same format and display applies for the CSV).
I hope this helps you out to have a better sense of the space requirements in your Workspace based upon your source files and the load times (and how to reduce them!)
Cheers!
Alex
Olivehorse
This is an interesting question.
I am not sure if there is a limit from the incoming CSV point of view, i have heard of 40Gb CSV files being uploaded into Anaplan, and it still went through, though it took a VERY long time.
And it's also possible that you can have CSV_A (with number formatted data) which is smaller than CSV_B (with text formatted data), but it ends up such that the loading of CSV_B results in an error because you've hit the 130Gb limit in Model B when you tried to load a lot of TEXT into Model B.
I would look at this problem from another angle, i.e. do we need to bring in so much data? I'm not saying Anaplan cannot handle large amount of data, but sometimes we need to make a challenge to the data warehouse or business side. Just because Anaplan can, doesn't mean it should.
Please refer to https://community.anaplan.com/t5/Best-Practices/Data-Hubs-Purpose-and-Peak-Performance/ta-p/48866
Especially the part below:
Thanks,
LipChean
@PujithaB
@LipChean_Soh is correct.
Also, here's another link where someone asked the same question (and it was also answered by @LipChean_Soh!)
There are some good statistics there about size limits.
As a matter of data integration, if you use the API or one of the connectors in Informatica, Dell Boomi, or Mulesoft, there is a recommended size of file.
I can't seem to find the link for that one but my recollection was around 20MB per chunk.
Lastly, here is the recommendations for large import files from Anaplan. Great suggestions, many that @LipChean_Soh recommended.
@AlejandroGomez I've never tried a zipped file before. Not sure how I missed that.
Thank you for sharing that - will be adding that to my personal best practices!
Thankyou AlejandroGomez . It helped me in reducing the Uploading time.
I am glad it came in handy.
See you around, guys
Hello folks, Does anyone know how to work around the Polaris limitation of using the Formula or Ratio summary method alongside the Closing Balance within the same line item? I have a Business Unit list where the total (All BUs) must display the Corporate (1 of 20 leaf items under All BUs total) number of Unique Customers.…
Hello community, I have an issue with anaplan report (functionality). The issue I have is within couple of slides. The data in that report is scrolling into a new page. There is a hierarchy in the rows and line items and time in the columns. My problem is, the hierarchy is getting broken in the second page. One of the node…
I want to create an S3 connection that will have monthly data stored in individual csv files that is connected to ADO. Around month end, data will be uploaded multiple times, replacing existing csvs. When this happens I want to replace the existing data for that month with the new file. I don't want to have to create a…
