Is there any limit on data load? i.e. If the data is importing from CSV, What could be the maximum size of the CSV File?
Hi @PujithaB
That is a simple and yet rather interesting question!
I would like to just add to @LipChean_Soh answer that in case of uploading large files, these can be zipped in order to reduce the time that it takes to load them into Anaplan's server. More info here.
I have tested it myself with a 2.26 Gb CSV and its zipped version (17.4 Mb) and the difference in terms of time to get the file loaded to Anaplan's server is huge (around 30 min vs a few seconds - with the same connection).
Finally, speaking about the risk of running out of memory when loading a large file. Assuming that the module receiving the data is not connected with any other module/list in any way (so we only have to worry about the module receiving the data itself), the same 2.26Gb CSV translates into a 3.57Gb module in Anaplan, i,e around 55% more space in Anaplan than in the CSV.
(The test module is just a table made of one list (3,989,350 items) + Line Items (140) across columns, all the LI formatted as text, same format and display applies for the CSV).
I hope this helps you out to have a better sense of the space requirements in your Workspace based upon your source files and the load times (and how to reduce them!)
Cheers!
Alex
Olivehorse
This is an interesting question.
I am not sure if there is a limit from the incoming CSV point of view, i have heard of 40Gb CSV files being uploaded into Anaplan, and it still went through, though it took a VERY long time.
And it's also possible that you can have CSV_A (with number formatted data) which is smaller than CSV_B (with text formatted data), but it ends up such that the loading of CSV_B results in an error because you've hit the 130Gb limit in Model B when you tried to load a lot of TEXT into Model B.
I would look at this problem from another angle, i.e. do we need to bring in so much data? I'm not saying Anaplan cannot handle large amount of data, but sometimes we need to make a challenge to the data warehouse or business side. Just because Anaplan can, doesn't mean it should.
Please refer to https://community.anaplan.com/t5/Best-Practices/Data-Hubs-Purpose-and-Peak-Performance/ta-p/48866
Especially the part below:
Thanks,
LipChean
@PujithaB
@LipChean_Soh is correct.
Also, here's another link where someone asked the same question (and it was also answered by @LipChean_Soh!)
There are some good statistics there about size limits.
As a matter of data integration, if you use the API or one of the connectors in Informatica, Dell Boomi, or Mulesoft, there is a recommended size of file.
I can't seem to find the link for that one but my recollection was around 20MB per chunk.
Lastly, here is the recommendations for large import files from Anaplan. Great suggestions, many that @LipChean_Soh recommended.
@AlejandroGomez I've never tried a zipped file before. Not sure how I missed that.
Thank you for sharing that - will be adding that to my personal best practices!
Thankyou AlejandroGomez . It helped me in reducing the Uploading time.
I am glad it came in handy.
See you around, guys
Hi ALL, I have a module where a line item (Assignments column) is formatted by a large ‘Projects’ list. Each record is assigned to a Project and this module is published to a UX page. When users apply a filter on the Assignments column, the filter dropdown displays the entire Projects list (since the column is…
Hi ALL, In my model, I have Need stored at a 'D3' RS Group level. I need to use this Need in another module at the G5 Location level. I currently have a mapping module where each G5 Location maps to a D3 RS Group. This Module uses D2 Role as dimension. but the line item I've to find the RS Group (Find D3 RS Group = 'FIND…
I couldn't find any formula to do this, but I want a way to put on the page/grid the connection details. This is so I know what connection I'm in rather than going into Connections details to see it. Is there an XL3 formula to return this value?
