Limitations on data load

Is there any limit on data load? i.e. If the data is importing from CSV, What could be the maximum size of the CSV File?

Tagged:

Best Answer

Answers

  • Hi @PujithaB 

     

    This is an interesting question.

     

    I am not sure if there is a limit from the incoming CSV point of view, i have heard of 40Gb CSV files being uploaded into Anaplan, and it still went through, though it took a VERY long time.

     

    And it's also possible that you can have CSV_A (with number formatted data) which is smaller than CSV_B (with text formatted data), but it ends up such that the loading of CSV_B results in an error because you've hit the 130Gb limit in Model B when you tried to load a lot of TEXT into Model B.

     

    I would look at this problem from another angle, i.e. do we need to bring in so much data? I'm not saying Anaplan cannot handle large amount of data, but sometimes we need to make a challenge to the data warehouse or business side. Just because Anaplan can, doesn't mean it should.

     

    Please refer to https://community.anaplan.com/t5/Best-Practices/Data-Hubs-Purpose-and-Peak-Performance/ta-p/48866

    Especially the part below:

    Screen Shot 2020-01-07 at 09.22.28.png

     

    Thanks,

    LipChean

  • @PujithaB 

    @LipChean_Soh is correct.

    Also, here's another link where someone asked the same question (and it was also answered by @LipChean_Soh!)

    There are some good statistics there about size limits.

     

    As a matter of data integration, if you use the API or one of the connectors in Informatica, Dell Boomi, or Mulesoft, there is a recommended size of file.

    I can't seem to find the link for that one but my recollection was around 20MB per chunk.

     

    Lastly, here is the recommendations for large import files from Anaplan. Great suggestions, many that @LipChean_Soh recommended.

     

  • @AlejandroGomez I've never tried a zipped file before. Not sure how I missed that.

    Thank you for sharing that - will be adding that to my personal best practices!

  • Thankyou  AlejandroGomez . It helped me in reducing the Uploading time.

  • I am glad it came in handy. 

     

    See you around, guys