Delta Load

govindaagrawal
govindaagrawal
in Integrations and extensibility

Is it possible to do a delta load in the Anaplan model? Lets say one flat file is uploaded at 10.00 AM with 5 million records and the same file is uploaded at 11.00 AM with 6 million records. I just want to upload the new 1 million records into my model. How can we achieve that? 

 

Thank you in advance!

Tagged:

Best Answers

  • LipChean_Soh
    LipChean_Soh
    Answer ✓

    Hi,

     

    Assuming you have 2 models, i.e. Data Hub and Spoke Model.

     

    You do the initial load into Data Hub, there's now 5 million transactions ID in the Transaction# List, and you have 2 modules, i.e.

    1. Transaction# Properties with line items of CODE, NAME, and any properties of Transaction#

    2. Transaction Module with the transaction data of Sales, Cost, etc etc

     

    You can add 2 line items in the Transaction# Properties Module, i.e.

    1. True Flag = TRUE, format = Boolean

    2. Not Exported?, format = Boolean

     

    Create a Saved View in the Transaction# Properties Module called Delta, with a filter where 'Not Exported? = TRUE'.

    Create a Saved View in Transaction Module, with a filter where 'Transaction# Properties.Not Exported? = TRUE'.

     

    Then perform the 3 actions below:

    1. Import Transaction# List to Spoke Model using Saved View 'Transaction# Properties.Delta'.

    2. Import Transaction data to Spoke Model using Saved View 'Transaction Module.Delta'.

    3. Import 'True Flag' to 'Not Exported?' in 'Transaction# Properties' module.

     

    The idea is that any data that has been sent from the Data Hub to the Spoke Model will be flagged, so that only new data will have a blank 'Not Exported?' and is filtered out in the Saved View ready for the next export.

     

    Having said that, you should consider 2 items below:

    1. 5 million data point sounds like a lot of data, is it daily transactional data? Do you REALLY need this in Anaplan? (Follow 'N' of PLANS, is it necessary?)

    2. If you only need an aggregation, do have a discussion with your database colleague, and get them to aggregate before sending the DELTA to Anaplan, so that you don't even need to do the steps above.

     

    For more info on the best practices on Data Hub, you can refer to https://community.anaplan.com/t5/Knowledge/Building-a-Data-Hub/ta-p/33593

     

    Thanks,

    LipChean

  • LipChean_Soh
    LipChean_Soh
    Answer ✓

    Hi,

     

    Cool. Please mark as solution, so that this topic is closed.

     

    Thanks

Comments

  • govindaagrawal
    govindaagrawal

    Hi LipChean,

     

    Thank you for your reply. 5 M records were just the assumption as I was trying to understand the concept. It worked for me. Thank you again!

Categories