Delta Load

govindaagrawal

Is it possible to do a delta load in the Anaplan model? Lets say one flat file is uploaded at 10.00 AM with 5 million records and the same file is uploaded at 11.00 AM with 6 million records. I just want to upload the new 1 million records into my model. How can we achieve that? 

 

Thank you in advance!

LipChean_Soh

Hi,

 Assuming you have 2 models, i.e. Data Hub and Spoke Model.

 You do the initial load into Data Hub, there's now 5 million transactions ID in the Transaction# List, and you have 2 modules, i.e.

1. Transaction# Properties with line items of CODE, NAME, and any properties of Transaction#

2. Transaction Module with the transaction data of Sales, Cost, etc etc

You can add 2 line items in the Transaction# Properties Module, i.e.

1. True Flag = TRUE, format = Boolean

2. Not Exported?, format = Boolean

 Create a Saved View in the Transaction# Properties Module called Delta, with a filter where 'Not Exported? = TRUE'.

Create a Saved View in Transaction Module, with a filter where 'Transaction# Properties.Not Exported? = TRUE'.

Then perform the 3 actions below:

1. Import Transaction# List to Spoke Model using Saved View 'Transaction# Properties.Delta'.

2. Import Transaction data to Spoke Model using Saved View 'Transaction Module.Delta'.

3. Import 'True Flag' to 'Not Exported?' in 'Transaction# Properties' module.

 

The idea is that any data that has been sent from the Data Hub to the Spoke Model will be flagged, so that only new data will have a blank 'Not Exported?' and is filtered out in the Saved View ready for the next export.

Having said that, you should consider 2 items below:

1. 5 million data point sounds like a lot of data, is it daily transactional data? Do you REALLY need this in Anaplan? (Follow 'N' of PLANS, is it necessary?)

2. If you only need an aggregation, do have a discussion with your database colleague, and get them to aggregate before sending the DELTA to Anaplan, so that you don't even need to do the steps above.

 

Thanks,

LipChean