1.05-10 Never delete list and reload the list as a daily occurrence

[#1]

Clearing and reloading a list increases the structural changes within a model and will increase the likelihood of a of model save, thus increasing the import time.  It also removes pre-allocated memory block designed to increase efficiency. Use a unique key to update to values rather than delete and re-load. Also consider adding a TRUE field to the data source which will allow you to know which records have been imported

Best Practices articles

Data Hubs Purpose and Peak Performance
Pre-Allocation in Lists and Impacts to Model Performance

[Misbah]

Rule 1.05-10 Never delete list and reload the list as daily occurrence. Using a unique key to update to values rather than delete and re-load. It goes against P & N elements of PLANS.

Here is how it was done in Pre Planual Era: When we had to delete the transactional data & reload the data we used to delete/wipe out the entire list items of the lists

What is wrong with this method? There are a lot of disadvantages

1. Deleting the list items is model specific and not module specific when there was a need to add/update data to few modules of a model.
2. There are lot of changes that the system has to do – first delete the list items, create the change log and record it in history, update the lists, create the change log again and record it in history. When the certain threshold is surpassed model will require a save and thus taking more time. On top of that model has to recalculate and re-aggregate the data across the whole model. That is a lot of work for Hyperblock and it surely impacts the performance of the model
3. Deleting and reloading the lists calls for an additional maintenance by resetting the lists once it nears 999,999,999 items

Here is how it should be done in Planual Way: All the transactional data should flow into modules and NOT lists, which is a Thumb Rule.

Now if your client operates in such a way that you have to delete the transactional data then you delete it from the modules without touching the list items. You can do it by creating a process which will wipe out the data first and then reload it back

1. Create line item formatted as “No Data”
2. Create a process with two imports
3. Import No Data formatted line item to Transactional data line item
4. Import data into the same line item

Now there can be many scenarios where you need to load data for all historical periods (i.e., restating the history) or load data for few periods (i.e., current month and previous month). These all can be tackled through filters and having one saved view for each scenario

Ideally what should happen is that only the updated/changed data should make its way to Anaplan Data Hub and not the entire data set which probably may remain unchanged. Two ways to achieve this -

1. From the source system, request IT to only send the updated information, not the full load every time. Additionally, request IT to create a column in the source file with a hardcoded value of “TRUE.” This will tell Anaplan which row is new or has been updated and can be used as a filter for an export. Just know, before the import of the source data gets loaded, make sure the first action within the process clears out the previous true records within a module (set this up via a view using a filter where the view only shows members with a value of true).
2. Utilize the current period function to only export the current period data from Hub. In the SYS Time Filter module, create a line item named Current Period with the formula CURRENTPERIODSTART(). In the export views, filter the data on this line item.