Delta Load

PujithaB

Scenario: I changed the parent of a child then how to do incremental load when there is a change in parent child hierarchy?

 

DavidSmith

@PujithaB 

This is a "depends" answer, but I'll run through a number of scenarios

 

Assuming you have a data hub (which you should have!), you should be used flat lists at the lowest level (not hierarchies - Planual 5.07-03 ).  You should also have a module by the child list with the parent attributes.

There are two ways to get the delta load from the hub to the source model(s).

 

1. This one is dealing with the specific issue of a change in the parent.

1a. Create a new line item in the module as above, to hold the parent attribute again.

1b. Create a boolean line item in this module too to act as a filter for a saved view used for the downstream export

1c. Set the formula to check the parent attribute vs the new line item from 1a

1d. Before you import the data from source, run an internal import from the parent attribute to the line item from1a

1e. When you import the data from source, the line item from 1b will highlight any differences in the parent

 

You could view these differences in a dashboard for validation purposes, because if you have any logic based on the parent in the downstream model, the calculations on the child level will change

If you need to track month by month changes in the parents in the downstream model, you can also create a module by month and hold the parent values for the flat child list and aggregate the data using that

 

2. A more general approach to delta loads which works well if the source file only contains new records is:

2a. Add a column to the source data with TRUE for all rows

2b. In the data hub, create 2 boolean line items

• Reset - formula = FALSE
• Export?

2c.  Prior to the import, run an internal import from Reset to Exports? to clear out any previous values

2d. Import the source data and bring in the TRUE flag to the Export? line item.  This can then be used to export the data downstream and only records in the source will be flagged to export

 

I hope this helps

David