incremental synchronization

Is there any way to  implement incremental synchronization of data loads?

which will only synchronizes the data in the application that has changed since the last sync from the data source?

Community Boss

Re: incremental synchronization

There are a few ways to go about only loading deltas. If you can track this in your data source (MySQL, SAP, etc), it offloads the burden to your database. If not, you can do a full load into a data hub, where locking the model during an import isn't consequential because users wouldn't be performing any work in that model - from there it's easy to perform a calculation to look for changes from the previous load. Create a boolean line item that is true when there are changes, and a saved view based on that from which you import into your spoke model(s).




Jesse Wilson
Solution Architect - Data Integrations
Community Boss

Re: incremental synchronization

To add on to what Jesse is saying, create a new row in the data load set to True or 1.  The process to load the data would be:

  • Run and action to set the boolean to False
  • Load the data (thus it will switch the boolean to a true where new records are created)
  • Load only the trues to the spoke.

Hope this helps




Re: incremental synchronization

@jesse_wilson  @rob_marshall 

Thanks for the Solution, It helped!