Hi All,
I have an issue which needs to be addressed for a client. I do have a solution however would like to reach out to amazing community and get a different viewpoint on this if possible .
Problem statement : we need to clear an item from a module however there is no simple way to identify the item to clear
Below is a screenshot of what we get let say today ( I have also attached an excel version)
Import file :- This is the file we get from source system , It goes into data staging module . in Datahub
Data staging :- In this module we create an id concatenation of customer store product date and price as below (i know price should not be part of id however there is no other way to achieve uniqueness as price changes within day and we need to capture QTY / price )
Data final :- This is a list where we load the ID's . So essentially we keep history of old records . We do have processes to remove old records however lets not go there ( in spoke model)

Tomorrow we receive below
Please notice that #3 transaction has updated price so think of this as yesterday there was an error and today its fixed . We cant have a flag which says what has been updated or ref to old record by IT in the file .

Issue :- We need to some how clear the transaction which was loaded yesterday i.e
Solution :

Any suggestion ideas will be really appreciated .
Karan