ALM Explained—Part 3: Avoiding Data Loss

[#1]

Introduction

In part two of the new ALM Explained series, we discussed the best practices for testing and validating changes prior to deploying them to production models.

However, we also know that testing protocols are not always followed, especially if the change is an emergency patch, and sometimes, the synchronization can cause issues in the production model. There is a defined best practice for “hot fixes”.

If the change needs to be reversed, see part two section on ‘reverting a revision’. However, the worst-case scenario is when a synchronize causes users’ data to be removed, and, in this case, just reverting to the previous revision is not going to restore the data (because data never moves from development to production).

The following are some of the most common examples of how stored data might be overwritten following a synchronization:

• Changing the ‘applies to’ of a module, including subsets of the original list.
• Changing the timescale or calendar settings.
• Moving the switchover period backward from the current setting.
• Adding a formula to a line item that was previously a data entry line item.
• Changing the list from production to structural.

The first four items are largely self-explanatory, but let's dive deeper into the last one, as this is probably the most common reason for data loss—and the least understood.

Production Lists vs Structural Lists

Before we look at how data loss occurs, let’s recap the concept of production lists.

Structural information refers to the configuration of the model:

• Lists.
• Modules.
• Actions.
• Model Calendar. 
• etc.

These are elements that cannot be edited in deployed mode; there is no expectation that end users will need to amend these structures.

Production information generally refers to the operational data within the model; for the most part, this will be data entered by users and the users list. When it comes to lists, there are times when we expect that the list's members will need to be amended within a deployed model:

• Importing from a Data Hub.
• As part of the operational process, such as promotional planning, or employee assignments to projects.

Setting the list as production data prior to setup or a synchronization allows the list members to be added, deleted, or amended within the production model in deployed mode. When the synchronization takes place, only the structural information is transferred; production data never moves from development to production.

For information on Production Lists, see here.

Why Do These Settings Matter?

One of the most common causes of loss of data in a deployed model is changing the list setting from production data back to structural. We need to look in detail at what is actually happening under the covers to explain why.

Each list member has an associated internal id. If the list is set to structural, when the initial synchronization takes place, the lists are different, but the list members are "effectively" the same and have the same ids. For simplicity, I have assumed the id is four alphanumeric characters; in reality, it is 12 digits.

When a new member is added to the list in development, on the next synchronization, this member is added to the list in the production model. Note that no data values are transferred across, as the data remains as production data.

Let’s assume that P1 Products is now set to production data. On the next synchronization, the link between the two lists is broken, but to maintain integrity, the list members and associated data are retained.

When a new item is added into the development model, because the list is production data, this member is not transferred over on the synchronization.

Users are able to add items into the list in production, and these items are not added to the development model.

Now, let’s assume that the list is re-set back to structural. This is where the problem occurs. The production list members are effectively deleted and re-added, they are given new ids, and any data associated with the list members is deleted.

How To Recover Data

Hopefully, this process will not be needed, but just in case the worst happens, the following steps are a way back. The steps below are able to correct basic issues, such as simple structural changes, or where formulas are added to 'data' line items. Depending on the level of complexity, it may not be possible to restore the data, so it is important to follow the correct testing protocols as outlined in part two of the new ALM Explained series.

As part one explained, models are connected through revision tags, and deployed mode ensures that the compatibility is maintained. This process does mean breaking the golden rule of ALM (‘never take the model out of deployed mode’), but this is an exception. Follow the process below, do not deviate, and you can restore the data lost through an erroneous change while maintaining compatibility.

1. Correct the issue. Identify the cause of the issue and correct it in the development model. This could be a simple change, or quite involved depending on the complexity of the issue.
2. Set a new revision tag for the correction in the development model.
3. Test, test, test. Ensure that this correction fixes the issue and also does not cause any regression issues in other areas.
4. Assuming the test results are as expected and the issue is fixed, synchronize the production model with the latest revision tag set in 2. This will not correct the data loss at this point, but it is a necessary step to correct the structural integrity of the production model.

Now, we have to break the rules!

1. In the production model, change the mode to standard, i.e. taking it out of deployed mode.
2. Refer to the history and find the point that the issue occurred. You should be able to see the revision tag and synchronization point in history. Look for a history id immediately below the “Begin sync revision” row.
3. Restore the model to this point; this will remove the structural changes made in the revision that caused the issue. At this point, the data and structures should be restored to a point where the data was correct (so not including the changes are part of the latest revision or the revision that caused the issue).
4. It is tempting now to copy the model to be safe – DO NOT DO THIS – As outlined in part one, copying a model with unsaved structural changes will create a new revision tag to crystallize the changes, and this will break the compatibility.
5. Navigate to the Revision Tag section of Model Settings. You should see the “Revert to Last Revision” button is available.
6. Click this button. This will correct the issue by restoring the structures while leaving the data as it was.
7. As a final step, change the model mode to deployed mode.

Conclusion

Remember, “with great power comes great responsibility”. Follow the rules, test thoroughly, and adhere to best practices to ensure a seamless Application Lifecycle Management process. I hope this three-part series has helped you develop a deeper understanding of the fundamental concepts of Application Lifecycle Management so that you can fully utilize the functionality and avoid any pitfalls.

[einas.ibrahim]

@DavidSmith 

What a great series. Thank you very much for the effort and the detailed explanation. The graphics you provided in all 3 series are fantastic and helped a lot. 

Love your summary "with great power comes great responsibility"

[dhruv411]

Thanks @DavidSmith 

Data Recovery steps are really helpful !