Imports and import data sources as production data

Misbah

This article explains the different scenarios and the implications of marking imports and import data sources as production data.

Before we begin, let's define production data. In the simplest terms, production data is data or artifacts that can be changed directly in a production environment without pushing the change via the development environment.

In order for the production environment to remain up to date, it is extremely important that it is connected to the source system. Since all the transactional data and metadata have to get loaded into the production environment directly, all the lists storing this data have to be marked as production data lists. In contrast to production data, there is structural data that doesn’t get changed directly in the production environment, and any change has to be pushed via the development environment through revision tags. On the other hand, imports and import data sources don’t necessarily have to be marked as production data. By default, they are marked as non- production data or structural data. Here are a few scenarios that this article will explain in detail, along with their pros and cons.

1. Imports marked as production data and import data source marked as non-production data.
2. Imports marked as non-production data and import data source as production data.
3. Both import and import data source marked as production data.
4. Both import and import data source marked as non-production data.

I - Imports marked as production data and import data source marked as non-production or data

In this scenario, import mappings and other import-related options can be changed from the production environment directly.

Let’s say we want to import from source module to target module. Both have similar dimensions. We take the following steps.

Step 1: Create a saved view in source module.

Step 2: Create an import, select mappings, and run it.

Step 3: Mark the import as production data.

Step 4: Push this change to the production environment. All is good until now.

Come back to the development model and change the saved view of the source module – in this case, “Sales” line item has been renamed to “Revenue”.

Run the same import action that you created in Step 2. You will notice that the import will run with failures; that is because Anaplan has not mapped renamed line item automatically.

Here is the mapping that is thrown off due to the above change.

In order to correct this, you edit the Import action in the development environment and re-select the mappings manually. Once done, you re-run the import action, and it will complete successfully.

Since this import action had been marked as production data earlier, whatever change you made in the dev environment will not flow to the production environment. Hence, there is a need to change the mappings in the production environment as well.

Advantages 

1. Admins can reorder the positioning of an import directly in the production model.
2. Admins can upload the new import data source file, provided the naming convention of the file doesn’t change. However, admins can’t save if they wish to edit import data source.
3. Sometimes clients want additional control over the import mappings for debugging the import mappings.

Disadvantages 

1. Your client might not want to do this change directly in the production environment.
2. This particular change can be made in production after the revision tag has been pushed from development, so there is a dependency on the development model.
3. Due to segregation of duties, you might not be able to access the production environment, and therefore, you may need to ask production wwners to make that change for you—hence the dependency.
4. You can’t test the change in test environment without pushing the revision tag to test.
5. It is repetitive, meaning whatever you are doing in development has to be done in test and production separately.                        

 II - Imports marked as non-production data or structural and import data source as production data

Import data source, as the name suggests, is the data source for any import. This data source can be in the form of a flat-file or model-to-model import.

When the import data source is a flat-file: In this scenario, admins will be able to edit the file and change the file options of an import. All the starred options can be changed in the import data source of production environment.

When the import data source is a saved view: In this scenario, admins can change the import data source from one saved view to another saved view directly in the production environment.

 

Advantages 

1. File options of import data source can be changed directly in the production environment.
2. Saved views of import data source can be changed directly in the production environment.
3. Admins of the prod environment can reorder the positioning of import data sources directly in the production environment.

Disadvantages 

1. Imports will fail if the import data source is not structurally the same.
2. Again, it is repetitive, meaning whatever you are doing in dev has to be done in test and prod separately.

III - Both imports and import data sources marked as production data

This scenario is the combination of scenario 1 and scenario 2 described above. This scenario gives more control to the admin users of the production environment. For example, if some of the parameters of the source file, like text encoding, column separators, delimiters, etc. have changed along with the column headers, admins can load the updated file directly and change the file options. After this, they can go to the imports and correct the mappings. This also applies when the source data comes from any saved view.

Advantages 

1. Greater control with production admins and better position to be in than the first two scenarios.
2. Don’t have to wait for the development team to push the change.
3. Both imports and import data sources marked as production data can be reordered.

Disadvantages 

    1. Changes go untested and unchecked directly in the production environment.

IV- Both imports and import data sources marked as non-production data

This is the default selection of any model. All the imports and import data sources are non-production data. Most of the use cases follow this approach and do not change the settings.

Advantages 

1. More control over how the data comes into the model.
2. It goes through multiple rounds of testing before making its way to the production environment.

Disadvantages

1. Every change has to come from the development environment (blessing in disguise).

 

JaredDolich

You're a true master @Misbah Great article - thank you for taking the time to write this! I learn something new every time.

Misbah

Thank you @JaredDolich Glad that you found it informative

andrewtye

I just wonder why you would want to have either as "production", in my time using Anaplan i've not thought that I would need either to be so. Don't know if you've had cause to set them as such?

 

Personally i'd prefer data loads and their respective processes to fail if something has changed in Production rather than allow end users (who may well know what they're doing) to make sure these things work. Afterall who knows what might happen and then the developers get the blame!

 

And as the first line of the Planual states: "Just because you CAN doesn’t mean you SHOULD"

Misbah

@andrewtye 

 

Just wanted to put it out there if anybody wants to play out with the options what are Pros/Cons. I have worked on multiple use cases where client wanted to have those as Production data assuming it would give them more flexibility to change the data files and their formats -which end users were uploading. Hence the need.

 

On your last line : Hahaha:) Are you asking that to Anaplan Platform Developers? Why would they build such a feature which is not tot be used? Nevertheless apt usage of the Zen.

 

Cheers

Misbah

jasonblinn

@andrewtye I have used these features many times. In some of my projects, we have one DEV with multiple production Environments, split out by geography or business. For sake of this example, I will say I have 3 regions, AMER, EMEA, APAC.

 

I have one DEV model, and one Data Hub that feeds a PROD AMER, PROD EMEA, and PROD APAC models.

To get data from my Data Hub into my PROD models I have saved views set up in the Data Hub that have specific filters to show only data that is applicable to that particular Region:

- Actuals Data Export - AMER

- Actuals Data Export - EMEA

- Actuals Data Export - APAC

 

In my AMER Model I would set the data source to be the  'Actuals Data Export - AMER' saved view, and change the others respectively as well. That way I have the same import running from specific saved views depending on the model. 

 

Could I create different actions for each one of them? Sure. But the goal was to make the user experience the same for all users, and not have customized buttons for everyone. Also, we had 28 different models split out by Geography, so having 28 buttons wouldn't have been a very viable option!

 

I have split models like this for Size and Security reasons in the past, Hyper Models could help with the size part, but the Security part might still exist making this a valid feature for me.

 

Hoping that gives you a real-world example of where this would be a useful feature!

 

Jason

 

JaredDolich

Nice @jasonblinn great use case. You should write that up as a best practice. I think there are a lot of people that could benefit from that setup.

andrewtye

Thanks @jasonblinn. Nice example.

karank_*4578

Good Article @Misbah 

andrewtye

And of course ironically i've had cause to set an import to production data. Normally only data for one item on a list would be updated but occasionally this might change.

andre.lie

@jasonblinn Good use case! 

In a few occasions, I marked Import Data Sources of Saved View as Production Data to workaround issue due to model import from one workspace to another. Despite of the Source Models being mapped correctly, somehow the action failed to execute with error message indicating failure to invoke API or illegal argument. Making the Import Data Sources as Production Data and re-pointing the view to correct workspace solved the issue.  

 

michael.chefer

@Misbah 

Nice illustration of the options and when it's best to exercise them in a production environment. 

 

On a related note, what's the best practice if I have a production list that is being updated regularly via a csv file. Do I (a) load the file directly to the list or (b) create a staging module that then has a saved view to update the list?

 

I've been wrestling with option (a) because if the file name changes then it seems impossible to update the IDS for the original import action.