Exporting multiple modules using API takes too much time

AHmedLattali

Hello Community, 

 

I'm trying to export multiple modules (7 modules) using the API. but it takes up to 4 hours to do that.

The size of the biggest module is about 600Mo.

 

My questions are the following :

 

Is there way to do a full export of the Anaplan modules (like creating a dump) ?

Is there some possibilities to accelerate the extraction using the API ?

 

Any help will be appreciated.

 

Best regards

Ahmed

 

jairamkamath

@AHmedLattali Kindly first revisit and validate if an entire module has to be exported. If not mandatory, below are few options you can consider:

Option 1: Something that we have implemented is split the models & data hubs into 15 smaller models. 
This allows APIs to run parallel job on all 15 models thereby reduced the run time by approximately 15-20 times; as in some cases it run much faster. 

Option 2: If you are continuing to use only one model - another options would be to use boolean and filter our any ZERO values in the export saved view. Also ensuring you do not export summaries and only one level from each list. And then exporting data through saved view. If there is lot of sparsity in the data being exported, this process can improve and may complete within an hour.

Option 3: If the model is bulky or memory intense, the model opening time may be also included in the total time of export. You may also want to check if you can use a dummy job and have model up before export job is triggered, that way saving another 15-20 minutes.

Option 4: Check if you can reduce the list values in export module by creating a new module by using list subsets. This can also reduce the exports by good numbers (we have seen unto 50% benefit due to reduced sparsity/cell count).

 

Option 5: The last option to save time and if the data is not very important; where if it is just being exported as a backup is to archive the model itself using "Copy and Archive" feature which can complete in 2 minutes on 100GB model.

There may be more options available, these are something I can think on top of my mind.

ben_speight

A multi-column tabulated text/csv export (with certain restrictions on pivoting/filters) is one of the fastest (least-blocking) way of getting data out of a model, the other being a model-to-model import. When I see the same large data set being shunted around repeatedly my first questions are whether it is necessary, and if only a small fraction has changed whether an incremental update of only the changed data can be performed by filtering out the rest.

AHmedLattali

Hi @jairamkamath 

Thanks for your reply

 

Option 5: The last option to save time and if the data is not very important; where if it is just being exported as a backup is to archive the model itself using "Copy and Archive" feature which can complete in 2 minutes on 100GB model.

For this option can we transfer the archive outside Anaplan server ? my goal is load this archive in a relational database

 

Thanks

jairamkamath

@AHmedLattali The "Copy and Archive" does not allow you to export data outside of Anaplan. It will be a point in time snapshot of entire model to access only on Anaplan Platform itself.

However if none of the options work for you - another possible work around:

In case you have additional workspace you may also be able to import the model into another workspace (takes 2-3 minutes approx.); and then run the batch to export data out of the newly imported model.

 

That way your users in application would not get impacted for the duration of 2-3 hours while the extracts take place. It requires some re-engineering and work.

david.savarin

I would be much interested in how to just automatically tag what data changed to narrow in the data to export ?