Import failure based on module cell count?
I have just submitted this to Anaplan support, but wanted to run it by the Hive Mind. Any insight into what's going on here would be appreciated. I will update this thread with any answer I receive from Anaplan support!
We had a very puzzling issue last week that prevented loading of data. We resolved the issue, but the required change to our model makes it significantly less user friendly. After internal investigation, it turned out that the issue was a user filter module with a single Boolean line item (shown below). We deleted the filter and the data could load.
Key facts:
This user filter acted on a list used as a dimension on a data module.
The list used for the filter was large … 1,908,818 prior to the import (and increased to 1,956,913 after the import was successfully completed).
The user list is also large – 1,091 users.
Prior to the failed data load, the User Filter module had 2,082,520,438 cells.
When troubleshooting the issue, we received the following message that eventually led us to the user filter noted above. The only reference I could find to a cell limit of this general value in Anaplan is the 2^31 (2,147,483,648) threshold that was previously a maximum value for exporting cells in the Excel Add-in 3.1. Since the failed process is completely unrelated to an Excel process, this really threw us for a loop. We were previously not aware of this limit.
This seems like a bug. Any advice us on a different method to have a user filter (with lots of users) on a list (with lots of records) that does not prevent data from loading? I look forward to hearing from you…
Best Answer
-
That 2^31 -1 is not only for the export, but it is the maximum cell limit of a line item at its most granular block of data. If you had time or if you had summaries turned on, that number could be bigger, but not at its most granular block (think month level). So, if your line item was using time, call it one year at the month level and your summaries are turned off, then take the total cells for that line item and divide by 12 (we divide by 12 because the month is the most granular block since we store the data three different blocks: time, versions, and custom lists).
Does that help?
Rob
4
Answers
-
How you discovered the issue is mind-blowing! When you say you removed the filter, do you mean the filter on the saved view or that you deleted the Boolean line item? If its the filter then that would be very puzzling because imports are agnostic to any view. Also, if it's a filter, is it on the default view or a saved view?
As for the Excel limit, @MagaliP will soon be announcing the next Excel Add-In upgrade will have an Excel limit increase to 5MM from 1MM rows that may also help with the export challenge you've got.
Anxious to hear what others have to say and what support comes up with. Thanks for posting this!
1 -
Not so mind-blowing. 😉 Working in IS Operations for a fortune 100 company with a boatload of complex business requirements and processes gives ample chances for things like this to happen.... LOL.
Removed the filter = deleted the filter module (e.g. the filter module no longer existed to be impacted when the list size increased.
This was an import, not an export but the only answer I could locate that used the 2^31 value was related to excel. The answer from @rob_marshall below is the answer. WOW. Didn't know this. 😁
Stacey Gibbens
1 -
As this was a filter module, the module had no relationship to time, versions, etc., but the conjunction of more than 1,000 users and nearly 2,000,000 list members caused the SINGLE line item in the filter module to **** up.
Yes, this is exactly the answer. I did not know this. Thank you so much!
I guess the question now is, when is Anaplan going to increase that limit? 🤣
Stacey Gibbens
0 -
I was looking for this picture and finally found it...
In this example, the green block is the most detailed and since it is NOT using Native Time nor Native Versions, there would be no division of the total cell count of that line item since the lists are all custom lists (Channel, Product, and Region). In this case, turn summaries off and that would be the size of the most granular block, which again, can not exceed 2^31 - 1.
Hope that clarifies as well.
3 -
In the current engine, I don't think that limitation will change, but again, that is in the "current" engine.
Rob
2
