Sort Lists greater than 1 million items
Order Lists is extremely helpful (https://help.anaplan.com/anapedia/Content/Modeling/Build%20Models/Actions/Order-list.html) but we have a couple key lists that are over 1 million items, so Order Lists does not work for them. We are looking at around 11 million items for current year and prior year data.
We have transactions that are feeding into HUB from our source system, that come in based upon a transaction number. This module/list is what we use to create our hierarchy levels. It was created with First Occurrence as a boolean to determine what the first instance was to create the various hierarchy levels (L1, L2, L3 and L4).
The issue is that there are some L3 items we want to exclude from going creating a hierarchy; however that same item may also be the first instance of L1 and L2 - meaning if we exclude it, our hierarchy for L1 and L2 is not set up, so any "valid" hierarchies at those levels will not be created.
What are some other options to get the proper hierarchy created?; while not creating hierarchy levels for items that will ultimately have no activity because they are being excluded by the L3 exclusion.
See attached for visual examples of what we want to accomplish.
Example 1 shows excess hierarchies being created that will never be used - increasing sparcity and space consumption. Example 2 shows issue if first item is first occurrence of L1 - no hierarchies created. Example 3 shows how L2 that are needed would not be created. Example 4 shows what we want to happen - hierarchies without L3 exclusions would be created and no unnecessary hierarchies would be created.