In as much detail as possible, describe the problem or experience related to your idea. Please provide the context of what you were trying to do and include specific examples or workarounds:
In our company, we operate with models that are significantly large, both in terms of data volume and the number of users interacting with them. These models generate a high number of transactions and records per minute. A key aspect of our operations involves numerous Product Assortments that are regularly updated, alongside frequent outbound exports from these models. The core problem we face is the current impracticality of effectively checking or downloading the model history. Due to the sheer volume of data generated by these processes, the model history becomes overwhelmingly large and difficult to navigate. For instance, an automated Product Assortment (PA) list creation process runs hourly for export purposes, generating millions of records each time. While this process is essential, its entries flood the model history, making it extremely challenging to identify and review meaningful changes or user-driven actions. This lack of effective history management hinders our ability to trace specific changes, troubleshoot issues, or understand the evolution of critical data points.
How often is this impacting your users?
This issue is impacting users constantly and frequently. Given that Product Assortments are "regularly updated" and automated processes like the PA list creation run "hourly," the problem of an unmanageable model history is a continuous challenge. Every time a user needs to review historical changes or understand data lineage, they encounter this difficulty.
Who is this impacting? (ex. model builders, solution architects, partners, admins, integration experts, business/end users, executive-level business users)
This problem primarily impacts:
- Model Builders: Those responsible for designing, maintaining, and troubleshooting the models. They need to understand changes and their impact.
- Solution Architects: Individuals who oversee the overall structure and data flow within the models and need to ensure data integrity and traceability.
- Integration Experts: Those managing outbound exports and data synchronization, who need to verify data consistency and changes.
- Business/End Users: Users who rely on the accuracy and historical context of the data within the models for reporting, analysis, or decision-making. While they might not directly interact with the raw history, the inability to effectively manage it can lead to delays or inaccuracies in their data.
What would your ideal solution be? How would it add value to your current experience?
Our ideal solution would involve enhanced control and filtering capabilities for the model history. Specifically, we would like:
- Ability to "Flag" or "Select" specific modules or actions for tracking/exclusion: This would allow us to define what is truly meaningful in the history.
- Exclusion Example: We could exclude automated processes like the hourly PA list creation that generates millions of records, as these are predictable and less critical for individual change tracking.
- Inclusion Example: We could specifically flag input or user-related modules/lists where tracking is highly meaningful, such as manual data entries, configuration changes, or critical business rule modifications.
- Ability to filter the model history when downloading: This would enable us to retrieve only relevant historical data. For example, we should be able to download history related to a specific module, a particular user, or within a defined time frame.
This solution would add significant value by:
- Improving Efficiency: Users would no longer waste time sifting through millions of irrelevant automated entries to find critical information.
- Enhancing Traceability: It would make the model history a much more meaningful and actionable tool for understanding changes, debugging issues, and ensuring data governance.
- Reducing Noise: By excluding high-volume, automated entries, the history would become cleaner and easier to interpret.
- Better Decision Making: With clearer and more focused historical data, users can make more informed decisions based on the evolution of key model components.
Please include any images to help illustrate your experience.
