Unit Testing is for finding bugs, not show correctness

vanreign

Architect, Builders;

Having worked projects with hundreds of bugs, the natural question is how did this happen?

The solution passes User Acceptance Testing means that it was adequately tested.

This could be a paradigm shift for some in thinking about unit testing. Doing a quality job of it adds time to get a solution to Production. However, development is the best and optimal time within the software development life cycle to spend the time, providing most value to a client.

This discussion thread is written with great comradery and respect for our Anaplan Community to express support and gratitude for those that put time into quality unit testing. It benefits all of us and usually goes unrecognized.

Software testing principles are published by the International Software Testing Qualifications Board:

• Testing reduces the probability of undiscovered defects, but if there are no defects found, that is not a validation of correctness.
• Exhaustive testing is impossible so conducted weighing risks and priorities.
• Early testing makes a bug much cheaper to fix.
• 80% of the problems are found in 20% of the modules.
• If the same tests are repeatedly done, eventually, the same set of test cases will no longer identify any new bugs in the system.
• The risk associated with each type of application is different, so using the same method is ineffective.
• Finding and fixing defects does not help if the system is unusable and does not fulfill users’ needs and requirements.

This discussion is addressed to Architects and Builders because a client may never see or validate integration points, understand how scaling will impact a model, recognize how a culmination of formula inefficiencies could impact optimization, have a clear understanding of processes, understand how design decisions will impact maintainability, and the list goes on. Since sufficiently complex models may have an infinite number of possible bugs, it is logically impossible to show the correctness of software through testing.

Those that have heard the phrase that Anaplan is Excel on steroids are sometimes caught off guard by the realization that building and maintaining this platform has a much steeper learning curve than Excel. And we guide them through that journey, reach a good degree of proficiency, and sometimes have complacency in overlooking some fundamental development principles like good unit testing. There are bugs that we may be the only ones able to identify so share the testing responsibility equally with the client.

Cheers,

James, CTFL

Stacey_Gibbens

I admit I laughed out loud at this one.

The solution passes User Acceptance Testing means that it was adequately tested.

Over the past several months, I have been working through stabilizing and completing a fairly lengthy list of incomplete enhancements that were the result a multi-model "optimization and enhancement" project done by a consulting partner with my current teammates (prior to my arrival to the team)… the solution was certainly NOT adequately tested nor adequately defined, IMHO.

I feel that a good deal of the issue derived from inadequate communication/user stories and the hyper-compressed timelines that are expected from Anaplan projects. There's often not nearly enough time to really understand the environment as a whole and there's this expectation that in 90 days (or less!) a fully baked model will be available. I laugh and cry at this.

Unless clients have a robust and DOCUMENTED understanding of what they need and how it fits into the existing framework, they bring the auxiliary system owners into the process, there are in-house integration specialists ready and able to devote time, the client has also invested in learning Anaplan, they client team isn't already stretched thin with their "day jobs" and so are actually able to participate properly in the project (including UAT) … the list is enormous. Only THEN will anything substantial and of QUALITY come from any software project, Anaplan included.

I feel your pain.

In my experience, the way to avoid data loss in scenarios like you describe is to have a Prod, Prod-Support, and Dev environments. Prod and Non-Prod are connected via ALM and Dev is not. New development is vetted out (in general) in Dev and migrated to Non-prod. There is a Non-Prod DataHUB connected to a non-prod auxiliary data source system in which to test end-to-end integrations. That's the only way I've confidently gone live with new data sources or integration processes.

Having a truly complete non-prod environment (all data sources and targets available to share between) is the only way to avoid it. Otherwise, the system is inherently at risk. The client/owning team will need to make the decision on how much risk they are willing to tolerate and how catastrophic a data loss they're willing to accept if the worst happens. There's always the ability to roll-back model history… <shrug>

Best wishes,

Stacey

vanreign

@Stacey_Gibbens Thank you Stacey! I appreciate your feedback and relate very well with your experiences. Best wishes to you as well :) -James