Test Principles

The following Testing principles have been designed to be in line with Industry best practices.

Each test case tests one thing - and only one thing.

The overarching principles of testing at A&G are to:

• Ensure testing is based on mitigating risks to the business

• Automate tests as a priority

• Demonstrate the software meets the Acceptance Criteria

• Verify that the solution meets the stated business and technical requirements

• Ensure that all defects/omissions outstanding at the end of the testing cycle are documented, and an action plan is in place for their resolution.

• Reduce the risk of software defects reaching production.

Principle	Description
Testing shows the presence of defects, not their absence	Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, testing is not a proof of correctness.
Exhaustive testing is impossible	Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Rather than attempting to test exhaustively, risk analysis, test techniques, and priorities should be used to focus test efforts.
Early testing saves time and money	To find defects early, both static and dynamic test activities should be started as early as possible in the software development lifecycle. Early testing is sometimes referred to as shift left. Testing early in the software development lifecycle helps reduce or eliminate costly changes.
Defects cluster together	A few modules usually contain most of the defects discovered during pre-release testing or are responsible for most of the operational failures. Predicted defect clusters, and the actually observed defect clusters in a test or operation, are important inputs into a risk analysis used to focus the test effort.
Beware of the pesticide paradox	If the same tests are repeated over and over again, eventually these tests no longer find any new defects. To detect new defects, existing tests and test data may need changing, and new tests may need to be written. (Tests are no longer effective at finding defects, just as pesticides are no longer effective at killing insects after a while.) In some cases, such as automated regression testing, the pesticide paradox has a beneficial outcome, which is the relatively low number of regression defects.
Testing is context dependent	Testing is done differently in different contexts.
Absence-of-errors is a fallacy	Some organizations expect that testers can run all possible tests and find all possible defects, but principles 2 and 1, respectively, tell us that this is impossible. Further, it is a fallacy (i.e., a mistaken belief) to expect that just finding and fixing a large number of defects will ensure the success of a system. For example, thoroughly testing all specified requirements and fixing all defects found could still produce a system that is difficult to use, that does not fulfill the users’ needs and expectations, or that is inferior compared to other competing systems.
Collaborative Approach	Where possible a Test-Driven Development approach will be taken. Unit tests will be written by the developers and run before publishing any new build candidates for deployment. Test Analysts will also be able to use the unit test framework to expand their API level tests.
Shared Responsibility	Testers are NOT the only group responsible for quality. Quality is everyone’s responsibility. Developers should validate their own work to the best of their ability and strive to get better at doing so over time. Quality begins at story writing/elaboration and building high quality acceptance criteria and flows right through the Software Delivery Life Cycle (SDLC) through to User Acceptance Testing. It involves all roles, including Product Owners, Business Analysts, Developers and Testers. Strong collaboration is the channel to high quality and the earlier quality issues are identified; the more cost effective it is to address them.
Define Quality Early	Quality must be defined early in the development lifecycle. This means that acceptance criteria, supporting test cases and definitions of done are defined in the earliest stages of the project life cycle to ensure everyone involved has a clear and common understanding of the level of quality that is acceptable and how, as a team, this will be achieved.
Create Fast Feedback Loops	A feedback loop in the context of application software development and change is the time it takes to verify or validate the deployed code to ensure it meets expectations and requirements and for this feedback to make its way back to the developer of the application software. This cycle is known as the Feedback Loop. The longer the period of time between deploying the new or updated application code and receiving feedback as to its acceptability, the longer time the defect has had to propagate through the system, the more time it has had to create additional issues and the more expensive it becomes to correct the original defect and remediate the subsequent consequences. The sooner feedback on a defect is passed back to the developer, the sooner they can correct the issue. Consequently, a key motivator of the Test Strategy is to speed up this feedback loop so that defects do not get time to propagate and cause subsequent and possibly expensive issues. Test early, test often and test appropriately.
Automation Testing	Automate high value repetitive and manual testing activities and tasks as much as possible.
Tests are independent	Tests should be independent of other tests so they can be run in parallel to increase feedback. Tests should also be responsible for managing their own data, an ideal test creates its state and tears it down upon completion
Testers Test as Users Use	We should always aim to interact with the system under test in the same way that the user would. If a portion of your business logic is on the client side (e.g. browser) you must test through the UI for that functionality. Only if the application is service architected can you confidently test through the service layers to mimic that user interaction. UI tests should still be run to detect presentation layer defects.
Risk Based Testing	A risk-based approach will be used to determine test coverage. A risk-based testing approach brings formal risk assessment methods to the testing process to prioritize the tests for suitable coverage and allocate testing effort. Risk-based testing provides higher confidence that the right level and type of coverage on the right things at the right time to prevent probable issues that carry high impact and high cost given the introduced changes under test.
Data Management	Production data must be obfuscated before being used for testing in non-production environments. When testing in production environments the following guidelines must be followed.
Segregation of Duties	As an agile technology team, every delivery team member is expected to contribute to delivering their agreed work. However, to ensure quality, duties are expected to be segregated so that an author of a piece of work will never be the reviewer.