Intro to Agile Testing

Agile testing overview

Principles

The following Testing principles have been designed to be in line with Industry best practices.

Each test case tests one thing - and only one thing.

The overarching principles of testing at A&G are to:

Ensure testing is based on mitigating risks to the business
Automate tests as a priority
Demonstrate the software meets the Acceptance Criteria
Verify that the solution meets the stated business and technical requirements
Ensure that all defects/omissions outstanding at the end of the testing cycle are documented, and an action plan is in place for their resolution.
Reduce the risk of software defects reaching production.

Principle	Description
Testing shows the presence of defects, not their absence	Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, testing is not a proof of correctness.
Exhaustive testing is impossible	Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Rather than attempting to test exhaustively, risk analysis, test techniques, and priorities should be used to focus test efforts.
Early testing saves time and money	To find defects early, both static and dynamic test activities should be started as early as possible in the software development lifecycle. Early testing is sometimes referred to as shift left. Testing early in the software development lifecycle helps reduce or eliminate costly changes.
Defects cluster together	A few modules usually contain most of the defects discovered during pre-release testing or are responsible for most of the operational failures. Predicted defect clusters, and the actually observed defect clusters in a test or operation, are important inputs into a risk analysis used to focus the test effort.
Beware of the pesticide paradox	If the same tests are repeated over and over again, eventually these tests no longer find any new defects. To detect new defects, existing tests and test data may need changing, and new tests may need to be written. (Tests are no longer effective at finding defects, just as pesticides are no longer effective at killing insects after a while.) In some cases, such as automated regression testing, the pesticide paradox has a beneficial outcome, which is the relatively low number of regression defects.
Testing is context dependent	Testing is done differently in different contexts.
Absence-of-errors is a fallacy	Some organizations expect that testers can run all possible tests and find all possible defects, but principles 2 and 1, respectively, tell us that this is impossible. Further, it is a fallacy (i.e., a mistaken belief) to expect that just finding and fixing a large number of defects will ensure the success of a system. For example, thoroughly testing all specified requirements and fixing all defects found could still produce a system that is difficult to use, that does not fulfill the users’ needs and expectations, or that is inferior compared to other competing systems.
Collaborative Approach	Where possible a Test-Driven Development approach will be taken. Unit tests will be written by the developers and run before publishing any new build candidates for deployment. Test Analysts will also be able to use the unit test framework to expand their API level tests.
Shared Responsibility	Testers are NOT the only group responsible for quality. Quality is everyone’s responsibility. Developers should validate their own work to the best of their ability and strive to get better at doing so over time. Quality begins at story writing/elaboration and building high quality acceptance criteria and flows right through the Software Delivery Life Cycle (SDLC) through to User Acceptance Testing. It involves all roles, including Product Owners, Business Analysts, Developers and Testers. Strong collaboration is the channel to high quality and the earlier quality issues are identified; the more cost effective it is to address them.
Define Quality Early	Quality must be defined early in the development lifecycle. This means that acceptance criteria, supporting test cases and definitions of done are defined in the earliest stages of the project life cycle to ensure everyone involved has a clear and common understanding of the level of quality that is acceptable and how, as a team, this will be achieved.
Create Fast Feedback Loops	A feedback loop in the context of application software development and change is the time it takes to verify or validate the deployed code to ensure it meets expectations and requirements and for this feedback to make its way back to the developer of the application software. This cycle is known as the Feedback Loop. The longer the period of time between deploying the new or updated application code and receiving feedback as to its acceptability, the longer time the defect has had to propagate through the system, the more time it has had to create additional issues and the more expensive it becomes to correct the original defect and remediate the subsequent consequences. The sooner feedback on a defect is passed back to the developer, the sooner they can correct the issue. Consequently, a key motivator of the Test Strategy is to speed up this feedback loop so that defects do not get time to propagate and cause subsequent and possibly expensive issues. Test early, test often and test appropriately.
Automation Testing	Automate high value repetitive and manual testing activities and tasks as much as possible.
Tests are independent	Tests should be independent of other tests so they can be run in parallel to increase feedback. Tests should also be responsible for managing their own data, an ideal test creates its state and tears it down upon completion
Testers Test as Users Use	We should always aim to interact with the system under test in the same way that the user would. If a portion of your business logic is on the client side (e.g. browser) you must test through the UI for that functionality. Only if the application is service architected can you confidently test through the service layers to mimic that user interaction. UI tests should still be run to detect presentation layer defects.
Risk Based Testing	A risk-based approach will be used to determine test coverage. A risk-based testing approach brings formal risk assessment methods to the testing process to prioritize the tests for suitable coverage and allocate testing effort. Risk-based testing provides higher confidence that the right level and type of coverage on the right things at the right time to prevent probable issues that carry high impact and high cost given the introduced changes under test.
Data Management	Production data must be obfuscated before being used for testing in non-production environments. When testing in production environments the following guidelines must be followed.
Segregation of Duties	As an agile technology team, every delivery team member is expected to contribute to delivering their agreed work. However, to ensure quality, duties are expected to be segregated so that an author of a piece of work will never be the reviewer.

Principle

Description

Testing shows the presence of defects, not their absence

Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, testing is not a proof of correctness.

Exhaustive testing is impossible

Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Rather than attempting to test exhaustively, risk analysis, test techniques, and priorities should be used to focus test efforts.

Early testing saves time and money

To find defects early, both static and dynamic test activities should be started as early as possible in the software development lifecycle. Early testing is sometimes referred to as shift left. Testing early in the software development lifecycle helps reduce or eliminate costly changes.

Defects cluster together

A few modules usually contain most of the defects discovered during pre-release testing or are responsible for most of the operational failures. Predicted defect clusters, and the actually observed defect clusters in a test or operation, are important inputs into a risk analysis used to focus the test effort.

Beware of the pesticide paradox

If the same tests are repeated over and over again, eventually these tests no longer find any new defects. To detect new defects, existing tests and test data may need changing, and new tests may need to be written. (Tests are no longer effective at finding defects, just as pesticides are no longer effective at killing insects after a while.) In some cases, such as automated regression testing, the pesticide paradox has a beneficial outcome, which is the relatively low number of regression defects.

Testing is context dependent

Testing is done differently in different contexts.

Absence-of-errors is a fallacy

Some organizations expect that testers can run all possible tests and find all possible defects, but principles 2 and 1, respectively, tell us that this is impossible. Further, it is a fallacy (i.e., a mistaken belief) to expect that just finding and fixing a large number of defects will ensure the success of a system. For example, thoroughly testing all specified requirements and fixing all defects found could still produce a system that is difficult to use, that does not fulfill the users’ needs and expectations, or that is inferior compared to other competing systems.

Collaborative Approach

Where possible a Test-Driven Development approach will be taken. Unit tests will be written by the developers and run before publishing any new build candidates for deployment. Test Analysts will also be able to use the unit test framework to expand their API level tests.

Shared Responsibility

Testers are NOT the only group responsible for quality. Quality is everyone’s responsibility. Developers should validate their own work to the best of their ability and strive to get better at doing so over time.

Quality begins at story writing/elaboration and building high quality acceptance criteria and flows right through the Software Delivery Life Cycle (SDLC) through to User Acceptance Testing. It involves all roles, including Product Owners, Business Analysts, Developers and Testers. Strong collaboration is the channel to high quality and the earlier quality issues are identified; the more cost effective it is to address them.

Define Quality Early

Quality must be defined early in the development lifecycle. This means that acceptance criteria, supporting test cases and definitions of done are defined in the earliest stages of the project life cycle to ensure everyone involved has a clear and common understanding of the level of quality that is acceptable and how, as a team, this will be achieved.

Create Fast Feedback Loops

A feedback loop in the context of application software development and change is the time it takes to verify or validate the deployed code to ensure it meets expectations and requirements and for this feedback to make its way back to the developer of the application software. This cycle is known as the Feedback Loop.

The longer the period of time between deploying the new or updated application code and receiving feedback as to its acceptability, the longer time the defect has had to propagate through the system, the more time it has had to create additional issues and the more expensive it becomes to correct the original defect and remediate the subsequent consequences.

The sooner feedback on a defect is passed back to the developer, the sooner they can correct the issue. Consequently, a key motivator of the Test Strategy is to speed up this feedback loop so that defects do not get time to propagate and cause subsequent and possibly expensive issues.

Test early, test often and test appropriately.

Automation Testing

Automate high value repetitive and manual testing activities and tasks as much as possible.

Tests are independent

Tests should be independent of other tests so they can be run in parallel to increase feedback. Tests should also be responsible for managing their own data, an ideal test creates its state and tears it down upon completion

Testers Test as Users Use

We should always aim to interact with the system under test in the same way that the user would. If a portion of your business logic is on the client side (e.g. browser) you must test through the UI for that functionality. Only if the application is service architected can you confidently test through the service layers to mimic that user interaction. UI tests should still be run to detect presentation layer defects.

Risk Based Testing

A risk-based approach will be used to determine test coverage. A risk-based testing approach brings formal risk assessment methods to the testing process to prioritize the tests for suitable coverage and allocate testing effort. Risk-based testing provides higher confidence that the right level and type of coverage on the right things at the right time to prevent probable issues that carry high impact and high cost given the introduced changes under test.

Data Management

Production data must be obfuscated before being used for testing in non-production environments. When testing in production environments the following guidelines must be followed.

Segregation of Duties

As an agile technology team, every delivery team member is expected to contribute to delivering their agreed work. However, to ensure quality, duties are expected to be segregated so that an author of a piece of work will never be the reviewer.

Test Automation Principles

The following Test Automation principles have been designed to be in line with industry best practices.

Principle	Description
Tests should be expressive	It should be transparent what is being tested by an automated test. Let's leave the guesswork out of this and make the tests expressive and transparent. This helps other Technical Testers, Functional Tests, Developers and stakeholders.
Tests should be independent	Tests should not depend on other tests for the state. Tightly coupled tests erode the value of building and running the tests as a continuous regression safety net due to intermittency issues caused by tests having to run in a certain sequence. Ideally, a test should also be responsible for its own (test) data. Meaning it prepares the test data required and cleans it up again.
Tests should be fast	Only when automated tests run fast, we can run them at any stage and as often as necessary or required. The aim here is to allow for fast feedback. Having to wait hours for automated tests to complete, defeats the purpose. This also means keeping long running e2e tests to a minimum.
Tests should be repeatable	With no changes to a system under test, automated tests should produce the same results consistently. Tests that are not repeatable, or unstable, do not provide a lot of confidence.
Tests should be maintainable	Software products change and so will automated tests. Only a test that is easy to read and understand is easy to maintain. Simplicity is key. See KISS Principle
Tests should be easy to run	If tests are hard to run, we discourage others to utilise them. Goal is to have tests that can be run anywhere at any time. Tests that are easy to run will allow for simpler pipelines as well.
Keep the lights on	Keep the tests maintained and efficient. This also applies to the framework itself by reducing tech debt and keeping it and its dependencies up to date. Having healthy pipelines is part of this as well. Pipelines that look like a Christmas tree (red / green) don't build confidence.
Don't repeat yourself	This principle can be applied to two areas - code and tests. Don't duplicate code to keep it easier to read and maintain, and don't duplicate test coverage. Something that has been verified by one test, doesn't need to be tested again.
No shortcuts	Manual testing is no shortcut if automated tests are broken or don't work. Invest the time to keep the test maintained and healthy. Also, don't run the tests locally if there are (and there should be) pipelines for those tests. If the pipelines take too long to complete, look at optimising them.

Principle

Description

Tests should be expressive

It should be transparent what is being tested by an automated test. Let's leave the guesswork out of this and make the tests expressive and transparent. This helps other Technical Testers, Functional Tests, Developers and stakeholders.

Tests should be independent

Tests should not depend on other tests for the state. Tightly coupled tests erode the value of building and running the tests as a continuous regression safety net due to intermittency issues caused by tests having to run in a certain sequence. Ideally, a test should also be responsible for its own (test) data. Meaning it prepares the test data required and cleans it up again.

Tests should be fast

Only when automated tests run fast, we can run them at any stage and as often as necessary or required. The aim here is to allow for fast feedback. Having to wait hours for automated tests to complete, defeats the purpose. This also means keeping long running e2e tests to a minimum.

Tests should be repeatable

With no changes to a system under test, automated tests should produce the same results consistently. Tests that are not repeatable, or unstable, do not provide a lot of confidence.

Tests should be maintainable

Software products change and so will automated tests. Only a test that is easy to read and understand is easy to maintain. Simplicity is key. See KISS Principle

Tests should be easy to run

If tests are hard to run, we discourage others to utilise them. Goal is to have tests that can be run anywhere at any time. Tests that are easy to run will allow for simpler pipelines as well.

Keep the lights on

Keep the tests maintained and efficient. This also applies to the framework itself by reducing tech debt and keeping it and its dependencies up to date. Having healthy pipelines is part of this as well. Pipelines that look like a Christmas tree (red / green) don't build confidence.

Don't repeat yourself

This principle can be applied to two areas - code and tests. Don't duplicate code to keep it easier to read and maintain, and don't duplicate test coverage. Something that has been verified by one test, doesn't need to be tested again.

No shortcuts

Manual testing is no shortcut if automated tests are broken or don't work. Invest the time to keep the test maintained and healthy. Also, don't run the tests locally if there are (and there should be) pipelines for those tests. If the pipelines take too long to complete, look at optimising them.

4 Aces Model

An extension to the AAA pattern of testing that looks beyond the original intended unit testing methods. The additional A stands for "Annihilate", which is the important clean-up step after tests have been run to allow them to be run over and over.

For unit tests, this is simple, meaning that any potential disk writing or state change actions are mocked out or only affected within a short-lived memory space. But, for tests higher up in the test triangle, this may mean that transactions can be reversed or not committed or that the entire environment can be replaced (through methods like IAC).

♦ Arrange

Crystalise the environment into the perfect state.

♠ Act

Dig in and do the work.

♥ Assert

Love the outcomes - and find the heartbreakers.

♣ Annihilate

Club the environment back to what is was like before the test was run.

Last updated 1 year ago