Test Case Prioritization

Test case prioritization is a test implementation process where test cases to be executed are ordered from first to last, based on certain criteria. This criteria could be risk assessment, business requirements, time, availability of resources.

What test case prioritization hopes to achieve is to provide an order in which test cases are executed from the most significant to the least significant, so that in a worst case scenario where testing has to be abruptly stopped or when time and available resources run short, already executed tests could offer some level of quality assurance.

Test Case Prioritization Techniques

1. Risk Based Prioritization

In this technique, test cases that cover the highest product risks are ordered first, followed by the test cases covering the second highest risk, until the least. The risk level here is determined by product risk analysis. This consists of risk identification and assessment. In risk identification, potential product bugs and defects are recognised, while in risk assessment, the likelihood of a bug or defect occurring is determined and multiplied by the impact that bug or defect would have on a system if it occurs. When this has been determined, test cases that cover parts of the system with the highest risks are executed first.

For example, in a patient data management system,

Test1 tests user access and prevents unauthorised access.

Test2 tests that patient data entered into the system matches what is saved in the database.

Test1 and Test2 both cover high risk functionalities but after risk assessment, it was ascertained that a failure of Test2 posses a higher risk and it should be executed before Test1. This is because, while unauthorised users having access to the system would have a high risk impact in the case of cyber attacks, the likelihood of that happening is low, but having a mismatch with patient data could lead to incorrect medication prescriptions, the likelihood of that happening is high and the risk impact is also high.

Risk based prioritization offers the benefit of testing critical parts of the system first so in the event where testing has to be abruptly stopped or time and resources run out, the system failure in production would have less adverse effects.

2. Coverage Based Prioritization

Coverage-based prioritization orders test cases based on code coverage and ensures that the most critical parts of the code are tested first.

This test case prioritization technique is based on executing test cases with the highest code coverage first up until the least. The more robust test cases that cover the most functionalities are given a higher priority. Coverage based technique is based on :

• Total Statement Coverage : Test cases are ordered based on the number of statements they cover, with the highest first.

• Total Branch Coverage: Test cases are ordered based on number of branch coverage the achieve, with the highest first.

For example, consider the table below

Test1 covers 5 statements, Test2 covers 7 statements and Test3 covers 2 statements, based on Total Statement Coverage, the test case prioritization would go- Test2, Test1 and Test3.

Test1 covers 3 branches, while Test2 and Test3 cover, 4 and 5 branches respectively, based on Total Branch Coverage, the test case execution would go -Test3, Test2 then Test1.

3. Dependency Based Prioritization

In this prioritization technique, test cases are ordered based on how many prerequisite dependencies a test cases has. The test cases with the least dependencies are executed first. Take this table below,

From table above, the test case prioritization would go- Test4, Test2, Test1 and Test3.

In dependency based technique, some test cases may depend on others, so the ones with no dependencies have to be executed first before the dependent test cases.

Importance of Test Case Prioritization

Test case prioritization is important as it helps keep testing organised, and an established order can be followed for subsequent regression testing with similar conditions. It also ensures the most relevant test cases are executed first, thus bringing the product closer to it’s desired quality goals.

Factors to consider when determining the method of test case prioritization to apply

Testing is context based ,so before choosing what prioritisation technique to apply, you should consider the business requirements, the available resources, the critical functionalities of the product, the product risks and software development lifecycle being used.