Regression Testing: Best practices

Regression Testing: "What" to test and "When"

Regression testing is often seen as an area in which companies are reluctant to allocate resources. We often hear phrases like: "The promoter said the defect is fixed. Do I need to try again?" And the answer should be: "Well, probably, the developer said the product is faultless to begin with." The truth of the matter is that in today's world of extremely complex devices and software applications, the quantity and quality of regression tests that are performed in a product are directly proportional to the commitment of suppliers to its customer base. This does not mean regression test more the better. It simply means that we must ensure that regression testing is done in the right quantity and with the right approach.

The two main challenges presented by the regression tests are:

1. What is tested?

2. When we tested this?

The purpose of this paper is to outline some techniques that will help us answer these questions. The first question to consider is the fact that it is not necessary to run our regression at the end of our test cycle. Much of the effort of regression can be performed simultaneously with all other testing activities. The alleged support of this approach is:

"Do not wait until all testing is done to correct our shortcomings."

Therefore, much of the effort of regression can be achieved well before the completion of the project if the project has a reasonable duration. Test whether our efforts will only last a week, the following techniques may have to be modified. However, it is unusual for a product to be tested in such a short period of time. Moreover, studying the techniques described below, you will see that with increasing the length of the project, the benefits of these techniques also increase.

To answer the questions of what to try and when, let's start with a simple set of ten tests. In the real world, this suite, it would obviously be much greater, and not necessarily static, which means that the number of tests may increase or decrease as needed. After our first test with the first beta version (which we call "DROP 1") of our hypothetical software product, our matrix looks like this.

In the matrix above, we have a cross reference between the defects we found, with evidence that caused them. As you can see, the number of defects 1 was caused by test 2, but also occurred in test 3. Failures caused defects remaining unique.

As we prepare to run our second test run (Code Drop 2), we must decide what evidence will be implemented. The rules only apply to use our efforts regression. There are rules that can apply to a subset of tests that have passed, in order to find out what we re-run. However, this will be the subject of another article.

The fundamental question we must now ask is: "Did any of the defects found have been fixed?" Suppose that the defects of 1, 2 and 3 have in fact been reported fixed by our developers. Suppose further that three tests were added to our test. After "Drop Code 2", our matrix is as follows:

Some key points of the communication are:

The evidence that previously failed, run only the tests associated with defects that were supposedly fixed. Test number 9, number of defects caused 4, was not executed Drop Code 2, because the number 4 is not fixed defects.

Defect # 1 is fixed, for tests 2 and 3 have been finally approved.

Test No. 7 yet. Therefore, the defect persists.

Test number 13 is a new test, prompting a new defect.

We chose not to run the tests had happened in the code Drop 1. This can often be the case, as the turmoil in the code or the importance of the area (as a new feature, an improvement of an old property or an item as a key selling point of the product) can lead us to re-run these tests.

This approach is simple, but effective, ensures that our parent will never look like the array below (in order to demonstrate more clearly the problem, we will omit the defects of column # code after each fall). Code also considers reduction of 5 to be our final regression pass.

We discuss tests 2, 7, and 9, later, but here are some key points to notice about this matrix:

Why the tests were 1, 4, 5, 6, 10, 11 and 12 carried five times? They passed each time.

Why were the tests of 3 and 8 performed five times? First failed and were fixed. What they need to be executed in every drop of code after the failure?

If the test 13, was the test team said it had been erroneously set at each drop of code? If not, why was executed four times with the same result? We may also ask the question: "Why is not fixed?" But we will not deal with this issue, since we are only dealing with the issue of regression.

In conclusion, we list some general rules that we apply to our testing effort to ensure our efforts are justified and accurate regression. These rules are:

1. Evidence that has happened twice to be considered a regression, unless the turmoil in the code (or for other reasons stated above, as a feature of importance) indicates otherwise. By this we mean that the only time a test was run over twice is whether changes in the code in the area of the exercise test (or the importance of the particular feature) sufficiently justified concerns about the state of the test or the condition of the property.

2. A test that has failed once again should not be executed unless the sponsor informs the test team that the defect has been corrected. This is the case of tests of 7 and 9. That should not have been re-executed until drops Code 4 and 5, respectively.

3. We apply the exact algorithms to determine what evidence it has already happened once to be re-run, so be aware of situations such as the number 2 of the test. This test passed twice after its initial failure, and failed again in the code Drop 4. As an additional note of caution: "When in doubt, run".

4. For the tests that have already happened once, the second execution should be reserved for the final step of regression, unless the turmoil in the code suggests otherwise, or that we have enough evidence to execute. However, we must be careful. While this allows us to obtain some of the efforts of the regression of the form earlier in the project, may limit our ability to find defects introduced later in the project.

5. The final regression step should not consist of more than 30% to 40% of the total number of tests in our series. This subset is assigned by the following priorities:

a. All tests that have failed more than once. By this we mean tests that failed, the developer informed of them as fixed, and yet again failed, either immediately after they were fixed or at any time during the rest of the tests.

b. All tests failed once and then passed, after they were reported as fixed.

c. Carefully chosen subset or all of the evidence that has happened only once.

d. If there is still room for more testing, perform any other tests that do not meet the above criteria, but you feel however, should be executed.

These common sense rules will ensure that regression testing is done with elegance and in the correct amount. In an ideal world, we would have the time and resources to test our product completely. However, today's world is a world of tight deadlines and budgets further. Wise resources costs today will ensure our ability to continue to develop reliable products tomorrow.